Before revision:  Second, we develop a probe to measure how much can a model cheat (via non-multifact reasoning) on existing tests and our sufficiency test. Third, we conduct experiments using a transformer based model (XLNet), demonstrating that the sufficiency transform not only reduces the amount of non-multifact reasoning in this model by 6.5\% but is also harder to cheat -- a non-multifact model sees a 20.8\% (absolute) reduction in score compared to previous metrics .
After revision:  Third, our experiments demonstrate that there hasn't been much progress in multifact reasoning. For a recent large-scale model (XLNet), demonstrating that the sufficiency transform not only reduces the amount of non-multifact reasoning in this model by 6.5\% but is also harder to cheat -- a non-multifact model sees a 20.8\% (absolute) reduction in score compared to previous metrics .