Before revision:  Third, we conduct experiments using a transformer based model (XLNet), demonstrating that the sufficiency transform not only reduces the amount of non-multifact reasoning in this model by 6.5\% but is also harder to cheat -- a non-multifact model sees a 20.8\% (absolute) reduction in score compared to previous metrics .
After revision:  Third, we conduct experiments using a transformer based model (XLNet), we show that only 18\% of its answer score is obtained through multifact reasoning, roughly the same as that of a simpler RNN baseline. Our transformation shows a substantial reduction in disconnected reasoning (nearly 19 points in answer F1). It is complementary to adversarial approaches, yielding further reductions in conjunction .