Before revision:  We  provide the first  demonstration that this is possible, and characterize the extent of this capability. To do this, we use a collection of synthetic datasets that test increasing levels of reasoning complexity (number of rules, presence of negation, and depth of chaining). We find transformers appear to learn rule-based reasoning with high (99\%) accuracy on these datasets, and in a way that generalizes to test data requiring substantially deeper chaining than in the training data (95\%+ scores).
After revision:  We provide the first demonstration that this kind of soft reasoning over language is learnable, can achieve high (99\%) accuracy on these datasets, and in a way that generalizes to test data requiring substantially deeper chaining than in the training data (95\%+ scores).