text: 415,6,1446143854588035083,280403336,Sean Welleck,"new paper: ""Symbolic Brittleness in Sequence Models: on Systematic Generalization in Symbolic Mathematics"" Sequence models show amazing performance on many tasks. Does perfect test accuracy tell the full story? w/ @PeterWestTM, @JizeCao, @YejinChoinka  <LINK> <LINK> We consider symbolic integration, as it requires generalizing systematically beyond the test set and is verifiable. Despite high test accuracy, we find deficiencies in robustness, compositionality, and OOD generalization in a state-of-the-art MLE seq2seq model for this task. <LINK> We develop a genetic algorithm 🧬 (SAGGA) which automatically discovers (thousands of) failures that highlight each type of generalization, and test suites that perturb and compose validation problems & simple functions. <LINK> Robustness tells us whether the model systematically solves all problems in a neighborhood, typically governed by a generalizable pattern. The model is surprisingly brittle when test problems or simple functions are slightly changed. <LINK> Regarding compositionality, successfully integrating two functions did not imply that the model learned to integrate their sum  (recall the sum rule of integration <LINK>) <LINK> Moving further from the training distribution: Performance degrades for integers and problem sizes larger than those typically encountered in training (extrapolation) And functions not covered in the training set (""exploits"") <LINK> We also study the effect of increasing the search budget and whether it is a search problem alone -- check out the paper! <LINK> Stay tuned for code, which we plan to release.",https://arxiv.org/abs/2109.13986,"Neural sequence models trained with maximum likelihood estimation have led to breakthroughs in many tasks, where success is defined by the gap between training and test performance. However, their ability to achieve stronger forms of generalization remains unclear. We consider the problem of symbolic mathematical integration, as it requires generalizing systematically beyond the test set. We develop a methodology for evaluating generalization that takes advantage of the problem domain's structure and access to a verifier. Despite promising in-distribution performance of sequence-to-sequence models in this domain, we demonstrate challenges in achieving robustness, compositionality, and out-of-distribution generalization, through both carefully constructed manual test suites and a genetic algorithm that automatically finds large collections of failures in a controllable manner. Our investigation highlights the difficulty of generalizing well with the predominant modeling and learning approach, and the importance of evaluating beyond the test set, across different aspects of generalization. ","Symbolic Brittleness in Sequence Models: on Systematic Generalization in