text:   Symbolic Mathematics",8,"['new paper:\n\n""Symbolic Brittleness in Sequence Models: on Systematic Generalization in Symbolic Mathematics""\n\nSequence models show amazing performance on many tasks. Does perfect test accuracy tell the full story?\n\nw/ @PeterWestTM, @JizeCao, @YejinChoinka \n\n<LINK> <LINK>', 'We consider symbolic integration, as it requires generalizing systematically beyond the test set and is verifiable.\n\nDespite high test accuracy, we find deficiencies in robustness, compositionality, and OOD generalization in a state-of-the-art MLE seq2seq model for this task. https://t.co/ErGkN7ve9w', 'We develop a genetic algorithm 🧬 (SAGGA) which automatically discovers (thousands of) failures that highlight each type of generalization, and test suites that perturb and compose validation problems &amp; simple functions. https://t.co/ofikc0H0HQ', 'Robustness tells us whether the model systematically solves all problems in a neighborhood, typically governed by a generalizable pattern.\n\nThe model is surprisingly brittle when test problems or simple functions are slightly changed. https://t.co/qWAQ51ZPIC', 'Regarding compositionality, successfully integrating two functions did not imply that the model learned to integrate their sum \n\n(recall the sum rule of integration https://t.co/Oo2XOVaUKv) https://t.co/78QyBDe1sf', 'Moving further from the training distribution:\n\nPerformance degrades for integers and problem sizes larger than those typically encountered in training (extrapolation)\n\nAnd functions not covered in the training set (""exploits"") https://t.co/I57XDM5KrH', 'We also study the effect of increasing the search budget and whether it is a search problem alone -- check out the paper! https://t.co/wIpDcDRL0r', 'Stay tuned for code, which we plan to release.']",21,09,1605