text: 361,331,1312120926985576449,446694758,Julian Eisenschlos,"Entailment has been studied in depth for textual premises, but the case with structured data like tables or even HTML can have many applications in the wild.  We tackle this in our latest #EMNLP2020 Findings paper <LINK> with Syrine Krichene and @muelletm 1/5 We extend TAPAS (Herzig et al, 2020), originally pretrained with MLM, to predict if a table entails or refutes a sentence and eval on TabFact (Chen et al, 2020). We introduce 2 novel pretraining binary-classification tasks called Counterfactual and Synthetic, shown in image. 2/5 <LINK> Counterfactual examples are created by swapping entities that appear in both a table and a sentence for a plausible alternative: they are realistic but simple. Synthetic ones are sampled from a small pCFG based on the values of a real table: they improve numerical reasoning. 3/5 <LINK> Pretraining with these 2 tasks, we improve SOTA by ~10pts on TabFact and, interestingly, also get a new SOTA on the table QA task SQA (Iyyer et al, 2017). This results hold even with fraction of the data, and is only 2 points below a strong baseline with no data at all! 4/5 <LINK> Finally, we investigate how to deal with large tables by selecting which parts of the input to pass through the model using simple heuristics. We can can get 2x speed-ups with ~1pt acc drop, or 4x still above prior art. Code and models coming soon at <LINK> 5/5",http://arxiv.org/abs/2010.00571,"Table entailment, the binary classification task of finding if a sentence is supported or refuted by the content of a table, requires parsing language and table structure as well as numerical and discrete reasoning. While there is extensive work on textual entailment, table entailment is less well studied. We adapt TAPAS (Herzig et al., 2020), a table-based BERT model, to recognize entailment. Motivated by the benefits of data augmentation, we create a balanced dataset of millions of automatically created training examples which are learned in an intermediate step prior to fine-tuning. This new data is not only useful for table entailment, but also for SQA (Iyyer et al., 2017), a sequential table QA task. To be able to use long examples as input of BERT models, we evaluate table pruning techniques as a pre-processing step to drastically improve the training and prediction efficiency at a moderate drop in accuracy. The different methods set the new state-of-the-art on the TabFact (Chen et al., 2020) and SQA datasets. ",Understanding tables with intermediate pre-training,5,"['Entailment has been studied in depth for textual premises, but the case with structured data like tables or even HTML can have many applications in the wild. \n\nWe tackle this in our latest #EMNLP2020 Findings paper <LINK> with Syrine Krichene and @muelletm\n\n1/5', 'We extend TAPAS (Herzig et al, 2020), originally pretrained with MLM, to predict if a table entails or refutes a sentence and eval on TabFact (Chen et al, 2020). We introduce 2 novel pretraining binary-classification tasks called Counterfactual and Synthetic, shown in image.\n\n2/5 https://t.co/4s84vzcut1', 'Counterfactual examples are created by swapping entities that appear in both a table and a sentence for a plausible alternative: they are realistic but simple. Synthetic ones are sampled from a small pCFG based on the values of a real table: they improve numerical reasoning.\n\n3/5 https://t.co/f17pVpWqiu', 'Pretraining with these 2 tasks, we improve SOTA by ~10pts on TabFact and, interestingly, also get a new SOTA on the table QA task SQA (Iyyer et al, 2017). This results hold even with fraction of the data, and is only 2 points below a strong baseline with no data at all!\n\n4/5 https://t.co/RAakCIRKGn', 'Finally, we investigate how to deal with large tables by selecting which parts of the input to pass through the model using simple heuristics. We can can get 2x speed-ups with ~1pt acc drop, or 4x still above prior art. Code and models coming soon at https://t.co/1VxCSrSunr\n\n5/5']",20,10,1377