Issue 1:
🌟 TAPAS (Weakly Supervised Table Parsing via Pre-Training)

Model description

TAPAS ArXiV Paper Google AI Blog, an approach to question answering over tables without generating logical forms. TAPAS trains from weak supervision, and predicts the denotation by selecting table cells and optionally applying a corresponding aggregation operator to such selection. TAPAS extends BERT’s architecture to encode tables as input, initializes from an effective joint pre-training of text segments and tables crawled from Wikipedia, and is trained end-to-end. We experiment with three different semantic parsing datasets, and find that TAPAS outperforms or rivals semantic parsing models by improving state-of-the-art accuracy on SQA from 55.1 to 67.2 and performing on par with the state-of-the-art on WIKISQL and WIKITQ, but with a simpler model architecture. We additionally find that transfer learning, which is trivial in our setting, from WIKISQL to WIKITQ, yields 48.7 accuracy, 4.2 points above the state-of-the-art.

Open source status

[ X ] the model implementation is available: GitHub repo

[ X ] the model weights are available: in the "Data" section at their GitHub page

[ X ] who are the authors: Jonathan Herzig, Paweł Krzysztof Nowak, Thomas Muller, Francesco Piccinno, Julian Martin Eisenschlos

Issue 2:
Environment info

transformers version: 4.3.2

Platform: Linux-5.4.0-66-generic-x86_64-with-debian-buster-sid

Python version: 3.7.9

PyTorch version (GPU?): 1.7.1 (True)

Tensorflow version (GPU?): not installed (NA)

Using GPU in script?:

Using distributed or parallel set-up in script?: No

Who can help

@sgugger, @patil-suraj

Information

Model I am using (Bert, XLNet ...): Bert

The problem arises when using:

the official example scripts: (give details below)

my own modified scripts: (give details below)

The tasks I am working on is:

an official GLUE/SQUaD task: (give the name)

my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

Run run_mlm.py with pdb break point at line 409.

see value for tokenized_datasets['train'][0]

ipdb> tokenized_datasets['train'][0]
{'attention_mask': ..., 'input_ids': [2, 11699, 6139, 23923, 6354, 6216, 3, 2, 124, 6149, 6228, 6164, 5125, 27479, 6228, 11699, 6139, 23923, 6354, 6216, 11961, 9121, 9804, 10602, 10293, 5328, 6721, 121, 6997, 15520, 16117, 10602, 11302, 5328, 6721, 121, 6997, 13014, 6177, 22111, 25147,
6189, 6106, 6315, 6110, 5084, 5158, 6291, 12836, 6108, 15512, 6726, 18139, 25596, 12701, 6291, 6106, 6315, 6616, 112, 5171, 113, 4363, 6380, 14946, 13769, 13928, 17518, 10216, 12299, 12571, 12850, 26355, 5315, 6457, 6117, 6303, 6213, 19358, 117, 122, 5201, 6361, 6211, 6377, 6312, 22259, 6631, 9268, 112, 10538, 113, 10728, 22278, 117, 14870, 13905, 142, 15214, 112, 10538, 113, 10728, 22278, 117, 14870, 14934, 7575, 10524, 186, 14921, 30912, 10758, 118, 10022, 680, 6275, 117, 181, 9860, 186, 14921, 30912, 10758, 136, 20107, 18973, 6358, 118, 3, 2, 10022, 7539, 25147, 6189, 116, 24690, 6915, 128, 116, 11134, 14216, 15650, 15373, 117, 13531, 20100, 117, 10028, 6132, 117, 127, 112, 124, 3866, 113, 15798, 15650, 15373, 117,
13531, 20100, 117, 9840, 6403, 117, 23999, 25006, 6131, 112, 14604, 113, 5084, 15466, 112, 5171, 113, 4363, 6380, 14946, 6213, 13579, 10393, 11023, 6187, 9218, 13014, 6236, 23534, 4587, 12827, 11069, 9422, 25686, 9112, 9220, 12112, 13538, 10112, 9427, 9215, 9260, 19036, 10393, 13514, 6187, 10112, 14882, 6130, 20150, 9279, 118, 3, 2, 14233, 15466, 18609, 16080, 118, 3, 2, 25147, 6189, 24864, 28007, 13581, 6149, 6228, 6164, 5125, 27479, 6228, 124, 5134, 16109, 28372, 3814, 6224, 20116, 6158, 12221, 6595, 105, 3, 2, 5466, 11794, 10393, 4700, 6224, 12819, 10694, 6187, 4671, 6628, 6119, 5502, 24468, 5743, 6125, 7111, 18452, 105, 3, 2, 14368, 6164, 5125, 27479, 6228, 6309, 6221, 6139, 4174, 6428, 6243, 167, 11699, 6139, 13295, 16589, 18619, 15924, 6131, 22573, 19515, 11914, 23850, 11914, 11512, 11346, 25763, 5134, 16109, 28372, 5134, 16109, 28372, 7031, 6114, 6114, 6626, 7020, 118, 3, 2, 5476, 6214, 116, 4121, 7788, 6107, 7788, 118, 4822, 6503, 6236, 15053, 4606, 9117, 118, 3, 2, 5676, 26156, 4973, 7088, 6114, 23122, 6114, 25444, 6422, 4218, 14246, 11920, 6147, 12097, 4011, 9117, 118, 3, 2, 4417, 25703, 9205, 9271, 9165, 19235, 4202, 6115, 14810, 6187, 19915, 6164, 4839, 6361, 11721, 4378, 7063, 15482, 9156, 11976, 30627, 9291, 3788, 19018, 20146, 4202, 9172, 118, 9868, 16712, 29634, 6115, 5206, 6203, 4469, 5294, 11019, 10250, 4973, 6284, 6203, 9691, 118, 3, 2, 9310, 10574, 5330, 9799, 11042, 13237, 6149, 9237, 118, 3, 2, 4378, 7063, 9126, 9271, 9242, 9822, 6236, 15472, 23041, 16135, 18119, 15314, 118, 3, 2, 5134, 6341, 6187, 9159, 14990, 5656, 4241, 14059, 6139, 4913, 12802, 9822, 9181, 9841, 4788, 18037, 116, 14059, 10825, 5087, 5178, 11699, 6213, 15171, 6333, 9242, 4645, 10212, 9691, 118, 3, 2, 9620, 10021, 11699, 6139, 17626, 6236, 12258, 4378, 7063, 6185, 16269, 26623, 30683, 12901, 118, 3, 2, 18276, 6130, 4378, 7063, 9126, 11699, 6187, 9159, 9242, 9319, 13793, 17451, 6260, 4184, 9242, 17561, 10724, 20756, 12126, 4789, 9172, 118, 9567, 4144, 12062, 10780, 5466, 6803, 4202, 4378, 7063, 9126, 4422, 12303, 6164, 9165, 10350, 571, 9467, 20853, 7177, 11947, 10441, 9270, 18480, 3795, 9207, 12098, 11725, 118], 'special_tokens_mask': ... , 'token_type_ids': ...}

When group_texts is maped to tokenized_datasets, whose examples already contain special tokens (e.g. [CLS] and [SEP]), the mapped results have the following format: [CLS] ... [SEP][CLS] ... [SEP][CLS] ... [SEP]

Expected behavior

The output of group_texts applied tokenized_datasets has the following format [CLS] ... [SEP] or [CLS] ... [SEP] ... [SEP].

The current input format is different from the original implementation, ELECTRA for example. Is this a trivial issue? I think the downstream task performance of the model pretrained with the current script could tell if this is a serious bug or not. Could someone share the results?