Getting Error from Default Data Collator while training Bert on SQUAD 2.0
Author: yanchao-yu
Pull request: no
Labels: name: wontfix

Hello, I'm a fresher playing with the transformers. I suppose to train the BERT model `bert-base-cased` on SQUAD 2.0, but having an error in `data_collator`: It shows an error of `TypeError: an integer is required`

Here is the detail:
```
Epoch:   0%|                                              | 0/2 [00:00<?, ?it/s]
Iteration:   0%|                                       | 0/4135 [00:00<?, ?it/s]
Epoch:   0%|                                              | 0/2 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/scratch/yyu/codebase/odqa/odqa/reader/reader_trainer.py", line 138, in <module>
    trainer.train()
  File "/scratch/yyu/codebase/odqa/odqa/reader/reader_trainer.py", line 119, in train
    self._trainer.train()
  File "/scratch/yyu/codebase/odqa/ENV/lib/python3.6/site-packages/transformers/trainer.py", line 456, in train
    for step, inputs in enumerate(epoch_iterator):
  File "/scratch/yyu/codebase/odqa/ENV/lib/python3.6/site-packages/tqdm/std.py", line 1130, in __iter__
    for obj in iterable:
  File "/scratch/yyu/codebase/odqa/ENV/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 363, in __next__
    data = self._next_data()
  File "/scratch/yyu/codebase/odqa/ENV/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 403, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/scratch/yyu/codebase/odqa/ENV/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/scratch/yyu/codebase/odqa/ENV/lib/python3.6/site-packages/transformers/data/data_collator.py", line 62, in default_data_collator
    batch[k] = torch.tensor([f[k] for f in features], dtype=torch.long)
TypeError: an integer is required (got type dict)
```



Here is my trainer configuration:
```
training_args = TrainingArguments(
            output_dir=self.output_dir,
            overwrite_output_dir=True,
            num_train_epochs=2,
            per_gpu_train_batch_size=32,
            # per_device_eval_batch_size=64,
            warmup_steps=500,
            weight_decay=0.01,
            # evaluate_during_training=True,
            save_steps=10_000,
            logging_dir='./logs',
        )
        self._trainer = Trainer(
            model=self._model,
            args=training_args,
            compute_metrics=self.compute_metrics,
            train_dataset=self.train_dataset,
            eval_dataset=self.test_dataset
        )
```

Does anyone know why it happens? And how can I fix this error? I don't know if this is one mistake in my code or from the transformers :-( Thanks for all help. 

Best

Comments:
First things first, please use ``` when copy-pasting stack traces or code (I've edited your post to use that) otherwise it's not really readable.

It's hard to know what's going on without knowing how you built your `train_dataset`. The data collator seems to have problems with it. The items should be dictionaries of list of ints/tensors. It seems there is some nested dictionary here.

Sorry about that I will remember to use it :-) And regarding the `train_dataset` I used `SquadV2Processor` to get examples for `train` and `eval`, and then convert them using `squad_convert_examples_to_features`: 

```
_processor = SquadV2Processor()
train_examples = _processor.get_train_examples(squad_dir, filename='SQuAD-v2.0-train.json')
train_dataset = squad_convert_examples_to_features(train_examples, self._tokenizer, max_seq_length=384, doc_stride=128, threads=2,max_query_length=64, is_training=True)
```


Is this correct? 

Could you print the result of `self.train_dataset[0]`? It would be helpful to see what the items look like.

i can't find any clues from the `self.train_dataset[0]`. It is a `SquadFeature` object, like: 

```
2020-08-12 20:48:57,663 -- [__main__:57][INFO]: train_dataset[0] input_ids: [101, 1706, 2292, 1225, 1103, 6567, 2090, 9273, 2845, 1107, 8109, 1107, 10111, 20500, 1699, 136, 102, 22182, 1193, 117, 1103, 1278, 1144, 170, 2336, 1959, 119, 1335, 4184, 1103, 4304, 4334, 112, 188, 2284, 10945, 1110, 170, 5404, 5921, 1104, 1103, 6567, 2090, 119, 13301, 1107, 1524, 1104, 1103, 4304, 4334, 1105, 4749, 1122, 117, 1110, 170, 7335, 5921, 1104, 4028, 1114, 1739, 1146, 14089, 5591, 1114, 1103, 7051, 107, 159, 21462, 1566, 24930, 2508, 152, 1306, 3965, 107, 119, 5893, 1106, 1103, 4304, 4334, 1110, 1103, 19349, 1104, 1103, 11373, 4641, 119, 13301, 1481, 1103, 171, 17506, 9538, 1110, 1103, 144, 10595, 2430, 117, 170, 14789, 1282, 1104, 8070, 1105, 9284, 119, 1135, 1110, 170, 16498, 1104, 1103, 176, 10595, 2430, 1120, 10111, 20500, 117, 1699, 1187, 1103, 6567, 2090, 25153, 1193, 1691, 1106, 2216, 17666, 6397, 3786, 1573, 25422, 13149, 1107, 8109, 119, 1335, 1103, 1322, 1104, 1103, 1514, 2797, 113, 1105, 1107, 170, 2904, 1413, 1115, 8200, 1194, 124, 11739, 1105, 1103, 3487, 17917, 114, 117, 1110, 170, 3014, 117, 2030, 2576, 5921, 1104, 2090, 119, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
2020-08-12 20:48:57,663 -- [__main__:58][INFO]: train_dataset[0] attention_mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
2020-08-12 20:48:57,663 -- [__main__:59][INFO]: train_dataset[0] token_type_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
2020-08-12 20:48:57,663 -- [__main__:60][INFO]: train_dataset[0] cls_index: 0
2020-08-12 20:48:57,663 -- [__main__:61][INFO]: train_dataset[0] p_mask: [0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
2020-08-12 20:48:57,663 -- [__main__:62][INFO]: train_dataset[0] example_index: 0
2020-08-12 20:48:57,663 -- [__main__:63][INFO]: train_dataset[0] unique_id: 1000000000
2020-08-12 20:48:57,663 -- [__main__:64][INFO]: train_dataset[0] paragraph_len: 163
2020-08-12 20:48:57,663 -- [__main__:65][INFO]: train_dataset[0] token_is_max_context: {17: True, 18: True, 19: True, 20: True, 21: True, 22: True, 23: True, 24: True, 25: True, 26: True, 27: True, 28: True, 29: True, 30: True, 31: True, 32: True, 33: True, 34: True, 35: True, 36: True, 37: True, 38: True, 39: True, 40: True, 41: True, 42: True, 43: True, 44: True, 45: True, 46: True, 47: True, 48: True, 49: True, 50: True, 51: True, 52: True, 53: True, 54: True, 55: True, 56: True, 57: True, 58: True, 59: True, 60: True, 61: True, 62: True, 63: True, 64: True, 65: True, 66: True, 67: True, 68: True, 69: True, 70: True, 71: True, 72: True, 73: True, 74: True, 75: True, 76: True, 77: True, 78: True, 79: True, 80: True, 81: True, 82: True, 83: True, 84: True, 85: True, 86: True, 87: True, 88: True, 89: True, 90: True, 91: True, 92: True, 93: True, 94: True, 95: True, 96: True, 97: True, 98: True, 99: True, 100: True, 101: True, 102: True, 103: True, 104: True, 105: True, 106: True, 107: True, 108: True, 109: True, 110: True, 111: True, 112: True, 113: True, 114: True, 115: True, 116: True, 117: True, 118: True, 119: True, 120: True, 121: True, 122: True, 123: True, 124: True, 125: True, 126: True, 127: True, 128: True, 129: True, 130: True, 131: True, 132: True, 133: True, 134: True, 135: True, 136: True, 137: True, 138: True, 139: True, 140: True, 141: True, 142: True, 143: True, 144: True, 145: True, 146: True, 147: True, 148: True, 149: True, 150: True, 151: True, 152: True, 153: True, 154: True, 155: True, 156: True, 157: True, 158: True, 159: True, 160: True, 161: True, 162: True, 163: True, 164: True, 165: True, 166: True, 167: True, 168: True, 169: True, 170: True, 171: True, 172: True, 173: True, 174: True, 175: True, 176: True, 177: True, 178: True, 179: True}
2020-08-12 20:48:57,663 -- [__main__:66][INFO]: train_dataset[0] tokens: ['[CLS]', 'To', 'whom', 'did', 'the', 'Virgin', 'Mary', 'allegedly', 'appear', 'in', '1858', 'in', 'Lou', '##rdes', 'France', '?', '[SEP]', 'Architectural', '##ly', ',', 'the', 'school', 'has', 'a', 'Catholic', 'character', '.', 'At', '##op', 'the', 'Main', 'Building', "'", 's', 'gold', 'dome', 'is', 'a', 'golden', 'statue', 'of', 'the', 'Virgin', 'Mary', '.', 'Immediately', 'in', 'front', 'of', 'the', 'Main', 'Building', 'and', 'facing', 'it', ',', 'is', 'a', 'copper', 'statue', 'of', 'Christ', 'with', 'arms', 'up', '##rai', '##sed', 'with', 'the', 'legend', '"', 'V', '##eni', '##te', 'Ad', 'Me', 'O', '##m', '##nes', '"', '.', 'Next', 'to', 'the', 'Main', 'Building', 'is', 'the', 'Basilica', 'of', 'the', 'Sacred', 'Heart', '.', 'Immediately', 'behind', 'the', 'b', '##asi', '##lica', 'is', 'the', 'G', '##rot', '##to', ',', 'a', 'Marian', 'place', 'of', 'prayer', 'and', 'reflection', '.', 'It', 'is', 'a', 'replica', 'of', 'the', 'g', '##rot', '##to', 'at', 'Lou', '##rdes', ',', 'France', 'where', 'the', 'Virgin', 'Mary', 'reputed', '##ly', 'appeared', 'to', 'Saint', 'Bern', '##ade', '##tte', 'So', '##ubi', '##rous', 'in', '1858', '.', 'At', 'the', 'end', 'of', 'the', 'main', 'drive', '(', 'and', 'in', 'a', 'direct', 'line', 'that', 'connects', 'through', '3', 'statues', 'and', 'the', 'Gold', 'Dome', ')', ',', 'is', 'a', 'simple', ',', 'modern', 'stone', 'statue', 'of', 'Mary', '.', '[SEP]']
2020-08-12 20:48:57,663 -- [__main__:67][INFO]: train_dataset[0] token_to_orig_map: {17: 0, 18: 0, 19: 0, 20: 1, 21: 2, 22: 3, 23: 4, 24: 5, 25: 6, 26: 6, 27: 7, 28: 7, 29: 8, 30: 9, 31: 10, 32: 10, 33: 10, 34: 11, 35: 12, 36: 13, 37: 14, 38: 15, 39: 16, 40: 17, 41: 18, 42: 19, 43: 20, 44: 20, 45: 21, 46: 22, 47: 23, 48: 24, 49: 25, 50: 26, 51: 27, 52: 28, 53: 29, 54: 30, 55: 30, 56: 31, 57: 32, 58: 33, 59: 34, 60: 35, 61: 36, 62: 37, 63: 38, 64: 39, 65: 39, 66: 39, 67: 40, 68: 41, 69: 42, 70: 43, 71: 43, 72: 43, 73: 43, 74: 44, 75: 45, 76: 46, 77: 46, 78: 46, 79: 46, 80: 46, 81: 47, 82: 48, 83: 49, 84: 50, 85: 51, 86: 52, 87: 53, 88: 54, 89: 55, 90: 56, 91: 57, 92: 58, 93: 58, 94: 59, 95: 60, 96: 61, 97: 62, 98: 62, 99: 62, 100: 63, 101: 64, 102: 65, 103: 65, 104: 65, 105: 65, 106: 66, 107: 67, 108: 68, 109: 69, 110: 70, 111: 71, 112: 72, 113: 72, 114: 73, 115: 74, 116: 75, 117: 76, 118: 77, 119: 78, 120: 79, 121: 79, 122: 79, 123: 80, 124: 81, 125: 81, 126: 81, 127: 82, 128: 83, 129: 84, 130: 85, 131: 86, 132: 87, 133: 87, 134: 88, 135: 89, 136: 90, 137: 91, 138: 91, 139: 91, 140: 92, 141: 92, 142: 92, 143: 93, 144: 94, 145: 94, 146: 95, 147: 96, 148: 97, 149: 98, 150: 99, 151: 100, 152: 101, 153: 102, 154: 102, 155: 103, 156: 104, 157: 105, 158: 106, 159: 107, 160: 108, 161: 109, 162: 110, 163: 111, 164: 112, 165: 113, 166: 114, 167: 115, 168: 115, 169: 115, 170: 116, 171: 117, 172: 118, 173: 118, 174: 119, 175: 120, 176: 121, 177: 122, 178: 123, 179: 123}
```


Ok. You need to remove some keys from it as it has way too many attributes (the failure comes from the fact the data collator is trying to build a tensor from the `token_is_max_context` fields).

The easiest way is probably to use the `SquadDataset` in `data.datasets.squad`, or you can just copy its `__getitem__` method and put it on your own dataset class.

Thanks Sylvain. But I found that my transformers library doesn't contain `data.datasets.squad` at all. I'm using `transformers==3.0.2`. It only contains `glue` and `language_model` two classes. 

Would you mind to explain how the error comes from the data collator? The `token_is_max_context` only contains a list of `True` flags. Is there a certain order (keys) of these features?

The data collator should receive a dictionary string to list of ints/tensors. The value associated to `token_is_max_context` is neither a list of int or a tensor, hence the error.
Note that you dataset should have items that are dictionaries with keys that are argument names your model will accept, which another reason why `token_is_max_context` needs to be removed.

Okay, it seems to make sense. There would be another question: does `transformers` process the original version of SQUAD dataset? I download the data from the SQUAD website, which shouldn't have any errors if it is the exact same as the one `transformers` used. Can I simply remove `token_is_max_context` from the `SquadFeatures` to solve this error? 

Also which version of `transformers` contains data.datasets.squad? I can't find it in 3.0.0, 2.9.0 and even 2.5.0. 

You need to only extract : input_ids, attention_mask and token_type_ids as the rest is probably not helpful to your model. AFAICT the file [data.datasets.squad](https://github.com/huggingface/transformers/blob/master/src/transformers/data/datasets/squad.py) has been there for a while, so you should have it in those versions.

Thanks for help, Sylvain. Let me give a try today. Hopefully, it works fine.  

Hi Sylvain,

Sorry for the interruption again. I've created a new class that only contains `input_ids`, `attention_mask` and `token_type_ids`, then the system gives an error: 

```
Traceback (most recent call last):
  File "/scratch/yyu/codebase/odqa/odqa/reader/reader_trainer.py", line 190, in <module>
    trainer.train()
  File "/scratch/yyu/codebase/odqa/odqa/reader/reader_trainer.py", line 168, in train
    self._trainer.train()
  File "/scratch/yyu/codebase/odqa/ENV/lib/python3.6/site-packages/transformers/trainer.py", line 375, in train
    for step, inputs in enumerate(epoch_iterator):
  File "/scratch/yyu/codebase/odqa/ENV/lib/python3.6/site-packages/tqdm/std.py", line 1130, in __iter__
    for obj in iterable:
  File "/scratch/yyu/codebase/odqa/ENV/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 363, in __next__
    data = self._next_data()
  File "/scratch/yyu/codebase/odqa/ENV/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 403, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/scratch/yyu/codebase/odqa/ENV/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/scratch/yyu/codebase/odqa/ENV/lib/python3.6/site-packages/transformers/data/data_collator.py", line 91, in collate_batch
    batch = self._tensorize_batch(examples)
  File "/scratch/yyu/codebase/odqa/ENV/lib/python3.6/site-packages/transformers/data/data_collator.py", line 99, in _tensorize_batch
    length_of_first = examples[0].size(0)
AttributeError: 'SimpleSquadFeature' object has no attribute 'size'
```

But I didn't find the `size()` function in the original class (SquadFeature) either. Do you know why it happens like that? 

And also I've installed transformers from 2.1.0 to 3.0.2 but can't import a class called `SquadDataset`, there is not a path [data.datasets.squad](https://github.com/huggingface/transformers/blob/master/src/transformers/data/datasets/squad.py), but `data.processor.squad`. When did transformers delete the `SquadDataset`? 

Thanks for help. 
Yanchao

Don't create a special class, just return a dictionary with those fields and it should work.

Hi,

I've had nearly the same problem as yy147 (the TypeError related to the default_data_collator), even though my train_dataset had items that were precisely a dictionary mapping the keys: 'input_ids', 'attention_mask', 'token_type_ids', to lists of ints, and 'label' to a torch.LongTensor.

Strangely I managed to solve the problem by copying the transformers.data.data_collator.default_data_collator into my own code and letting the Trainer use that.

Python version 3.6.8
torch version 1.6.0
transformers version 3.0.2

Hope it helps,
Gijs


> Don't create a special class, just return a dictionary with those fields and it should work.

Thanks Sylvain. I've found the `SquadDateset`, which is surprisingly not included in any transformers versions if you run `pip install transfermers==3.0.2`. I can only find it if I install `transformers` from the source. It seems something needs to be fixed.

Best,
Yanchao 

Thanks for help, Gijs. I will try to copy and paste it later. It is a wired situation. 

Hi @yy147 , I am also getting a similar error:
```    
length_of_first = examples[0].size(0)
AttributeError: 'dict' object has no attribute 'size
```

Have you managed to fix your error?

Hi @gungor2, I found another source code extended from transformers example `https://github.com/kamalkraj/BERT-SQuAD/blob/master/utils.py`. It gives a great example to solve the problem of `squad_examples_to_features`. It works well for me. Hope it is helpful for you. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hi, I have a related issue I am trying to train the [**TrOCR**](https://huggingface.co/microsoft/trocr-large-handwritten) model on my own data what I tried: 
```
# the class for loading data has function :
def __getitem__(self, idx):
        file_name = self.df['file_name'][idx]
        text = self.df['text'][idx]
        # prepare image (i.e. resize + normalize)
        image = Image.open(self.root_dir + file_name).convert("RGB")
        pixel_values = self.processor(image, return_tensors="pt").pixel_values
        labels = self.processor.tokenizer(text, padding="max_length",
                                                 max_length=self.max_target_length).input_ids    
        labels = [label if label != self.processor.tokenizer.pad_token_id else -100 for label in
                  labels]

        encoding = {"pixel_values": pixel_values.squeeze(), "labels": torch.tensor(labels)}
        return encoding
training_args = Seq2SeqTrainingArguments(
        num_train_epochs=25, 
        learning_rate=5e-5,
        predict_with_generate=True,
        evaluation_strategy="steps",
        per_device_train_batch_size=64,
        per_device_eval_batch_size=64,
        fp16=True,    
        output_dir="/1/large/",    
        logging_steps=100,
        save_steps=2000,
        eval_steps=5000,
        )

    trainer = Seq2SeqTrainer( model=model, tokenizer=processor.feature_extractor,
        args=training_args, compute_metrics=compute_metrics, train_dataset=train_dataset,
        eval_dataset=eval_dataset,  data_collator=default_data_collator, )
```
the feed input image to the model  has a `height of 64 `fixed for all and different `width`  
The issue I see is: where the training stops after a few hours
```
Traceback (most recent call last):
  File "train.py", line 191, in <module>
    main()
  File "train.py", line 173, in main
    trainer.train()
  File "/home/user/venv/lib/python3.8/site-packages/transformers/trainer.py", line 1521, in train
    return inner_training_loop(
  File "/home/user/venv/lib/python3.8/site-packages/transformers/trainer.py", line 1737, in _inner_training_loop
    for step, inputs in enumerate(epoch_iterator):
  File "/home/user/venv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/home/user/venv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 475, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/user/venv/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/home/user/venv/lib/python3.8/site-packages/transformers/trainer_utils.py", line 696, in __call__
    return self.data_collator(features)
  File "/home/user/venv/lib/python3.8/site-packages/transformers/data/data_collator.py", line 67, in default_data_collator
    return torch_default_data_collator(features)
  File "/home/user/venv/lib/python3.8/site-packages/transformers/data/data_collator.py", line 129, in torch_default_data_collator
    batch[k] = torch.stack([f[k] for f in features])
RuntimeError: stack expects each tensor to be equal size, but got [128] at entry 0 and [139] at entry 19
  1%|▊                                                                                                         | 1356/166025 [40:59<82:58:15,  1.81s/it]
```
Transformer version: 4.22.2
@sgugger  @NielsRogge  @



Hi,

It looks like your target texts aren't having the same length. You need to not only pad but also set `truncation=True` to make sure all texts have 128 tokens.