Distilling zero-shot classification: Assertion `srcIndex < srcSelectDimSize` failed.
Author: dbs700
Pull request: no

## Environment info

tokenizers          0.10.2
torch               1.8.1+cu111
transformers        4.5.1
datasets            1.6.1
IPython             7.19.0
jupyter_client      6.1.7
jupyter_core        4.6.3
jupyterlab          2.2.6
notebook            6.1.4

Python 3.8.5 (default, Sep  3 2020, 21:29:08) [MSC v.1916 64 bit (AMD64)]
Windows-10-10.0.17763-SP0

## Issue
I'm trying to train a distill_classifier on a custom dataset with 40.000 rows of text, 10 labels. 

```
 !python ./distill_classifier.py \
--overwrite_output_dir \
--data_file email.txt \
--class_names_file class_names.txt \
--hypothesis_template "This text is about {}." \
--student_name_or_path distilbert-base-uncased \
--output_dir ./distilbert-base-uncased-student-large
```

At first, I got an error regarding max_length, I changed the script by adding truncation=True to tokenizer, and it resulted in another error. Strange thing is that I'm able to train it on a small custom dataset (~200 rows of text data, 10 labels) but with bigger data, it doesn't work so well. The output:

``` 
[INFO|configuration_utils.py:491] 2021-06-30 10:00:54,363 >> loading configuration file https://huggingface.co/roberta-large-mnli/resolve/main/config.json from cache at C:\Users\d/.cache\huggingface\transformers\fab42bdbd5cb5e6ff7cabeb9bcc12728f56022f50b9644a3079904564f2bc704.ddc5961cccf081d6ca7f4f58ee119c21895aa9b19f0044f01954cd2ff42fefcb
[INFO|configuration_utils.py:527] 2021-06-30 10:00:54,364 >> Model config RobertaConfig {
  "_num_labels": 3,
  "architectures": [
    "RobertaForSequenceClassification"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "eos_token_id": 2,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 1024,
  "id2label": {
    "0": "CONTRADICTION",
    "1": "NEUTRAL",
06/30/2021 10:00:53 - WARNING - __main__ -   Process rank: -1, device: cuda:0, n_gpu: 3distributed training: False, 16-bits training: False
06/30/2021 10:00:53 - INFO - __main__ -   Training/evaluation parameters DistillTrainingArguments(output_dir='./distilbert-base-uncased-notino-student-large', overwrite_output_dir=True, do_train=True, do_eval=True, do_predict=False, evaluation_strategy=<IntervalStrategy.NO: 'no'>, prediction_loss_only=False, per_device_train_batch_size=32, per_device_eval_batch_size=128, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, eval_accumulation_steps=None, learning_rate=5e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=1.0, max_steps=-1, lr_scheduler_type=<SchedulerType.LINEAR: 'linear'>, warmup_ratio=0.0, warmup_steps=0, logging_dir='runs\\Jun30_10-00-53_dcvmdwhanl03', logging_strategy=<IntervalStrategy.STEPS: 'steps'>, logging_first_step=False, logging_steps=500, save_strategy=<IntervalStrategy.STEPS: 'steps'>, save_steps=500, save_total_limit=0, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', fp16_backend='auto', fp16_full_eval=False, local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=500, dataloader_num_workers=0, past_index=-1, run_name='./distilbert-base-uncased-notino-student-large', disable_tqdm=False, remove_unused_columns=True, label_names=None, load_best_model_at_end=False, metric_for_best_model=None, greater_is_better=None, ignore_data_skip=False, sharded_ddp=[], deepspeed=None, label_smoothing_factor=0.0, adafactor=False, group_by_length=False, length_column_name='length', report_to=['tensorboard', 'wandb'], ddp_find_unused_parameters=None, dataloader_pin_memory=True, skip_memory_metrics=False, mp_parameters='')
06/30/2021 10:00:53 - INFO - __main__ -   Generating predictions from zero-shot teacher model
06/30/2021 10:03:10 - INFO - __main__ -   Initializing student model
06/30/2021 10:03:58 - INFO - __main__ -   Training student model on teacher predictions



    "2": "ENTAILMENT"
  },
  "initializer_range": 0.02,
  "intermediate_size": 4096,
  "label2id": {
    "CONTRADICTION": 0,
    "ENTAILMENT": 2,
    "NEUTRAL": 1
  },
  "layer_norm_eps": 1e-05,
  "max_position_embeddings": 514,
  "model_type": "roberta",
  "num_attention_heads": 16,
  "num_hidden_layers": 24,
  "pad_token_id": 1,
  "position_embedding_type": "absolute",
  "transformers_version": "4.5.1",
  "type_vocab_size": 1,
  "use_cache": true,
  "vocab_size": 50265
}

[INFO|modeling_utils.py:1052] 2021-06-30 10:00:54,790 >> loading weights file https://huggingface.co/roberta-large-mnli/resolve/main/pytorch_model.bin from cache at C:\Users\o/.cache\huggingface\transformers\63cbd98723b89863bcd86a8002e823de3004a139513559246690c65521cdc9b9.38ef55c51c84ab2e78e5a0e2ea9c25830fd074df70d2f10071eb9a1bc1586ca0
[WARNING|modeling_utils.py:1159] 2021-06-30 10:01:01,810 >> Some weights of the model checkpoint at roberta-large-mnli were not used when initializing RobertaForSequenceClassification: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']
- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
[INFO|modeling_utils.py:1176] 2021-06-30 10:01:01,810 >> All the weights of RobertaForSequenceClassification were initialized from the model checkpoint at roberta-large-mnli.
If your task is similar to the task the model of the checkpoint was trained on, you can already use RobertaForSequenceClassification for predictions without further training.
[INFO|configuration_utils.py:491] 2021-06-30 10:01:04,545 >> loading configuration file https://huggingface.co/roberta-large-mnli/resolve/main/config.json from cache at C:\Users\o/.cache\huggingface\transformers\fab42bdbd5cb5e6ff7cabeb9bcc12728f56022f50b9644a3079904564f2bc704.ddc5961cccf081d6ca7f4f58ee119c21895aa9b19f0044f01954cd2ff42fefcb
[INFO|configuration_utils.py:527] 2021-06-30 10:01:04,546 >> Model config RobertaConfig {
  "_num_labels": 3,
  "architectures": [
    "RobertaForSequenceClassification"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "eos_token_id": 2,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 1024,
  "id2label": {
    "0": "CONTRADICTION",
    "1": "NEUTRAL",
    "2": "ENTAILMENT"
  },
  "initializer_range": 0.02,
  "intermediate_size": 4096,
  "label2id": {
    "CONTRADICTION": 0,
    "ENTAILMENT": 2,
    "NEUTRAL": 1
  },
  "layer_norm_eps": 1e-05,
  "max_position_embeddings": 514,
  "model_type": "roberta",
  "num_attention_heads": 16,
  "num_hidden_layers": 24,
  "pad_token_id": 1,
  "position_embedding_type": "absolute",
  "transformers_version": "4.5.1",
  "type_vocab_size": 1,
  "use_cache": true,
  "vocab_size": 50265
}

[INFO|tokenization_utils_base.py:1707] 2021-06-30 10:01:07,087 >> loading file https://huggingface.co/roberta-large-mnli/resolve/main/vocab.json from cache at C:\Users\o/.cache\huggingface\transformers\64a1d72b2bd05b0aff1a4dd9e7a90a6eea0312b4f914e80b0a923aa8f72219bd.d67d6b367eb24ab43b08ad55e014cf254076934f71d832bbab9ad35644a375ab
[INFO|tokenization_utils_base.py:1707] 2021-06-30 10:01:07,087 >> loading file https://huggingface.co/roberta-large-mnli/resolve/main/merges.txt from cache at C:\Users\o/.cache\huggingface\transformers\425529714b758f50b6d3f93f8093d859856fd41cf1cec7c8edf2ab44aee632b6.5d12962c5ee615a4c803841266e9c3be9a691a924f72d395d3a6c6c81157788b
[INFO|tokenization_utils_base.py:1707] 2021-06-30 10:01:07,087 >> loading file https://huggingface.co/roberta-large-mnli/resolve/main/tokenizer.json from cache at C:\Users\o/.cache\huggingface\transformers\d077eac6b48c43618a441cba6eab600a5cc6383b98e7eada6d1ad4d3f3cc457e.fc9576039592f026ad76a1c231b89aee8668488c671dfbe6616bab2ed298d730
[INFO|tokenization_utils_base.py:1707] 2021-06-30 10:01:07,087 >> loading file https://huggingface.co/roberta-large-mnli/resolve/main/added_tokens.json from cache at None
[INFO|tokenization_utils_base.py:1707] 2021-06-30 10:01:07,087 >> loading file https://huggingface.co/roberta-large-mnli/resolve/main/special_tokens_map.json from cache at None
[INFO|tokenization_utils_base.py:1707] 2021-06-30 10:01:07,087 >> loading file https://huggingface.co/roberta-large-mnli/resolve/main/tokenizer_config.json from cache at None

  0%|          | 0/147 [00:00<?, ?it/s]
...
100%|##########| 147/147 [02:03<00:00,  1.19it/s]
[INFO|configuration_utils.py:491] 2021-06-30 10:03:10,735 >> loading configuration file https://huggingface.co/distilbert-base-uncased/resolve/main/config.json from cache at C:\Users\o/.cache\huggingface\transformers\23454919702d26495337f3da04d1655c7ee010d5ec9d77bdb9e399e00302c0a1.d423bdf2f58dc8b77d5f5d18028d7ae4a72dcfd8f468e81fe979ada957a8c361
[INFO|configuration_utils.py:527] 2021-06-30 10:03:10,736 >> Model config DistilBertConfig {
  "activation": "gelu",
  "architectures": [
    "DistilBertForMaskedLM"
  ],
  "attention_dropout": 0.1,
  "dim": 768,
  "dropout": 0.1,
  "hidden_dim": 3072,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1",
    "2": "LABEL_2",
    "3": "LABEL_3",
    "4": "LABEL_4",
    "5": "LABEL_5",
    "6": "LABEL_6",
    "7": "LABEL_7",
    "8": "LABEL_8",
    "9": "LABEL_9",
    "10": "LABEL_10"
  },
  "initializer_range": 0.02,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1,
    "LABEL_10": 10,
    "LABEL_2": 2,
    "LABEL_3": 3,
    "LABEL_4": 4,
    "LABEL_5": 5,
    "LABEL_6": 6,
    "LABEL_7": 7,
    "LABEL_8": 8,
    "LABEL_9": 9
  },
  "max_position_embeddings": 512,
  "model_type": "distilbert",
  "n_heads": 12,
  "n_layers": 6,
  "pad_token_id": 0,
  "qa_dropout": 0.1,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": false,
  "tie_weights_": true,
  "transformers_version": "4.5.1",
  "vocab_size": 30522
}

[INFO|modeling_utils.py:1052] 2021-06-30 10:03:11,160 >> loading weights file https://huggingface.co/distilbert-base-uncased/resolve/main/pytorch_model.bin from cache at C:\Users\o/.cache\huggingface\transformers\9c169103d7e5a73936dd2b627e42851bec0831212b677c637033ee4bce9ab5ee.126183e36667471617ae2f0835fab707baa54b731f991507ebbb55ea85adb12a
[WARNING|modeling_utils.py:1159] 2021-06-30 10:03:12,327 >> Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: ['vocab_transform.weight', 'vocab_transform.bias', 'vocab_layer_norm.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight', 'vocab_projector.bias']
- This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
[WARNING|modeling_utils.py:1170] 2021-06-30 10:03:12,328 >> Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['pre_classifier.weight', 'pre_classifier.bias', 'classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
[INFO|configuration_utils.py:491] 2021-06-30 10:03:12,750 >> loading configuration file https://huggingface.co/distilbert-base-uncased/resolve/main/config.json from cache at C:\Users\o/.cache\huggingface\transformers\23454919702d26495337f3da04d1655c7ee010d5ec9d77bdb9e399e00302c0a1.d423bdf2f58dc8b77d5f5d18028d7ae4a72dcfd8f468e81fe979ada957a8c361
[INFO|configuration_utils.py:527] 2021-06-30 10:03:12,750 >> Model config DistilBertConfig {
  "activation": "gelu",
  "architectures": [
    "DistilBertForMaskedLM"
  ],
  "attention_dropout": 0.1,
  "dim": 768,
  "dropout": 0.1,
  "hidden_dim": 3072,
  "initializer_range": 0.02,
  "max_position_embeddings": 512,
  "model_type": "distilbert",
  "n_heads": 12,
  "n_layers": 6,
  "pad_token_id": 0,
  "qa_dropout": 0.1,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": false,
  "tie_weights_": true,
  "transformers_version": "4.5.1",
  "vocab_size": 30522
}

[INFO|tokenization_utils_base.py:1707] 2021-06-30 10:03:14,877 >> loading file https://huggingface.co/distilbert-base-uncased/resolve/main/vocab.txt from cache at C:\Users\o/.cache\huggingface\transformers\0e1bbfda7f63a99bb52e3915dcf10c3c92122b827d92eb2d34ce94ee79ba486c.d789d64ebfe299b0e416afc4a169632f903f693095b4629a7ea271d5a0cf2c99
[INFO|tokenization_utils_base.py:1707] 2021-06-30 10:03:14,877 >> loading file https://huggingface.co/distilbert-base-uncased/resolve/main/tokenizer.json from cache at C:\Users\o/.cache\huggingface\transformers\75abb59d7a06f4f640158a9bfcde005264e59e8d566781ab1415b139d2e4c603.7f2721073f19841be16f41b0a70b600ca6b880c8f3df6f3535cbc704371bdfa4
[INFO|tokenization_utils_base.py:1707] 2021-06-30 10:03:14,877 >> loading file https://huggingface.co/distilbert-base-uncased/resolve/main/added_tokens.json from cache at None
[INFO|tokenization_utils_base.py:1707] 2021-06-30 10:03:14,877 >> loading file https://huggingface.co/distilbert-base-uncased/resolve/main/special_tokens_map.json from cache at None
[INFO|tokenization_utils_base.py:1707] 2021-06-30 10:03:14,877 >> loading file https://huggingface.co/distilbert-base-uncased/resolve/main/tokenizer_config.json from cache at C:\Users\o/.cache\huggingface\transformers\8c8624b8ac8aa99c60c912161f8332de003484428c47906d7ff7eb7f73eecdbb.20430bd8e10ef77a7d2977accefe796051e01bc2fc4aa146bc862997a1a15e79

  0%|          | 0/1282 [00:00<?, ?ex/s][WARNING|tokenization_utils_base.py:3136] 2021-06-30 10:03:14,926 >> Token indices sequence length is longer than the specified maximum sequence length for this model (527 > 512). Running this sequence through the model will result in indexing errors

 20%|#9        | 250/1282 [00:00<00:00, 2475.26ex/s]
 38%|###7      | 484/1282 [00:00<00:00, 2433.06ex/s]
 57%|#####6    | 729/1282 [00:00<00:00, 2438.11ex/s]
 78%|#######7  | 999/1282 [00:00<00:00, 2504.20ex/s]
 94%|#########4| 1206/1282 [00:00<00:00, 2355.94ex/s]
100%|##########| 1282/1282 [00:00<00:00, 2387.33ex/s]
[INFO|trainer.py:490] 2021-06-30 10:03:58,139 >> The following columns in the training set  don't have a corresponding argument in `DistilBertForSequenceClassification.forward` and have been ignored: text.
[INFO|trainer.py:1013] 2021-06-30 10:03:58,433 >> ***** Running training *****
[INFO|trainer.py:1014] 2021-06-30 10:03:58,439 >>   Num examples = 1282
[INFO|trainer.py:1015] 2021-06-30 10:03:58,444 >>   Num Epochs = 1
[INFO|trainer.py:1016] 2021-06-30 10:03:58,449 >>   Instantaneous batch size per device = 32
[INFO|trainer.py:1017] 2021-06-30 10:03:58,455 >>   Total train batch size (w. parallel, distributed & accumulation) = 96
[INFO|trainer.py:1018] 2021-06-30 10:03:58,461 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:1019] 2021-06-30 10:03:58,466 >>   Total optimization steps = 14
[INFO|integrations.py:586] 2021-06-30 10:03:59,225 >> Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true"
wandb: Currently logged in as: dbs700 (use `wandb login --relogin` to force relogin)
wandb: wandb version 0.10.33 is available!  To upgrade, please run:
wandb:  $ pip install wandb --upgrade
wandb: Tracking run with wandb version 0.10.30
wandb: Syncing run ./distilbert-base-uncased-notino-student-large
wandb:  View project at https://wandb.ai/dbs700/huggingface
wandb:  View run at https://wandb.ai/dbs700/huggingface/runs/3aegl7qi
wandb: Run data is saved locally in C:\Users\o\wandb\run-20210630_100419-3aegl7qi
wandb: Run `wandb offline` to turn off syncing.

  0%|          | 0/14 [00:00<?, ?it/s]C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/Indexing.cu:662: block: [251,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/Indexing.cu:662: block: [251,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/Indexing.cu:662: block: [251,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/Indexing.cu:662: block: [251,0,0], thread: [35,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
```
## Who can help

@LysandreJik
@sgugger
@joeddav

Comments:
Found a solution by reducing the input size of the text, and limiting it by the model's max_position_embeddings. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.