IndexError when training with GLUE dataset using pretrained from scratch ELECTRA.
Author: saiefulEZO
Pull request: no

Hi guys, 

I have a problem in which i'm not sure how to solve. Short story is I pretrained ELECTRA from scratch, now I wanted to train and test with GLUE. I converted ELECTRA tf checkpoint to pytorch using `transformers/src/transformers/models/electra/convert_electra_original_tf_checkpoint_to_pytorch.py` then I run the GLUE test with this `transformers/examples/pytorch/text-classification/run_glue.py`. At 1st I ran with electra_small model, its working fine. However when I ran it with my model that I have pretrained from scratch it produced this error. 

python /Users/nlplabo/tensorflow-test/transformers/examples/pytorch/text-classification/run_glue.py\
  --model_name_or_path "/Users/nlplabo/Desktop/electra_pos" \
  --task_name $TASK_NAME \
  --ignore_mismatched_sizes true \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_gpu_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3.0 \
  --output_dir "/Users/nlplabo/Desktop/electra_pos/result/testing"\


05/28/2023 18:03:09 - WARNING - __main__ - Process rank: 0, device: cpu, n_gpu: 0distributed training: True, 16-bits training: False
Downloading and preparing dataset glue/cola to /Users/nlplabo/.cache/huggingface/datasets/glue/cola/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad...
Dataset glue downloaded and prepared to /Users/nlplabo/.cache/huggingface/datasets/glue/cola/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad. Subsequent calls will reuse this data.
100%|███████████████████████████████████████████| 3/3 [00:00<00:00, 2262.71it/s]
[WARNING|modeling_utils.py:3175] 2023-05-28 18:03:13,062 >> Some weights of the model checkpoint at /Users/nlplabo/Desktop/retrying/electra_pos were not used when initializing ElectraForSequenceClassification: ['discriminator_predictions.dense.weight', 'discriminator_predictions.dense_prediction.bias', 'discriminator_predictions.dense.bias', 'discriminator_predictions.dense_prediction.weight']
- This IS expected if you are initializing ElectraForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ElectraForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
[WARNING|modeling_utils.py:3187] 2023-05-28 18:03:13,062 >> Some weights of ElectraForSequenceClassification were not initialized from the model checkpoint at /Users/nlplabo/Desktop/retrying/electra_pos and are newly initialized: ['classifier.out_proj.bias', 'classifier.dense.weight', 'classifier.out_proj.weight', 'classifier.dense.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
/Users/nlplabo/opt/miniconda3/envs/tftransformer/lib/python3.8/site-packages/transformers/optimization.py:407: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning
  warnings.warn(
  0%|                                           | 1/804 [00:01<16:47,  1.25s/it]Traceback (most recent call last):
  File "/Users/nlplabo/tensorflow-test/transformers/examples/pytorch/text-classification/run_glue.py", line 622, in <module>
    main()
  File "/Users/nlplabo/tensorflow-test/transformers/examples/pytorch/text-classification/run_glue.py", line 530, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/Users/nlplabo/opt/miniconda3/envs/tftransformer/lib/python3.8/site-packages/transformers/trainer.py", line 1664, in train
    return inner_training_loop(
  File "/Users/nlplabo/opt/miniconda3/envs/tftransformer/lib/python3.8/site-packages/transformers/trainer.py", line 1940, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/Users/nlplabo/opt/miniconda3/envs/tftransformer/lib/python3.8/site-packages/transformers/trainer.py", line 2735, in training_step
    loss = self.compute_loss(model, inputs)
  File "/Users/nlplabo/opt/miniconda3/envs/tftransformer/lib/python3.8/site-packages/transformers/trainer.py", line 2767, in compute_loss
    outputs = model(**inputs)
  File "/Users/nlplabo/opt/miniconda3/envs/tftransformer/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/nlplabo/opt/miniconda3/envs/tftransformer/lib/python3.8/site-packages/transformers/models/electra/modeling_electra.py", line 1004, in forward
    discriminator_hidden_states = self.electra(
  File "/Users/nlplabo/opt/miniconda3/envs/tftransformer/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/nlplabo/opt/miniconda3/envs/tftransformer/lib/python3.8/site-packages/transformers/models/electra/modeling_electra.py", line 908, in forward
    hidden_states = self.embeddings(
  File "/Users/nlplabo/opt/miniconda3/envs/tftransformer/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/nlplabo/opt/miniconda3/envs/tftransformer/lib/python3.8/site-packages/transformers/models/electra/modeling_electra.py", line 201, in forward
    inputs_embeds = self.word_embeddings(input_ids)
  File "/Users/nlplabo/opt/miniconda3/envs/tftransformer/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/nlplabo/opt/miniconda3/envs/tftransformer/lib/python3.8/site-packages/torch/nn/modules/sparse.py", line 162, in forward
    return F.embedding(
  File "/Users/nlplabo/opt/miniconda3/envs/tftransformer/lib/python3.8/site-packages/torch/nn/functional.py", line 2210, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self
  0%|                                           | 1/804 [00:01<17:31,  1.31s/it]


I tried to search for solutions out there, but I couldn't get it to work. Last choice is to pretrained again from scratch, but that would take too much time as my pc was not that strong. So I hope to make this model works. Thank you. 

Comments:
The problem seems to be that you are using token IDs that are not accepted by your model. Are you sure your tokenzier length and model embedding size match?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.