How can I initialize RobertaForSequenceClassification empty?
Author: raj5287
Pull request: no

I am having my own custom dataset which is completely different from the one with roberta. Now using ByteLevelBPETokenizer, I am creating vocab.json and merges.txt . I am using these two to files to initialize RobertaTokenizerFast for encoding my corpus. Now I am training a RobertaForSequenceClassification for a binary classification problem. When I am initializing RobertaForSequenceClassification with any of the pre-trained models I am getting 
`IndexError: index out of range in self` on the RAM while on GPU I am getting `RuntimeError: CUDA error: an illegal memory access was encountered` . I have followed other [issues ](https://github.com/pytorch/pytorch/issues/21819) but of no help. My understanding since I am creating my own vocabulary, some of the tokens are not in pre-trained model. So is there a way to initialize RobertaForSequenceClassification empty or to train this classification model for my dataset?

Comments:
Hi! If you want to initialize a new `RobertaForSequenceClassification` model you can do so as such:

```py
from transformers import RobertaForSequenceClassification, RobertaConfig

config = RobertaConfig(
    # put the args you need, like the vocab size
    vocab_size=100
)

model = RobertaForSequenceClassification(config)
```

I had the same problem when i try to finetune RoBERTa model with my own tokenizer as @raj5287 did. I have already updated all packages (pytorch, transformers)

Code:

> tokenizer = RobertaTokenizerFast.from_pretrained(tokenizer_path)
> model = RobertaForMaskedLM.from_pretrained(self.config)
>         dataset = LineByLineTextDataset(
>             tokenizer=tokenizer,
>             file_path=input_text_path,
>             block_size=512,
>         )
> 
>         data_collator = DataCollatorForLanguageModeling(
>             tokenizer=tokenizer, mlm=True, mlm_probability=0.15
>         )
>         training_args = TrainingArguments(
>             output_dir=output_path,
>             overwrite_output_dir=True,
>             num_train_epochs=3,
>             per_gpu_train_batch_size=8,
>             save_steps=50_000,
>             save_total_limit=2,
>         )
> 
>         trainer = Trainer(
>             model=model,
>             args=training_args,
>             data_collator=data_collator,
>             train_dataset=dataset,
>             prediction_loss_only=True
>         )
>         trainer.train()
>         trainer.save_model(output_path)
> 

Output:

File "Fine_tuning_tokenizer.py", line 106, in <module>
    main(sys.argv[1:])
  File "Fine_tuning_tokenizer.py", line 94, in main
    modelRoberta.fine_tune_LM(inputfile, outputDir,tokenizer_path=outputDir)
  File "/sentiment-embeddings/projeto-modularizar/src/uff/ic/mell/sentimentembedding/modelos/modelo_roberta.py", line 58, in fine_tune_LM
    trainer.train()
  File "/home/apps/anaconda3/lib/python3.7/site-packages/transformers/trainer.py", line 549, in train
    tr_loss += self.training_step(model, inputs)
  File "/home/apps/anaconda3/lib/python3.7/site-packages/transformers/trainer.py", line 762, in training_step
    return loss.item()
RuntimeError: CUDA error: an illegal memory access was encountered

> I had the same problem when i try to finetune RoBERTa model with my own tokenizer as @raj5287 did. I have already updated all packages (pytorch, transformers)
> 
> Code:
> 
> > tokenizer = RobertaTokenizerFast.from_pretrained(tokenizer_path)
> > model = RobertaForMaskedLM.from_pretrained(self.config)
> > dataset = LineByLineTextDataset(
> > tokenizer=tokenizer,
> > file_path=input_text_path,
> > block_size=512,
> > )
> > ```
> >     data_collator = DataCollatorForLanguageModeling(
> >         tokenizer=tokenizer, mlm=True, mlm_probability=0.15
> >     )
> >     training_args = TrainingArguments(
> >         output_dir=output_path,
> >         overwrite_output_dir=True,
> >         num_train_epochs=3,
> >         per_gpu_train_batch_size=8,
> >         save_steps=50_000,
> >         save_total_limit=2,
> >     )
> > 
> >     trainer = Trainer(
> >         model=model,
> >         args=training_args,
> >         data_collator=data_collator,
> >         train_dataset=dataset,
> >         prediction_loss_only=True
> >     )
> >     trainer.train()
> >     trainer.save_model(output_path)
> > ```
> 
> Output:
> 
> File "Fine_tuning_tokenizer.py", line 106, in 
> main(sys.argv[1:])
> File "Fine_tuning_tokenizer.py", line 94, in main
> modelRoberta.fine_tune_LM(inputfile, outputDir,tokenizer_path=outputDir)
> File "/sentiment-embeddings/projeto-modularizar/src/uff/ic/mell/sentimentembedding/modelos/modelo_roberta.py", line 58, in fine_tune_LM
> trainer.train()
> File "/home/apps/anaconda3/lib/python3.7/site-packages/transformers/trainer.py", line 549, in train
> tr_loss += self.training_step(model, inputs)
> File "/home/apps/anaconda3/lib/python3.7/site-packages/transformers/trainer.py", line 762, in training_step
> return loss.item()
> RuntimeError: CUDA error: an illegal memory access was encountered

Is it ok to use "DataCollatorForLanguageModeling" in a classification task? 

> I had the same problem when i try to finetune RoBERTa model with my own tokenizer as @raj5287 did. I have already updated all packages (pytorch, transformers)
> 
> Code:
> 
> > tokenizer = RobertaTokenizerFast.from_pretrained(tokenizer_path)
> > model = RobertaForMaskedLM.from_pretrained(self.config)
> > dataset = LineByLineTextDataset(
> > tokenizer=tokenizer,
> > file_path=input_text_path,
> > block_size=512,
> > )
> > ```
> >     data_collator = DataCollatorForLanguageModeling(
> >         tokenizer=tokenizer, mlm=True, mlm_probability=0.15
> >     )
> >     training_args = TrainingArguments(
> >         output_dir=output_path,
> >         overwrite_output_dir=True,
> >         num_train_epochs=3,
> >         per_gpu_train_batch_size=8,
> >         save_steps=50_000,
> >         save_total_limit=2,
> >     )
> > 
> >     trainer = Trainer(
> >         model=model,
> >         args=training_args,
> >         data_collator=data_collator,
> >         train_dataset=dataset,
> >         prediction_loss_only=True
> >     )
> >     trainer.train()
> >     trainer.save_model(output_path)
> > ```
> 
> Output:
> 
> File "Fine_tuning_tokenizer.py", line 106, in
> main(sys.argv[1:])
> File "Fine_tuning_tokenizer.py", line 94, in main
> modelRoberta.fine_tune_LM(inputfile, outputDir,tokenizer_path=outputDir)
> File "/sentiment-embeddings/projeto-modularizar/src/uff/ic/mell/sentimentembedding/modelos/modelo_roberta.py", line 58, in fine_tune_LM
> trainer.train()
> File "/home/apps/anaconda3/lib/python3.7/site-packages/transformers/trainer.py", line 549, in train
> tr_loss += self.training_step(model, inputs)
> File "/home/apps/anaconda3/lib/python3.7/site-packages/transformers/trainer.py", line 762, in training_step
> return loss.item()
> RuntimeError: CUDA error: an illegal memory access was encountered

Hey @SergioBarretoJr ,I guess one of the causes is when the model encounters an unseen token. and no we cannot use DataCollatorForLanguageModeling for classification task @Chandrak1907. will have to loop through epochs for training and fine tuning.

Thks guys @raj5287  @Chandrak1907 ! I solved this problems! When I trained my BPE tokenizer, I defined my vocabulary size of 52.000 and I used RoBERTa config from hugging face that has a different vocabulary size. This caused this error.

@raj5287 , @SergioBarretoJr  Can you share github  link for your code? I could not find proper reference on huggingface.co. Thanks.

@SergioBarretoJr I am also facing the same issue, can you please help me?

> @SergioBarretoJr I am also facing the same issue, can you please help me?

@amandalmia14 Have you ever checked if you set vocab size equal in tokenizer and RoBERTa.config file? 

    def train_tokenizer(self,file_path,outDir):
        # Initialize a tokenizer
        tokenizer = ByteLevelBPETokenizer()
        # Customize training
        tokenizer.train(files=file_path, **vocab_size=52_000**, min_frequency=2, special_tokens=[
            "
",
            "<pad>",
            "
",
            "<unk>",
            "<mask>",
        ])
        self.tokenizer=tokenizer
        tokenizer.save(outDir)


This was how I solver my issue. Can I help with something more?

@SergioBarretoJr Thanks for the snippet, I am able to train the custom language model with the help of [this](https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb). However, after creating the custom Roberta Model, I want to use it for creating a multi-label sentence classifier. 

Can you help me with that snippet of code? Thanks