Using run_glue.py on external datasets for fine-tuning a RoBERTa classification model --> Is this possible?
Author: seyonechithrananda
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->
I recently uploaded the model weights for RoBERTa trained on a chemical benchmark dataset called ZINC15K for masked-language modelling of individual atoms in each molecule. The model performed pretty decently, so I thought it would be interesting to apply it to a downstream task of toxicity prediction on the Tox21 dataset (balanced dataset I created here: https://github.com/seyonechithrananda/bert-loves-chemistry/blob/master/tox21_balanced_revised.csv)

^The link above is in the repo with all the HuggingFace notebooks relating to this task I have created for more details. 

As you can see in the CSV above, the 'SR-p53' value represents the labels for the dataset, whereas the 'SMILES' column represents the text representation for each molecule. Is there a way that `run_glue.py` can be repurposed alongside `RobertaforSequenceClassification` to train a pre-trained model for this classification task? And if so, could anyone give me a couple pointers on where to start? I'm relatively new to HuggingFace (more familiar with CV + graph NN's for chemistry) but have been enjoying using it so far!

Link to model weights (in Huggingface hub): https://huggingface.co/seyonec/ChemBERTa-zinc-base-v1

Thanks for the help!

*Given that this was a more library-centric question and not a bug I felt it would be better to post here then on SO.

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->


Comments:
Hi @seyonechithrananda, do you have a `torch.data.Dataset` for your classification dataset?

If you do, this will be pretty easy following #3800 (i.e. the amount of code required should be pretty minimal)

Thanks for the response @julien-c! I originally grab a list of SMILES sequences and their corresponding labels into two separate lists, before tokenizing the sequences and converting them into a tensor (following the `RobertaForSequenceClassification` docs). Will look into `torch.data.Dataset`. 

Link to code I was using previously (I originally tried to fine-tune for classification without `run_glue.py`): https://t.co/lqVqh3L1oA?amp=1

> Hi @seyonechithrananda, do you have a `torch.data.Dataset` for your classification dataset?
> 
> If you do, this will be pretty easy following #3800 (i.e. the amount of code required should be pretty minimal)



Hi @julien-c! Followed your tips and created a Dataset class following a similar tutorial. However I ran into an issue with CUDA in the training pipeline:

```
AttributeError                            Traceback (most recent call last)
<ipython-input-15-ea2a288fbd03> in <module>()
      8         if torch.cuda.is_available():
      9           sent = sent.cuda()
---> 10           label = labels.cuda()
     11         output = model.forward(sent)[0]
     12         _, predicted = torch.max(output, 1)

AttributeError: 'list' object has no attribute 'cuda'

```

Do you know why this issue is occurring? I use a pre-trained RoBERTA model (trained on MLM for a diff. dataset). [Here](https://colab.research.google.com/drive/1Q9pvFQoEe_4NIO853-tDy0ERZ3pyNzwT) is the notebook. Also, can we utilize the config from a MLM RoBERTa model for sequence classification or should it be the `roberta-base` config?

Thanks for the help! 



Managed to create a variant of `run_glue.py` and get it working. Thanks for the help1