How to output labels for GLUE test set
Author: JetRunner
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Thanks for your great work.

I have done some modification based on your library. I'd like to test it on GLUE test set (not dev set). Is there any way I can do it? Do you have a script for output GLUE submission files?

Thanks.

Comments:
Hi, if you're working on a clone/fork of this library, then you can freely change the following lines in `transformers/data/processors/glue.py` (lines 254-262 on the current master):

```py
    def get_train_examples(self, data_dir):
        """See base class."""
        return self._create_examples(
            self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")

    def get_dev_examples(self, data_dir):
        """See base class."""
        return self._create_examples(
            self._read_tsv(os.path.join(data_dir, "dev.tsv")), "dev")
```

to read the `test.tsv` instead of the `dev.tsv`.

Hi @LysandreJik,

Thanks for your reply!
However, the format of test set is different from the dev set. It has an extra `id` field and does not have ground truth labels.  Thus, the simple change you suggested wouldn't work in this case.   Besides, GLUE leaderboard also requires a specific output format for submission.

I have noticed that in Huggingface's workshop paper [DistilBERT](https://arxiv.org/abs/1910.01108), you also use DEV set instead of TEST result from GLUE leaderboard. I guess it may be somehow related to the lack of a submission script. To facilitate further research, I hope you can make a script for GLUE submission. As far as I know, many researchers are using huggingface/transformers for their research. I really appreciate it if you can provide a script for the community.

Thanks!

Hi @LysandreJik,
I've just realized that you are one of the authors of DistilBERT so I suppose you know exactly what I mean. It can truly benefit the research community if you do this. Thanks.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


hi did u get the script? 

@yuchenlin My co-author wrote one himself. I added you on Wechat but you didn't respond. 

Hey, I think this would be helpful for many in the community. Can you maybe share the script you wrote for this via a pull request?

@Breakend Sure, I will verify the script and open a pull request.

Hello @JetRunner , no news about this script?
Thank you in advance

> Hello @JetRunner , no news about this script?
> Thank you in advance

Sorry, I totally forgot about this stuff. I'll notify you as soon as we upload it.

cc @michaelzhouwang

> Hello @JetRunner , no news about this script?
> Thank you in advance

Hi I have uploaded the script for prediction on GLUE benchmarks at:
https://github.com/JetRunner/BERT-of-Theseus/tree/master/glue_script
You can first replace the glue.py in src/transformers/data/processor/ and then use run_prediction.py.

> > Hello @JetRunner , no news about this script?
> > Thank you in advance
> 
> Hi I have uploaded the script for prediction on GLUE benchmarks at:
> https://github.com/JetRunner/BERT-of-Theseus/tree/master/glue_script
> You can first replace the glue.py in src/transformers/data/processor/ and then use run_prediction.py.

cc @Breakend @jibay 

@JetRunner  @MichaelZhouwang  Thank you for the quick answer, i will check asap :)