Repository: huggingface/transformers

Issue:
metric is uninitialized when csv data is supplied to example/pytorch/text-classification/run_glue_no_trainer.py
## Environment info

- `transformers` version: 4.5.1
- Platform: Darwin-18.7.0-x86_64-i386-64bit
- Python version: 3.7.0
- PyTorch version (GPU?): 1.8.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: distributed
- OS type and version: Mac OSX 10.14.6

### Who can help
- maintained examples (not research project or legacy): @sgugger, @patil-suraj


## Information

Model I am using (Bert, XLNet ...): distilbert-base-uncased

The problem arises when using:
* [ x] the official example scripts: (give details below)
Running the script: transformers/examples/pytorch/text-classification/run_glue_no_trainer.py
With parameters: --model_name_or_path distilbert-base-uncased --per_device_train_batch_size 32 --learning_rate 2e-5 --num_train_epochs 3 --train_file piracy_train.csv --validation_file piracy_validation.csv --output_dir /data/output/distilbert-base-uncased-piracy-no-trainer

Yields the error:

`
Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: ['vocab_transform.weight', 'vocab_transform.bias', 'vocab_layer_norm.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight', 'vocab_projector.bias']
- This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['pre_classifier.weight', 'pre_classifier.bias', 'classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
100%|██████████| 2/2 [00:00<00:00, 12.75ba/s]
100%|██████████| 1/1 [00:00<00:00, 45.35ba/s]
04/22/2021 14:47:33 - INFO - __main__ -   Sample 598 of the training set: {'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'input_ids': [101, 6084, 1997, 16298, 1024, 2006, 5641, 2233, 2418, 2012, 5511, 19961, 11396, 1999, 2597, 2410, 1011, 4720, 2078, 1011, 28714, 1011, 2322, 2063, 2019, 19842, 2988, 2048, 10756, 19801, 2018, 7333, 2176, 8301, 21807, 2008, 5411, 1996, 19842, 2000, 2306, 1015, 5830, 1012, 27120, 4273, 3036, 2136, 3662, 4255, 1998, 8301, 21807, 11672, 1012, 6258, 2003, 3647, 1012, 102], 'labels': 1}.
04/22/2021 14:47:33 - INFO - __main__ -   Sample 65 of the training set: {'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'input_ids': [101, 2105, 2260, 2078, 2213, 22064, 1997, 11937, 9148, 1011, 11937, 9148, 2479, 5137, 1012, 2048, 10027, 2152, 1011, 3177, 6242, 5411, 1037, 9625, 6839, 14128, 1012, 3040, 2992, 1996, 8598, 3626, 21900, 1998, 8878, 2811, 28405, 2543, 21290, 2015, 1012, 5137, 3212, 2038, 2042, 11925, 2011, 27527, 2557, 1012, 1996, 10027, 6242, 5411, 2000, 1037, 3292, 1997, 3156, 5563, 2013, 1996, 2911, 1998, 2333, 2185, 1012, 1996, 2911, 7943, 2014, 6019, 2000, 1996, 2279, 3417, 1997, 7688, 1012, 102], 'labels': 1}.
04/22/2021 14:47:33 - INFO - __main__ -   Sample 877 of the training set: {'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'input_ids': [101, 7387, 1024, 2006, 2570, 2254, 1037, 9625, 6839, 2988, 2108, 2628, 2379, 2597, 5709, 1011, 2321, 2078, 4002, 2509, 1011, 2410, 2063, 3155, 3963, 13221, 2148, 1997, 16738, 1012, 1996, 2911, 2001, 7283, 2628, 2012, 1037, 3292, 1997, 1021, 2661, 1998, 2439, 1996, 10027, 6258, 2044, 1037, 3177, 3623, 1998, 2607, 2689, 1012, 102], 'labels': 1}.
04/22/2021 14:47:33 - INFO - __main__ -   ***** Running training *****
04/22/2021 14:47:33 - INFO - __main__ -     Num examples = 1173
04/22/2021 14:47:33 - INFO - __main__ -     Num Epochs = 3
04/22/2021 14:47:33 - INFO - __main__ -     Instantaneous batch size per device = 32
04/22/2021 14:47:33 - INFO - __main__ -     Total train batch size (w. parallel, distributed & accumulation) = 32
04/22/2021 14:47:33 - INFO - __main__ -     Gradient Accumulation steps = 1
04/22/2021 14:47:33 - INFO - __main__ -     Total optimization steps = 111
 33%|███▎      | 37/111 [07:48<14:24, 11.69s/it]Traceback (most recent call last):
  File "/Users/daraghhartnett/Projects/D3M/neural_text/code/transformers/examples/pytorch/text-classification/run_glue_no_trainer.py", line 441, in <module>
    main()
  File "/Users/daraghhartnett/Projects/D3M/neural_text/code/transformers/examples/pytorch/text-classification/run_glue_no_trainer.py", line 406, in main
    metric.add_batch(
UnboundLocalError: local variable 'metric' referenced before assignment
 33%|███▎      | 37/111 [07:49<15:39, 12.69s/it]
`

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x ] my own task or dataset: (give details below)

Simple single sentence text classification

## To reproduce

Steps to reproduce the behavior:

1. Pick any csv dataset with a train and validation files and run the transformers/examples/pytorch/text-classification/run_glue_no_trainer.py script using the following parameters:
2. --model_name_or_path distilbert-base-uncased --per_device_train_batch_size 32 --learning_rate 2e-5 --num_train_epochs 3 --train_file piracy_train.csv --validation_file piracy_validation.csv --output_dir /data/output/distilbert-base-uncased-piracy-no-trainer
3. This will yield an error as the metric variable is not initialized when an optional args.task_name is not specified.


`
Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: ['vocab_transform.weight', 'vocab_transform.bias', 'vocab_layer_norm.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight', 'vocab_projector.bias']
- This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['pre_classifier.weight', 'pre_classifier.bias', 'classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
100%|██████████| 2/2 [00:00<00:00, 12.75ba/s]
100%|██████████| 1/1 [00:00<00:00, 45.35ba/s]
04/22/2021 14:47:33 - INFO - __main__ -   Sample 598 of the training set: {'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'input_ids': [101, 6084, 1997, 16298, 1024, 2006, 5641, 2233, 2418, 2012, 5511, 19961, 11396, 1999, 2597, 2410, 1011, 4720, 2078, 1011, 28714, 1011, 2322, 2063, 2019, 19842, 2988, 2048, 10756, 19801, 2018, 7333, 2176, 8301, 21807, 2008, 5411, 1996, 19842, 2000, 2306, 1015, 5830, 1012, 27120, 4273, 3036, 2136, 3662, 4255, 1998, 8301, 21807, 11672, 1012, 6258, 2003, 3647, 1012, 102], 'labels': 1}.
04/22/2021 14:47:33 - INFO - __main__ -   Sample 65 of the training set: {'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'input_ids': [101, 2105, 2260, 2078, 2213, 22064, 1997, 11937, 9148, 1011, 11937, 9148, 2479, 5137, 1012, 2048, 10027, 2152, 1011, 3177, 6242, 5411, 1037, 9625, 6839, 14128, 1012, 3040, 2992, 1996, 8598, 3626, 21900, 1998, 8878, 2811, 28405, 2543, 21290, 2015, 1012, 5137, 3212, 2038, 2042, 11925, 2011, 27527, 2557, 1012, 1996, 10027, 6242, 5411, 2000, 1037, 3292, 1997, 3156, 5563, 2013, 1996, 2911, 1998, 2333, 2185, 1012, 1996, 2911, 7943, 2014, 6019, 2000, 1996, 2279, 3417, 1997, 7688, 1012, 102], 'labels': 1}.
04/22/2021 14:47:33 - INFO - __main__ -   Sample 877 of the training set: {'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'input_ids': [101, 7387, 1024, 2006, 2570, 2254, 1037, 9625, 6839, 2988, 2108, 2628, 2379, 2597, 5709, 1011, 2321, 2078, 4002, 2509, 1011, 2410, 2063, 3155, 3963, 13221, 2148, 1997, 16738, 1012, 1996, 2911, 2001, 7283, 2628, 2012, 1037, 3292, 1997, 1021, 2661, 1998, 2439, 1996, 10027, 6258, 2044, 1037, 3177, 3623, 1998, 2607, 2689, 1012, 102], 'labels': 1}.
04/22/2021 14:47:33 - INFO - __main__ -   ***** Running training *****
04/22/2021 14:47:33 - INFO - __main__ -     Num examples = 1173
04/22/2021 14:47:33 - INFO - __main__ -     Num Epochs = 3
04/22/2021 14:47:33 - INFO - __main__ -     Instantaneous batch size per device = 32
04/22/2021 14:47:33 - INFO - __main__ -     Total train batch size (w. parallel, distributed & accumulation) = 32
04/22/2021 14:47:33 - INFO - __main__ -     Gradient Accumulation steps = 1
04/22/2021 14:47:33 - INFO - __main__ -     Total optimization steps = 111
 33%|███▎      | 37/111 [07:48<14:24, 11.69s/it]Traceback (most recent call last):
  File "/Users/daraghhartnett/Projects/D3M/neural_text/code/transformers/examples/pytorch/text-classification/run_glue_no_trainer.py", line 441, in <module>
    main()
  File "/Users/daraghhartnett/Projects/D3M/neural_text/code/transformers/examples/pytorch/text-classification/run_glue_no_trainer.py", line 406, in main
    metric.add_batch(
UnboundLocalError: local variable 'metric' referenced before assignment
 33%|███▎      | 37/111 [07:49<15:39, 12.69s/it]
`


## Expected behavior

Since providing your own csv files is allowed, the metric object should be initialized when no task_name is provided.



Discussion:

Resolving patch:
diff --git a/examples/pytorch/text-classification/run_glue_no_trainer.py b/examples/pytorch/text-classification/run_glue_no_trainer.py
--- a/examples/pytorch/text-classification/run_glue_no_trainer.py
+++ b/examples/pytorch/text-classification/run_glue_no_trainer.py
@@ -367,6 +367,8 @@ def preprocess_function(examples):
     # Get the metric function
     if args.task_name is not None:
         metric = load_metric("glue", args.task_name)
+    else:
+        metric = load_metric("accuracy")
 
     # Train!
     total_batch_size = args.per_device_train_batch_size * accelerator.num_processes * args.gradient_accumulation_steps


Test patch: