Evaluation of labelled test set?
Author: Mahmedturk
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->
Hi, 

I have a labelled  test set of QQP data set. What arguments do i need to input if i want to report accuracy and F1 on test set not just the development set?
<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 

Comments:
Hi, you could use model.eval() or load pre-tuned model and run against test set
	# Load a trained model and vocabulary that you have fine-tuned
	model = model_class.from_pretrained(output_dir)
	tokenizer = tokenizer_class.from_pretrained(output_dir)
See this post: https://mccormickml.com/2019/07/22/BERT-fine-tuning/#a1-saving--loading-fine-tuned-model


+1

Is there something similar for testing to the `--do_train` or `--do_eval` flag in the glue examples?

@Mahmedturk I tried it like this now:

```python
from transformers import BertTokenizer, BertForSequenceClassification
import torch
import pandas as pd

tokenizer = BertTokenizer.from_pretrained('./my-model')
model = BertForSequenceClassification.from_pretrained('./my-model')
labels = [ ... ]

df_test = pd.read_csv('./my-data/test.tsv', sep='\t', names=['label', 'sentence'])
df_test['prediction'] = None

for row in df_test.itertuples():
    inputs = tokenizer.encode_plus(row.sentence, add_special_tokens=True, return_tensors='pt')
    pred = model(inputs['input_ids'], token_type_ids=inputs['token_type_ids'])[0].argmax().item()
    df_test.loc[row.Index, 'prediction'] = labels[pred]
```

Then you can filter the pandas dataframe, i.e. `df_test[df_test['label'] == df_test['prediction']]` to see the true positives.

@olastor is there a way to print confusion matrix?


Also for the evaluation set, i get the following three metrics.  What is "acc_and_f1" in the below?


acc = 0.9455445544554455
acc_and_f1 = 0.8709204253758709
f1 = 0.7962962962962963


@Mahmedturk Not with a built in function in my example, but manually. Let's say you have the labels `True` and `False`, then the correct way to calculate the absolute values of the confusion matrix would be like this I think:

- number of true positves: `len(df[(df['label'] == True]) & (df['prediction'] == True)])`
- number of false positves: `len(df[(df['label'] == False]) & (df['prediction'] == True)])`
- number of false negatives: `len(df[(df['label'] == True]) & (df['prediction'] == False)])`
- number of true negatives: `len(df[(df['label'] == False]) & (df['prediction'] == False)])`

@Mahmedturk From [here](https://github.com/huggingface/transformers/blob/master/src/transformers/data/metrics/__init__.py#L41):

```python
def acc_and_f1(preds, labels):
        acc = simple_accuracy(preds, labels)
        f1 = f1_score(y_true=labels, y_pred=preds)
        return {
            "acc": acc,
            "f1": f1,
            "acc_and_f1": (acc + f1) / 2, # <---
        }
```

@sunyangfu 
After loading the saved model and vocabulary how do i run against the test set?
Sorry if this sounds silly. I am very new to PyTorch and deep learning.
The given link shows how to test on CoLa dataset which has only one sentence. Whereas in QQP dataset there are two sentences. What changes do i need to make in the code to test it with QQP dataset?


@Mahmedturk Here you can run against the test set:
After loading the model, you can code whatever you want to fit the model.
```python
output_dir = './saved_model_dir/'

MODEL_CLASSES = {
    'bert': (BertConfig, BertForSequenceClassification, BertTokenizer),
    'xlnet': (XLNetConfig, XLNetForSequenceClassification, XLNetTokenizer),
    'xlm': (XLMConfig, XLMForSequenceClassification, XLMTokenizer),
    'roberta': (RobertaConfig, RobertaForSequenceClassification, RobertaTokenizer),
    'distilbert': (DistilBertConfig, DistilBertForSequenceClassification, DistilBertTokenizer),
    'albert': (AlbertConfig, AlbertForSequenceClassification, AlbertTokenizer)
}	

# Config class and load a trained model and vocabulary 
config_class, model_class, tokenizer_class = MODEL_CLASSES['bert']	
model = model_class.from_pretrained(output_dir)
tokenizer = tokenizer_class.from_pretrained(output_dir)

# Copy the model to the GPU.
model.to(device)

# Put model in evaluation mode
model.eval()

#Then you do test data pre-processing and apply the model on test data
with torch.no_grad():
    outputs = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask)


```

Hi @olastor,

I have tried your method, it returns the below error

 df_test.loc[row.Index, 'prediction'] = labels[pred]
IndexError: list index out of range. 

@Mahmedturk Did you update the list of labels in my example for your task?

labels = df_test.label.values

> labels = df_test.label.values

It needs to be the same set of labels used for training.

@olastor thanks.