Does huggingface allow to perform n-fold cross validation and custom loss function using Hugging face Trainer and save best model?
Author: pratikchhapolika
Pull request: no

I am solving binary classification problem using `Roberta-Toxic` model.  My classes are highly skewed. ( `2% positive sample` )

I thought to perform `n-fold cross validation`. First thing that came to my mind is used Trainer in loop.

`I initialise the model every-time in the loop.`

Here is the simple code:

```

kf = KFold(n_splits=10, shuffle=True, random_state=0)

def compute_metrics(eval_pred):
    logits, labels = eval_pred
   

    predictions = np.argmax(logits, axis=-1)
    
    acc = np.sum(predictions == labels) / predictions.shape[0]
    return {"accuracy" : acc}

training_args = tr.TrainingArguments(
    #report_to = 'wandb',
    output_dir='/home/pc/results_new1',          # output directory
    overwrite_output_dir = True,
    num_train_epochs=2,              # total number of training epochs
    per_device_train_batch_size=32,  # batch size per device during training
    per_device_eval_batch_size=32,   # batch size for evaluation
    learning_rate=2e-5,
    warmup_steps=1000,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='./logs_new1',            # directory for storing logs
    logging_steps=220,
    evaluation_strategy="epoch"
    ,save_strategy="epoch"
    ,load_best_model_at_end=True
)


counter = 0
results_lst = []

train_text=train["text"]
train_label=train["label"]

for train_idx, val_idx in kf.split(train_text):
    print("Starting fold", counter)

    # split data
    train_texts_base = train_text.iloc[train_idx].tolist()
    train_labels_base = train_label.iloc[train_idx].tolist()

    val_texts = train_text.iloc[val_idx].tolist()
    val_labels = train_label.iloc[val_idx].tolist()

    # do tokenization
    train_encodings = tokenizer(train_texts_base, truncation=True, padding=True, max_length=512, return_tensors="pt")
    val_encodings = tokenizer(val_texts, truncation=True, padding=True, max_length=512, return_tensors="pt")
    
    # make datasets
    train_data = SEDataset(train_encodings, train_labels_base)
    val_data = SEDataset(val_encodings, val_labels)
    
    # train
    model = tr.RobertaForSequenceClassification.from_pretrained("/home/unbiased_toxic_roberta",problem_type="single_label_classification", num_labels=2, ignore_mismatched_sizes=True)

    model.to(device)
    
    trainer = tr.Trainer(
        model=model,                         # the instantiated 🤗 Transformers model to be trained
        args=training_args,                  # training arguments, defined above
        train_dataset=train_data,         # training dataset
        eval_dataset=val_data,             # evaluation dataset
        compute_metrics=compute_metrics
    )
    trainer.train()
    
    # eval
    preds = trainer.predict(val_data)
    
    result_df = pd.DataFrame({
        "text" :val_texts,
        "score" : softmax(preds[0], axis=1)[:,1]
    })
    results_lst.append(result_df)
    
    counter+=1
    
trainer.save_model("/home/pc/result_toxic_model_new1")

```

`The possible drawback I see in this approach is, the model saved will be last model in the loop which might not be robust model.?`

`Is there any capability which saves the average model or best model? Or does `Trainer provides` n-fold cross validation automatically .`

Also, apart from `accuracy ` which `compute metrics` can I add for highly skewed data?

```
def compute_metrics(eval_pred):
    logits, labels = eval_pred
   

    predictions = np.argmax(logits, axis=-1)
    
    acc = np.sum(predictions == labels) / predictions.shape[0]
    return {"accuracy" : acc}
```

**Does the above function only capture accuracy on evaluation data?** How can I capture same for training loss?

What is the loss function used here?

I get this output:

```
{'train_loss': 0.12655824422836304, 'train_accuracy': 0.9573275368950921, 'train_runtime': 49.3499, 'train_samples_per_second': 177.123, 'train_steps_per_second': 2.776, 'epoch': 3.0}

{'eval_loss': 0.07398280501365662, 'eval_accuracy': 0.9754901960784313, 'eval_runtime': 0.9742, 'eval_samples_per_second': 418.785, 'eval_steps_per_second': 7.185, 'epoch': 3.0}

So the `train and eval accuracy` is coming from `my compute_metrics` function?

Where from is `train_loss` and `eval_loss` coming?
```





Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

cc @sgugger 

Thanks!

Hello @pratikchhapolika , 
I am using your simple code for a 10-fold cross validation on BERT model using hugging face if the code is public can you share the repository. I am having hard time with **SEDataset** function converting tensors to dataframe and then training.

Thanks!


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.