Different intermediate results given different number of epochs
Author: DolevAdas
Pull request: no

### System Info

- `transformers` version: 4.31.0
- Platform: Linux-4.18.0-477.15.1.el8_8.x86_64-x86_64-with-glibc2.28
- Python version: 3.9.6
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (False)
- - Using GPU in script?: no
- Using distributed or parallel set-up in script?:no

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

We are using Hugging Face API to fine-tune a pretrained model ( BertForSequenceClassification).
We see differences in the first five epochs between 5 and 15 epoch runs and do not understand why they would not be (nearly) identical given that only the number of epochs is different between those runs. ( the seed and other parameters are all the same). 

**For example:**
### Seed 7
**5 epochs :**
,loss,learning_rate,epoch,step
0,**24.6558**,4.955555555555556e-05,0.04,500,,,,,,,,,
1,19.9439,4.9111111111111114e-05,0.09,1000,,,,,,,,,
2,19.2654,4.866666666666667e-05,0.13,1500,,,,,,,,,
3,20.4078,4.8222222222222225e-05,0.18,2000,,,,,,,,,
4,20.3372,4.7777777777777784e-05,0.22,2500,,,,,,,,,
5,20.0602,4.7333333333333336e-05,0.27,3000,,,,,,,,,
6,19.6761,4.6888888888888895e-05,0.31,3500,,,,,,,,,
7,20.193,4.644444444444445e-05,0.36,4000,,,,,,,,,
8,19.1265,4.600000000000001e-05,0.4,4500,,,,,,,,,
9,19.1949,4.555555555555556e-05,0.44,5000,,,,,,,,,
10,19.5078,4.511111111111112e-05,0.49,5500,,,,,,,,,
11,20.7165,4.466666666666667e-05,0.53,6000,,,,,,,,,
12,20.1907,4.422222222222222e-05,0.58,6500,,,,,,,,,
13,19.6967,4.377777777777778e-05,0.62,7000,,,,,,,,,
14,19.6693,4.3333333333333334e-05,0.67,7500,,,,,,,,,
15,20.011,4.2888888888888886e-05,0.71,8000,,,,,,,,,
16,19.516,4.2444444444444445e-05,0.76,8500,,,,,,,,,
17,18.9949,4.2e-05,0.8,9000,,,,,,,,,

**15 epochs:**
,loss,learning_rate,epoch,step
0,**18.9326**,4.9851851851851855e-05,0.04,500,,,,,,,,,
1,5.6773,4.970370370370371e-05,0.09,1000,,,,,,,,,
2,4.6515,4.955555555555556e-05,0.13,1500,,,,,,,,,
3,4.2881,4.940740740740741e-05,0.18,2000,,,,,,,,,
4,3.641,4.925925925925926e-05,0.22,2500,,,,,,,,,
5,3.2491,4.9111111111111114e-05,0.27,3000,,,,,,,,,
6,3.012,4.896296296296297e-05,0.31,3500,,,,,,,,,
7,2.8161,4.881481481481482e-05,0.36,4000,,,,,,,,,
8,2.7497,4.866666666666667e-05,0.4,4500,,,,,,,,,
9,2.6776,4.851851851851852e-05,0.44,5000,,,,,,,,,
10,2.5254,4.837037037037037e-05,0.49,5500,,,,,,,,,
11,2.6059,4.8222222222222225e-05,0.53,6000,,,,,,,,,
12,2.5966,4.807407407407408e-05,0.58,6500,,,,,,,,,
13,2.2252,4.792592592592593e-05,0.62,7000,,,,,,,,,
14,2.3321,4.7777777777777784e-05,0.67,7500,,,,,,,,,
15,2.23,4.762962962962963e-05,0.71,8000,,,,,,,,,
16,2.3754,4.7481481481481483e-05,0.76,8500,,,,,,,,,


### Seed 0 :
**5 epochs:**
,loss,learning_rate,epoch,step
0,**17.7629**,4.955555555555556e-05,0.04,500,,,,,,,,,
1,5.6264,4.9111111111111114e-05,0.09,1000,,,,,,,,,
2,4.9429,4.866666666666667e-05,0.13,1500,,,,,,,,,
3,4.5756,4.8222222222222225e-05,0.18,2000,,,,,,,,,
4,4.4063,4.7777777777777784e-05,0.22,2500,,,,,,,,,
5,3.9688,4.7333333333333336e-05,0.27,3000,,,,,,,,,
6,3.6656,4.6888888888888895e-05,0.31,3500,,,,,,,,,
7,3.6779,4.644444444444445e-05,0.36,4000,,,,,,,,,
8,3.2495,4.600000000000001e-05,0.4,4500,,,,,,,,,
9,3.2306,4.555555555555556e-05,0.44,5000,,,,,,,,,
10,3.1333,4.511111111111112e-05,0.49,5500,,,,,,,,,
11,2.7543,4.466666666666667e-05,0.53,6000,,,,,,,,,
12,3.1086,4.422222222222222e-05,0.58,6500,,,,,,,,,
13,3.0666,4.377777777777778e-05,0.62,7000,,,,,,,,,
14,3.156,4.3333333333333334e-05,0.67,7500,,,,,,,,,
15,2.5553,4.2888888888888886e-05,0.71,8000,,,,,,,,,
16,2.7727,4.2444444444444445e-05,0.76,8500,,,,,,,,,
17,2.651,4.2e-05,0.8,9000,,,,,,,,,

**15 epochs:**
,loss,learning_rate,epoch,step
0,**14.8927**,4.9851851851851855e-05,0.04,500,,,,,,,,,
1,5.4558,4.970370370370371e-05,0.09,1000,,,,,,,,,
2,4.065,4.955555555555556e-05,0.13,1500,,,,,,,,,
3,3.8751,4.940740740740741e-05,0.18,2000,,,,,,,,,
4,3.4581,4.925925925925926e-05,0.22,2500,,,,,,,,,
5,3.1641,4.9111111111111114e-05,0.27,3000,,,,,,,,,
6,2.8896,4.896296296296297e-05,0.31,3500,,,,,,,,,
7,2.8967,4.881481481481482e-05,0.36,4000,,,,,,,,,
8,2.5912,4.866666666666667e-05,0.4,4500,,,,,,,,,
9,2.5563,4.851851851851852e-05,0.44,5000,,,,,,,,,
10,2.482,4.837037037037037e-05,0.49,5500,,,,,,,,,
11,2.1695,4.8222222222222225e-05,0.53,6000,,,,,,,,,
12,2.447,4.807407407407408e-05,0.58,6500,,,,,,,,,
13,2.4438,4.792592592592593e-05,0.62,7000,,,,,,,,,
14,2.2014,4.7777777777777784e-05,0.67,7500,,,,,,,,,
15,2.2,4.762962962962963e-05,0.71,8000,,,,,,,,,

The only difference in the experiments is the number of epochs.
We also saved the train and validation split to a file and read it from there. To make sure we are reading in the same order.

**My environment**: python 3.9.6,  cuda 12.2.0, pytorch 2.0.1
**Here is part of my code:**

from transformers import (AutoTokenizer, DataCollatorWithPadding, TrainingArguments, 
                          BertForSequenceClassification, Trainer, AutoConfig)
import datasets
import numpy as np
import torch
import torch.nn as nn
import random

random.seed(cseed)
np.random.seed(cseed)
torch.manual_seed(cseed)
torch.cuda.manual_seed_all(cseed) 
os.environ['CUBLAS_WORKSPACE_CONFIG']=":16:8"

tokenizer = AutoTokenizer.from_pretrained(checkpoint, model_max_length=max_token_len)
training_args = TrainingArguments(out_path,
                                  save_total_limit = 10,
                                  #load_best_model_at_end = True,
                                  report_to=None,
                                  evaluation_strategy="steps",
                                  eval_steps=11250,
                                  do_eval=True,
                                  num_train_epochs=epochs_num,
                                  seed = cseed
                                  )

from transformers import set_seed
set_seed(cseed)


trian_data_from_disk = datasets.Dataset.load_from_disk(tokenized_datasets_path+"/train" , keep_in_memory=True)
validation_data_from_disk = datasets.Dataset.load_from_disk(tokenized_datasets_path+"/validation" , keep_in_memory=True)

model = BertForSequenceClassification.from_pretrained(checkpoint, num_labels=1)
loss_fn = nn.MSELoss()

trainer = CustomTrainer(
    model,
    training_args,
    train_dataset=trian_data_from_disk,
    eval_dataset=validation_data_from_disk,
    data_collator=data_collator,
    tokenizer=tokenizer,
)
training_results = trainer.train()




Comments:
Hi @DolevAdas, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

For the difference between 5 and 15 epochs, you will see that the learning rate at each step is different. This will be due to the learning rate scheduler.

For the difference between the same number of epochs and different seeds, this is likely due to randomness between runs. I would suggest running with the same seed to see if you observe the same loss values at each step, and then running with  a few other seeds to see how different the loss values are across runs. 

Thank you 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.