Different intermediate results given different number of epochs

username: dolevA
cooked: <p>We are using Hugging Face API to Fine-tune a pretrained model ( BertForSequenceClassification).<br>
We see differences in the first five epochs between 5 and 15 epoch runs and do not understand why they would not be (nearly) identical given that only the number of epochs is different between those runs. ( the seed and other parameters are all the same).<br>
For example:<br>
Seed 7<br>
5 epochs :<br>
,loss,learning_rate,epoch,step<br>
0,24.6558,4.955555555555556e-05,0.04,500,<br>
1,19.9439,4.9111111111111114e-05,0.09,1000,<br>
2,19.2654,4.866666666666667e-05,0.13,1500,<br>
3,20.4078,4.8222222222222225e-05,0.18,2000,<br>
4,20.3372,4.7777777777777784e-05,0.22,2500,<br>
5,20.0602,4.7333333333333336e-05,0.27,3000,<br>
6,19.6761,4.6888888888888895e-05,0.31,3500,<br>
7,20.193,4.644444444444445e-05,0.36,4000,<br>
8,19.1265,4.600000000000001e-05,0.4,4500,<br>
9,19.1949,4.555555555555556e-05,0.44,5000,<br>
10,19.5078,4.511111111111112e-05,0.49,5500,<br>
11,20.7165,4.466666666666667e-05,0.53,6000,<br>
12,20.1907,4.422222222222222e-05,0.58,6500,<br>
13,19.6967,4.377777777777778e-05,0.62,7000,<br>
14,19.6693,4.3333333333333334e-05,0.67,7500,<br>
15,20.011,4.2888888888888886e-05,0.71,8000,<br>
16,19.516,4.2444444444444445e-05,0.76,8500,<br>
17,18.9949,4.2e-05,0.8,9000,<br>
15 epochs:<br>
,loss,learning_rate,epoch,step<br>
0,18.9326,4.9851851851851855e-05,0.04,500,<br>
1,5.6773,4.970370370370371e-05,0.09,1000,<br>
2,4.6515,4.955555555555556e-05,0.13,1500,<br>
3,4.2881,4.940740740740741e-05,0.18,2000,<br>
4,3.641,4.925925925925926e-05,0.22,2500,<br>
5,3.2491,4.9111111111111114e-05,0.27,3000,<br>
6,3.012,4.896296296296297e-05,0.31,3500,<br>
7,2.8161,4.881481481481482e-05,0.36,4000,<br>
8,2.7497,4.866666666666667e-05,0.4,4500,<br>
9,2.6776,4.851851851851852e-05,0.44,5000,<br>
10,2.5254,4.837037037037037e-05,0.49,5500,<br>
11,2.6059,4.8222222222222225e-05,0.53,6000,<br>
12,2.5966,4.807407407407408e-05,0.58,6500,<br>
13,2.2252,4.792592592592593e-05,0.62,7000,<br>
14,2.3321,4.7777777777777784e-05,0.67,7500,<br>
15,2.23,4.762962962962963e-05,0.71,8000,<br>
16,2.3754,4.7481481481481483e-05,0.76,8500,</p>
<p>Seed 0 :<br>
5 epochs:<br>
,loss,learning_rate,epoch,step<br>
0,17.7629,4.955555555555556e-05,0.04,500,<br>
1,5.6264,4.9111111111111114e-05,0.09,1000,<br>
2,4.9429,4.866666666666667e-05,0.13,1500,<br>
3,4.5756,4.8222222222222225e-05,0.18,2000,<br>
4,4.4063,4.7777777777777784e-05,0.22,2500,<br>
5,3.9688,4.7333333333333336e-05,0.27,3000,<br>
6,3.6656,4.6888888888888895e-05,0.31,3500,<br>
7,3.6779,4.644444444444445e-05,0.36,4000,<br>
8,3.2495,4.600000000000001e-05,0.4,4500,<br>
9,3.2306,4.555555555555556e-05,0.44,5000,<br>
10,3.1333,4.511111111111112e-05,0.49,5500,<br>
11,2.7543,4.466666666666667e-05,0.53,6000,<br>
12,3.1086,4.422222222222222e-05,0.58,6500,<br>
13,3.0666,4.377777777777778e-05,0.62,7000,<br>
14,3.156,4.3333333333333334e-05,0.67,7500,<br>
15,2.5553,4.2888888888888886e-05,0.71,8000,<br>
16,2.7727,4.2444444444444445e-05,0.76,8500,<br>
17,2.651,4.2e-05,0.8,9000,</p>
<p>15 epochs:<br>
,loss,learning_rate,epoch,step<br>
0,14.8927,4.9851851851851855e-05,0.04,500,<br>
1,5.4558,4.970370370370371e-05,0.09,1000,<br>
2,4.065,4.955555555555556e-05,0.13,1500,<br>
3,3.8751,4.940740740740741e-05,0.18,2000,<br>
4,3.4581,4.925925925925926e-05,0.22,2500,<br>
5,3.1641,4.9111111111111114e-05,0.27,3000,<br>
6,2.8896,4.896296296296297e-05,0.31,3500,<br>
7,2.8967,4.881481481481482e-05,0.36,4000,<br>
8,2.5912,4.866666666666667e-05,0.4,4500,<br>
9,2.5563,4.851851851851852e-05,0.44,5000,<br>
10,2.482,4.837037037037037e-05,0.49,5500,<br>
11,2.1695,4.8222222222222225e-05,0.53,6000,<br>
12,2.447,4.807407407407408e-05,0.58,6500,<br>
13,2.4438,4.792592592592593e-05,0.62,7000,<br>
14,2.2014,4.7777777777777784e-05,0.67,7500,<br>
15,2.2,4.762962962962963e-05,0.71,8000,</p>
<p>The only difference in the experiments is the number of epochs.</p>
<p>We also saved the train and validation split to a file and read it from there. To make sure we are reading in the same order.</p>
<p>My environment: python 3.9.6,  cuda 12.2.0, pytorch 2.0.1</p>
<p>Here is part of my code:</p>
<p>from transformers import (AutoTokenizer, DataCollatorWithPadding, TrainingArguments,<br>
BertForSequenceClassification, Trainer, AutoConfig)<br>
import datasets<br>
import numpy as np<br>
import torch<br>
import torch.nn as nn<br>
import random</p>
<p>random.seed(cseed)<br>
np.random.seed(cseed)<br>
torch.manual_seed(cseed)<br>
torch.cuda.manual_seed_all(cseed)<br>
os.environ[‘CUBLAS_WORKSPACE_CONFIG’]=“:16:8”</p>
<p>tokenizer = AutoTokenizer.from_pretrained(checkpoint, model_max_length=max_token_len)<br>
training_args = TrainingArguments(out_path,<br>
save_total_limit = 10,<br>
<span class="hashtag-raw">#load_best_model_at_end</span> = True,<br>
report_to=None,<br>
evaluation_strategy=“steps”,<br>
eval_steps=11250,<br>
do_eval=True,<br>
num_train_epochs=epochs_num,<br>
seed = cseed<br>
)</p>
<p>from transformers import set_seed<br>
set_seed(cseed)</p>
<p>trian_data_from_disk = datasets.Dataset.load_from_disk(tokenized_datasets_path+“/train” , keep_in_memory=True)<br>
validation_data_from_disk = datasets.Dataset.load_from_disk(tokenized_datasets_path+“/validation” , keep_in_memory=True)</p>
<p>model = BertForSequenceClassification.from_pretrained(checkpoint, num_labels=1)<br>
loss_fn = nn.MSELoss()</p>
<p>trainer = CustomTrainer(<br>
model,<br>
training_args,<br>
train_dataset=trian_data_from_disk,<br>
eval_dataset=validation_data_from_disk,<br>
data_collator=data_collator,<br>
tokenizer=tokenizer,<br>
)<br>
training_results = trainer.train()</p>