Repository: foundation-model-stack/fms-hf-tuning (language: python)

Issue:
Performance regression in fms-hf-tuning image
## Describe the bug

As part of our performance evaluation of the fms-hf-tuning fine-tuning stack, we observed a regression between the images we tested:

*    quay.io/modh/fms-hf-tuning:01b3824c9aba22d9d0695399681e6f0507840e7f --> fast (April 17)
```
{'train_runtime': 829.2865, 'train_samples_per_second': 62.707, 'train_steps_per_second': 3.919, 'train_tokens_per_second': 12141.512, 'train_loss': 2.287814903846154, 'epoch': 1.0}
```
*    quay.io/modh/fms-hf-tuning:bd8bf628cd739c7a201a976bc3c1096785353f1a --> slow (May 26)
```
{'train_runtime': 1391.6176, 'train_samples_per_second': 37.368, 'train_steps_per_second': 2.335, 'train_tokens_per_second': 7235.315, 'train_loss': 2.287858623798077, 'epoch': 1.0}
```
*    quay.io/modh/fms-hf-tuning:84b0337b7baee119e909d4e901b6dadfe34c1f9a --> slow (May 21, planned for delivery)
```
{'train_runtime': 1404.3311, 'train_samples_per_second': 37.03, 'train_steps_per_second': 2.314, 'train_tokens_per_second': 7169.813, 'train_loss': 2.2878201622596155, 'epoch': 1.0}
```
Additionally, we observed that with the bd8bf628cd739c7a201a976bc3c1096785353f1a image, the multi-model tuning performance (multiple independent fine-tuning jobs, running on a dedicated GPU) are degraded compared to the reference run (one job running alone in the cluster). This wasn't the case in our first tests with 84b0337b7baee119e909d4e901b6dadfe34c1f9a.

* fine tuning of gpt_bigcode-santacoder against the Alpaca dataset

## Platform

OpenShift AI

## Steps to reproduce

1. Create these 3 objects in a namespace,
    * [pytorchjob.yaml.log](https://github.com/user-attachments/files/15900894/pytorchjob.yaml.log)
    * [cm_entrypoint.yaml.log](https://github.com/user-attachments/files/15900895/cm_entrypoint.yaml.log)
    * [cm_config.yaml.log](https://github.com/user-attachments/files/15900896/cm_config.yaml.log)
2. See the Pod being executed with the `quay.io/modh/fms-hf-tuning:release` image, currently hitting the regression
3. When the Pod finishes, create an interactive Pod with root privileges (to be able to install the new package)
```
oc debug fine-tuning-master-0 --as-root
```
4. Prepare the environment, by running the script a first time
```
$ bash /mnt/entrypoint/entrypoint.sh
+ grep transformers
transformers==4.41.2
...
{'train_runtime': 146.0027, 'train_samples_per_second': 35.616, 'train_steps_per_second': 2.226, 'train_tokens_per_second': 6719.494, 'train_loss': 2.377412109375, 'epoch': 1.0}
```
5. install the 4.40.2 version of transformers and tell Python to use it
```
export HOME=/tmp
pip install transformers==4.40.2 --user
export PYTHONPATH=/tmp/.local/lib/python3.11/site-packages/:$PYTHONPATH
```
6. Run the script again, with the 4.40 version of `transformers`
```
$ bash /mnt/entrypoint/entrypoint.sh
+ grep transformers
transformers==4.40.2
...
{'train_runtime': 88.4812, 'train_samples_per_second': 58.77, 'train_steps_per_second': 3.673, 'train_tokens_per_second': 11087.825, 'train_loss': 2.377412109375, 'epoch': 1.0}
```
7. **Compare the `train_runtime` and `train_samples_per_second` etc and notice the difference.**

Note that:
* you can configure the length of the dataset with `DATASET_REPLICATION`. Set to `0.1` to get results faster.
* the version pinning has been tested without the fiddling above, only required to make a simple reproducer.

## Expected behavior

![image](https://github.com/foundation-model-stack/fms-hf-tuning/assets/7559202/675e09ee-ca20-4779-aa1a-dcba8d5708ed)

## Observed behavior

![image](https://github.com/foundation-model-stack/fms-hf-tuning/assets/7559202/57f99e86-4d97-478f-af86-6d6de9f0f188)

## See also

[RHOAIENG-8551](https://issues.redhat.com/browse/RHOAIENG-8551)




Discussion / hints:

Fix patch:
diff --git a/examples/prompt_tuning_twitter_complaints/README.md b/examples/prompt_tuning_twitter_complaints/README.md
index cd8e95233..c8383cd57 100644
--- a/examples/prompt_tuning_twitter_complaints/README.md
+++ b/examples/prompt_tuning_twitter_complaints/README.md
@@ -51,7 +51,7 @@ tuning/sft_trainer.py  \
 --per_device_train_batch_size 1  \
 --per_device_eval_batch_size 1  \
 --gradient_accumulation_steps 1  \
---eval_strategy "no"  \
+--evaluation_strategy "no"  \
 --save_strategy "epoch"  \
 --learning_rate 1e-5  \
 --weight_decay 0.  \
diff --git a/pyproject.toml b/pyproject.toml
index 744090676..830e3d273 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -28,7 +28,7 @@ classifiers=[
 dependencies = [
 "numpy>=1.26.4,<2.0",
 "accelerate>=0.20.3,<0.40",
-"transformers>=4.34.1,<5.0,!=4.38.2",
+"transformers>=4.34.1,<=4.40.2,!=4.38.2",
 "torch>=2.2.0,<3.0",
 "sentencepiece>=0.1.99,<0.3",
 "tokenizers>=0.13.3,<1.0",


Test patch:
diff --git a/tests/test_sft_trainer.py b/tests/test_sft_trainer.py
index 7041bd136..d7508d8de 100644
--- a/tests/test_sft_trainer.py
+++ b/tests/test_sft_trainer.py
@@ -304,7 +304,7 @@ def test_run_causallm_pt_with_validation():
     with tempfile.TemporaryDirectory() as tempdir:
         train_args = copy.deepcopy(TRAIN_ARGS)
         train_args.output_dir = tempdir
-        train_args.eval_strategy = "epoch"
+        train_args.evaluation_strategy = "epoch"
         data_args = copy.deepcopy(DATA_ARGS)
         data_args.validation_data_path = TWITTER_COMPLAINTS_DATA
 
@@ -317,7 +317,7 @@ def test_run_causallm_pt_with_validation_data_formatting():
     with tempfile.TemporaryDirectory() as tempdir:
         train_args = copy.deepcopy(TRAIN_ARGS)
         train_args.output_dir = tempdir
-        train_args.eval_strategy = "epoch"
+        train_args.evaluation_strategy = "epoch"
         data_args = copy.deepcopy(DATA_ARGS)
         data_args.validation_data_path = TWITTER_COMPLAINTS_DATA
         data_args.dataset_text_field = None