Different batch sizes result in inconsistent predicted probabilities when using the trainer for regression task predictions.
Author: jiahuanluo
Pull request: no
Labels: name: Usage

name: bug

### System Info

Different batch sizes result in inconsistent predicted probabilities when using the trainer for regression task predictions.
How to get the consistent result?
Also, How to get consistent prediction results between using [run_classification.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-classification/run_classification.py) and the below:
```
class Predictor:
    def __init__(self, model_path):
        self.device = get_device()
        self.tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
        self.model = AutoModelForSequenceClassification.from_pretrained(model_path, trust_remote_code=True).to(
            self.device)
        self.model.eval()

    def classify_text(self, text, max_length):
        inputs = self.tokenizer(text, return_tensors="pt", truncation=True, padding=True, max_length=max_length)

        inputs = {k: v.to(self.device) for k, v in inputs.items()}

        with torch.no_grad():
            outputs = self.model(**inputs)

        logits = outputs.logits.squeeze()

        return logits
```

Thanks!

### Who can help?

@muellerzr @SunMarc

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

1. torchrun --nnodes=1 --nproc-per-node=8 run_classification.py --per_device_eval_batch_size 8/2

### Expected behavior

When using batch size 8 and 2, the results are consistent.

Comments:
Hey @jiahuanluo, it's going to be hard to help without some more details, can you share some exact reproducible scripts so that we may see what seems to be the issue?

Thanks @LysandreJik 
1. sft and get the trained model
```
dataset="nyu-mll/glue"
subset="cola"
python run_classification.py \
    --model_name_or_path  google-bert/bert-base-uncased \
    --dataset_name ${dataset} \
    --dataset_config_name ${subset} \
    --shuffle_train_dataset \
    --text_column_name "sentence" \
    --label_column_name label \
    --do_train \
    --do_eval \
    --do_regression \
    --max_seq_length 512 \
    --per_device_train_batch_size 32 \
    --learning_rate 2e-5 \
    --num_train_epochs 3 \
    --output_dir ./debug/model
```

2. change `writer.write(f"{index}\t{item:3.3f}\n")` in [L730](https://github.com/huggingface/transformers/blob/295a90cb40468ddd607577448e674348cff55adc/examples/pytorch/text-classification/run_classification.py#L730) to `writer.write(f"{index}\t{item}\n")` , predict with different batch sizes (eg. 4 and 32), and get the predictions.
```
dataset="nyu-mll/glue"
subset="cola"
python run_classification.py \
    --model_name_or_path  ./debug/model \
    --dataset_name ${dataset} \
    --dataset_config_name ${subset} \
    --shuffle_train_dataset \
    --text_column_name "sentence" \
    --label_column_name label \
    --do_predict \
    --do_regression \
    --max_seq_length 512 \
    --per_device_eval_batch_size 4 or 32 \
    --learning_rate 2e-5 \
    --num_train_epochs 3 \
    --output_dir ./debug/result_bs_4 or 32
```
3. diff two prediction files

```
diff ./debug/result_bs_4/predict_results.txt ./debug/result_bs_32/predict_results.txt
```
![image](https://github.com/user-attachments/assets/84fc908d-b655-4539-8f87-64a1f9f9b5b6)


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.