When using --eval_do_concat_batches=False with run_glue.py example, I get "ValueError: Predictions and/or references don't match the expected format."
Author: jeffhataws
Pull request: no
Labels: name: bug

### System Info

Ran on c5.4xlarge CPU instance with Ubuntu22

- `transformers` version: 4.44.0
- Platform: Linux-6.8.0-1024-aws-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.30.2
- Safetensors version: 0.5.3
- Accelerate version: 1.6.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.6.0+cu124 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>



### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Setup:

```
export HF_VER=4.51.3
pip install -U transformers==$HF_VER datasets evaluate scikit-learn
cd ~/
git clone https://github.com/huggingface/transformers --branch v$HF_VER
cd ~/transformers/examples/pytorch/text-classification
```

Then run the following script:

```
#!/usr/bin/env bash
set -eExuo
export TASK_NAME=mrpc
torchrun --nproc_per_node=1 ./run_glue.py \
--model_name_or_path gaunernst/bert-tiny-uncased \
--task_name $TASK_NAME \
--do_eval \
--max_seq_length 128 \
--per_device_train_batch_size 16 \
--max_eval_samples 1000 \
--learning_rate 2e-5 \
--num_train_epochs 5 \
--save_total_limit 1 \
--eval_do_concat_batches False \
--overwrite_output_dir \
--output_dir /tmp/$TASK_NAME/ |& tee log_run

```

You will see the error:

```

***** Running Evaluation *****                                                                                                                                        [INFO|trainer.py:3831] 2025-04-17 18:48:47,507 >>   Num examples = 408                                                                                                [INFO|trainer.py:3834] 2025-04-17 18:48:47,507 >>   Batch size = 8                                                                                                    100%|██████████| 51/51 [02:22<00:00,  2.84s/it][rank0]: Traceback (most recent call last):                                                                            [rank0]:   File "/home/ubuntu/transformers/examples/pytorch/text-classification/./run_glue.py", line 637, in <module>                                                 
[rank0]:     main()                                                                                                                                                   
[rank0]:   File "/home/ubuntu/transformers/examples/pytorch/text-classification/./run_glue.py", line 575, in main                                                     
[rank0]:     metrics = trainer.evaluate(eval_dataset=eval_dataset)
[rank0]:   File "/home/ubuntu/test_venv_py310/lib/python3.10/site-packages/transformers/trainer.py", line 3676, in evaluate
[rank0]:     output = eval_loop(
[rank0]:   File "/home/ubuntu/test_venv_py310/lib/python3.10/site-packages/transformers/trainer.py", line 3966, in evaluation_loop
[rank0]:     metrics = self.compute_metrics(EvalPrediction(predictions=all_preds, label_ids=all_labels))
[rank0]:   File "/home/ubuntu/transformers/examples/pytorch/text-classification/./run_glue.py", line 513, in compute_metrics
[rank0]:     result = metric.compute(predictions=preds, references=p.label_ids)
[rank0]:   File "/home/ubuntu/test_venv_py310/lib/python3.10/site-packages/evaluate/module.py", line 455, in compute
[rank0]:     self.add_batch(**inputs)
[rank0]:   File "/home/ubuntu/test_venv_py310/lib/python3.10/site-packages/evaluate/module.py", line 546, in add_batch
[rank0]:     raise ValueError(error_msg) from None
[rank0]: ValueError: Predictions and/or references don't match the expected format. 
[rank0]: Expected format: {'predictions': Value(dtype='int64', id=None), 'references': Value(dtype='int64', id=None)},
[rank0]: Input predictions: [[1 4]
[rank0]:  [0 4]
[rank0]:  [0 1]
[rank0]:  [1 1]
[rank0]:  [0 1]
[rank0]:  [3 2]
[rank0]:  [1 2]
[rank0]:  [0 0]
[rank0]:  [2 1]
[rank0]:  [0 3]
[rank0]:  [0 0]
[rank0]:  [1 0]
[rank0]:  [1 1]
[rank0]:  [3 4]
[rank0]:  [0 3]
[rank0]:  [0 1]
[rank0]:  [0 3]
[rank0]:  [6 0]
[rank0]:  [0 3]
[rank0]:  [0 4]
[rank0]:  [0 0]
[rank0]:  [4 1]
[rank0]:  [4 5]
[rank0]:  [0 0]
[rank0]:  [0 2]
[rank0]:  [0 0]
[rank0]:  [0 2]
[rank0]:  [7 7]
[rank0]:  [0 1]
[rank0]:  [0 3]
[rank0]:  [0 3]
[rank0]:  [0 1]
[rank0]:  [0 1]
[rank0]:  [0 1]
[rank0]:  [2 1]
[rank0]:  [0 3]
[rank0]:  [7 7]
[rank0]:  [0 1]
[rank0]:  [0 4]
[rank0]:  [0 0]
[rank0]:  [1 3]
[rank0]:  [2 0]
[rank0]:  [7 0]
[rank0]:  [1 4]
[rank0]:  [0 3]
[rank0]:  [1 0]
[rank0]:  [0 1]
[rank0]:  [7 4]
[rank0]:  [0 3]
[rank0]:  [4 3]
[rank0]:  [0 2]],
[rank0]: Input references: [array([1, 0, 0, 1, 0, 1, 0, 1]), array([1, 1, 1, 0, 0, 1, 1, 1]), array([1, 0, 1, 0, 0, 1, 0, 1]), ..., array([1, 0, 1, 0, 1, 1, 1, 1]), a
rray([1, 1, 1, 1, 1, 1, 1, 1]), array([0, 1, 1, 0, 1, 1, 0, 1])]
100%|██████████| 51/51 [02:22<00:00,  2.79s/it]
E0417 18:51:13.767000 4267 torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: 1) local_rank: 0 (pid: 4285) of binary: /home/ubuntu/test_venv_py31
0/bin/python3
Traceback (most recent call last):
  File "/home/ubuntu/test_venv_py310/bin/torchrun", line 8, in <module>
    sys.exit(main())


```

### Expected behavior

No error

Comments:
We need this feature for torch-xla to avoid frequent recompilations during evaluation (due to nested concatenation).

The following change in run_glue.py would work. I also have it in my own branch. However, would need HF expert to look at other tests/examples and update them.

```
diff --git a/examples/pytorch/text-classification/run_glue.py b/examples/pytorch/text-classification/run_glue.py
index c29303808..37c472d84 100755
--- a/examples/pytorch/text-classification/run_glue.py
+++ b/examples/pytorch/text-classification/run_glue.py
@@ -507,9 +507,16 @@ def main():
     # You can define your custom compute_metrics function. It takes an `EvalPrediction` object (a namedtuple with a
     # predictions and label_ids field) and has to return a dictionary string to float.
     def compute_metrics(p: EvalPrediction):
-        preds = p.predictions[0] if isinstance(p.predictions, tuple) else p.predictions
-        preds = np.squeeze(preds) if is_regression else np.argmax(preds, axis=1)
-        result = metric.compute(predictions=preds, references=p.label_ids)
+        if training_args.eval_do_concat_batches:
+            preds = p.predictions[0] if isinstance(p.predictions, tuple) else p.predictions
+            preds = np.squeeze(preds) if is_regression else np.argmax(preds, axis=1)
+            labels = p.label_ids
+        else:
+            preds = p.predictions
+            preds = np.squeeze(preds) if is_regression else np.argmax(preds, axis=2)
+            preds = preds.flatten()
+            labels = np.array(p.label_ids).flatten()
+        result = metric.compute(predictions=preds, references=labels)
         if len(result) > 1:
             result["combined_score"] = np.mean(list(result.values())).item()
         return result

```

Thanks for the report @jeffhataws ! To make it more clearer, I think it will be easier if we do the following changes. For the other examples, we will have to do something similar as well. Would you like to submit a PR to first fix this example ? 

```python
    def compute_metrics(p: EvalPrediction):
        preds = p.predictions[0] if isinstance(p.predictions, tuple) else p.predictions
        if not training_args.eval_do_concat_batches:
            preds = np.concatenate(preds, axis=0)
            labels = np.concatenate(p.label_ids, axis=0)
        preds = np.squeeze(preds) if is_regression else np.argmax(preds, axis=1)
        result = metric.compute(predictions=preds, references=labels)
        if len(result) > 1:
            result["combined_score"] = np.mean(list(result.values())).item()
        return result
```

I will check that. For run_ner.py, the following works:

```
@@ -530,8 +539,11 @@

     def compute_metrics(p):
         predictions, labels = p
+        if not training_args.eval_do_concat_batches:
+            predictions = np.hstack(predictions)
+            labels = np.hstack(labels)
+            
         predictions = np.argmax(predictions, axis=2)
-

```

There maybe other examples that need fixing. Will send patches as needed.