[tests] run one test but got 2 test results
Author: faaany
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.47.0.dev0
- Platform: Linux-4.18.0-425.3.1.el8.x86_64-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.26.5
- Safetensors version: 0.4.5
- Accelerate version: 1.1.0.dev0
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: bf16
        - use_cpu: False
        - debug: False
        - num_processes: 2
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: all
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - enable_cpu_affinity: False
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
- PyTorch version (GPU?): 2.5.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA A100 80GB PCIe

### Who can help?

@ydshieh
 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

with the following test command:
```bash
pytest -rA tests/models/mra/test_modeling_mra.py::MraModelTest::test_load_with_mismatched_shapes
``` 
I will get 2 test results:
```bash
======================================================== short test summary info =========================================================
PASSED tests/models/mra/test_modeling_mra.py::MraModelTest::test_load_with_mismatched_shapes
[Testing <class 'transformers.models.mra.modeling_mra.MraForSequenceClassification'>] SUBFAIL tests/models/mra/test_modeling_mra.py::MraModelTest::test_load_with_mismatched_shapes - ValueError: sequence length must be divisible by the block_size.
================================================ 1 failed, 1 passed, 3 warnings in 6.27s =================================================
```

### Expected behavior

I expect that this command gives_ me only one test result rather than two. 

Below are the experiments I tried:

- with pytest-subtests installed, comment out "with self.subTest(msg=f"Testing {model_class}"):", test pass with 1 test case
- with pytest-subtests uninstalled, comment out "with self.subTest(msg=f"Testing {model_class}"):", test pass with 1 test case
- with pytest-subtests uninstalled, test fails with 1 test case 

I also saw your other PR #34806. Maybe this is another issue with `pytest-subtests`?

Comments:
Hi @faaany !

`pytest-subtests` is designed to report multiple states (regarding subtests) even for a single test method.
So far in `transformers`, the CI env. doesn't have `pytest-subtests` installed (yet).
Also `pytest-subtests` has some issue regarding the reporting, see [this](https://github.com/pytest-dev/pytest-subtests/pull/169).

 `transformers` might install `pytest-subtests` in the CI env. soon and we definitely need to work on some issues that haven't been observed so far. But `reporting multiple states for a single test` is from `pytest-subtests` and not something `transformers` could change.





This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.