Issues counting passing rates on tests which use subTest()
Author: dvrogozh
Pull request: no

As we discussed in https://github.com/huggingface/transformers/pull/34723#issuecomment-2479470083, we spotted that tests using `with self.subTest` along with `self.skipTest()` report confusing passing rates. We need to understand how we should handle this. Below is breakdown of behavior on few syntetic tests.

Note:
* Behavior depends on whether `pytest-subtest` is installed or not
* Accelerate brings in `pytest-subtest` into environment, see https://github.com/huggingface/accelerate/blob/c0552c9012a9bae7f125e1df89cf9ee0b0d250fd/setup.py#L25
* Transformers do not depend on `pytest-subtest` at the moment
* Transformers ci does not have `pytest-subtest` in the environment at the moment (according to below breakdown this means that if first subtest is skipped other subtests won't be executed)

Synthetic tests:
* `ex0.py` - `subTest()` without skipping:

```
$ cat ex0.py
import unittest
class T(unittest.TestCase):
    def test_foo(self):
        for i in range(7):
            with self.subTest(i=i):
                self.assertLess(i, 3)
```

* `ex1.py` - `subTest()` with skipping:

```
$ cat ex1.py
import unittest
class T(unittest.TestCase):
    def test_foo(self):
        for i in range(7):
            with self.subTest(i=i):
                if i < 3:
                    self.skipTest(f"bon {i}")
                self.assertLess(i, 3)

```

| | | Failed | Passed | Skipped |
| --- | --- | --- | --- | --- |
| No `pytest-subtests` | `ex0.py` | 1 | 0 | 0 |
| No `pytest-subtests` | `ex1.py` | 0 | 0 | 1 |
| With `pytest-subtests` | `ex0.py` | 4 | 1 | 0 |
| With `pytest-subtests` | `ex1.py` | 1 | 1 | 3 |

Logs:
* With `pytest-subtests`:

```
$ python -m pytest ex0.py
...
(i=3) SUBFAIL ex0.py::T::test_foo - AssertionError: 3 not less than 3
(i=4) SUBFAIL ex0.py::T::test_foo - AssertionError: 4 not less than 3
(i=5) SUBFAIL ex0.py::T::test_foo - AssertionError: 5 not less than 3
(i=6) SUBFAIL ex0.py::T::test_foo - AssertionError: 6 not less than 3
=================================== 4 failed, 1 passed in 0.11s ===================================

$ python -m pytest ex1.py
...
(i=6) SUBFAIL ex1.py::T::test_foo - AssertionError: 6 not less than 3
============================= 1 failed, 1 passed, 3 skipped in 0.11s ===
```

* No pytest-subtests:

```
$ python -m pytest ex0.py
FAILED ex0.py::T::test_foo - AssertionError: 3 not less than 3
======================================== 1 failed in 0.11s ========================================


$ python -m pytest ex1.py
...
======================================= 1 skipped in 0.07s ========================================
```

CC: @ydshieh @ArthurZucker 

Comments:
It turns out this is related to pytest interacting with unittest and subTest/skipTest.

If running with `python3 ex1.py` (or `ex0.py`) with those files with ending

```
if __name__ == "__main__":
    unittest.main()

```
then `subTest/skipTest` works well together ...



There are couple open issues which probably worth mentioning:
* https://github.com/pytest-dev/pytest/issues/1367 - it's opened, long thread, last update in 2021
* https://github.com/pytest-dev/pytest/issues/13756 - it's also opened, it partially describes concerns we have, last commented 2023, no replies from maintainers

Opening an pull request in 

https://github.com/pytest-dev/pytest-subtests/pull/169

Might take some time to be merged (if it happens)

Closed as https://github.com/pytest-dev/pytest-subtests/pull/169 is merged