Make CI failure summary more useful
Author: amyeroberts
Pull request: yes
Labels: name: WIP

# What does this PR do?


The current 'Failed Tests' summary output of tests isn't useful e.g. here: https://app.circleci.com/pipelines/github/huggingface/transformers/95692/workflows/2c1a2ff4-1182-46f4-b331-33d9fdb0a2c4/jobs/1260625

Output:
```   
1 failed because `AssertionError` -> torch.Size([1, 512, 37890]) != (1, 512, 1026)
2 failed because `AssertionError` -> False is not true
Number of failed tests: 3
```

As: 
* There is no record of the test which have failed. This means, to address failing tests, one still needs to go the main test output and parse yourself
* Without the test name messages like: "`AssertionError` -> False is not true" is not informative (pretty much all tests fail this way) 

Additional nit: "splitted" isn't a word. splitted -> split

Example of an updated output [for this run](https://app.circleci.com/pipelines/github/huggingface/transformers/95689/workflows/106ed803-3f9a-41d4-b6db-709929cc7db9/jobs/1260576): 

```
tests/models/mask2former/test_image_processing_mask2former.py::Mask2FormerImageProcessingTest::test_call_numpy failed because `AssertionError: Tuples differ` -> (1, 3, 32, 91) != (1, 3, 4032, 32)
tests/models/mask2former/test_image_processing_mask2former.py::Mask2FormerImageProcessingTest::test_call_numpy_4_channels failed because `AssertionError: Tuples differ` -> (1, 4, 32, 32) != (1, 4, 1320, 32)
tests/models/swin2sr/test_image_processing_swin2sr.py::Swin2SRImageProcessingTest::test_call_numpy failed because `AssertionError: Tuples differ` -> (1, 3, 136, 384) != (1, 3, 384, 8)
tests/models/swin2sr/test_image_processing_swin2sr.py::Swin2SRImageProcessingTest::test_call_numpy_4_channels failed because `AssertionError: Tuples differ` -> (1, 266, 104, 8) != (1, 4, 104, 8)
tests/models/yolos/test_image_processing_yolos.py::YolosImageProcessingTest::test_call_numpy failed because `AssertionError: Tuples differ` -> (1, 3, 16, 48) != (1, 3, 1376, 0)
tests/models/yolos/test_image_processing_yolos.py::YolosImageProcessingTest::test_call_numpy_4_channels failed because `AssertionError: Tuples differ` -> (1, 4, 48, 16) != (1, 4, 432, 16)
tests/models/vit/test_image_processing_vit.py::ViTImageProcessingTest::test_fast_is_faster_than_slow failed because `AssertionError` -> 0.003969907760620117 not less than or equal to 0.003456274668375651
tests/models/idefics2/test_image_processing_idefics2.py::Idefics2ImageProcessingTest::test_call_numpy_4_channels failed because `TypeError: Cannot handle this data type` -> (1, 1, 157), |u1
```



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31426). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Example of an updated output [for this run](https://app.circleci.com/pipelines/github/huggingface/transformers/95689/workflows/106ed803-3f9a-41d4-b6db-709929cc7db9/jobs/1260576):

From this link, I see (under `Failed Tests` section)

```bash
   1 failed because `AssertionError: Tuples differ` -> (1, 3, 48, 24) != (1, 3, 48, 0)
   1 failed because `AssertionError: Tuples differ` -> (1, 4, 48, 16) != (1, 4, 48, 2)
   1 failed because `AssertionError: Tuples differ` -> (1, 3, 22, 18) != (1, 3, 1692, 18)
   1 failed because `AssertionError: Tuples differ` -> (1, 4, 18, 31) != (1, 4, 1039, 18)
   1 failed because `AssertionError: Tuples differ` -> (1, 3, 18, 54) != (1, 3, 1692, 18)
   1 failed because `AssertionError: Tuples differ` -> (1, 4, 48, 18) != (1, 4, 607, 18)
Number of failed tests: 6

Exited with code exit status 1
```

Is this the right link, or is the above output what you expect ..? Or I am looking the wrong section?



> From this link, I see (under Failed Tests section)

Ah, yes, sorry, I wasn't completely clear. The link I shared was for a run with the old formatting. I took the output of `tests_outputs.txt` for that run and passed that to the updated `parse_test_ouptuts.py` script locally with `python3 .circleci/parse_test_outputs.py --file tests_output.txt --fail` which returns the output: 


```
tests/models/mask2former/test_image_processing_mask2former.py::Mask2FormerImageProcessingTest::test_call_numpy failed because `AssertionError: Tuples differ` -> (1, 3, 32, 91) != (1, 3, 4032, 32)
tests/models/mask2former/test_image_processing_mask2former.py::Mask2FormerImageProcessingTest::test_call_numpy_4_channels failed because `AssertionError: Tuples differ` -> (1, 4, 32, 32) != (1, 4, 1320, 32)
tests/models/swin2sr/test_image_processing_swin2sr.py::Swin2SRImageProcessingTest::test_call_numpy failed because `AssertionError: Tuples differ` -> (1, 3, 136, 384) != (1, 3, 384, 8)
tests/models/swin2sr/test_image_processing_swin2sr.py::Swin2SRImageProcessingTest::test_call_numpy_4_channels failed because `AssertionError: Tuples differ` -> (1, 266, 104, 8) != (1, 4, 104, 8)
tests/models/yolos/test_image_processing_yolos.py::YolosImageProcessingTest::test_call_numpy failed because `AssertionError: Tuples differ` -> (1, 3, 16, 48) != (1, 3, 1376, 0)
tests/models/yolos/test_image_processing_yolos.py::YolosImageProcessingTest::test_call_numpy_4_channels failed because `AssertionError: Tuples differ` -> (1, 4, 48, 16) != (1, 4, 432, 16)
tests/models/vit/test_image_processing_vit.py::ViTImageProcessingTest::test_fast_is_faster_than_slow failed because `AssertionError` -> 0.003969907760620117 not less than or equal to 0.003456274668375651
tests/models/idefics2/test_image_processing_idefics2.py::Idefics2ImageProcessingTest::test_call_numpy_4_channels failed because `TypeError: Cannot handle this data type` -> (1, 1, 157), |u1
Number of failed tests: 8
```


Got it. I like this format of information (I feel the same as you regarding the current output).

But I think @ArthurZucker would also like to keep the information about counting of each type of failure. So probably it would be better to show **both** these 2 kinds of information.

Note: the counting like  

> 1 failed because `AssertionError: Tuples differ` -> (1, 3, 48, 24) != (1, 3, 48, 0)
is not so great as the shape may different for different tests.

But for some failure outputs, they are nice, like

> 10 failed because `ReadTimeout: (...same error message ...)

we can wait what Arthur say though.

> But for some failure outputs, they are nice, like

Importantly only for some! In the example of ReadTimeouts, I agree just knowing the number of tests can be useful, because the tests aren't actually failing because of the code. This way we can know the global effect something external has on our test suite. I'd argue that, even if we have a more informative error, this is only useful if we actually know which test is comes from. 

I'll update, so that we get the summary of the number of failures at the bottom and you can let me know what you think of the output :) We can also decide to split it up so we have two CI stages, one with the counts and one with the failures 

`ReadTimeout` is just one example pop up in my mind, but I believe there are some types of failure worth a counting.

I do agree that it's very useful to actually know/show which tests failed with the error information.

> I'll update, so that we get the summary of the number of failures at the bottom and you can let me know what you think of the output :) We can also decide to split it up so we have two CI stages, one with the counts and one with the failures

Thank you!

I've added back counts in the form of a summary at the bottom of the output. I've also sorted them so they're in descending order (most common error on-top). For [this run](https://app.circleci.com/pipelines/github/huggingface/transformers/95683/workflows/e813dea2-7607-4c46-994d-7a2e8bb0bd68/jobs/1260447)

Old output:
```
   1 failed because `huggingface_hub.utils._errors.RepositoryNotFoundError: 404 Client Error. (Request ID` -> Root=1-666c5df0-44ed53b967e5209427bc7629;8de677a5-ac62-43c9-a890-08f5ae5a5f73)
   1 failed because `OSError: remote` -> Repository not found
   1 failed because `huggingface_hub.utils._errors.RepositoryNotFoundError: 404 Client Error. (Request ID` -> Root=1-666c5dfb-22f3b85118386c6569284476;b005a2af-8b8e-4277-9fa3-3c5fad50fa26)
   1 failed because `huggingface_hub.utils._errors.RepositoryNotFoundError: 404 Client Error. (Request ID` -> Root=1-666c5dfc-0cb35eda6b8aeb14720f32b1;fa6c49c7-a8c2-42c5-b72f-ebff1c0ded71)
   1 failed because `huggingface_hub.utils._errors.HfHubHTTPError: 500 Server Error: Internal Server Error for url: https://hub-ci.huggingface.co/api/models/__DUMMY_TRANSFORMERS_USER__/test-image-processor/preupload/main (Request ID` -> Root=1-666c5e02-012aafb1444f4fb50f5330d5;34edf9ae-787c-4240-b714-e72a2a1226db)
   1 failed because `huggingface_hub.utils._errors.RepositoryNotFoundError: 404 Client Error. (Request ID` -> Root=1-666c5e05-7004de021c4377c600920b4d;2b3cd217-17e0-48a8-be84-f3686a9d104b)
   4 failed because `KeyError` -> 'url'
Number of failed tests: 10
```

New output: 
```
Failures:
tests/trainer/test_trainer.py::TrainerIntegrationWithHubTester::test_push_to_hub_with_saves_each_n_steps failed because `huggingface_hub.utils._errors.RepositoryNotFoundError: 404 Client Error. (Request ID` -> Root=1-666c5df0-44ed53b967e5209427bc7629;8de677a5-ac62-43c9-a890-08f5ae5a5f73)
tests/test_modeling_utils.py::ModelPushToHubTester::test_push_to_hub_dynamic_model failed because `KeyError` -> 'url'
tests/models/auto/test_processor_auto.py::ProcessorPushToHubTester::test_push_to_hub failed because `KeyError` -> 'url'
tests/models/auto/test_processor_auto.py::ProcessorPushToHubTester::test_push_to_hub_dynamic_processor failed because `OSError: remote` -> Repository not found
tests/test_configuration_utils.py::ConfigPushToHubTester::test_push_to_hub failed because `huggingface_hub.utils._errors.RepositoryNotFoundError: 404 Client Error. (Request ID` -> Root=1-666c5dfb-22f3b85118386c6569284476;b005a2af-8b8e-4277-9fa3-3c5fad50fa26)
tests/test_configuration_utils.py::ConfigPushToHubTester::test_push_to_hub_dynamic_config failed because `huggingface_hub.utils._errors.RepositoryNotFoundError: 404 Client Error. (Request ID` -> Root=1-666c5dfc-0cb35eda6b8aeb14720f32b1;fa6c49c7-a8c2-42c5-b72f-ebff1c0ded71)
tests/test_image_processing_utils.py::ImageProcessorPushToHubTester::test_push_to_hub failed because `huggingface_hub.utils._errors.HfHubHTTPError: 500 Server Error: Internal Server Error for url: https://hub-ci.huggingface.co/api/models/__DUMMY_TRANSFORMERS_USER__/test-image-processor/preupload/main (Request ID` -> Root=1-666c5e02-012aafb1444f4fb50f5330d5;34edf9ae-787c-4240-b714-e72a2a1226db)
tests/test_tokenization_utils.py::TokenizerPushToHubTester::test_push_to_hub failed because `huggingface_hub.utils._errors.RepositoryNotFoundError: 404 Client Error. (Request ID` -> Root=1-666c5e05-7004de021c4377c600920b4d;2b3cd217-17e0-48a8-be84-f3686a9d104b)
tests/generation/test_configuration_utils.py::ConfigPushToHubTester::test_push_to_hub failed because `KeyError` -> 'url'
tests/generation/test_configuration_utils.py::ConfigPushToHubTester::test_push_to_hub_in_organization failed because `KeyError` -> 'url'

Summary:
   4 failed because of `KeyError` -> 'url'
   1 failed because of `huggingface_hub.utils._errors.RepositoryNotFoundError: 404 Client Error. (Request ID` -> Root=1-666c5df0-44ed53b967e5209427bc7629;8de677a5-ac62-43c9-a890-08f5ae5a5f73)
   1 failed because of `OSError: remote` -> Repository not found
   1 failed because of `huggingface_hub.utils._errors.RepositoryNotFoundError: 404 Client Error. (Request ID` -> Root=1-666c5dfb-22f3b85118386c6569284476;b005a2af-8b8e-4277-9fa3-3c5fad50fa26)
   1 failed because of `huggingface_hub.utils._errors.RepositoryNotFoundError: 404 Client Error. (Request ID` -> Root=1-666c5dfc-0cb35eda6b8aeb14720f32b1;fa6c49c7-a8c2-42c5-b72f-ebff1c0ded71)
   1 failed because of `huggingface_hub.utils._errors.HfHubHTTPError: 500 Server Error: Internal Server Error for url: https://hub-ci.huggingface.co/api/models/__DUMMY_TRANSFORMERS_USER__/test-image-processor/preupload/main (Request ID` -> Root=1-666c5e02-012aafb1444f4fb50f5330d5;34edf9ae-787c-4240-b714-e72a2a1226db)
   1 failed because of `huggingface_hub.utils._errors.RepositoryNotFoundError: 404 Client Error. (Request ID` -> Root=1-666c5e05-7004de021c4377c600920b4d;2b3cd217-17e0-48a8-be84-f3686a9d104b)
Number of failed tests: 10
```

TLDR, let's leave the parsing to junit-xlm 😉 
and let's educate people on how to use the test tab 

@ArthurZucker I'm hoping that you can try and see this from my and others' perspective here, and I feel very strongly about this. 

I'm happy for this information to be in a separate, expanded section that's shown on the main CI page, rather than bundled with the grouped errors, but this should easily available i.e. shown immediately without requiring further parsing or steps.

The reason I opened this PR is I'm becoming increasingly frustrated with trying to navigate the CI and efficiently getting the information I need to do both for my maintenance responsibilities and PR development.

If I encounter failures on the CI the first thing I want to know is how to resolve them. In order to resolve them I need to have the list of failed tests. Having these means I can develop and test locally, quickly without needing repeated pushes to CI which will be slow. 

The current ways to get this information are cumbersome: 
* Copy all the verbose information in `Run Tests` tab and then parse to extract the test names (remove verbose failure information, traceback etc.). If there are many failed tests with long tracebacks this is even more annoying to stroll to extract what you need. 
* Go to tests tab (extra click). Copy/paste from HTML sections, parse to get the test names
* Go to artefacts (extra click), go through every failures file for each of the parallel runs (many clicks) and combine to get the list of failures. 

Regarding your reply: 
> the goal of this is not to tell you exactly what happened

The goal of "what" are we talking about here? 

I'd argue telling you what happened is the point of the CI outputs. Otherwise, how are we suppose to efficiently debug failures? We might as well have a CI that just says "failed/not failed". 

> You have the tab test for this and I want to keep it this way.
> Let's push people to use the test tab, everything is parsed here, 
> and let's educate people on how to use the test tab

I know how to use the tests tab; it's clunky and makes development slower

As noted above - this is fine for looking at insights on the circle CI webpage, but it isn't good for trying to fix tests locally and makes information less accessible.

> TLDR, let's leave the parsing to junit-xlm 😉

No, this is not a satisfactory solution. I shouldn't have to take extra steps to get information which can be easily provided.