Github actions: more readable/informative output
Author: sshleifer
Pull request: no
Labels: name: Help wanted

name: Tests

### TLDR
The goal of this Issue is to make the output of https://github.com/huggingface/transformers/runs/1269744369?check_suite_focus=true more readable, without changing the fact that the job failed.

#### Current Workflow:

I randomly remember to check github actions every 3 days, scroll a long way to see what broke, figure out who to assign (sometimes easy/sometimes hard), and then make issues or slack that person. The goal of this PR is to try to make that process easier by attacking the first two steps, but ideas for automating parts of the second two steps are welcome.



#### Goals:
- user can click on github actions and see what broke quickly. 
- (Optional) user can see 50 slowest tests
- (HARD/Optional) user can subscribe to slack message/email that pings them when action breaks (Related: https://github.community/t/subscribing-to-actions-notifications/135328)


#### How to attack this PR
I would make a new github action file that runs only on your branch and makes the output/artifacts files you desire. Then, once it is working, apply similar logic to `.github/workflows/self-scheduled.yml`.  Once PR is merged, send follow up PR applying same logic to `.github/workflows/self-push.yml`


Comments:
I will surely give it a try, @sshleifer 

So the idea of running my own gh action didn't quite work https://github.com/huggingface/transformers/pull/7890 as I don't have the right permissions to do so.

But that doesn't mean this issue can't be worked on.

# Making information easier/quicker to find 

Basically we have the logs of the tests run and nothing else. Currently finding something in those logs is not trivial, so this part can definitely be improved. We could `tee` the logs into a file and then do a postprocessing on those (will need to figure out how to run a subsequent task despite the test suite task failure, but I'm sure in the worst case some bash magic could work around it). So we can make various clickable tasks that will show just specific outputs. I can also check whether there are some pytest extensions that might help here. Or we might write our own extension that will do just that - log different interesting things into different files.

You can tell me which info chunks you want and I can work on generating those from the log file. Currently you mentioned:
- show slow tests (already in the log file, just need to make them a separate one-click accessible item, I think pytest has an option to log that into a separate report)
- show just the errors

anything else?

This part would be just as useful to normal CIs.

# Communicating with specific user

a. the first problem is which user? a scheduled job may have dozens of merges in it - how do you find out which of them made the commit that caused a failure? 

   Unless you make some kind of contact-table and look up the contact based on which test file failed. I was under the impression you wanted to contact not the specific maintainer, but the original PR creator.

b. any push communications require some kind of auth, which would be very tricky with public repo. it could probably use some webhooks to ping some external server that could do something with that info. e.g. it could have the right auth and post on failure to a slack channel which devs can watch - easier than watching git hub actions (pull model).


@sgugger suggested that CircleCI has an Artifacts tab, where we could add various bit of info.

I want exactly the artifacts file of the circleci job (just uses `|tee output.txt`) available in github actions, but it sounds like that is hard.

Re, communicating with specific user > can be just lysandre for now. I already added the SLACK_WEBHOOK_URL to secrets, but this should definitely not be your task given privs.


To automate the finding of the specific user we will need to `git bisect` for the range of commits since the last check with a rerun of just the failing tests , which will give us the first hash, and thus the PR that caused it. 

This efficient approach may not work 100% of the time, if the failing test happens due to some other test (test dependency) affecting it, so if this speedy reduction doesn't work, then the full test suite will need to be re-run in conjunction with `git bisect`.

WIP https://github.com/huggingface/transformers/pull/7995