rfc: integration tests need non-example application for testing
Author: stas00
Pull request: no
Labels: name: Tests

name: Benchmarks

# 🚀 Feature request

We have an ongoing conflict with some of the core integration tests needing a serious program to be tested with. The only place these can be found is under `examples/` - and so the tests - e.g. deepspeed/apex/fairscale reside under `examples/` because of that.
The problem is that because they are under `examples/` they are being treated as such, but they are not examples.

I propose we have at least one complex representative example turned into a serious program that is being supported like any other core function. Such program or programs can then be used for integration testing and what we keep on discussing but not getting to do is performance regression. You can't do performance regression on mock ups. 

The few attempts to measure bleu scores are being killed as well with the current `seq2seq` wipeout. How can one do regression testing if there is nothing to measure. The regressions can be subtle and not detected by general common tests. It's a way easier to know that this input should give this bleu score on this model and if it doesn't then something is wrong.

@patrickvonplaten, @sgugger, @LysandreJik, @patil-suraj  

Comments:
I'm all for having core integration tests to do regression testing. As you have said, these tests should not be under `examples/` as that is a dedicated `examples/` folder, but should be under `tests/`.

I'm not 100% sure whether we would want that in existing testing files (for example a BART regression test in `test_modeling_bart.py`), or if we would want to create new files.

We could also create a new file `test_modeling_common_integration.py` that would serve a similar purpose to `test_modeling_common.py`, but for integration tests, if these can be shared among models simply.

---

Regarding how to approach this, these would be quite heavy tests to run, and take a long time. Do we want to run them daily, like other slow tests, or should we create a weekly suite? We'll need to create a weekly suite eventually, as some TensorFlow tests take 3+ hours *per* model to test `saved_models` (cc @jplu)

---

I believe you've proposed a similar approach to the registry system you've built for fastai, I think this would be a good approach to tackle the issue. Happy to help set this up/work with you on that front to keep the current performance regression tests you have created.

Thank you for your feedback, @LysandreJik 

I think one thing that was missing is that this need is not only for performance regression testing, but also for normal testing of deepspeed/apex/fairscale which are part of the core. So I think I wasn't clear at communicating that for that I need a real program, such as the ones we have under examples. So the tests run this program. As compared to a normal test that has all the logic contained within. This is for functionality testing. So we have 2 unrelated things:

1. performance+quality regression testing - is our core getting slower? is it delivering worse quality?
2. 3rd party component integration functionality testing - can we run HF Trainer w/ DeepSpeed on a gpu with only 3 legs?

They are common only as such that they should be placed somewhere under the core tests.

Wrt number 1 - yes, we started discussing at how to implement that practically (there was an idea of reusing the registry), but again I'm returning to the main need which is not that, but perhaps adapting one of the example scripts to be a testing tool and not an example:
- not facing users - clean refactored code
- probably needs to have several different functionalities - so that different aspects can be tested - probably it needs to cover all the main NLP tasks (not exhaustively, but say one translation, one summarization, etc.) So that the different main logic paths can be tested.

Once we have the tool then we can see how to start recording and validating results. Of course, it can be an organically need-based grown tool, and my first question is where such tool would live.

It'll also be used for posting public benchmarks - so users should be able to use it too to reproduce reported results, but not try to read its code as they would with an example, just as an opaque tool.

I won't worry at the moment at how often we run those things, the schedule will evolve once we have something in place and then we can see what the requirements are.

We don't need to do one hour training to detect a quality or performance regression, while we can - we should instead design optimized scenarios where bad things are detected within a much quicker time span. 

At the moment these are just feeler notes, I'd be happy to start compiling a detailed proposal once others get a chance to voice their inspirations and of course we need to see if there is a group's desire to go in that direction.

we agreed to copy what's needed for the benchmarking/testing which may happen down the road.