[setup] require torch>=1.4
Author: stas00
Pull request: yes
Labels: name: WIP

I run the non-slow test suite on lower torch versions:

* torch-1.2 and below is definitely a no go - a gazillion of errors in the test suite.

* torch-1.3+ is mostly OK, but:
```
FAILED tests/test_modeling_bart.py::BartHeadTests::test_generate_fp16 - RuntimeError: "argmax_cuda" not implemented for 'Half'
FAILED tests/test_modeling_funnel.py::FunnelModelIntegrationTest::test_inference_tiny_model - OSError: Unable to load weights from pytorch checkpoi...
FAILED tests/test_modeling_gpt2.py::GPT2ModelTest::test_model_outputs_equivalence - RuntimeError: Expected object of scalar type Float but got scal...
FAILED tests/test_modeling_lxmert.py::LxmertModelTest::test_lxmert_pretraining - RuntimeError: Expected object of scalar type Float but got scalar ...
FAILED tests/test_modeling_openai.py::OpenAIGPTModelTest::test_model_outputs_equivalence - RuntimeError: Expected object of scalar type Float but g...
FAILED tests/test_modeling_reformer.py::ReformerLocalAttnModelTest::test_reformer_model_fp16_generate - RuntimeError: "argmax_cuda" not implemented...
FAILED tests/test_modeling_reformer.py::ReformerLSHAttnModelTest::test_reformer_model_fp16_forward - RuntimeError: "argmax_cuda" not implemented fo...
FAILED tests/test_modeling_reformer.py::ReformerLSHAttnModelTest::test_reformer_model_fp16_generate - RuntimeError: "argmax_cuda" not implemented f...
```
which could be fixed in the core if desired, but it won't pass as is right now.

* torch-1.4 - has mostly serialization issues (files saved with new pytorch can't be read by the old one) 

Hence changing to `torch>=1.4`

@sgugger, @LysandreJik

Comments:
@patrickvonplaten @sshleifer @sgugger Could we solve the errors here to have `torch>=1.3` instead?

May be it's simpler to set it to 1.4+ and only if someone asks for it to bother with 1.3?

After discussion with the team, we'll take a look at supporting v1.3+ in the coming days, and if it requires too many efforts we'll stick with v1.4+. We'll take this as an opportunity to test the versions we say we support as well (1.3, 1.4, 1.5, 1.6, 1.7) so that the README isn't full of empty promises :slightly_smiling_face:.

Could you keep your PR as-is for the coming days, and let me come back to you when we've reached a consensus?

That's an excellent and clear proposition, @LysandreJik - thank you!

> We'll take this as an opportunity to test the versions we say we support as well 

If I may propose a scheduled CI that runs all tests for each of the supported versions, say, once a week or so. Probably `tf` too.

I trust you will have the best plan. 

ping

will try to work on it today - are we sticking to torch 1.3 @LysandreJik ?

Maybe we could discuss also whether we can do some more general optimizations in the lib then (I think we can safely change the attention masks to bools then)

Yes, we are! There's a branch in progress here: https://github.com/huggingface/transformers/tree/previous-torch
Feel free to push fixes onto it directly. I've been planning on doing so right after the TAPAS merge.

The branch tests out torch versions going back to v1.3. It's not setup for slow tests right now, and it tests on every commit. I haven't really thought about if this is the best way to do so, but it's certainly easier to debug the failing tests this way.

I don't suppose there is a point at resolving the conflict, right? 

ping

I'd like to get to it as soon as I have some availability.

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.

ping

It's on the roadmap!

It seems wasteful trying to keep up with the conflicts for more than 6 months. Since it's on the roadmap I think it's safe to close this one now.