Ci test tf super slow
Author: LysandreJik
Pull request: yes

Enable the slow TF suite on GPU. Below are the current failing tests. 

Investigating whether they're actually failing or if it's for some other reason.

- [x] ~FAILED tests/test_modeling_marian.py::ModelManagementTests::test_model_names~
- [x] ~FAILED tests/test_modeling_prophetnet.py::ProphetNetModelIntegrationTest::test_cnndm_inference~
- [x] ~FAILED tests/test_modeling_prophetnet.py::ProphetNetModelIntegrationTest::test_pretrained_checkpoint_hidden_states~
- [x] ~FAILED tests/test_modeling_prophetnet.py::ProphetNetModelIntegrationTest::test_question_gen_inference~
- [x] ~FAILED tests/test_modeling_roberta.py::RobertaModelIntegrationTest::test_inference_classification_head~
- [x] ~FAILED tests/test_modeling_roberta.py::RobertaModelIntegrationTest::test_inference_masked_lm~
- [x] ~FAILED tests/test_modeling_roberta.py::RobertaModelIntegrationTest::test_inference_no_head~
- [x] ~FAILED tests/test_modeling_squeezebert.py::SqueezeBertModelIntegrationTest::test_inference_classification_head~
- [x] ~FAILED tests/test_modeling_tf_albert.py::TFAlbertModelTest::test_saved_model_with_attentions_output~
- [x] ~FAILED tests/test_modeling_tf_albert.py::TFAlbertModelTest::test_saved_model_with_hidden_states_output~
- [x] ~FAILED tests/test_modeling_tf_camembert.py::TFCamembertModelIntegrationTest::test_output_embeds_base_model~
- [x] ~FAILED tests/test_modeling_tf_electra.py::TFElectraModelTest::test_saved_model_with_attentions_output~
- [x] ~FAILED tests/test_modeling_tf_electra.py::TFElectraModelTest::test_saved_model_with_hidden_states_output~
- [x] ~FAILED tests/test_modeling_tf_flaubert.py::TFFlaubertModelTest::test_saved_model_with_attentions_output~
- [x] ~FAILED tests/test_modeling_tf_flaubert.py::TFFlaubertModelTest::test_saved_model_with_hidden_states_output~
- [x] ~FAILED tests/test_modeling_tf_funnel.py::TFFunnelModelTest::test_saved_model_with_attentions_output~
- [x] ~FAILED tests/test_modeling_tf_funnel.py::TFFunnelModelTest::test_saved_model_with_hidden_states_output~
- [x] ~FAILED tests/test_modeling_tf_funnel.py::TFFunnelBaseModelTest::test_saved_model_with_attentions_output~
- [x] ~FAILED tests/test_modeling_tf_funnel.py::TFFunnelBaseModelTest::test_saved_model_with_hidden_states_output~
- [x] FAILED tests/test_modeling_tf_longformer.py::TFLongformerModelTest::test_saved_model_with_attentions_output
- [x] ~FAILED tests/test_modeling_tf_lxmert.py::TFLxmertModelTest::test_saved_model_with_attentions_output~
- [x] ~FAILED tests/test_modeling_tf_lxmert.py::TFLxmertModelTest::test_saved_model_with_hidden_states_output~
- [x] ~FAILED tests/test_modeling_tf_mobilebert.py::TFMobileBertModelTest::test_model_from_pretrained~
- [x] FAILED tests/test_modeling_tf_t5.py::TFT5ModelTest::test_saved_model_with_attentions_output
- [x] FAILED tests/test_modeling_tf_t5.py::TFT5ModelTest::test_saved_model_with_hidden_states_output
- [x] ~FAILED tests/test_modeling_tf_xlm_roberta.py::TFFlaubertModelIntegrationTest::test_output_embeds_base_model~
- [x] ~FAILED tests/test_modeling_tf_xlnet.py::TFXLNetModelLanguageGenerationTest::test_lm_generate_xlnet_base_cased~
- [x] ~FAILED tests/test_modeling_transfo_xl.py::TransfoXLModelLanguageGenerationTest::test_lm_generate_transfo_xl_wt103~
- [x] ~FAILED tests/test_modeling_xlm_prophetnet.py::XLMProphetNetModelIntegrationTest::test_ntg_hidden_states~
- [x] ~FAILED tests/test_modeling_xlm_prophetnet.py::XLMProphetNetModelIntegrationTest::test_pretrained_checkpoint_hidden_states~
- [x] ~FAILED tests/test_modeling_xlm_prophetnet.py::XLMProphetNetModelIntegrationTest::test_xprophetnet_ntg_inference~
- [x] ~FAILED tests/test_modeling_xlm_roberta.py::XLMRobertaModelIntegrationTest::test_xlm_roberta_base~
- [x] ~FAILED tests/test_modeling_xlm_roberta.py::XLMRobertaModelIntegrationTest::test_xlm_roberta_large~
- [x] FAILED tests/test_pipelines.py::PipelineCommonTests::test_tf_defaults - Value...
- [x] ~FAILED tests/test_tokenization_fsmt.py::FSMTTokenizationTest::test_match_encode_decode~

And while we're at it here are the remaining failing tests in PyTorch slow multi-gpu tests:

- [x] FAILED tests/test_data_collator.py::DataCollatorIntegrationTest::test_nsp - K...
- [x] FAILED tests/test_modeling_common.py::ModelUtilsTest::test_model_from_pretrained
- [x] ~FAILED tests/test_modeling_rag.py::RagModelIntegrationTests::test_rag_sequence_generate_batch~
- [x] ~FAILED tests/test_modeling_rag.py::RagModelIntegrationTests::test_rag_sequence_generate_beam~
- [x] ~FAILED tests/test_modeling_rag.py::RagModelIntegrationTests::test_rag_token_generate_batch~
- [x] ~FAILED tests/test_modeling_rag.py::RagModelIntegrationTests::test_rag_token_generate_beam~
- [x] ~FAILED tests/test_modeling_rag.py::RagModelIntegrationTests::test_rag_token_inference~
- [x] FAILED tests/test_pipelines.py::MonoColumnInputTestCase::test_integration_torch_conversation
- [x] FAILED tests/test_pipelines.py::MonoColumnInputTestCase::test_integration_torch_conversation_truncated_history
- [x] FAILED tests/test_pipelines.py::DialoguePipelineTests::test_torch_conversation
- [x] FAILED tests/test_pipelines.py::PipelineCommonTests::test_pt_defaults - Value...

The RAG integration tests seem to happen because of OOM errors, deactivated them in a multi-gpu setup as the GPUs have less memory.

Status: 

Done! Waiting for the green tests to merge.

Comments:
`tests/test_tokenization_fsmt.py::FSMTTokenizationTest::test_match_encode_decode` fixed in https://github.com/huggingface/transformers/pull/8031

I checked it off in your list.

@patrickvonplaten can you take a look at the failing TF-Longformer and TF-T5 tests?

> @patrickvonplaten can you take a look at the failing TF-Longformer and TF-T5 tests?

Will take a look at the TF-Longformer Test :-) I tried a bit unsuccessfully to debug the `TF-T5` test for 2h a week ago and I didn't manage to get rid of `cast_bool_....` for `TFT5` at the moment. I think the hacky `cast_bool_...` function is the reason for the T5 Test Failure. Not sure if it's worth it spending a lot of time here again. :-/ I could comment out the test for now? Or @jplu @LysandreJik do you have a nice TF insight to solve it? 

For now I propose to comment them out. I will do a pass over it later in a couple of weeks. I planned to go through each TF model anyway.

I commented both TF T5 tests out ATM, same for the longformer. I think the Longformer test can be fixed, but the T5 tests cannot, at least not without introducing breaking changes.

I think that we would need to have an additional layer between the `TFT5Model` and the encoder/decoder models, same as we do with every other model (the xxxMainLayer), otherwise we won't be able to use the saved model. Even with the `cast_bool_....` they're not currently usable as saved models.

It's the same issue with BART, and why those two tests are commented in BART as well imo.

All tests are passing now. The only error on the scheduled is because of something on the hub with one tiny models, which I'm fixing right now.