[DEV Testing] Issues with `test_modeling_common`
Author: muellerzr
Pull request: no

I was writing some new tests to help us fix GA down in the modeling code and propagate the fix manually, since everything is breaking constantly with each fix and taking the dive. 

When writing a test for this, which should be limiting to all ForCausalLM models, I was finding some strange errors on models that realistically shouldn't be trying to run these tests.

One of the general failures:
```
E       ValueError: Unrecognized configuration class <class 'transformers.models.vipllava.configuration_vipllava.VipLlavaConfig'> for this kind of AutoModel: AutoModelForCausalLM.
E       Model type should be one of AriaTextConfig
```

Said test:
```python

    def test_training_gradient_accumulation(self):
        if not self.model_tester.is_training:
            self.skipTest(reason="ModelTester is not configured to run training tests")

        if len(self.all_generative_model_classes) == 0:
            self.skipTest(f"No generative model classes for {self.__class__.__name__}")

        for model_class in self.all_generative_model_classes:
            config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()

            with tempfile.TemporaryDirectory() as tmpdir:
                with torch.device(torch_device):
                    model_eager = AutoModelForCausalLM.from_config(
                        config, torch_dtype=torch.float32
                    )

                model_eager.save_pretrained(tmpdir)
                with torch.device(torch_device):
                    model = AutoModelForCausalLM.from_pretrained(
                        tmpdir, torch_dtype=torch.float32
                    )
                    inputs_dict["num_items_in_batch"] = inputs_dict["input_ids"].shape[0]
                    res = model(**inputs_dict, return_dict=False)[0]
```

List of tests:
```
FAILED tests/models/aria/test_modeling_aria.py::AriaForConditionalGenerationModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.aria.configuration_aria.AriaConfig'> for this kind of AutoModel: AutoMod...
FAILED tests/models/autoformer/test_modeling_autoformer.py::AutoformerModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.autoformer.configuration_autoformer.AutoformerConfig'> for this kind of ...
FAILED tests/models/bart/test_modeling_bart.py::BartModelTest::test_training_gradient_accumulation - TypeError: BartForCausalLM.forward() got an unexpected keyword argument 'decoder_input_ids'
FAILED tests/models/bigbird_pegasus/test_modeling_bigbird_pegasus.py::BigBirdPegasusModelTest::test_training_gradient_accumulation - TypeError: BigBirdPegasusForCausalLM.forward() got an unexpected keyword argument 'decoder_input_ids'
FAILED tests/models/blenderbot/test_modeling_blenderbot.py::BlenderbotModelTest::test_training_gradient_accumulation - TypeError: BlenderbotForCausalLM.forward() got an unexpected keyword argument 'decoder_input_ids'
FAILED tests/models/blenderbot_small/test_modeling_blenderbot_small.py::BlenderbotSmallModelTest::test_training_gradient_accumulation - TypeError: BlenderbotSmallForCausalLM.forward() got an unexpected keyword argument 'decoder_input_ids'
FAILED tests/models/blip_2/test_modeling_blip_2.py::Blip2ForConditionalGenerationDecoderOnlyTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.blip_2.configuration_blip_2.Blip2Config'> for this kind of AutoModel: Au...
FAILED tests/models/bloom/test_modeling_bloom.py::BloomModelTest::test_training_gradient_accumulation - ValueError: Got unexpected arguments: {'num_items_in_batch': 14}
FAILED tests/models/flava/test_modeling_flava.py::FlavaImageModelTest::test_training_gradient_accumulation - AttributeError: 'FlavaImageModelTester' object has no attribute 'is_training'
FAILED tests/models/flava/test_modeling_flava.py::FlavaMultimodalModelTest::test_training_gradient_accumulation - AttributeError: 'FlavaMultimodalModelTester' object has no attribute 'is_training'
FAILED tests/models/flava/test_modeling_flava.py::FlavaImageCodebookTest::test_training_gradient_accumulation - AttributeError: 'FlavaImageCodebookTester' object has no attribute 'is_training'
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsForVisionText2TextTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.idefics.configuration_idefics.IdeficsConfig'> for this kind of AutoModel...
FAILED tests/models/idefics2/test_modeling_idefics2.py::Idefics2ForConditionalGenerationModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.idefics2.configuration_idefics2.Idefics2Config'> for this kind of AutoMo...
FAILED tests/models/idefics3/test_modeling_idefics3.py::Idefics3ForConditionalGenerationModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.idefics3.configuration_idefics3.Idefics3Config'> for this kind of AutoMo...
FAILED tests/models/imagegpt/test_modeling_imagegpt.py::ImageGPTModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.imagegpt.configuration_imagegpt.ImageGPTConfig'> for this kind of AutoMo...
FAILED tests/models/informer/test_modeling_informer.py::InformerModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.informer.configuration_informer.InformerConfig'> for this kind of AutoMo...
FAILED tests/models/instructblip/test_modeling_instructblip.py::InstructBlipForConditionalGenerationDecoderOnlyTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.instructblip.configuration_instructblip.InstructBlipConfig'> for this ki...
FAILED tests/models/instructblipvideo/test_modeling_instructblipvideo.py::InstructBlipVideoForConditionalGenerationDecoderOnlyTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.instructblipvideo.configuration_instructblipvideo.InstructBlipVideoConfi...
FAILED tests/models/kosmos2/test_modeling_kosmos2.py::Kosmos2ModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.kosmos2.configuration_kosmos2.Kosmos2Config'> for this kind of AutoModel...
FAILED tests/models/led/test_modeling_led.py::LEDModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.led.configuration_led.LEDConfig'> for this kind of AutoModel: AutoModelF...
FAILED tests/models/llava/test_modeling_llava.py::LlavaForConditionalGenerationModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.llava.configuration_llava.LlavaConfig'> for this kind of AutoModel: Auto...
FAILED tests/models/llava_next/test_modeling_llava_next.py::LlavaNextForConditionalGenerationModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.llava_next.configuration_llava_next.LlavaNextConfig'> for this kind of A...
FAILED tests/models/llava_next_video/test_modeling_llava_next_video.py::LlavaNextVideoForConditionalGenerationModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.llava_next_video.configuration_llava_next_video.LlavaNextVideoConfig'> f...
FAILED tests/models/llava_onevision/test_modeling_llava_onevision.py::LlavaOnevisionForConditionalGenerationModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.llava_onevision.configuration_llava_onevision.LlavaOnevisionConfig'> for...
FAILED tests/models/longt5/test_modeling_longt5.py::LongT5ModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.longt5.configuration_longt5.LongT5Config'> for this kind of AutoModel: A...
FAILED tests/models/longt5/test_modeling_longt5.py::LongT5TGlobalModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.longt5.configuration_longt5.LongT5Config'> for this kind of AutoModel: A...
FAILED tests/models/m2m_100/test_modeling_m2m_100.py::M2M100ModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.m2m_100.configuration_m2m_100.M2M100Config'> for this kind of AutoModel:...
FAILED tests/models/marian/test_modeling_marian.py::MarianModelTest::test_training_gradient_accumulation - TypeError: MarianForCausalLM.forward() got an unexpected keyword argument 'decoder_input_ids'
FAILED tests/models/mbart/test_modeling_mbart.py::MBartModelTest::test_training_gradient_accumulation - TypeError: MBartForCausalLM.forward() got an unexpected keyword argument 'decoder_input_ids'
FAILED tests/models/mllama/test_modeling_mllama.py::MllamaForCausalLMModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.mllama.configuration_mllama.MllamaTextConfig'> for this kind of AutoMode...
FAILED tests/models/mllama/test_modeling_mllama.py::MllamaForConditionalGenerationModelTest::test_training_gradient_accumulation - ValueError: The checkpoint you are trying to load has model type `mllama_text_model` but Transformers does not recognize this architecture. This ...
FAILED tests/models/moshi/test_modeling_moshi.py::MoshiTest::test_training_gradient_accumulation - TypeError: MoshiForCausalLM.forward() got an unexpected keyword argument 'moshi_audio_codes'
FAILED tests/models/mt5/test_modeling_mt5.py::MT5ModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.mt5.configuration_mt5.MT5Config'> for this kind of AutoModel: AutoModelF...
FAILED tests/models/mvp/test_modeling_mvp.py::MvpModelTest::test_training_gradient_accumulation - TypeError: MvpForCausalLM.forward() got an unexpected keyword argument 'decoder_input_ids'
FAILED tests/models/nllb_moe/test_modeling_nllb_moe.py::NllbMoeModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.nllb_moe.configuration_nllb_moe.NllbMoeConfig'> for this kind of AutoMod...
FAILED tests/models/paligemma/test_modeling_paligemma.py::PaliGemmaForConditionalGenerationModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.paligemma.configuration_paligemma.PaliGemmaConfig'> for this kind of Aut...
FAILED tests/models/patchtsmixer/test_modeling_patchtsmixer.py::PatchTSMixerModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.patchtsmixer.configuration_patchtsmixer.PatchTSMixerConfig'> for this ki...
FAILED tests/models/pegasus/test_modeling_pegasus.py::PegasusModelTest::test_training_gradient_accumulation - TypeError: PegasusForCausalLM.forward() got an unexpected keyword argument 'decoder_input_ids'
FAILED tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.pegasus_x.configuration_pegasus_x.PegasusXConfig'> for this kind of Auto...
FAILED tests/models/pix2struct/test_modeling_pix2struct.py::Pix2StructModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.pix2struct.configuration_pix2struct.Pix2StructConfig'> for this kind of ...
FAILED tests/models/plbart/test_modeling_plbart.py::PLBartModelTest::test_training_gradient_accumulation - TypeError: PLBartForCausalLM.forward() got an unexpected keyword argument 'decoder_input_ids'
FAILED tests/models/prophetnet/test_modeling_prophetnet.py::ProphetNetModelTest::test_training_gradient_accumulation - TypeError: ProphetNetForCausalLM.forward() got an unexpected keyword argument 'decoder_input_ids'
FAILED tests/models/qwen2_5_vl/test_modeling_qwen2_5_vl.py::Qwen2_5_VLModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.qwen2_5_vl.configuration_qwen2_5_vl.Qwen2_5_VLConfig'> for this kind of ...
FAILED tests/models/qwen2_vl/test_modeling_qwen2_vl.py::Qwen2VLModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.qwen2_vl.configuration_qwen2_vl.Qwen2VLConfig'> for this kind of AutoMod...
FAILED tests/models/sam/test_modeling_sam.py::SamModelTest::test_training_gradient_accumulation - AttributeError: 'SamModelTester' object has no attribute 'is_training'
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelWithSpeechInputTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.seamless_m4t.configuration_seamless_m4t.SeamlessM4TConfig'> for this kin...
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelWithTextInputTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.seamless_m4t.configuration_seamless_m4t.SeamlessM4TConfig'> for this kin...
FAILED tests/models/seamless_m4t_v2/test_modeling_seamless_m4t_v2.py::SeamlessM4Tv2ModelWithSpeechInputTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.seamless_m4t_v2.configuration_seamless_m4t_v2.SeamlessM4Tv2Config'> for ...
FAILED tests/models/seamless_m4t_v2/test_modeling_seamless_m4t_v2.py::SeamlessM4Tv2ModelWithTextInputTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.seamless_m4t_v2.configuration_seamless_m4t_v2.SeamlessM4Tv2Config'> for ...
FAILED tests/models/speech_to_text/test_modeling_speech_to_text.py::Speech2TextModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.speech_to_text.configuration_speech_to_text.Speech2TextConfig'> for this...
FAILED tests/models/superglue/test_modeling_superglue.py::SuperGlueModelTest::test_training_gradient_accumulation - AttributeError: 'SuperGlueModelTester' object has no attribute 'is_training'
FAILED tests/models/superpoint/test_modeling_superpoint.py::SuperPointModelTest::test_training_gradient_accumulation - AttributeError: 'SuperPointModelTester' object has no attribute 'is_training'
FAILED tests/models/switch_transformers/test_modeling_switch_transformers.py::SwitchTransformersModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.switch_transformers.configuration_switch_transformers.SwitchTransformers...
FAILED tests/models/t5/test_modeling_t5.py::T5ModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.t5.configuration_t5.T5Config'> for this kind of AutoModel: AutoModelForC...
FAILED tests/models/time_series_transformer/test_modeling_time_series_transformer.py::TimeSeriesTransformerModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.time_series_transformer.configuration_time_series_transformer.TimeSeries...
FAILED tests/models/udop/test_modeling_udop.py::UdopModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.udop.configuration_udop.UdopConfig'> for this kind of AutoModel: AutoMod...
FAILED tests/models/umt5/test_modeling_umt5.py::UMT5ModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.umt5.configuration_umt5.UMT5Config'> for this kind of AutoModel: AutoMod...
FAILED tests/models/video_llava/test_modeling_video_llava.py::VideoLlavaForConditionalGenerationModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.video_llava.configuration_video_llava.VideoLlavaConfig'> for this kind o...
FAILED tests/models/vipllava/test_modeling_vipllava.py::VipLlavaForConditionalGenerationModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.vipllava.configuration_vipllava.VipLlavaConfig'> for this kind of AutoMo...
FAILED tests/models/whisper/test_modeling_whisper.py::WhisperModelTest::test_training_gradient_accumulation - KeyError: 'input_ids'
```

cc @ydshieh @ArthurZucker 

Comments:
Hello @muellerzr . Do you already have a branch that you can share so we can run and check 🙏 ?

@ydshieh kinda. It's my own testing branch but basically add that test to `test_modeling_common` (my testing branch is about to get much worse for the bug I'm looking into 😓 )

OK, taking a look!

I think it's because 


`self.all_generative_model_classes` doesn't only means `AutoModelForCausalLM`.

If restricting only to `xxxForCausalLM` models is an option, you can filter out within (at the beginning of )  `for model_class in self.all_generative_model_classes:` by checking against `MODEL_FOR_CAUSAL_LM_MAPPING`.

If you want to test for all kind of `all_generative_model_classes`, you can simply use `model_class(....)` I think.

@ydshieh that seems to somewhat work, but then leads me to:

```python
______________________________________________ FlavaImageModelTest.test_training_gradient_accumulation _______________________________________________

self = <tests.models.flava.test_modeling_flava.FlavaImageModelTest testMethod=test_training_gradient_accumulation>

    def test_training_gradient_accumulation(self):
>       if not self.model_tester.is_training:
E       AttributeError: 'FlavaImageModelTester' object has no attribute 'is_training'
```

Which yes, I could just do:

```python

        if not getattr(self.model_tester, "is_training", False):
            self.skipTest(reason="ModelTester is not configured to run training tests")
```
But wanted to point out another difference in existing code

Yeah, thanks. Not sure how

> class FlavaImageModelTest(ModelTesterMixin)

is passing, maybe all tests that use `self.model_tester.is_training` for that model is skipped 😢 



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.