Fix some bugs for two stage training of deformable detr
Author: jypjypjypjyp
Pull request: yes

Hello  @amyeroberts:
There are some issues encountered when training with the two-stage method using Deformable DETR, for which I have made modifications.

Comments:
@jypjypjypjyp Thanks for opening this PR! Could you: share a minimal code snippet in the PR which triggers the errors on main and which this PR resolves? 

> @jypjypjypjyp Thanks for opening this PR! Could you: share a minimal code snippet in the PR which triggers the errors on main and which this PR resolves?

Hello @amyeroberts:

Using the original code might not result in errors, but there are apparent mistakes in several places in the source code. Let me explain my modifications:

1. Line 2002: In the original code, 'enc_outputs' is stored in outputs, but the computation of loss uses **outputs_loss**, which means enc_outputs will never be utilized.
2. Line 2267: Due to the first point, 'enc_outputs' will never be used, so this problem will never arise. However, if the first issue is remedied, this problem is revealed. Throughout the entire code, 'class_labels' is used as a key rather than 'labels'.
3. Line 2270: This piece of code is derived from the original Deformable Detr code, and there is no log functionality in our implementation, rendering this code meaningless.
4. Line 1972: This segment of code involves the calculation of the auxiliary loss; if the original code is used, an error will be thrown during the calculation of the auxiliary loss because the shape of the tensor passed in is incorrect.

In my usage, the modified code can be correctly trained. It's possible that my understanding is incorrect, and I hope you can verify these issues.

@jypjypjypjyp Thank you for explaining so clearly and with such detail, it really helps :) 

All of these changes make sense. My only request is that we add a test to make sure the model can train when `config.two_stage` is `True` e.g. somethlng like `test_training_two_stage`, similar to [test_training](https://github.com/huggingface/transformers/blob/03f98f96836477f6f5b86957d3ce98778cad5d94/tests/test_modeling_common.py#L508). 

@amyeroberts Sorry, I'm not very familiar with the test portion of the code. Could you help me complete it, or explain it in more detail? I'm unsure about what specifically I should do.

@jypjypjypjyp Of course :) 

All models are tested to make sure that they can be trained i.e. do a forwards / backwards pass. Most models' tests are implemented in [test_modeling_common.py](https://github.com/huggingface/transformers/blob/d27e4c18fe2970abcb9a48dcb8a824e48083b15f/tests/test_modeling_common.py#L4). 

By default, Deformable Detr has [`two_stage` set to False](https://github.com/huggingface/transformers/blob/d27e4c18fe2970abcb9a48dcb8a824e48083b15f/src/transformers/models/deformable_detr/configuration_deformable_detr.py#L184C1-L184C1), and only the [default model config value](https://github.com/huggingface/transformers/blob/d27e4c18fe2970abcb9a48dcb8a824e48083b15f/tests/models/deformable_detr/test_modeling_deformable_detr.py#L136) was used during testing. This is why the issues with two stage training were never uncovered. 

Each model has its own test module for its model logic e.g. [test_modeling_deformable_detr.py](https://github.com/huggingface/transformers/blob/d27e4c18fe2970abcb9a48dcb8a824e48083b15f/tests/models/deformable_detr/test_modeling_deformable_detr.py). Here we can add model specific tests which aren't covered by the tests in test_modeling_common.py. 

I'm suggesting that we add another test to specifically test two-stage training in `DeformableDetrModelTest` e.g. something like: 

```python

class DeformableDetrModelTest
    ...

    def test_two_stage_training(self):
        model_class =  DeformableDetrForObjectDetection
        config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()
        config.return_dict = True
        config.two_stage = True

        model = model_class(config)
        model.to(torch_device)
        model.train()
        inputs = self._prepare_for_class(inputs_dict, model_class, return_labels=True)
        loss = model(**inputs).loss
        loss.backward()
```


 

@amyeroberts Hello, Thank you for your tutorial! I have added a test_two_stage_training and discovered another issue - the feature dimension of get_proposal_pos_embed is fixed. I made modifications to address this (which also involved modifying the code in deta when using 'make fix-copies'). Please review again.

_The documentation is not available anymore as the PR was closed or merged._