Add DAB-DETR Object detection/segmentation model
Author: conditionedstimulus
Pull request: yes
Labels: name: New model

name: Vision

name: run-slow

# What does this PR do?

Add DAB-DETR Object detection model. Paper: https://arxiv.org/abs/2201.12329
Original code repo: https://github.com/IDEA-Research/DAB-DETR

Fixes # (issue)
[WIP] This model is part of how DETR models have evolved, alongside DN DETR (not part of this PR), to pave the way for newer and better models like Dino and Stable Dino in object detection


## Who can review?
@amyeroberts




Comments:
Hi @conditionedstimulus, thanks for opening a PR! 

Just skimming over the modeling files, it looks like all of the modules are copied from, or can be copied from conditional DETR. Are there any architectural changes this model brings? If not, then all we need to do is convert the checkpoints and upload those to the hub such that they can be loaded in ConditionalDETR directly 

> Hi @conditionedstimulus, thanks for opening a PR!
> 
> Just skimming over the modeling files, it looks like all of the modules are copied from, or can be copied from conditional DETR. Are there any architectural changes this model brings? If not, then all we need to do is convert the checkpoints and upload those to the hub such that they can be loaded in ConditionalDETR directly

Hi Amy,

I attached a photo comparing the cross-attention of the decoder in DETR, Conditional DETR, and DAB DETR, as this is the main architectural difference. I copied the code from Conditional DETR because this model is an extension/evolved version of Conditional DETR. I believe it would be cool and useful to include this model in the HF object detection collection.
![Screenshot 2024-05-15 at 22 25 15](https://github.com/huggingface/transformers/assets/37246112/0eefc8af-b6cd-447a-9c2f-d4dc714ba019)


@conditionedstimulus Thanks for sharing! OK, seems useful to have this available as an option as part of the DETR family in the library. Feel free to ping me when the PR is ready for review. 

cc @qubvel for reference 

Hi @amyeroberts,

I'm ready with version 0.1, but I've encountered some issues that require your assistance and guidance.

I trained two models: 
- One on the CPPE-5 dataset following the HF Object Detection tutorial ([link to the notebook](https://github.com/conditionedstimulus/hf_media/blob/main/CPPE_5.ipynb)) (Results could be better)
- The other on the balloon dataset as per the Transformers Tutorials [(link to the notebook)](https://github.com/conditionedstimulus/hf_media/blob/main/Fine_tuning_DAB_DETR_on_custom_dataset_(balloon).ipynb). (Results could be better)
- Additionally, I forked the original implementation of the model to compare the outputs (with my implementation in HF) [(link to the notebook)](https://github.com/conditionedstimulus/DAB-DETR/blob/comparing_original_model_w_hf_model/inference_and_visualize.ipynb).


**Issues**
1. Segmentation Head
Unfortunately, the segmentation head wasn't thoroughly tested. Due to this reason, it isn't currently included in the HF tests. (BTW, I left the copied code there.) I opened an [issue](https://github.com/IDEA-Research/DAB-DETR/issues/77) on their Github but I also found their [answer](https://github.com/IDEA-Research/DAB-DETR/issues/28) under a similar issue.

2. Shared Weights
DAB DETR utilizes shared weights, which has caused some challenges with the saving methods in HF. This has also led to similar saving issues in the HF trainer. I'm curious to know if you have any insights or solutions to address this problem. [This portion of the code is particularly relevant to this issue.](https://github.com/conditionedstimulus/transformers/blob/0151f654de3e7d058d53cb44e2ecf307f4fbc1db/src/transformers/models/dab_detr/modeling_dab_detr.py#L1520)

3. Github Checks are failing. 
I attempted to investigate further, but I lack the necessary permissions to access the reasons for the failures.

4. Image processor test fails.
I probably didn't need to implement the ImageProcessor since it's identical to the one used in Conditional DETR, and AutoProcessor should handle it. However, I implemented it just in case.

The image processor test for the DAB DETR model is encountering some inconsistencies. When executed as a .py file, it fails to produce the desired results. However, upon running the command "tests/models/dab_detr/test_image_processing_dab_detr.py::DABDETRImageProcessingTest," the test successfully passes. Interestingly, the regular DETR model's image processor test exhibits similar issues, indicating a potential broader problem. It's unclear whether this is specific to my local setup or a more widespread concern.

**Changes and open questions**

- Removed the DetrAttention module.
I wasn't sure why this module is part of all DETR-based model implementations. After testing the models, I realized that the reason might be to track the gradient values of the encoder model's attention weights when the optional parameter output_attentions is set to True. In the original implementation, they used a simple torch.nn.MultiHeadAttention. I did the same because the code is much simpler, and the attention weights are still extractable for each layer, just like in the Hugging Face approach. The only difference is that the gradient values are not available in the original PyTorch implementation. Is this a significant issue?

- Slightly 'hackish' transformation.
Since I didn't use the DetrAttention module, I also avoided the [_prepare_4d_attention_mask](https://github.com/conditionedstimulus/transformers/blob/0151f654de3e7d058d53cb44e2ecf307f4fbc1db/src/transformers/models/conditional_detr/modeling_conditional_detr.py#L26) method, which handles pixel_masks transformation. In the original implementation, the image processor managed the pixels and padding in the opposite manner compared to Hugging Face (HF). Specifically, HF uses pixel_values: 1 and padding: 0, whereas the original implementation uses pixel_values: 0 and padding: 1. Consequently, in the HF universe, you need to transform these values as I [did](https://github.com/conditionedstimulus/transformers/blob/0151f654de3e7d058d53cb44e2ecf307f4fbc1db/src/transformers/models/dab_detr/modeling_dab_detr.py#L1617) or as it's done in [_prepare_4d_attention_mask](https://github.com/conditionedstimulus/transformers/blob/0151f654de3e7d058d53cb44e2ecf307f4fbc1db/src/transformers/modeling_attn_mask_utils.py#L184). It's not the most elegant solution, but it works. (Side note: Mine is a boolean tensor, while the HF one is already an Int or FloatTensor.)

- Dropout in Encoder
There is a dropout step in line [1176](https://github.com/conditionedstimulus/transformers/blob/ffbb1dce4537f7dc8e74c90bddad668e7142cb91/src/transformers/models/dab_detr/modeling_dab_detr.py#L1176) that is not part of the original implementation (not even in Conditional DETR). I commented it out, but it might still be important for the training process.

- Bbox Embed Initialization
Although it's not the most elegant solution, it seems crucial to initialize these weights properly. Therefore, I added the [initialization](https://github.com/conditionedstimulus/transformers/blob/ffbb1dce4537f7dc8e74c90bddad668e7142cb91/src/transformers/models/dab_detr/modeling_dab_detr.py#L1514) to the self.post_init() method as shown, rather than leaving it solely in the DABDETRModel init function.

- return_dict=False is messy.
When there are many return variables then it gets messy.

Hi @conditionedstimulus, great! Glad the model is ready for its first review. 

With regards to the issues, could you provide more details? Specifically what is the expected behaviour, what is being experienced, and ideally any errors with full error traceback. Statements like: "which has caused some challenges with the saving methods in HF" are under specified (there's infinite ways something can not work) so unfortunately can't help without more information. 

Hi @amyeroberts,


Shared weights issue. I copied the full error traceback. The error is easy to reproduce since I skipped the following tests:
- test_save_load
- test_save_load_fast_init_from_base
- test_model_weights_reload_no_missing_tied_weights
- test_load_save_without_tied_weights

```

__________________________________________________________________________________________ DABDETRModelTest.test_save_load __________________________________________________________________________________________

self = <tests.models.dab_detr.test_modeling_dab_detr.DABDETRModelTest testMethod=test_save_load>

    def test_save_load(self):
        config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()
    
        def check_save_load(out1, out2):
            # make sure we don't have nans
            out_2 = out2.cpu().numpy()
            out_2[np.isnan(out_2)] = 0
    
            out_1 = out1.cpu().numpy()
            out_1[np.isnan(out_1)] = 0
            max_diff = np.amax(np.abs(out_1 - out_2))
            self.assertLessEqual(max_diff, 1e-5)
    
        for model_class in self.all_model_classes:
            model = model_class(config)
            model.to(torch_device)
            model.eval()
            with torch.no_grad():
                first = model(**self._prepare_for_class(inputs_dict, model_class))[0]
    
            with tempfile.TemporaryDirectory() as tmpdirname:
>               model.save_pretrained(tmpdirname)

tests/test_modeling_common.py:266: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = DABDETRModel(
  (query_refpoint_embeddings): Embedding(12, 4)
  (input_projection): Conv2d(40, 32, kernel_size=(1, 1),...ar(in_features=32, out_features=32, bias=True)
      (2): Linear(in_features=32, out_features=4, bias=True)
    )
  )
)
save_directory = '/var/folders/n9/k7bfnd3961v0l7nx7ld043sh0000gn/T/tmpq3y_y3ef', is_main_process = True
state_dict = OrderedDict({'query_refpoint_embeddings.weight': tensor([[-0.0194,  0.0063, -0.0108, -0.0017],
        [ 0.0267,  0.02... 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0.]]), 'bbox_embed.layers.2.bias': tensor([0., 0., 0., 0.])})
save_function = <function save at 0x1183b8cc0>, push_to_hub = False, max_shard_size = '5GB', safe_serialization = True, variant = None, token = None, save_peft_format = True, kwargs = {}, use_auth_token = None
ignore_metadata_errors = False, _hf_peft_config_loaded = False, hf_quantizer = None, quantization_serializable = False

    def save_pretrained(
        self,
        save_directory: Union[str, os.PathLike],
        is_main_process: bool = True,
        state_dict: Optional[dict] = None,
        save_function: Callable = torch.save,
        push_to_hub: bool = False,
        max_shard_size: Union[int, str] = "5GB",
        safe_serialization: bool = True,
        variant: Optional[str] = None,
        token: Optional[Union[str, bool]] = None,
        save_peft_format: bool = True,
        **kwargs,
    ):
        """
        Save a model and its configuration file to a directory, so that it can be re-loaded using the
        [`~PreTrainedModel.from_pretrained`] class method.
    
        Arguments:
            save_directory (`str` or `os.PathLike`):
                Directory to which to save. Will be created if it doesn't exist.
            is_main_process (`bool`, *optional*, defaults to `True`):
                Whether the process calling this is the main process or not. Useful when in distributed training like
                TPUs and need to call this function on all processes. In this case, set `is_main_process=True` only on
                the main process to avoid race conditions.
            state_dict (nested dictionary of `torch.Tensor`):
                The state dictionary of the model to save. Will default to `self.state_dict()`, but can be used to only
                save parts of the model or if special precautions need to be taken when recovering the state dictionary
                of a model (like when using model parallelism).
            save_function (`Callable`):
                The function to use to save the state dictionary. Useful on distributed training like TPUs when one
                need to replace `torch.save` by another method.
            push_to_hub (`bool`, *optional*, defaults to `False`):
                Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the
                repository you want to push to with `repo_id` (will default to the name of `save_directory` in your
                namespace).
            max_shard_size (`int` or `str`, *optional*, defaults to `"5GB"`):
                The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size
                lower than this size. If expressed as a string, needs to be digits followed by a unit (like `"5MB"`).
                We default it to 5GB in order for models to be able to run easily on free-tier google colab instances
                without CPU OOM issues.
    
                <Tip warning={true}>
    
                If a single weight of the model is bigger than `max_shard_size`, it will be in its own checkpoint shard
                which will be bigger than `max_shard_size`.
    
                </Tip>
    
            safe_serialization (`bool`, *optional*, defaults to `True`):
                Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).
            variant (`str`, *optional*):
                If specified, weights are saved in the format pytorch_model.<variant>.bin.
            token (`str` or `bool`, *optional*):
                The token to use as HTTP bearer authorization for remote files. If `True`, or not specified, will use
                the token generated when running `huggingface-cli login` (stored in `~/.huggingface`).
            save_peft_format (`bool`, *optional*, defaults to `True`):
                For backward compatibility with PEFT library, in case adapter weights are attached to the model, all
                keys of the state dict of adapters needs to be pre-pended with `base_model.model`. Advanced users can
                disable this behaviours by setting `save_peft_format` to `False`.
            kwargs (`Dict[str, Any]`, *optional*):
                Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method.
        """
        use_auth_token = kwargs.pop("use_auth_token", None)
        ignore_metadata_errors = kwargs.pop("ignore_metadata_errors", False)
    
        if use_auth_token is not None:
            warnings.warn(
                "The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.",
                FutureWarning,
            )
            if token is not None:
                raise ValueError(
                    "`token` and `use_auth_token` are both specified. Please set only the argument `token`."
                )
            token = use_auth_token
    
        if token is not None:
            kwargs["token"] = token
    
        _hf_peft_config_loaded = getattr(self, "_hf_peft_config_loaded", False)
    
        hf_quantizer = getattr(self, "hf_quantizer", None)
        quantization_serializable = (
            hf_quantizer is not None and isinstance(hf_quantizer, HfQuantizer) and hf_quantizer.is_serializable
        )
    
        if hf_quantizer is not None and not _hf_peft_config_loaded and not quantization_serializable:
            raise ValueError(
                f"The model is quantized with {hf_quantizer.quantization_config.quant_method} and is not serializable - check out the warnings from"
                " the logger on the traceback to understand the reason why the quantized model is not serializable."
            )
    
        if "save_config" in kwargs:
            warnings.warn(
                "`save_config` is deprecated and will be removed in v5 of Transformers. Use `is_main_process` instead."
            )
            is_main_process = kwargs.pop("save_config")
        if safe_serialization and not is_safetensors_available():
            raise ImportError("`safe_serialization` requires the `safetensors library: `pip install safetensors`.")
    
        if os.path.isfile(save_directory):
            logger.error(f"Provided path ({save_directory}) should be a directory, not a file")
            return
    
        os.makedirs(save_directory, exist_ok=True)
    
        if push_to_hub:
            commit_message = kwargs.pop("commit_message", None)
            repo_id = kwargs.pop("repo_id", save_directory.split(os.path.sep)[-1])
            repo_id = self._create_repo(repo_id, **kwargs)
            files_timestamps = self._get_files_timestamps(save_directory)
    
        # Only save the model itself if we are using distributed training
        model_to_save = unwrap_model(self)
    
        # save the string version of dtype to the config, e.g. convert torch.float32 => "float32"
        # we currently don't use this setting automatically, but may start to use with v5
        dtype = get_parameter_dtype(model_to_save)
        model_to_save.config.torch_dtype = str(dtype).split(".")[1]
    
        # Attach architecture to the config
        model_to_save.config.architectures = [model_to_save.__class__.__name__]
    
        # If we have a custom model, we copy the file defining it in the folder and set the attributes so it can be
        # loaded from the Hub.
        if self._auto_class is not None:
            custom_object_save(self, save_directory, config=self.config)
    
        # Save the config
        if is_main_process:
            if not _hf_peft_config_loaded:
                model_to_save.config.save_pretrained(save_directory)
            if self.can_generate():
                # generation config built from the model config + the model config holds generation kwargs -> generate
                # may revert to legacy behavior if the two don't match
                if (
                    model_to_save.generation_config._from_model_config
                    and model_to_save.config._has_non_default_generation_parameters()
                ):
                    new_generation_config = GenerationConfig.from_model_config(model_to_save.config)
                    if new_generation_config != model_to_save.generation_config:
                        logger.warning(
                            "Your generation config was originally created from the model config, but the model "
                            "config has changed since then. Unless you pass the `generation_config` argument to this "
                            "model's `generate` calls, they will revert to the legacy behavior where the base "
                            "`generate` parameterization is loaded from the model config instead. "
                            "To avoid this behavior and this warning, we recommend you to overwrite the generation "
                            "config model attribute before calling the model's `save_pretrained`, preferably also "
                            "removing any generation kwargs from the model config. This warning will be raised to an "
                            "exception in v4.41."
                        )
                model_to_save.generation_config.save_pretrained(save_directory)
    
            if _hf_peft_config_loaded:
                logger.info(
                    "Detected adapters on the model, saving the model in the PEFT format, only adapter weights will be saved."
                )
                state_dict = model_to_save.get_adapter_state_dict()
    
                if save_peft_format:
                    logger.info(
                        "To match the expected format of the PEFT library, all keys of the state dict of adapters will be pre-pended with `base_model.model`."
                    )
                    peft_state_dict = {}
                    for key, value in state_dict.items():
                        peft_state_dict[f"base_model.model.{key}"] = value
                    state_dict = peft_state_dict
    
                active_adapter = self.active_adapters()
    
                if len(active_adapter) > 1:
                    raise ValueError(
                        "Multiple active adapters detected, saving multiple active adapters is not supported yet. You can save adapters separately one by one "
                        "by iteratively calling `model.set_adapter(adapter_name)` then `model.save_pretrained(...)`"
                    )
                active_adapter = active_adapter[0]
    
                current_peft_config = self.peft_config[active_adapter]
                current_peft_config.save_pretrained(save_directory)
    
        # for offloaded modules
        module_map = {}
    
        # Save the model
        if state_dict is None:
            # if any model parameters are offloaded to the disk, make module map
            if hasattr(self, "hf_device_map") and (
                "cpu" in self.hf_device_map.values() or "disk" in self.hf_device_map.values()
            ):
                warnings.warn(
                    "Attempting to save a model with offloaded modules. Ensure that unallocated cpu memory exceeds the `shard_size` (5GB default)"
                )
                for name, module in model_to_save.named_modules():
                    if name == "":
                        continue
                    module_state_dict = module.state_dict()
    
                    for key in module_state_dict:
                        module_map[name + f".{key}"] = module
    
            state_dict = model_to_save.state_dict()
    
        # Translate state_dict from smp to hf if saving with smp >= 1.10
        if IS_SAGEMAKER_MP_POST_1_10:
            for smp_to_hf, _ in smp.state.module_manager.translate_functions:
                state_dict = smp_to_hf(state_dict)
    
        # Handle the case where some state_dict keys shouldn't be saved
        if self._keys_to_ignore_on_save is not None:
            for ignore_key in self._keys_to_ignore_on_save:
                if ignore_key in state_dict.keys():
                    del state_dict[ignore_key]
        if safe_serialization:
            # Safetensors does not allow tensor aliasing.
            # We're going to remove aliases before saving
            ptrs = collections.defaultdict(list)
            for name, tensor in state_dict.items():
                # Sometimes in the state_dict we have non-tensor objects.
                # e.g. in bitsandbytes we have some `str` objects in the state_dict
                if isinstance(tensor, torch.Tensor):
                    ptrs[id_tensor_storage(tensor)].append(name)
                else:
                    # In the non-tensor case, fall back to the pointer of the object itself
                    ptrs[id(tensor)].append(name)
    
            # These are all the pointers of shared tensors
            if hasattr(self, "hf_device_map"):
                # if the model has offloaded parameters, we must check using find_tied_parameters()
                tied_params = find_tied_parameters(self)
                if tied_params:
                    tied_names = tied_params[0]
                    shared_ptrs = {
                        ptr: names for ptr, names in ptrs.items() if any(name in tied_names for name in names)
                    }
                else:
                    shared_ptrs = {}
            else:
                shared_ptrs = {ptr: names for ptr, names in ptrs.items() if len(names) > 1}
    
            # Recursively descend to find tied weight keys
            _tied_weights_keys = _get_tied_weight_keys(self)
            error_names = []
            to_delete_names = set()
            for names in shared_ptrs.values():
                # Removing the keys which are declared as known duplicates on
                # load. This allows to make sure the name which is kept is consistent.
                if _tied_weights_keys is not None:
                    found = 0
                    for name in sorted(names):
                        matches_pattern = any(re.search(pat, name) for pat in _tied_weights_keys)
                        if matches_pattern and name in state_dict:
                            found += 1
                            if found < len(names):
                                to_delete_names.add(name)
            # We are entering a place where the weights and the transformers configuration do NOT match.
            shared_names, disjoint_names = _find_disjoint(shared_ptrs.values(), state_dict)
            # Those are actually tensor sharing but disjoint from each other, we can safely clone them
            # Reloaded won't have the same property, but it shouldn't matter in any meaningful way.
            for name in disjoint_names:
                state_dict[name] = state_dict[name].clone()
    
            # When not all duplicates have been cleaned, still remove those keys, but put a clear warning.
            # If the link between tensors was done at runtime then `from_pretrained` will not get
            # the key back leading to random tensor. A proper warning will be shown
            # during reload (if applicable), but since the file is not necessarily compatible with
            # the config, better show a proper warning.
            shared_names, identical_names = _find_identical(shared_names, state_dict)
            # delete tensors that have identical storage
            for inames in identical_names:
                known = inames.intersection(to_delete_names)
                for name in known:
                    del state_dict[name]
                unknown = inames.difference(to_delete_names)
                if len(unknown) > 1:
                    error_names.append(unknown)
    
            if shared_names:
                error_names.append(set(shared_names))
    
            if len(error_names) > 0:
>               raise RuntimeError(
                    f"The weights trying to be saved contained shared tensors {error_names} that are mismatching the transformers base configuration. Try saving using `safe_serialization=False` or remove this tensor sharing.",
                )
E               RuntimeError: The weights trying to be saved contained shared tensors [{'bbox_embed.layers.0.weight', 'decoder.bbox_embed.layers.0.weight'}, {'bbox_embed.layers.0.bias', 'decoder.bbox_embed.layers.0.bias'}, {'bbox_embed.layers.1.weight', 'decoder.bbox_embed.layers.1.weight'}, {'decoder.bbox_embed.layers.1.bias', 'bbox_embed.layers.1.bias'}, {'bbox_embed.layers.2.weight', 'decoder.bbox_embed.layers.2.weight'}, {'bbox_embed.layers.2.bias', 'decoder.bbox_embed.layers.2.bias'}] that are mismatching the transformers base configuration. Try saving using `safe_serialization=False` or remove this tensor sharing.

src/transformers/modeling_utils.py:2616: RuntimeError
```

@amyeroberts 
4. Image processor test error.

As I mentioned earlier, it only fails if I run it as: 
`RUN_SLOW=1 pytest tests/models/dab_detr/test_image_processing_dab_detr.py`
If I run it as: 
`RUN_SLOW=1 pytest tests/models/dab_detr/test_image_processing_dab_detr.py::DABDETRImageProcessingTest`
, there are no issues


Error:
```
====================================================================================================== ERRORS =======================================================================================================
_____________________________________________________________________ ERROR collecting tests/models/dab_detr/test_image_processing_dab_detr.py ______________________________________________________________________
/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/unittest/case.py:511: in __hash__
    return hash((type(self), self._testMethodName))
E   AttributeError: 'DABDETRImageProcessingTester' object has no attribute '_testMethodName'
================================================================================================= warnings summary ==================================================================================================
.venv/lib/python3.12/site-packages/_pytest/config/__init__.py:1448
  /Users/davidhajdu/Desktop/Projects/transformers/.venv/lib/python3.12/site-packages/_pytest/config/__init__.py:1448: PytestConfigWarning: Unknown config option: doctest_glob
  
    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================================================================== short test summary info ==============================================================================================
ERROR tests/models/dab_detr/test_image_processing_dab_detr.py::DABDETRImageProcessingTester - AttributeError: 'DABDETRImageProcessingTester' object has no attribute '_testMethodName'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
```

Hi, @conditionedstimulus thanks for working on this model!

Regarding the shared weights issue, you probably should add `_tied_weights_keys = [r"bbox_embed\.\d+"]` as a model class variable.

For example DETA code:
https://github.com/huggingface/transformers/blob/dce253f645e3fce0e4f5d34740ba643ecd2e573d/src/transformers/models/deprecated/deta/modeling_deta.py#L1853

Regarding errors with image processor testing. There was a similar question in [PR](https://github.com/huggingface/transformers/pull/30566), see the [comment](https://github.com/huggingface/transformers/pull/30566#issuecomment-2088589295). 

I hope that helps 🤗 

Hi @qubvel and @amyeroberts,

Thanks to @qubvel's hints, the issues with shared weights and the local testing environment have been resolved. 

Hi @amyeroberts,

**EDIT: The issue might be due to using my forked transformers and the original transformers simultaneously.**

There is still an issue with saving the model in the safetensor format when using the Huggingface Trainer. This is puzzling because I was able to save the model without any issues, and all the save/load tests were successful.

I am fine-tuning the model again using the [HF Object Detection Tutorial Notebook](https://huggingface.co/docs/transformers/en/tasks/object_detection). The 'save_safetensors' parameter in TrainingArguments is set to 'True' by default.

Despite this, I encountered the following error:

```
RuntimeError                              Traceback (most recent call last)
[<ipython-input-19-4ca345dc2cd2>](https://localhost:8080/#) in <cell line: 13>()
     11 )
     12 
---> 13 trainer.train()



7 frames

[/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
   1874                 # Disable progress bars when uploading models during checkpoints to avoid polluting stdout
   1875                 hf_hub_utils.disable_progress_bars()
-> 1876                 return inner_training_loop(
   1877                     args=args,
   1878                     resume_from_checkpoint=resume_from_checkpoint,

[/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in _inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)
   2309 
   2310             self.control = self.callback_handler.on_epoch_end(args, self.state, self.control)
-> 2311             self._maybe_log_save_evaluate(tr_loss, grad_norm, model, trial, epoch, ignore_keys_for_eval)
   2312 
   2313             if DebugOption.TPU_METRICS_DEBUG in self.args.debug:

[/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in _maybe_log_save_evaluate(self, tr_loss, grad_norm, model, trial, epoch, ignore_keys_for_eval)
   2730 
   2731         if self.control.should_save:
-> 2732             self._save_checkpoint(model, trial, metrics=metrics)
   2733             self.control = self.callback_handler.on_save(self.args, self.state, self.control)
   2734 

[/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in _save_checkpoint(self, model, trial, metrics)
   2809         run_dir = self._get_output_dir(trial=trial)
   2810         output_dir = os.path.join(run_dir, checkpoint_folder)
-> 2811         self.save_model(output_dir, _internal_call=True)
   2812 
   2813         if not self.args.save_only_model:

[/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in save_model(self, output_dir, _internal_call)
   3353 
   3354         elif self.args.should_save:
-> 3355             self._save(output_dir)
   3356 
   3357         # Push to the Hub when `save_model` is called by the user.

[/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in _save(self, output_dir, state_dict)
   3424                 logger.info("Trainer.model is not a `PreTrainedModel`, only saving its state dict.")
   3425                 if self.args.save_safetensors:
-> 3426                     safetensors.torch.save_file(
   3427                         state_dict, os.path.join(output_dir, SAFE_WEIGHTS_NAME), metadata={"format": "pt"}
   3428                     )

[/usr/local/lib/python3.10/dist-packages/safetensors/torch.py](https://localhost:8080/#) in save_file(tensors, filename, metadata)
    282     ```
    283     """
--> 284     serialize_file(_flatten(tensors), filename, metadata=metadata)
    285 
    286 

[/usr/local/lib/python3.10/dist-packages/safetensors/torch.py](https://localhost:8080/#) in _flatten(tensors)
    478 
    479     if failing:
--> 480         raise RuntimeError(
    481             f"""
    482             Some tensors share memory, this will lead to duplicate memory on disk and potential differences when loading them again: {failing}.

RuntimeError: 
            Some tensors share memory, this will lead to duplicate memory on disk and potential differences when loading them again: [{'bbox_predictor.layers.0.weight', 'model.decoder.bbox_embed.layers.0.weight'}, {'model.decoder.bbox_embed.layers.0.bias', 'bbox_predictor.layers.0.bias'}, {'bbox_predictor.layers.1.weight', 'model.decoder.bbox_embed.layers.1.weight'}, {'model.decoder.bbox_embed.layers.1.bias', 'bbox_predictor.layers.1.bias'}, {'bbox_predictor.layers.2.weight', 'model.decoder.bbox_embed.layers.2.weight'}, {'model.decoder.bbox_embed.layers.2.bias', 'bbox_predictor.layers.2.bias'}].
            A potential way to correctly save your model is to use `save_model`.
            More information at https://huggingface.co/docs/safetensors/torch_shared_tensors
```

TrainingArguments (with save_safetensors=False it works):
```
training_args = TrainingArguments(
    output_dir="dab_detr_finetuned_cppe5",
    num_train_epochs=30,
    fp16=False,
    per_device_train_batch_size=8,
    dataloader_num_workers=4,
    learning_rate=5e-5,
    lr_scheduler_type="cosine",
    weight_decay=1e-4,
    max_grad_norm=0.01,
    metric_for_best_model="eval_map",
    greater_is_better=True,
    load_best_model_at_end=True,
    eval_strategy="epoch",
    save_strategy="epoch",
    save_total_limit=2,
    remove_unused_columns=False,
    eval_do_concat_batches=False,
    save_safetensors=False,
    push_to_hub=True,
)
```

Hi @amyeroberts,

Just wondering what's next for this pull request. I know there are higher priority tasks, so no rush. Whenever you get a chance, could you give me an update or an estimated timeline for the review?
Thank you!

Hi @conditionedstimulus ~~it seems like the tied weights issue is not resolved, probably you should also add "bbox_predictor" pattern to the list.~~

Oh, I see it's already there...

There are also some quality tests failing, can you, please, fix them?
```
ValueError: The model table in the `index.md` has not been updated. Run `make fix-copies` to fix this.
```

> Hi @conditionedstimulus ~it seems like the tied weights issue is not resolved, probably you should also add "bbox_predictor" pattern to the list.~
> 
> Oh, I see it's already there...
> 
> There are also some quality tests failing, can you, please, fix them?
> 
> ```
> ValueError: The model table in the `index.md` has not been updated. Run `make fix-copies` to fix this.
> ```

Hi @qubvel,

Thank you for your feedback. I really needed to fix some parts! It's all good now.

Thanks!

@qubvel For first review if you have time 

Hi @qubvel,

Thank you for your review, suggestions, and highlights!

I have addressed many of your comments and updated the code accordingly. However, there is still some confusion regarding DABDETRAttention. I’ve provided more details in the comments and hope it clarifies the issues.

Thanks again, and I look forward to your response!

Hi @qubvel ,

Thank you for your review once again!

I’ve made the code modifications and answered your suggestions and requests in the comments.

- I discovered an issue related to repo consistency and style. In the `DABDETRConfig` file, there are two variables essential for setting the backbone structure: `dilatation` and `num_channels`. Since these parameters are only used in the configuration file and not in the modeling code (which only utilizes `load_backbone`), the `repo-consistency` check fails because of this discrepancy. At the moment it fails on `make style` because I added them to DABDETRConvEncoder as unused parameters. 
Any suggestions, maybe I should refactor the whole backbone config in the configuration file?


- There are config parameters, such as query_scale_type, that don’t work with the pretrained structure. I'm wondering how you handle these issues, if at all.

Thanks again, and I look forward to your response!

Hi @qubvel ,

How are you doing?
I merged the main branch and updated the config file. However, I'm not sure how to pass the tests with the dilation and num_channels parameters, since the style check fails if these are only used in the config file and not in the main file. 
Do you have any hints on how to pass the tests? 
At the moment, they are hard-coded, but that's just a temporary solution to pass the tests.

Hi @conditionedstimulus will have a look today/tomorrow!

Hi @qubvel ,

Any update?:)

Hi @conditionedstimulus thanks for fixing the previous comments and sorry for the silence :)

1. Do we need `dilation` and `num_channels` then? How is it handled in RT-DETR?
2. Do you mean that `query_scale_type` will not work with other backbones? I suppose we might warn about it

Hi @qubvel 
> Hi @conditionedstimulus thanks for fixing the previous comments and sorry for the silence :)
> 
> 1. Do we need `dilation` and `num_channels` then? How is it handled in RT-DETR?
- Yes, we do need both parameters. They are set in the config file initialization method at line 234 (`dilation`) and line 238 (`num_channels`) with the current default setup.
- In RT-DETR, there is a ResNet backbone module separately, and that module has a separate config file in addition to the model file config.
> 2. Do you mean that `query_scale_type` will not work with other backbones? I suppose we might warn about it
- I meant that it is not compatible with the pre-trained weights ('davidhajdu/dab-detr-resnet-50'). There are no pre-trained versions with the other type of query_scale_type, only with the default value.



> I meant that it is not compatible with the pre-trained weights ('davidhajdu/dab-detr-resnet-50'). There are no pre-trained versions with the other type of query_scale_type, only with the default value.

Got it, thanks for the clarification! But the model can be created with another `query_scale_type` and fine-tuned, right?

I mean this works without error:
```python
model = DABDetr.from_pretrained("davidhajdu/dab-detr-resnet-50", query_scale_type="cond_scalar")
```

If so, it's fine to have a param that is different from pretrained settings.


> I discovered an issue related to repo consistency and style. In the DABDETRConfig file, there are two variables essential for setting the backbone structure: dilatation and num_channels. Since these parameters are only used in the configuration file and not in the modeling code (which only utilizes load_backbone), the repo-consistency check fails because of this discrepancy. At the moment it fails on make style because I added them to DABDETRConvEncoder as unused parameters.
Any suggestions, maybe I should refactor the whole backbone config in the configuration file?

Regarding the question with "dilation" in config let's see if @amyeroberts has something to suggest (related [commit](https://github.com/huggingface/transformers/pull/30803/commits/ce549c553160004b6b5f3a1c540bb48a87b0871d)) 

> Regarding the question with "dilation" in config let's see if @amyeroberts has something to suggest (related [commit](https://github.com/huggingface/transformers/pull/30803/commits/ce549c553160004b6b5f3a1c540bb48a87b0871d))

In this commit, I can see `dilation` and `num_channels` as previously set in the `DABDETRConvEncoder` class, but it's not clear to me how they were used at all? This will affect how they should be stored. @conditionedstimulus Could you maybe provide some pseudocode snippet to show how these values are being used to manipulate the backbones in the modeling code? 

> > Regarding the question with "dilation" in config let's see if @amyeroberts has something to suggest (related [commit](https://github.com/huggingface/transformers/pull/30803/commits/ce549c553160004b6b5f3a1c540bb48a87b0871d))
> 
> In this commit, I can see `dilation` and `num_channels` as previously set in the `DABDETRConvEncoder` class, but it's not clear to me how they were used at all? This will affect how they should be stored. @conditionedstimulus Could you maybe provide some pseudocode snippet to show how these values are being used to manipulate the backbones in the modeling code?

Hi @amyeroberts,

Both parameters are only part of the default backbone, which is loaded in the `DabDetrConvEncoder `class through the `load_backbone` method. In the basic backbone for RT-DETR, the `num_channels` (line 258 in configuration_rt_detr.py) variable is hardcoded (`dilation` is set to `False` by default in our case and isn't even a parameter in RT-DETR). This setup should work fine for us as well.

If the user wants to use a different backbone, they will need to specify it in the `backbone_config` parameter, as usual.

Hi @conditionedstimulus, 

Just to make sure we're on the same page, looking at the code, I don't think these arguments are used in `load_backbone` at all - AFAICT they're used to control the creation of `backbone_kwargs` in the config. `backbone_kwargs` are then used in `load_backbone`. Is this correct? 

Am I right in understanding that the issue comes from having these arguments in the config file, but they're not used within the model itself? 

There's a few different options available, but the best route will depend on the current models. For the pretrained checkpoints, do they all have different `dilation` and `num_channels` values?  

Hi @amyeroberts 

> Hi @conditionedstimulus,
> 
> Just to make sure we're on the same page, looking at the code, I don't think these arguments are used in `load_backbone` at all - AFAICT they're used to control the creation of `backbone_kwargs` in the config. `backbone_kwargs` are then used in `load_backbone`. Is this correct?

**Excatly!**
> 
> Am I right in understanding that the issue comes from having these arguments in the config file, but they're not used within the model itself?

**Yes!**
> 
> There's a few different options available, but the best route will depend on the current models. For the pretrained checkpoints, do they all have different `dilation` and `num_channels` values?

As a matter of fact, there are 5 different pre-trained models available. Three of these use the `dilation` parameter since they are ResNet50 DC5 versions. The `num_channels` can be hard-coded.

I uploaded these other versions as well and created a comparison notebook with the original implementation:

Original version (simple ResNet50):
You both have already seen this version.
[HF pre-trained weights](https://huggingface.co/davidhajdu/dab-detr-resnet-50)
[Notebook](https://github.com/conditionedstimulus/hf_media/blob/main/inference_and_visualize.ipynb)

R50-pat3 version.
In this version, the model architecture is extended with pattern embeddings to improve the object query design (3 patterns). An interesting point is that they changed the `random_refpoint_xy` parameter in the config file to `True.` This is notable because, when they trained the same pat3 version with DC5, they left it as `False`.
[HF pre-trained weights](https://huggingface.co/davidhajdu/dab-detr-resnet-50-pat3)
[Notebook](https://github.com/conditionedstimulus/hf_media/blob/main/inference_and_visualize_pat3.ipynb)

R50-DC5 version.
This is the DC5 ResNet version, where `dilation` is set to `True` in the config. Another interesting change is that the temperature values are set to 10 instead of 20.
[HF pre-trained weights](https://huggingface.co/davidhajdu/dab-detr-resnet-50-dc5)
[Notebook](https://github.com/conditionedstimulus/hf_media/blob/main/inference_and_visualize_dc5.ipynb)

R50-DC5-fixxy
_The term "fixxy" means we use random initialization of anchors and do not update their parameters during training (See Appendix H of [our paper](https://arxiv.org/abs/2201.12329) for more details)._
[HF pre-trained weights](https://huggingface.co/davidhajdu/dab-detr-resnet-50-dc5-fixxy)
[Notebook](https://github.com/conditionedstimulus/hf_media/blob/main/inference_and_visualize_dc5_fix_xy.ipynb)

R50-DC5-pat3
Same as the simple pat3 version, but in this case, the `random_refpoint_xy` parameter is `False`.
[HF pre-trained weights](https://huggingface.co/davidhajdu/dab-detr-resnet-50-dc5-pat3)
[Notebook](https://github.com/conditionedstimulus/hf_media/blob/main/inference_and_visualize_dc5_pat3.ipynb)


P.S.: The tests are failing because I committed a version where dilation is a parameter again in the config file. Thanks!






Hi @qubvel,

> > I meant that it is not compatible with the pre-trained weights ('davidhajdu/dab-detr-resnet-50'). There are no pre-trained versions with the other type of query_scale_type, only with the default value.
> 
> Got it, thanks for the clarification! But the model can be created with another `query_scale_type` and fine-tuned, right?
> 
> I mean this works without error:
> 
> ```python
> model = DABDetr.from_pretrained("davidhajdu/dab-detr-resnet-50", query_scale_type="cond_scalar")
> ```
> 
> If so, it's fine to have a param that is different from pretrained settings.

**Yes, it only affects the pre-trained versions; fine-tuning will still work.**

**I don't understand how it worked for you. I tried it and got the following error:**

```
Exception has occurred: RuntimeError
Error(s) in loading state_dict for DABDETRForObjectDetection:
	size mismatch for model.decoder.query_scale.layers.1.weight: copying a param with shape torch.Size([256, 256]) from checkpoint, the shape in current model is torch.Size([1, 256]).
	size mismatch for model.decoder.query_scale.layers.1.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([1]).
	You may consider adding `ignore_mismatched_sizes=True` in the model `from_pretrained` method.
```

**It makes sense since the query scale part of the architecture has different dimensions based on the different parameters. I copied the critical part of the architecture.**
```
# cond_elewise
(query_scale): MLP(
        (layers): ModuleList(
          (0-1): 2 x Linear(in_features=256, out_features=256, bias=True)
        )

# cond_scalar
 (query_scale): MLP(
        (layers): ModuleList(
          (0): Linear(in_features=256, out_features=256, bias=True)
          (1): Linear(in_features=256, out_features=1, bias=True)
        )

# fix_elewise
(query_scale): Embedding(6, 256)
```

**Thanks!**
> 
> > I discovered an issue related to repo consistency and style. In the DABDETRConfig file, there are two variables essential for setting the backbone structure: dilatation and num_channels. Since these parameters are only used in the configuration file and not in the modeling code (which only utilizes load_backbone), the repo-consistency check fails because of this discrepancy. At the moment it fails on make style because I added them to DABDETRConvEncoder as unused parameters.
> > Any suggestions, maybe I should refactor the whole backbone config in the configuration file?
> 
> Regarding the question with "dilation" in config let's see if @amyeroberts has something to suggest (related [commit](https://github.com/huggingface/transformers/pull/30803/commits/ce549c553160004b6b5f3a1c540bb48a87b0871d))



Hey @amyeroberts and @qubvel,

Any news?:)

Hi @conditionedstimulus, thanks for ping and sorry for slow responses, we a bit overwhelmed with tasks.

In case dilation is required there are a few options how to avoid unused argument error
1. Assign dilation in one of the modules `self.dilation = config.dilation`
2. Exclude this parameter in utils/check_config_unused_args file

I will have more time next week to review your PR again :hugs: thank you for your patience!

> Hi @conditionedstimulus, thanks for ping and sorry for slow responses, we a bit overwhelmed with tasks.
> 
> In case dilation is required there are a few options how to avoid unused argument error
> 
> 1. Assign dilation in one of the modules `self.dilation = config.dilation`
> 2. Exclude this parameter in utils/check_config_unused_args file
> 
> I will have more time next week to review your PR again 🤗 thank you for your patience!

Hi @qubvel,

No problem, I get it. :)
Next week sounds perfect! I'll make a commit with the modifications you just mentioned!

Thanks again!


Hi @qubvel ,

I've just updated the code + merged the main branch!

Thanks!

> Thanks for the update! Overall looks good to me, I left a few comments below, but I think it's already in pretty good shape!
> 
> For the next steps we have to:
> 
> * make sure all _slow_ tests are pass, you will need to create an empty commit with the message `[run_slow] dab_detr`
> * prepare all model checkpoints and **model cards** for transfer, as a reference you can take RT-DETR model card
> * make sure fine-tuning also works as expected (I can run some experiments)
> * get final approval from core-maintainers
> 
> At the moment I have an issue with the following snippet:
> 
> ```python
> from transformers import AutoModelForObjectDetection
> 
> checkpoint = "davidhajdu/dab-detr-resnet-50"
> 
> id2label = {0: "cat", 1: "dog"}
> label2id = {v: k for k, v in id2label.items()}
> 
> model = AutoModelForObjectDetection.from_pretrained(
>    checkpoint,
>    id2label=id2label,
>    label2id=label2id,
>    ignore_mismatched_sizes=True,
> )
> ```
> 
> Can you check if it is an issue of outdated config or weights on the hub or is there any bug in the code?

Hi @qubvel,

Thank you for your response and feedback.

I have addressed the modifications you requested and completed [a fine-tuning run with the basic model ](https://github.com/conditionedstimulus/hf_media/blob/main/CPPE_5.ipynb)on the CPPE-5 dataset, which[ is an object detection tutorial on the Hugging Face page](https://huggingface.co/docs/transformers/en/tasks/object_detection). How many additional fine-tuning examples would you like?

I plan to finish the model cards today or tomorrow, and I believe the checkpoints are in good shape.

Could you please let me know who the core maintainers are? A simple tag here would be enough to notify them about this PR?

Regarding the dog and cat error, I was unable to reproduce it. I’ve attached a [link](https://github.com/conditionedstimulus/hf_media/blob/main/Dogs_and_cats.ipynb) to my notebook, where you’ll see that it works well(Same example with the CPPE-5 dataset.).

Additionally, I ran the empty commit with the message `[run_slow] dab_detr`, and it worked successfully.

Thank you once again!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30803). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @ArthurZucker,

Thank you for your review and insights. I believe I understand your points, but just to clarify—are you suggesting that I remove any code paths, functions, or configuration variables not related to the pre-trained version? It seems that most users will likely utilize the pre-trained model rather than training one from scratch. There are also different code paths corresponding to various [pre-trained versions](https://github.com/huggingface/transformers/pull/30803#issuecomment-2341915119). Should you/someone select the best-performing models (or based on other criteria) and retain only those codepaths?

I'll refactor and address the other code parts where you left comments!

Thanks again!

Hi @ArthurZucker ,
CC @qubvel,

Thank you once again for the review! I've mostly refactored the parts you requested and removed most of the code paths as suggested since the pre-trained models are preferred. However, there are still a few points that remain unclear, and there are some areas where I don't fully agree. I've addressed nearly all of your comments.

I'm unsure what you meant regarding the loss functions, should we move them to a utils file, as they aren't directly related to the model? Additionally, I'm unclear about your suggestion on positional embeddings, RoPE generates a different type of embedding than the simple sinusoidal embeddings we currently use.

As for the attention functions in DETR-based models, I don’t entirely agree with your feedback. These attention mechanisms are somewhat standard, and a recent model, [RT-DETR](https://github.com/huggingface/transformers/blob/fa3f2db5c7405a742fcb8f686d3754f70db00977/src/transformers/models/rt_detr/modeling_rt_detr.py#L904) (deployed in July), employs a similar attention function.

Thanks again!

Okay let me try to answer on the comments directly! 

> are you suggesting that I remove any code paths, functions, or configuration variables not related to the pre-trained version? It seems that most users will likely utilize the pre-trained model rather than training one from scratch.

exactly. If someone want's to add complexity, he can copy past the model or monky patch it, we want to reflect the architecture of the PreTrainedModel as much as possiblew

For the pretrained checkpoints, if they exist for a certain path we kinda have no choice but to keep it that way! Tho what matters is the intention: we try to remove them, then we try to only have things that change the init and not the forward (ex: create 2 classes instead of 1 with if else) etc 🤗 

Feel free to png me when this is ready for another review 

> Feel free to png me when this is ready for another review

Hi, thanks! Sorry, I got caught up with other errands and couldn’t work on the code for the past two weeks. I’m back at it now and will ping you once I'm done!

> IMO only thing left to do:
> 
> * refactor the attention to make it as close as Llama/Gemma
> * small renaming -> d_model is hidden_size etc. Having a look at modeling llama for the correct standards!
> * move the losses to loss_utils or loss_dab_detr.py
> 
> Awesome refactoring otherwise! 🔥

Hi @ArthurZucker,

I could use your review and help here.

One test is failing, but it’s unrelated to my model. (TFResnet assert error)
- I refactored the attention mechanism, and it looks solid—I think it’s much closer to `mllama` now.
- I also did some renaming, do you suggest any further changes? 
- Additionally, I moved the loss function as you recommended.

The main issue is that the model isn't performing well. It barely learns, validation loss hardly decreases, metrics improve very little, and the final results are poor.
I suspect the problem might lie in the loss function or the attention, with the loss function being my main guess. 
While I implemented it following the approach in Conditional DETR, I’m not entirely sure it’s correct.

[Current fine-tuned model notebook](https://github.com/conditionedstimulus/hf_media/blob/main/CPPE_5_11_03.ipynb)
[Previous model notebook when the results were good](https://github.com/conditionedstimulus/hf_media/blob/main/CPPE_5.ipynb)

Could you help identify where the issue might be? 
Thank you!

Loss function seems good too.

Hi @ArthurZucker,

I tried to pinpoint why the model isn’t learning but haven't identified the issue yet. I’ve tested various model versions, including:

- [The last modeling file version, which previously produced good results](https://github.com/conditionedstimulus/hf_media/blob/main/CPPE_5_11_06_prereview_modelversion.ipynb)
- [Old attention functions](https://github.com/conditionedstimulus/hf_media/blob/main/CPPE_5_11_06_old_attentions.ipynb)
- [The previous loss function](https://github.com/conditionedstimulus/hf_media/blob/main/CPPE_5_11_06_oldloss.ipynb)
I still can't see where the problem lies. Even running the last modeling version (that performed well, before I made any modifications of this review), the results remain poor. This is odd because I also ran the [ConditionalDETR notebook](https://github.com/conditionedstimulus/hf_media/blob/main/CPPE_5_11_03_ConditionalDETR.ipynb), and that model performs well, so it's not the Trainer function.

Any idea? :)
Thanks for your help in finding the issue!

Will leave @qubvel take the review! 🤗 in general super hard for us to jump through and debug training at this stage, will see if it helps!

Hi @qubvel ,

First of all, Happy Holidays!

I’ve made all the modifications  @ArthurZucker requested, but I ran into an issue where the model wasn’t learning. After investigating, I found the problem was with the ImageProcessor. Since the DabDetrImageProcessor was identical to ConditionalDetr, Arthur recommended removing it and using a ready-made (Detr) one. However, it turned out that the DetrImageProcessor wasn’t compatible, while ConditionalDetr worked perfectly.

Everything is now functioning properly, and you can see the latest updates in the [CPPE-5 notebook](https://github.com/conditionedstimulus/hf_media/blob/main/CPPE_5__24_12_25.ipynb).
The model is learning and optimizing well, so it’s ready for another review and, hopefully, to be merged! 😊

Thanks!


Hi @qubvel ,

Any progress?:)

Hi @conditionedstimulus, sorry missed the ping among others! Will check tomorrow 🤗

Hi @conditionedstimulus code-wise everything looks great to me, thanks for refactoring 🤗
I will double-check the inference API and training tomorrow morning. Would you mind if we transfer checkpoints to [IDEA-Research](https://huggingface.co/IDEA-Research) org on the hub? If so, it would be great to add proper model cards (smth similar to RT-DETR [model card](https://huggingface.co/PekingU/rtdetr_r50vd_coco_o365) or DAB-DETR docs).

btw, I see there is a "DAB-Deformable-DETR-R50-v2" checkpoint on [github](https://github.com/IDEA-Research/DAB-DETR/tree/main?tab=readme-ov-file#model-zoo) which has the best metrics, is it possible to convert it as well?

Hi @qubvel,

Thank you for your feedback and review! Here's an update on my progress:

- I merged the main branch into the feature branch.
- I ran the slow tests.
- I updated the model cards.
- Yes, please copy the model cards to IDEA-Research as it is their work, I already updated the 'how to use' part with their source in the model cards.
- Unfortunately, I’m unable to convert the DAB-Deformable-DETR checkpoint, as it uses a different architecture. You can see the differences [here](https://github.com/IDEA-Research/DAB-DETR/blob/main/figure/model.png). (c, e ,f)


Model cards: 
[davidhajdu/dab-detr-resnet-50](https://huggingface.co/davidhajdu/dab-detr-resnet-50)
[davidhajdu/dab-detr-resnet-50-dc5](https://huggingface.co/davidhajdu/dab-detr-resnet-50-dc5)
[davidhajdu/dab-detr-resnet-50-dc5-fixxy](https://huggingface.co/davidhajdu/dab-detr-resnet-50-dc5-fixxy)
[davidhajdu/dab-detr-resnet-50-dc5-pat3](https://huggingface.co/davidhajdu/dab-detr-resnet-50-dc5-pat3)
[davidhajdu/dab-detr-resnet-50-pat3](https://huggingface.co/davidhajdu/dab-detr-resnet-50-pat3)

If you would like any modifications to the model cards, please let me know!

run-slow: dab_detr

This comment contains run-slow, running the specified jobs: ['models/dab_detr'] ...

I see these 3 tests are failing on CI:

```
FAILED tests/models/dab_detr/test_modeling_dab_detr.py::DabDetrModelTest::test_cpu_offload - RuntimeError: Attempted to call `variable.set_data(tensor)`, but `variable` and `tensor` have incompatible tensor type.
FAILED tests/models/dab_detr/test_modeling_dab_detr.py::DabDetrModelTest::test_disk_offload_safetensors - RuntimeError: Attempted to call `variable.set_data(tensor)`, but `variable` and `tensor` have incompatible tensor type.
FAILED tests/models/dab_detr/test_modeling_dab_detr.py::DabDetrModelTest::test_save_load_low_cpu_mem_usage - RuntimeError: Attempted to call `variable.set_data(tensor)`, but `variable` and `tensor` have incompatible tensor type.
```

Did they pass locally?

Also it seems like images are not loading in model cards (maybe my issue only). Feel free to open a upload them into [dataset](https://huggingface.co/datasets/huggingface/documentation-images/tree/main/transformers/model_doc) with "Add file" button and ping me to merge.

> test_cpu_offload

Hey, yes I don't have these kind of errors locally. I have one issue but that related to the model weight sources. I already left the new 'IDEA-Research' one inside, but that throws a different kind of error. <br>
![src_fail](https://github.com/user-attachments/assets/4834f4a5-9a19-4e7b-bec5-53cb6f247ffc)
<br>
But if I fix the source then it runs without errors:
![src_succ](https://github.com/user-attachments/assets/d70a5f2f-1d58-49c2-a3de-42222392a4aa)




Hi @qubvel,

I opened the[ PR on huggingface datasets](https://huggingface.co/datasets/huggingface/documentation-images/discussions/425).
When it will be ready I'll modify the image sources!

Thank you!

Thanks, merged!

Yeah, integration tests will be fixed as soon as we transfer checkpoints. `test_cpu_offload` is more interesting, are you running with GPU available?

> Thanks, merged!
> 
> 
> 
> Yeah, integration tests will be fixed as soon as we transfer checkpoints. `test_cpu_offload` is more interesting, are you running with GPU available?

Thanks! No i run actually on a really weak cpu only with 8gb ram (i5 dual core)

I think that's the reason, the test is just skipped for cpu-only env. Try to run with `pytest -v` to see.

Hi @qubvel ,

This test fails: FAILED tests/models/qwen2_5_vl/test_modeling_qwen2_5_vl.py::Qwen2_5_VLModelTest::test_generate_from_inputs_embeds_1_beam_search
but it's not related to DAB-DETR.
I found the error and fixed it and I also pulled the main branch into the feature branch.
I also updated the model cards.

Thanks!

run-slow: dab_detr

This comment contains run-slow, running the specified jobs: ['models/dab_detr'] ...

run-slow: dab_detr

@qubvel I can see my comment doesn't have the superpower to trigger the bot to run the tests from a comment. :)

Yeah, it's available for maintainers only 😄 

run-slow: dab_detr

This comment contains run-slow, running the specified jobs: ['models/dab_detr'] ...

Noticed we don't have approval from @ArthurZucker, waiting for his review

Hi @ArthurZucker and @qubvel,

I’ve made most of the required modifications. Where I didn’t, I left comments on your feedback.
I also updated the test file where needed and added some additional information to the model card markdown file.

Thanks!

@conditionedstimulus Thanks for the updates! Please update converted weights for other checkpoints on the Hub as well and I will ask for transfer

Hi @ArthurZucker  and @qubvel,

I’ve finalized the last modification—if I understand correctly, this should be the final version, and we’ll roll it out soon.
I also updated the converted weights, so I believe it’s ready to be moved under the new organization.
I merged main too and ofc: 
```
SKIPPED [1] tests/generation/test_utils.py:1458: The decoder-only derived from encoder-decoder models are not expected to support left-padding.
FAILED tests/models/qwen2_5_vl/test_modeling_qwen2_5_vl.py::Qwen2_5_VLModelTest::test_prompt_lookup_decoding_matches_greedy_search - IndexError: index 41 is out of bound
```

Thanks, for your review, guidance, and support! :)

Looking forward to the merge! 🤗

run-slow: dab_detr

This comment contains run-slow, running the specified jobs: ['models/dab_detr'] ...

run-slow: dab_detr


This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/dab_detr']
quantizations: [] ...

run-slow: dab_detr

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/dab_detr']
quantizations: [] ...

run-slow: dab_detr

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/dab_detr']
quantizations: [] ...

@conditionedstimulus Congratulations on merging the model! 🎉 It was a long journey, and we really appreciate you were able to finish it 💪 . Thank you for your contribution, and sorry for the delays on our side. Great job! 🚀

And feel free to share your achievement on social networks, we’d be happy to amplify it!

@qubvel, @ArthurZucker 

Thank you guys!
It was a long and fun journey, and I truly appreciate your support and guidance. I'm glad I could contribute! :)