Mask2Former get_num_masks does not average across the nodes
Author: tommiekerssies
Pull request: no

### System Info

The original Mask2Former does this:
`# Compute the average number of target boxes accross all nodes, for normalization purposes
        num_masks = sum(len(t["labels"]) for t in targets)
        num_masks = torch.as_tensor(
            [num_masks], dtype=torch.float, device=next(iter(outputs.values())).device
        )
        if is_dist_avail_and_initialized():
            torch.distributed.all_reduce(num_masks)
        num_masks = torch.clamp(num_masks / get_world_size(), min=1).item()`
        https://github.com/facebookresearch/Mask2Former/blob/main/mask2former/modeling/criterion.py
        
Whereas the HF implementation does this:
`    def get_num_masks(self, class_labels: torch.Tensor, device: torch.device) -> torch.Tensor:
        """
        Computes the average number of target masks across the batch, for normalization purposes.
        """
        num_masks = sum([len(classes) for classes in class_labels])
        num_masks_pt = torch.as_tensor([num_masks], dtype=torch.float, device=device)
        return num_masks_pt
`https://github.com/huggingface/transformers/blob/v4.34.0/src/transformers/models/mask2former/modeling_mask2former.py


### Who can help?

@alaradirik @shivalikasingh95

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

-

### Expected behavior

-

Comments:
cc @rafaelpadilla might be worth checking? 

Hi @tommiekerssies ,

We appreciate you bringing this to our attention.

The original Mask2Former code handles two losses (labels and masks) as seen [here](https://github.com/facebookresearch/Mask2Former/blob/main/mask2former/modeling/criterion.py#L204). In HF Transformers, we compute both losses separately using two distinct functions `loss_masks` and `loss_labels`, as shown [here](https://github.com/huggingface/transformers/blob/v4.34.0/src/transformers/models/mask2former/modeling_mask2former.py#L782).

Could you kindly provide an example illustrating when the results might differ between the two implementations? 

Hi @tommiekerssies,

I'm aware of this, I remember we had to pull out distributed-related code when adding Mask2Former to the Transformers library, as otherwise it wouldn't work with libraries like Accelerate or the Trainer class. So to make it work, I'm afraid you will need to fork the library and make the necessary changes.

cc'ing @ArthurZucker here, wondering whether pulling out distributed-related code makes sense cause it makes it harder for people to do distributed training

Yeah I don't think we allow distributed related code nowdays and rather output the loss if ask / leave the computation to the user ! 

I think the difference in performance is negligible so it shouldn't be a big deal. However, the docblock is confusing, because the code is not actually computing the average across the batch, but is instead computing the total number of masks in the batch.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.