Conversion of OWL-ViT model fails
Author: alexey-chaykin
Pull request: no

### System Info

I've trained OWL-ViT model on my data using [training code from original repo](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit#fine-tuning) and trying to use it in HuggingFace pytorch OWL-ViT implementation.

As far as I understand, I need to convert it using [convert_owlvit_original_flax_to_hf.py](https://github.com/huggingface/transformers/blob/main/src/transformers/models/owlvit/convert_owlvit_original_flax_to_hf.py), first. 

But, when I invoke:  
`python3 convert_owlvit_original_flax_to_hf.py --owlvit_version clip_b32  --owlvit_checkpoint ~/scenic/training/checkpoint_16000 --hf_config vit_b32 --pytorch_dump_folder_path .`
it fails with:  
```
Traceback (most recent call last):
  File "~/transformers/src/transformers/models/owlvit/convert_owlvit_original_flax_to_hf.py", line 406, in <module>
    variables = checkpoints.restore_checkpoint(args.owlvit_checkpoint, target=None)["optimizer"]["target"]
KeyError: 'optimizer'
```
How to fix that?

P.S. the dict, returned by checkpoints.restore_checkpoint() has next keys: ['opt_state', 'params', 'global_step', 'model_state', 'rng', 'metadata'




### Who can help?

@amyeroberts  @alaradirik

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

`python3 convert_owlvit_original_flax_to_hf.py --owlvit_version clip_b32  --owlvit_checkpoint ~/scenic/training/checkpoint_16000 --hf_config vit_b32 --pytorch_dump_folder_path .`



### Expected behavior

successful script running with converted model as output

Comments:
P.S. if that could help, when I'm changing ['optimizer']['target'] to ['params'], the script is failing with:
```
Traceback (most recent call last):
  File "~/transformers/src/transformers/models/owlvit/convert_owlvit_original_flax_to_hf.py", line 411, in <module>
    pt_backbone_params, clip_pt, attn_params = convert_clip_backbone(flax_params, torch_config)
  File "~/transformers/src/transformers/models/owlvit/convert_owlvit_original_flax_to_hf.py", line 281, in convert_clip_backbone
    flax_clip_params = flatten_nested_dict(flax_params["backbone"]["clip"])
  File "~/transformers/src/transformers/models/owlvit/convert_owlvit_original_flax_to_hf.py", line 85, in flatten_nested_dict
    if isinstance(v, collections.MutableMapping):
AttributeError: module 'collections' has no attribute 'MutableMapping'
```

You almost certainly will have to fork the Transformers library and adapt the conversion script a bit to make it work for your use case.

In this case, it seems that you're not properly reading the parameters of the model into a dictionary. I'd check which keys are in `checkpoints.restore_checkpoint(args.owlvit_checkpoint)` => you apparently already found that it should be 'params'. Next, you can check what's exactly in `flax_clip_params`, this should be in dictionary with key-value pairs of parameter names and their corresponding values.

Hi @alexey-chaykin, I can convert the official checkpoints but the training script was not available when I added OWL-ViT to transformers and their original script is probably a little different than the released one. You would need to find out what key the parameters are stored under and edit the conversion script.

As for the second error you're getting, I think it's just a version issue as `collections.MutableMapping` has been moved to `collections.abc.MutableMapping` in newer versions.

Thanks, Niels, Alara. Will do that way.

Do you have any plans to implement pytorch training for HuggingFace OWL-ViT?

No problem @alexey-chaykin, we are planning to implement PyTorch training as the training code is released. We will probably be releasing a tutorial / blog post on it in the next few weeks :)

Thanks! Looking forward to that.

@alaradirik Any update on the tutorial/blog or the training code in PyTorch?

I'm re-opening the issue #20091 for adding the training code for Owl-Vit for anyone in the community to contribute if they're interested cc @rafaelpadilla