runtime error clip_vision_bert TypeError: __init__() got an unexpected keyword argument '_do_init'
Author: guanhdrmq
Pull request: no

### System Info

transformer 4.25.1
python 3.9
torch 1.13
windows 10 or ubuntu 20

### Who can help?

@sanchit-gandhi

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

from transformers import CLIPProcessor, BertTokenizerFast
from multilingual_vqa.models.flax_clip_vision_bert.modeling_clip_vision_bert import (
 FlaxCLIPVisionBertForSequenceClassification,
)

clip_processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
tokenizer = BertTokenizerFast.from_pretrained("bert-base-multilingual-uncased")

dataset = VQADataset(
 questions=questions[:10],
 annotations=annotations[:10],
 image_processor=clip_processor,
 text_processor=tokenizer,
)

model = FlaxCLIPVisionBertForSequenceClassification.from_pretrained(
 "./pretrained/clip-vision-bert-vqa-ft-6k",
 num_labels=len(config.id2label),
 id2label=config.id2label,
 label2id=config.label2id,
)

print(model)
model.to(device)
model.eval()

test_dataloader = DataLoader(dataset, collate_fn=collate_fn, batch_size=1, shuffle=False)

for batch in tqdm(test_dataloader):

 batch = {k: [v.to](http://v.to/)(device) for k, v in batch.items()}
 outputs = model(**batch)
 preds = outputs.logits[0]
 sorted_indices = np.argsort(preds)[::-1] # Get reverse sorted scores
 top_5_indices = sorted_indices[:5]
 top_5_tokens = list(map(model.config.id2label.get, top_5_indices))
 top_5_scores = preds[top_5_indices]
 print(dict(zip(top_5_tokens, top_5_scores))Traceback (most recent call last):

### Expected behavior

fixt the do_init issue and can use clip_vision_bert for vqa inference on validation dataset:
  File "/media/wayne/data/clip_vision_bert/clip_vision_bert.py", line 210, in <module>
    model = FlaxCLIPVisionBertForSequenceClassification.from_pretrained(
  File "/media/wayne/data/clip_vision_bert/multilingual_vqa/models/flax_clip_vision_bert/modeling_clip_vision_bert.py", line 991, in from_pretrained
    return super().from_pretrained(*args, **kwargs)
  File "/home/wayne/anaconda3/envs/clip_vision_bert/lib/python3.8/site-packages/transformers/modeling_flax_utils.py", line 797, in from_pretrained
    model = cls(config, *model_args, _do_init=_do_init, **model_kwargs)
  File "/media/wayne/data/clip_vision_bert/multilingual_vqa/models/flax_clip_vision_bert/modeling_clip_vision_bert.py", line 857, in __init__
    module = self.module_class(config=config, dtype=dtype, **kwargs)
TypeError: __init__() got an unexpected keyword argument '_do_init'

Comments:
Hey! Can you update the version of transformers you are using ? 
Hey! Seems like an error with `multilingual_vqa` calling from pretrained. Would recommend you to upgrade to a more recent version but also isolate this to transformers or `multilingual_vqa` and post the issue there! 

Hi! @ArthurZucker. Thanks for your answering.  Both of your solutions already done and upgrade to transformers 4.35.2 but does not work. This issue is posted : https://github.com/gchhablani/multilingual-vqa/issues/2.

It seems like huggingface scripts problem here modellin_flax_utils.py

The traceback points to `"D:\multimodal_robustness\multilingual_vqa\models\flax_clip_vision_bert\modeling_clip_vision_bert.py` which does not exist in transformers! 😉 

Hi @ArthurZucker Haha I know that.  Can hugging face team help solve this issue? because people have the same issue but adding _do_init=False does not work. ihttps://github.com/huggingface/transformers/issues/12513
The author indeed used huggingface framework. I also contact him on git https://github.com/gchhablani but no answers. 

This author arises another problem on huggingface visualbert lower accuracy in VQA validation dataset.

Sincerely hope hugginface team can help fix this error? Thank you very much.

Sorry but I have 0 knowledge on this library, and not a lot of bandwidth 😓 You should be able to open a PR implementing a fix (probably just removing the `_do_init` kwarg from the call to `cls`)

Answered here: https://github.com/gchhablani/multilingual-vqa/issues/2#issuecomment-1843424353

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.