Can't load the model from your tutorial for inference
Author: kopyl
Pull request: no

### System Info

Linux. Doesn't matter

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

https://huggingface.co/docs/transformers/main/tasks/image_captioning

### Expected behavior

I tried this:

`model_trained = AutoModelForCausalLM.from_pretrained("/workspace/git-base-trainer/checkpoint-100")`

But when I ran
```
inputs = processor(images=image, return_tensors="pt").to("cuda")
pixel_values = inputs.pixel_values

generated_ids = model_trained.generate(pixel_values=pixel_values, max_length=50)
generated_caption = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
```

i got this error:

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[61], line 4
      1 inputs = processor(images=image, return_tensors="pt").to("cuda")
      2 pixel_values = inputs.pixel_values
----> 4 generated_ids = model_trained.generate(pixel_values=pixel_values, max_length=50)
      5 generated_caption = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
      6 # print(generated_caption)

File /usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py:115, in context_decorator.<locals>.decorate_context(*args, **kwargs)
    112 @functools.wraps(func)
    113 def decorate_context(*args, **kwargs):
    114     with ctx_factory():
--> 115         return func(*args, **kwargs)

File /usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py:1406, in GenerationMixin.generate(self, inputs, generation_config, logits_processor, stopping_criteria, prefix_allowed_tokens_fn, synced_gpus, **kwargs)
   1400         raise ValueError(
   1401             f"num_return_sequences has to be 1, but is {generation_config.num_return_sequences} when doing"
   1402             " greedy search."
   1403         )
   1405     # 11. run greedy search
-> 1406     return self.greedy_search(
   1407         input_ids,
   1408         logits_processor=logits_processor,
   1409         stopping_criteria=stopping_criteria,
   1410         pad_token_id=generation_config.pad_token_id,
   1411         eos_token_id=generation_config.eos_token_id,
   1412         output_scores=generation_config.output_scores,
   1413         return_dict_in_generate=generation_config.return_dict_in_generate,
   1414         synced_gpus=synced_gpus,
   1415         **model_kwargs,
   1416     )
   1418 elif is_contrastive_search_gen_mode:
   1419     if generation_config.num_return_sequences > 1:

File /usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py:2201, in GenerationMixin.greedy_search(self, input_ids, logits_processor, stopping_criteria, max_length, pad_token_id, eos_token_id, output_attentions, output_hidden_states, output_scores, return_dict_in_generate, synced_gpus, **model_kwargs)
   2198 model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)
   2200 # forward pass to get next token
-> 2201 outputs = self(
   2202     **model_inputs,
   2203     return_dict=True,
   2204     output_attentions=output_attentions,
   2205     output_hidden_states=output_hidden_states,
   2206 )
   2208 if synced_gpus and this_peer_finished:
   2209     continue  # don't waste resources running the code we don't need

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File /usr/local/lib/python3.10/dist-packages/transformers/models/git/modeling_git.py:1496, in GitForCausalLM.forward(self, input_ids, attention_mask, position_ids, pixel_values, head_mask, inputs_embeds, labels, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)
   1493 if labels is not None:
   1494     use_cache = False
-> 1496 outputs = self.git(
   1497     input_ids,
   1498     attention_mask=attention_mask,
   1499     position_ids=position_ids,
   1500     pixel_values=pixel_values,
   1501     head_mask=head_mask,
   1502     inputs_embeds=inputs_embeds,
   1503     past_key_values=past_key_values,
   1504     use_cache=use_cache,
   1505     output_attentions=output_attentions,
   1506     output_hidden_states=output_hidden_states,
   1507     return_dict=return_dict,
   1508 )
   1510 sequence_output = outputs[0]
   1511 logits = self.output(sequence_output)

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File /usr/local/lib/python3.10/dist-packages/transformers/models/git/modeling_git.py:1236, in GitModel.forward(self, input_ids, attention_mask, position_ids, pixel_values, head_mask, inputs_embeds, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)
   1233 if pixel_values is not None:
   1234     if pixel_values.ndim == 4:
   1235         # here we assume pixel_values is of shape (batch_size, num_channels, height, width)
-> 1236         visual_features = self.image_encoder(pixel_values).last_hidden_state
   1238     elif pixel_values.ndim == 5:
   1239         # here we assume pixel_values is of shape (batch_size, num_frames, num_channels, height, width)
   1240         visual_features = []

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File /usr/local/lib/python3.10/dist-packages/transformers/models/git/modeling_git.py:1039, in GitVisionModel.forward(self, pixel_values, output_attentions, output_hidden_states, return_dict)
   1016 r"""
   1017 Returns:
   1018 
   (...)
   1035 >>> last_hidden_state = outputs.last_hidden_state
   1036 ```"""
   1037 return_dict = return_dict if return_dict is not None else self.config.use_return_dict
-> 1039 return self.vision_model(
   1040     pixel_values=pixel_values,
   1041     output_attentions=output_attentions,
   1042     output_hidden_states=output_hidden_states,
   1043     return_dict=return_dict,
   1044 )

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File /usr/local/lib/python3.10/dist-packages/transformers/models/git/modeling_git.py:965, in GitVisionTransformer.forward(self, pixel_values, output_attentions, output_hidden_states, return_dict)
    962 if pixel_values is None:
    963     raise ValueError("You have to specify pixel_values")
--> 965 hidden_states = self.embeddings(pixel_values)
    966 hidden_states = self.pre_layrnorm(hidden_states)
    968 encoder_outputs = self.encoder(
    969     inputs_embeds=hidden_states,
    970     output_attentions=output_attentions,
    971     output_hidden_states=output_hidden_states,
    972     return_dict=return_dict,
    973 )

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File /usr/local/lib/python3.10/dist-packages/transformers/models/git/modeling_git.py:630, in GitVisionEmbeddings.forward(self, pixel_values)
    628 def forward(self, pixel_values: torch.FloatTensor) -> torch.Tensor:
    629     batch_size = pixel_values.shape[0]
--> 630     patch_embeds = self.patch_embedding(pixel_values)  # shape = [*, width, grid, grid]
    631     patch_embeds = patch_embeds.flatten(2).transpose(1, 2)
    633     class_embeds = self.class_embedding.expand(batch_size, 1, -1)

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/conv.py:463, in Conv2d.forward(self, input)
    462 def forward(self, input: Tensor) -> Tensor:
--> 463     return self._conv_forward(input, self.weight, self.bias)

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/conv.py:459, in Conv2d._conv_forward(self, input, weight, bias)
    455 if self.padding_mode != 'zeros':
    456     return F.conv2d(F.pad(input, self._reversed_padding_repeated_twice, mode=self.padding_mode),
    457                     weight, bias, self.stride,
    458                     _pair(0), self.dilation, self.groups)
--> 459 return F.conv2d(input, weight, bias, self.stride,
    460                 self.padding, self.dilation, self.groups)

RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same
```

I tried running this:
`processor_trained = AutoProcessor.from_pretrained("/workspace/git-base-trainer/checkpoint-100")`

But immediately got this error:

```
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
Cell In[63], line 3
      1 # tokenizer = AutoTokenizer.from_pretrained("/workspace/git-base-trainer/checkpoint-50")
----> 3 processor_trained = AutoProcessor.from_pretrained("/workspace/git-base-trainer/checkpoint-100")
      4 # model_trained = AutoModelForCausalLM.from_pretrained("/workspace/git-base-trainer/checkpoint-100")

File /usr/local/lib/python3.10/dist-packages/transformers/models/auto/processing_auto.py:276, in AutoProcessor.from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    274 # Last try: we use the PROCESSOR_MAPPING.
    275 if type(config) in PROCESSOR_MAPPING:
--> 276     return PROCESSOR_MAPPING[type(config)].from_pretrained(pretrained_model_name_or_path, **kwargs)
    278 # At this stage, there doesn't seem to be a `Processor` class available for this model, so let's try a
    279 # tokenizer.
    280 try:

File /usr/local/lib/python3.10/dist-packages/transformers/processing_utils.py:184, in ProcessorMixin.from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    153 @classmethod
    154 def from_pretrained(cls, pretrained_model_name_or_path, **kwargs):
    155     r"""
    156     Instantiate a processor associated with a pretrained model.
    157 
   (...)
    182             [`~tokenization_utils_base.PreTrainedTokenizer.from_pretrained`].
    183     """
--> 184     args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, **kwargs)
    185     return cls(*args)

File /usr/local/lib/python3.10/dist-packages/transformers/processing_utils.py:228, in ProcessorMixin._get_arguments_from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    225     else:
    226         attribute_class = getattr(transformers_module, class_name)
--> 228     args.append(attribute_class.from_pretrained(pretrained_model_name_or_path, **kwargs))
    229 return args

File /usr/local/lib/python3.10/dist-packages/transformers/models/auto/image_processing_auto.py:315, in AutoImageProcessor.from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    312 trust_remote_code = kwargs.pop("trust_remote_code", False)
    313 kwargs["_from_auto"] = True
--> 315 config_dict, _ = ImageProcessingMixin.get_image_processor_dict(pretrained_model_name_or_path, **kwargs)
    316 image_processor_class = config_dict.get("image_processor_type", None)
    317 image_processor_auto_map = None

File /usr/local/lib/python3.10/dist-packages/transformers/image_processing_utils.py:268, in ImageProcessingMixin.get_image_processor_dict(cls, pretrained_model_name_or_path, **kwargs)
    265 image_processor_file = IMAGE_PROCESSOR_NAME
    266 try:
    267     # Load from local folder or from cache or download from model Hub and cache
--> 268     resolved_image_processor_file = cached_file(
    269         pretrained_model_name_or_path,
    270         image_processor_file,
    271         cache_dir=cache_dir,
    272         force_download=force_download,
    273         proxies=proxies,
    274         resume_download=resume_download,
    275         local_files_only=local_files_only,
    276         use_auth_token=use_auth_token,
    277         user_agent=user_agent,
    278         revision=revision,
    279         subfolder=subfolder,
    280     )
    281 except EnvironmentError:
    282     # Raise any environment error raise by `cached_file`. It will have a helpful error message adapted to
    283     # the original exception.
    284     raise

File /usr/local/lib/python3.10/dist-packages/transformers/utils/hub.py:380, in cached_file(path_or_repo_id, filename, cache_dir, force_download, resume_download, proxies, use_auth_token, revision, local_files_only, subfolder, user_agent, _raise_exceptions_for_missing_entries, _raise_exceptions_for_connection_errors, _commit_hash)
    378 if not os.path.isfile(resolved_file):
    379     if _raise_exceptions_for_missing_entries:
--> 380         raise EnvironmentError(
    381             f"{path_or_repo_id} does not appear to have a file named {full_filename}. Checkout "
    382             f"'[https://huggingface.co/{path_or_repo_id}/{](https://huggingface.co/%7Bpath_or_repo_id%7D/%7Brevision)[revision](https://huggingface.co/%7Bpath_or_repo_id%7D/%7Brevision)}' for available files."
    383         )
    384     else:
    385         return None

OSError: /workspace/git-base-trainer/checkpoint-100 does not appear to have a file named preprocessor_config.json. Checkout 'https://huggingface.co//workspace/git-base-trainer/checkpoint-100/None' for available files.
```

But when i ran this (with old model):
```
inputs = processor(images=image, return_tensors="pt").to("cuda")
pixel_values = inputs.pixel_values

generated_ids = model.generate(pixel_values=pixel_values, max_length=50)
generated_caption = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(generated_caption)
```

, i was getting a different caption from raw `microsoft/git-base` model, meaning that it fine-tuned, but for some reason (why the hell?) it overriden a loaded model and stayed in memory.

Meaning you can spend thousand of dollars on training but can't just load the model so it works the same as after training.

Could you please provide a clear tutorial on how to be able to do the inference from the same fine-tuned model after a server was shut down?

Comments:
Honestly, there is just too much info in docs which does not help.

@kopyl, the first error due to GPU, you moved input into `.to("cuda")`, but model is still on `cpu`. The second error due to lack of preprocessing [file](https://huggingface.co/microsoft/git-base/blob/main/preprocessor_config.json), so you can download it or
`processor_trained = AutoProcessor.from_pretrained("microsoft/git-base")`

@Xrenya oh, thanks a lot, this helped ❤️ 

Are you a maintainer of Transformers? If so, could I please ask you to add this info to the guide?

And also do you know whether I should load a custom processor from training or the `microsoft/git-base` does the same?
Or do I just train only the model and don't interfere with the processor while training and leaving the old one does not have any side effects?

I think it is already contain that information in the following sections **Preprocess the dataset** and **Load a base model** in [doc](https://huggingface.co/docs/transformers/main/tasks/image_captioning). 
```
from transformers import AutoProcessor

checkpoint = "microsoft/git-base"
processor = AutoProcessor.from_pretrained(checkpoint)
```
```
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(checkpoint)
```
If your custom preprocessing is different (parameters, e.g. image_mean, image_std etc. ) from `microsoft/git-base` then you should your preprocessing for training and inference, otherwise, you can just use their preprocessing. 

@Xrenya thanks :)

Haven't seen this info in the doc :(

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.