[Fix] ViViT interpolate_pos_encoding
Author: RUFFY-369
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Fixes #33814 


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts 
 

Comments:
@amyeroberts All green and suggestions are pushed.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33815). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts Weirdly the slow test is passing locally but in the `PR Slow CI` that you started it is throwing out the following error:
```
=================================== FAILURES ===================================
______ VivitModelIntegrationTest.test_inference_interpolate_pos_encoding _______

checkpoint_file = '/mnt/cache/hub/models--google--vivit-b-16x2/snapshots/fc341053d36b42d446b3ffccdbd52452712a23f3/pytorch_model.bin'
is_quantized = False

    def load_state_dict(checkpoint_file: Union[str, os.PathLike], is_quantized: bool = False):
        """
        Reads a PyTorch checkpoint file, returning properly formatted errors if they arise.
        """
        if checkpoint_file.endswith(".safetensors") and is_safetensors_available():
            # Check format of the archive
            with safe_open(checkpoint_file, framework="pt") as f:
                metadata = f.metadata()
            if metadata.get("format") not in ["pt", "tf", "flax", "mlx"]:
                raise OSError(
                    f"The safetensors archive passed at {checkpoint_file} does not contain the valid metadata. Make sure "
                    "you save your model with the `save_pretrained` method."
                )
            return safe_load_file(checkpoint_file)
        try:
            if (
                (is_deepspeed_zero3_enabled() and torch.distributed.is_initialized() and torch.distributed.get_rank() > 0)
                or (is_fsdp_enabled() and not is_local_dist_rank_0())
            ) and not is_quantized:
                map_location = "meta"
            else:
                map_location = "cpu"
            extra_args = {}
            # mmap can only be used with files serialized with zipfile-based format.
            if (
                isinstance(checkpoint_file, str)
                and map_location != "meta"
                and version.parse(torch.__version__) >= version.parse("2.1.0")
                and is_zipfile(checkpoint_file)
            ):
                extra_args = {"mmap": True}
            weights_only_kwarg = {"weights_only": True} if is_torch_greater_or_equal_than_1_13 else {}
>           return torch.load(
                checkpoint_file,
                map_location=map_location,
                **weights_only_kwarg,
                **extra_args,
            )

src/transformers/modeling_utils.py:575: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/local/lib/python3.8/dist-packages/torch/serialization.py:1066: in load
    if _is_zipfile(opened_file):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

f = <_io.BufferedReader name='/mnt/cache/hub/models--google--vivit-b-16x2/snapshots/fc341053d36b42d446b3ffccdbd52452712a23f3/pytorch_model.bin'>

    def _is_zipfile(f) -> bool:
        # This is a stricter implementation than zipfile.is_zipfile().
        # zipfile.is_zipfile() is True if the magic number appears anywhere in the
        # binary. Since we expect the files here to be generated by torch.save or
        # torch.jit.save, it's safe to only check the start bytes and avoid
        # collisions and assume the zip has only 1 file.
        # See bugs.python.org/issue28494.
    
        start = f.tell()
        # Read the first few bytes and match against the ZIP file signature
        local_header_magic_number = b'PK\x03\x04'
>       read_bytes = f.read(len(local_header_magic_number))
E       OSError: [Errno 5] Input/output error

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:[206](https://github.com/huggingface/transformers/actions/runs/11105958398/job/30854864070#step:11:207): OSError

During handling of the above exception, another exception occurred:

self = <tests.models.vivit.test_modeling_vivit.VivitModelIntegrationTest testMethod=test_inference_interpolate_pos_encoding>

    @slow
    def test_inference_interpolate_pos_encoding(self):
        # Vivit models have an `interpolate_pos_encoding` argument in their forward method,
        # allowing to interpolate the pre-trained position embeddings in order to use
        # the model on higher resolutions. The DINO model by Facebook AI leverages this
        # to visualize self-attention on higher resolution images.
>       model = VivitModel.from_pretrained("google/vivit-b-16x2").to(torch_device)

tests/models/vivit/test_modeling_vivit.py:362: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/transformers/modeling_utils.py:3814: in from_pretrained
    state_dict = load_state_dict(resolved_archive_file)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

checkpoint_file = '/mnt/cache/hub/models--google--vivit-b-16x2/snapshots/fc341053d36b42d446b3ffccdbd52452712a23f3/pytorch_model.bin'
is_quantized = False

    def load_state_dict(checkpoint_file: Union[str, os.PathLike], is_quantized: bool = False):
        """
        Reads a PyTorch checkpoint file, returning properly formatted errors if they arise.
        """
        if checkpoint_file.endswith(".safetensors") and is_safetensors_available():
            # Check format of the archive
            with safe_open(checkpoint_file, framework="pt") as f:
                metadata = f.metadata()
            if metadata.get("format") not in ["pt", "tf", "flax", "mlx"]:
                raise OSError(
                    f"The safetensors archive passed at {checkpoint_file} does not contain the valid metadata. Make sure "
                    "you save your model with the `save_pretrained` method."
                )
            return safe_load_file(checkpoint_file)
        try:
            if (
                (is_deepspeed_zero3_enabled() and torch.distributed.is_initialized() and torch.distributed.get_rank() > 0)
                or (is_fsdp_enabled() and not is_local_dist_rank_0())
            ) and not is_quantized:
                map_location = "meta"
            else:
                map_location = "cpu"
            extra_args = {}
            # mmap can only be used with files serialized with zipfile-based format.
            if (
                isinstance(checkpoint_file, str)
                and map_location != "meta"
                and version.parse(torch.__version__) >= version.parse("2.1.0")
                and is_zipfile(checkpoint_file)
            ):
                extra_args = {"mmap": True}
            weights_only_kwarg = {"weights_only": True} if is_torch_greater_or_equal_than_1_13 else {}
            return torch.load(
                checkpoint_file,
                map_location=map_location,
                **weights_only_kwarg,
                **extra_args,
            )
        except Exception as e:
            try:
                with open(checkpoint_file) as f:
>                   if f.read(7) == "version":
E                   OSError: [Errno 5] Input/output error

src/transformers/modeling_utils.py:584: OSError
```
I pushed a commit which could fix it but still a little doubtful

Hey! 🤗 Thanks for your contribution to the `transformers` library!

Before merging this pull request, slow tests CI should be triggered. To enable this:
- Add the `run-slow` label to the PR
- When your PR is ready for merge and all reviewers' comments have been addressed, push an empty commit with the command `[run-slow]` followed by a comma separated list of all the models to be tested, i.e. `[run_slow] model_to_test_1, model_to_test_2`
  - If the pull request affects a lot of models, put at most 10 models in the commit message
- A `transformers` maintainer will then approve the workflow to start the tests

(For maintainers) The documentation for slow tests CI on PRs is [here](https://www.notion.so/huggingface2/CI-for-pull-requests-8335bd9217d24d1e8ed1d3a016f39804).