Fix mismatching loading in from_pretrained with/without accelerate
Author: fxmarty
Pull request: yes

It appears that passing a `device_map` may result in some parameters being not contiguous in the loaded model, which is not the case when loading a model without a `device_map`.

See for instance:

```python
from transformers import OwlViTProcessor, OwlViTForObjectDetection

model = OwlViTForObjectDetection.from_pretrained("google/owlvit-base-patch16")
print("is contig (no device_map):", model.owlvit.visual_projection.weight.is_contiguous())

model = OwlViTForObjectDetection.from_pretrained("google/owlvit-base-patch16", device_map="auto")
print("is contig (device_map):", model.owlvit.visual_projection.weight.is_contiguous())
```

printing
```
is contig (no device_map): True
is contig (device_map): False
```

A byproduct of this bug is that when a model is loaded with a `device_map`, then using

```python
model.save_pretrained("owlvit", safe_serialization=True)
```

results in

```
Traceback (most recent call last):
  File "<tmp 2>", line 13, in <module>
    model.save_pretrained("owlvit", save_config=True, safe_serialization=True)
  File "/home/fxmarty/hf_internship/transformers/src/transformers/modeling_utils.py", line 2406, in save_pretrained
    safe_save_file(shard, os.path.join(save_directory, shard_file), metadata={"format": "pt"})
  File "/home/fxmarty/anaconda3/envs/hf-inf/lib/python3.9/site-packages/safetensors/torch.py", line 281, in save_file
    serialize_file(_flatten(tensors), filename, metadata=metadata)
  File "/home/fxmarty/anaconda3/envs/hf-inf/lib/python3.9/site-packages/safetensors/torch.py", line 481, in _flatten
    "data": _tobytes(v, k),
  File "/home/fxmarty/anaconda3/envs/hf-inf/lib/python3.9/site-packages/safetensors/torch.py", line 396, in _tobytes
    raise ValueError(
ValueError: You are trying to save a non contiguous tensor: `owlvit.visual_projection.weight` which is not allowed. It either means you are trying to save tensors which are reference of each other in which case it's recommended to save only the full tensors, and reslice at load time, or simply call `.contiguous()` on your tensor to pack it before saving.
```

This bug stems from the fact that in the case we don't use accelerate, the weights are loaded through [`_load_from_state_dict`](https://github.com/huggingface/transformers/blob/357971ec367fecb9951ae3218feafece5f61416a/src/transformers/modeling_utils.py#L600), which makes use of [`param.copy_(input_param)`](https://github.com/pytorch/pytorch/blob/db79ceb110f6646523019a59bbd7b838f43d4a86/torch/nn/modules/module.py#L2040C29-L2040C29) which preserves the contiguity of the module's parameters. On the contrary, Accelerate's [`set_module_tensor_to_device`](https://github.com/huggingface/accelerate/blob/3969731ce827b088fcc56ea790935cdece12f800/src/accelerate/utils/modeling.py#L370) appears to override the existing value [simply by the one from the state dict](https://github.com/huggingface/transformers/blob/357971ec367fecb9951ae3218feafece5f61416a/src/transformers/modeling_utils.py#L716-L758), which may be not contiguous.

This PR fixes the issue.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28414). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@fxmarty Could you add a test that passes with these changes? 

@amyeroberts sure, where would be a good place for it?

@fxmarty I'd suggest putting a test in `modeling_utils` which does effectively the same thing as in the example in PR description with a small model. Happy for you to put it somewhere else if you think it's better 

There seem to be unrelated issues in the CI

```
Building wheels for collected packages: natten
  Building wheel for natten (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [80 lines of output]
      Building NATTEN for CPU ONLY.
      Number of workers: 9
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-38
      creating build/lib.linux-x86_64-cpython-38/natten
      copying src/natten/__init__.py -> build/lib.linux-x86_64-cpython-38/natten
      copying src/natten/flops.py -> build/lib.linux-x86_64-cpython-38/natten
      copying src/natten/functional.py -> build/lib.linux-x86_64-cpython-38/natten
      copying src/natten/natten1d.py -> build/lib.linux-x86_64-cpython-38/natten
      copying src/natten/natten2d.py -> build/lib.linux-x86_64-cpython-38/natten
      copying src/natten/natten3d.py -> build/lib.linux-x86_64-cpython-38/natten
      copying src/natten/nested.py -> build/lib.linux-x86_64-cpython-38/natten
      creating build/lib.linux-x86_64-cpython-38/natten/utils
      copying src/natten/utils/__init__.py -> build/lib.linux-x86_64-cpython-38/natten/utils
      copying src/natten/utils/tensor.py -> build/lib.linux-x86_64-cpython-38/natten/utils
      copying src/natten/utils/testing.py -> build/lib.linux-x86_64-cpython-38/natten/utils
      copying src/natten/utils/typing.py -> build/lib.linux-x86_64-cpython-38/natten/utils
      running build_ext
      Traceback (most recent call last):
        File "/tmp/pip-install-musj9gr5/natten_ea055ed283804ee0b0ef252dd4ad0d6c/setup.py", line 147, in build_extension
          subprocess.check_output(["cmake", "--version"])
        File "/home/circleci/.pyenv/versions/3.8.12/lib/python3.8/subprocess.py", line 415, in check_output
          return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
        File "/home/circleci/.pyenv/versions/3.8.12/lib/python3.8/subprocess.py", line 493, in run
          with Popen(*popenargs, **kwargs) as process:
        File "/home/circleci/.pyenv/versions/3.8.12/lib/python3.8/subprocess.py", line 858, in __init__
          self._execute_child(args, executable, preexec_fn, close_fds,
        File "/home/circleci/.pyenv/versions/3.8.12/lib/python3.8/subprocess.py", line 1704, in _execute_child
          raise child_exception_type(errno_num, err_msg, err_filename)
      FileNotFoundError: [Errno 2] No such file or directory: 'cmake'
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-musj9gr5/natten_ea055ed283804ee0b0ef252dd4ad0d6c/setup.py", line 210, in <module>
          setup(
        File "/home/circleci/.pyenv/versions/3.8.12/lib/python3.8/site-packages/setuptools/__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
        File "/home/circleci/.pyenv/versions/3.8.12/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/home/circleci/.pyenv/versions/3.8.12/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/home/circleci/.pyenv/versions/3.8.12/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/home/circleci/.pyenv/versions/3.8.12/lib/python3.8/site-packages/setuptools/dist.py", line 963, in run_command
          super().run_command(command)
        File "/home/circleci/.pyenv/versions/3.8.12/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/home/circleci/.pyenv/versions/3.8.12/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 368, in run
          self.run_command("build")
        File "/home/circleci/.pyenv/versions/3.8.12/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/home/circleci/.pyenv/versions/3.8.12/lib/python3.8/site-packages/setuptools/dist.py", line 963, in run_command
          super().run_command(command)
        File "/home/circleci/.pyenv/versions/3.8.12/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/home/circleci/.pyenv/versions/3.8.12/lib/python3.8/site-packages/setuptools/_distutils/command/build.py", line 131, in run
          self.run_command(cmd_name)
        File "/home/circleci/.pyenv/versions/3.8.12/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/home/circleci/.pyenv/versions/3.8.12/lib/python3.8/site-packages/setuptools/dist.py", line 963, in run_command
          super().run_command(command)
        File "/home/circleci/.pyenv/versions/3.8.12/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/home/circleci/.pyenv/versions/3.8.12/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 88, in run
          _build_ext.run(self)
        File "/home/circleci/.pyenv/versions/3.8.12/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
          self.build_extensions()
        File "/home/circleci/.pyenv/versions/3.8.12/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
          self._build_extensions_serial()
        File "/home/circleci/.pyenv/versions/3.8.12/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
          self.build_extension(ext)
        File "/tmp/pip-install-musj9gr5/natten_ea055ed283804ee0b0ef252dd4ad0d6c/setup.py", line 149, in build_extension
          raise RuntimeError("Cannot find CMake executable")
      RuntimeError: Cannot find CMake executable
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for natten
  Running setup.py clean for natten
Failed to build natten
ERROR: Could not build wheels for natten, which is required to install pyproject.toml-based projects
```

@ydshieh do you know if something was changed in the workflows?

Rebase on main should work - https://huggingface.slack.com/archives/C01NE71C4F7/p1704883154443469

Also cc @SunMarc 

@amyeroberts Thank you for the review. I modified the test to use a small model (1.6 MB).