Loading sharded model with torch_dtype='auto' causes TypeError
Author: koreyou
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.20.1
- Platform: macOS-12.4-arm64-arm-64bit
- Python version: 3.9.13
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.12.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Install latest version of `transformers` as above:

```bash
pip install pytorch transformers datasets
```

Load a sharded model (the following example creates sharded model for easier reproducibility, but this reproduces on other sharded models as well (at least for BLOOM)).

```python
import os, tempfile
from transformers import AutoModel


model = AutoModel.from_pretrained("distilbert-base-uncased")

with tempfile.TemporaryDirectory() as tmp_dir:
    model.save_pretrained(tmp_dir, max_shard_size="100MB")
    print(sorted(os.listdir(tmp_dir)))

    # Now, actually reproduce the problem
    AutoModel.from_pretrained(
        tmp_dir,
        torch_dtype='auto')
```

### Expected behavior

We expect it to load the model without an error. Instead it gives out a TypeError:

```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
File ~/.pyenv/versions/3.9.13/envs/transformers-bugfix/lib/python3.9/site-packages/torch/serialization.py:308, in _check_seekable(f)
    307 try:
--> 308     f.seek(f.tell())
    309     return True

AttributeError: 'list' object has no attribute 'seek'

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
File ~/.pyenv/versions/3.9.13/envs/transformers-bugfix/lib/python3.9/site-packages/transformers/modeling_utils.py:461, in load_state_dict(checkpoint_file)
    460 try:
--> 461     return torch.load(checkpoint_file, map_location="cpu")
    462 except Exception as e:

File ~/.pyenv/versions/3.9.13/envs/transformers-bugfix/lib/python3.9/site-packages/torch/serialization.py:699, in load(f, map_location, pickle_module, **pickle_load_args)
    697     pickle_load_args['encoding'] = 'utf-8'
--> 699 with _open_file_like(f, 'rb') as opened_file:
    700     if _is_zipfile(opened_file):
    701         # The zipfile reader is going to advance the current file position.
    702         # If we want to actually tail call to torch.jit.load, we need to
    703         # reset back to the original position.

File ~/.pyenv/versions/3.9.13/envs/transformers-bugfix/lib/python3.9/site-packages/torch/serialization.py:235, in _open_file_like(name_or_buffer, mode)
    234 elif 'r' in mode:
--> 235     return _open_buffer_reader(name_or_buffer)
    236 else:

File ~/.pyenv/versions/3.9.13/envs/transformers-bugfix/lib/python3.9/site-packages/torch/serialization.py:220, in _open_buffer_reader.__init__(self, buffer)
    219 super(_open_buffer_reader, self).__init__(buffer)
--> 220 _check_seekable(buffer)

File ~/.pyenv/versions/3.9.13/envs/transformers-bugfix/lib/python3.9/site-packages/torch/serialization.py:311, in _check_seekable(f)
    310 except (io.UnsupportedOperation, AttributeError) as e:
--> 311     raise_err_msg(["seek", "tell"], e)
    312 return False

File ~/.pyenv/versions/3.9.13/envs/transformers-bugfix/lib/python3.9/site-packages/torch/serialization.py:304, in _check_seekable.<locals>.raise_err_msg(patterns, e)
    301         msg = (str(e) + ". You can only torch.load from a file that is seekable."
    302                         + " Please pre-load the data into a buffer like io.BytesIO and"
    303                         + " try to load from it instead.")
--> 304         raise type(e)(msg)
    305 raise e

AttributeError: 'list' object has no attribute 'seek'. You can only torch.load from a file that is seekable. Please pre-load the data into a buffer like io.BytesIO and try to load from it instead.

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
Input In [10], in <cell line: 1>()
      3 print(sorted(os.listdir(tmp_dir)))
      5 # Now, actually reproduce the problem
----> 6 AutoModel.from_pretrained(
      7     Path (tmp_dir),
      8     torch_dtype='auto')

File ~/.pyenv/versions/3.9.13/envs/transformers-bugfix/lib/python3.9/site-packages/transformers/models/auto/auto_factory.py:446, in _BaseAutoModelClass.from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    444 elif type(config) in cls._model_mapping.keys():
    445     model_class = _get_model_class(config, cls._model_mapping)
--> 446     return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
    447 raise ValueError(
    448     f"Unrecognized configuration class {config.__class__} for this kind of AutoModel: {cls.__name__}.\n"
    449     f"Model type should be one of {', '.join(c.__name__ for c in cls._model_mapping.keys())}."
    450 )

File ~/.pyenv/versions/3.9.13/envs/transformers-bugfix/lib/python3.9/site-packages/transformers/modeling_utils.py:2148, in PreTrainedModel.from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   2146     torch_dtype = get_state_dict_dtype(state_dict)
   2147 else:
-> 2148     one_state_dict = load_state_dict(resolved_archive_file)
   2149     torch_dtype = get_state_dict_dtype(one_state_dict)
   2150     del one_state_dict  # free CPU memory

File ~/.pyenv/versions/3.9.13/envs/transformers-bugfix/lib/python3.9/site-packages/transformers/modeling_utils.py:464, in load_state_dict(checkpoint_file)
    462 except Exception as e:
    463     try:
--> 464         with open(checkpoint_file) as f:
    465             if f.read().startswith("version"):
    466                 raise OSError(
    467                     "You seem to have cloned a repository without having git-lfs installed. Please install "
    468                     "git-lfs and run `git lfs install` followed by `git lfs pull` in the folder "
    469                     "you cloned."
    470                 )

TypeError: expected str, bytes or os.PathLike object, not list
```

Comments:
I've created PR to fix the problem as it is an easy fix. The variable names clearly suggest the intended behavior, so I just followed it.

#18061 fixes this problem. Closing the issue.