Repository: huggingface/accelerate

Issue:
adapter-transformers: `IndexError` in `infer_auto_device_map`
### System Info

```Shell
- `Accelerate` version: 0.15.0.dev0
- Platform: Linux-3.10.0-1160.80.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.16+
- Numpy version: 1.24.0
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- `Accelerate` default config:
        Not found
```


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] One of the scripts in the examples/ folder of Accelerate or an officially supported `no_trainer` script in the `examples` folder of the `transformers` repo (such as `run_no_trainer_glue.py`)
- [X] My own task or dataset (give details below)

### Reproduction

1. `pip3 uninstall transformers`
2. `pip3 install adapter-transformers`
3. test.py:
```
import transformers
model = transformers.AutoAdapterModel.from_pretrained('google/flan-t5-base', device_map='auto')
```
Result:
```
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/user/scratch/test-2023-01-07.py:2 in <module>                                              │
│                                                                                                  │
│   1 import transformers                                                                          │
│ ❱ 2 model = transformers.AutoAdapterModel.from_pretrained('google/flan-t5-base', device_map=     │
│   3                                                                                              │
│                                                                                                  │
│ /home/user/.local/lib/python3.9/site-packages/transformers/models/auto/auto_factory.py:446 in    │
│ from_pretrained                                                                                  │
│                                                                                                  │
│   443 │   │   │   return model_class.from_pretrained(pretrained_model_name_or_path, *model_arg   │
│   444 │   │   elif type(config) in cls._model_mapping.keys():                                    │
│   445 │   │   │   model_class = _get_model_class(config, cls._model_mapping)                     │
│ ❱ 446 │   │   │   return model_class.from_pretrained(pretrained_model_name_or_path, *model_arg   │
│   447 │   │   raise ValueError(                                                                  │
│   448 │   │   │   f"Unrecognized configuration class {config.__class__} for this kind of AutoM   │
│   449 │   │   │   f"Model type should be one of {', '.join(c.__name__ for c in cls._model_mapp   │
│                                                                                                  │
│ /home/user/.local/lib/python3.9/site-packages/transformers/modeling_utils.py:2121 in             │
│ from_pretrained                                                                                  │
│                                                                                                  │
│   2118 │   │   │   no_split_modules = model._no_split_modules                                    │
│   2119 │   │   │   # Make sure tied weights are tied before creating the device map.             │
│   2120 │   │   │   model.tie_weights()                                                           │
│ ❱ 2121 │   │   │   device_map = infer_auto_device_map(                                           │
│   2122 │   │   │   │   model, no_split_module_classes=no_split_modules, dtype=torch_dtype, max_  │
│   2123 │   │   │   )                                                                             │
│   2124                                                                                           │
│                                                                                                  │
│ /shared/src/accelerate/src/accelerate/utils/modeling.py:545 in infer_auto_device_map             │
│                                                                                                  │
│   542 │   │   elif tied_param is not None:                                                       │
│   543 │   │   │   # Determine the sized occupied by this module + the module containing the ti   │
│   544 │   │   │   tied_module_size = module_size                                                 │
│ ❱ 545 │   │   │   tied_module_index = [i for i, (n, _) in enumerate(modules_to_treat) if n in    │
│   546 │   │   │   tied_module_name, tied_module = modules_to_treat[tied_module_index]            │
│   547 │   │   │   tied_module_size += module_sizes[tied_module_name] - module_sizes[tied_param   │
│   548 │   │   │   if current_max_size is not None and current_memory_used + tied_module_size >   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
IndexError: list index out of range
```

### Expected behavior

```Shell
An error or warning, or silent success.
```



Discussion:

Resolving patch:
diff --git a/src/accelerate/utils/modeling.py b/src/accelerate/utils/modeling.py
--- a/src/accelerate/utils/modeling.py
+++ b/src/accelerate/utils/modeling.py
@@ -509,7 +509,9 @@ def infer_auto_device_map(
             )
         # Assess size needed
         module_size = module_sizes[name]
-        tied_params = [v for k, v in tied_parameters.items() if name in k]
+        # We keep relevant tied parameters only: once of the tied parameters is inside the current module and the other
+        # is not.
+        tied_params = [v for k, v in tied_parameters.items() if name in k and name not in v]
         # We ignore parameters that are tied when they're tied to > 1 one
         tied_param = tied_params[0] if len(tied_params) == 1 else None
 


Test patch: