Issue loading some peft adapter model when revision is specified
Author: oOraph
Pull request: no

### System Info

transformers v4.39
peft v0.9.0

### Who can help?

@younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Not sure if I'm just misusing the transformers lib here :)

Here is a mini script to reproduce what I think is an issue:

```
from transformers import AutoModelForCausalLM

model_id = 'abhishek/autotrain-n0ni3-n7xzu'
sha = '9a08426f829f3a3f921b2858bbb23ff2801c38c6'
# sha=None
model = AutoModelForCausalLM.from_pretrained(model_id, revision=sha)
```

If we do not specify a revision then it works properly. But if we specify a revision then it fails with the following error:

```
OSError: openai-community/gpt2 does not appear to have a file named config.json. Checkout 'https://huggingface.co/openai-community/gpt2/9a08426f829f3a3f921b2858bbb23ff2801c38c6' for available files.
```

We see that the provided revision is used as a revision for the base model as well (not only the adapter), here:

https://github.com/huggingface/transformers/blob/8e9a2207b3e77eb586b2488aced2a748dde865ff/src/transformers/models/auto/auto_factory.py#L523 (stored in **hub_kwargs I think)

### Expected behavior

The adapter should load properly even when specifying a revision, and the revision should not be used to fetch the base model (or am I missing sth ?)

Comments:
cc @pacman100 and @younesbelkada ! Good news is this is not a regression.  Might just not be how it's supposed to be used! 

here is a fix proposal, this probably does not work as provided (the kwarg for the adapter is probably not the right one), need to check in peft how to provide adapter kwargs properly :)

https://github.com/oOraph/transformers/tree/dev/fix_29849_proposal

```
diff --git a/src/transformers/models/auto/auto_factory.py b/src/transformers/models/auto/auto_factory.py
index 98c0e851b..fb7ea3e34 100644
--- a/src/transformers/models/auto/auto_factory.py
+++ b/src/transformers/models/auto/auto_factory.py
@@ -506,9 +506,12 @@ class _BaseAutoModelClass:
             if maybe_adapter_path is not None:
                 with open(maybe_adapter_path, "r", encoding="utf-8") as f:
                     adapter_config = json.load(f)
-
+                    # FIXME: did not check if the key was right, just a suggestion of what the fix might look like
+                    # (or not)
+                    adapter_kwargs["_adapter_hub_kwargs"] = copy(hub_kwargs)
                     adapter_kwargs["_adapter_model_path"] = pretrained_model_name_or_path
                     pretrained_model_name_or_path = adapter_config["base_model_name_or_path"]
+                    hub_kwargs.pop("revision", None)
 
         if not isinstance(config, PretrainedConfig):
             kwargs_orig = copy.deepcopy(kwargs)
```



hi @oOraph 
thanks for the issue !
to pass a revision , you need to use `adapter_kwargs={"revision": xxx}` and pass that to `from_pretrained`, does that works for you?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.