proposed fix for https://github.com/huggingface/transformers/issues/29259
Author: rl337
Pull request: yes

# What does this PR do?
In the issue: https://github.com/huggingface/transformers/issues/29259 I raised a problem about from_config not being usable when loading models directly from disk when the current path doesn't happen to have the model to be loaded. 

This PR is solution (2) in that issue report.

Creating this PR as I needed to make this change to unblock my code anyway.  Will change to whatever the ultimate solution is if it's not this one.

Here's how it's used in my code:

```
try:
    model = AutoModel.from_pretrained(args.base_model_dir, config=conf, local_files_only=True, trust_remote_code=True)
    logger.info(f"Loaded pre-trained model from {args.base_model_dir}")
except (OSError, ValueError) as e:
    logger.info(f"No pretrained weights found creating a fresh model from {args.base_model_dir}")
    model = AutoModel.from_config(conf, name_or_path=args.base_model_dir, local_files_only=True, trust_remote_code=True)
```
cc @Rocketknight1 
<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # https://github.com/huggingface/transformers/issues/29259


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Hey @Rocketknight1 not sure if you are working on this since you assigned yourself? 

I'd really like this change merged OR propose a different change that accomplishes the same thing. I'll have time to update the PR soon if this isn't the solution we want. 

Hi, firstly I'm very sorry for the long delay - I totally lost track of this one!

However, I'm struggling to reproduce this myself. Here's what I ran locally. I used [InternLM2](https://huggingface.co/internlm/internlm2-1_8b), which is a remote-code only model on the HF Hub, and I cloned it to a local dir:

```python
from transformers import AutoConfig, AutoModelForCausalLM

config = AutoConfig.from_pretrained("./internlm2-1_8b", trust_remote_code=True)
model = AutoModelForCausalLM.from_config(config, trust_remote_code=True)
```

This seemed to run correctly for me - can you clarify what exactly goes wrong for you when you do this? How are you generating the config that you're passing to `AutoModel.from_config()`?

@Rocketknight1 I go into the details of what happens in the associated issue but specifically, the error manifests as remote files not being found.  There are a couple of situations that work and will mask the issue.  
1. If the model is loaded relative to the current directory and the subdirectory happens to be the same as the model name.  2. If the model is in a subdirectory of the current directory that happens to be the same as the model id specified in the `model_id||classname` format. 

I have models stored on a drive array and access them from a path that's not in the current directory structure.  So for example, my base path to my models. is /mnt/model_storage where as the code I'm running is relative to my home directory. 

The root cause is, we don't keep track of the physical location that the original config was loaded from and instead start from scratch when locating subsequent files.  We need to either pass it in as I did in this PR or infer it from the physical location of the loaded config (if one exists)

Hi @rl337, I tried moving the directory around and renaming it and still can't reproduce it! I ran the code from my home directory, and loaded from a directory that's outside my home dir, using a new name, and it still worked fine for me.

```python
>>> from transformers import AutoConfig, AutoModelForCausalLM
>>> config = AutoConfig.from_pretrained("/tmp/renamed_internlm2-1_8b", trust_remote_code=True)
>>> model = AutoModelForCausalLM.from_config(config, trust_remote_code=True)
```

> Hi @rl337, I tried moving the directory around and renaming it and still can't reproduce it! I ran the code from my home directory, and loaded from a directory that's outside my home dir, using a new name, and it still worked fine for me.
> 
> ```python
> >>> from transformers import AutoConfig, AutoModelForCausalLM
> >>> config = AutoConfig.from_pretrained("/tmp/renamed_internlm2-1_8b", trust_remote_code=True)
> >>> model = AutoModelForCausalLM.from_config(config, trust_remote_code=True)
> ```

when I look at internlm2-1_8b, it's not using the `model_id||class` but only has the class name in its AutoConfig.  Here's an excerpt from https://huggingface.co/internlm/internlm2-1_8b/blob/main/config.json

```
  "auto_map": {
    "AutoConfig": "configuration_internlm2.InternLM2Config",
    "AutoModelForCausalLM": "modeling_internlm2.InternLM2ForCausalLM",
    "AutoModel": "modeling_internlm2.InternLM2ForCausalLM"
  },
```

If we change this to be of the format `model_id||class` does it still succeed?
```
  "auto_map": {
    "AutoConfig": "internlm/internlm2-1_8b||configuration_internlm2.InternLM2Config",
    "AutoModelForCausalLM": "internlm/internlm2-1_8b||modeling_internlm2.InternLM2ForCausalLM",
    "AutoModel": "internlm/internlm2-1_8b||modeling_internlm2.InternLM2ForCausalLM"
  },
````



Hm, this might be a stupid question (I'm not totally familiar with that part of the codebase), but how would values like that get into the `auto_map`? Are there other repos using that format? Most repos I've looked at just use a simple `modeling_xxx.xxxConfig` structure in the `auto_map`

sorry the model_id/class separator is `--` not `||` You can see an example of it in the config.json here:
https://huggingface.co/togethercomputer/StripedHyena-Nous-7B/blob/main/config.json#L26-L28

When I was in the code, the model_id ends up getting prepended to the path in the event that the model_or_path doesn't exist.  If the prepended model_id/path new path doesn't exist then it tries a remote lookup against hugging face where it tries to download the needed files from `username/model` which forms the model_id. 

It feels like this expanding fallback logic probably grew organically to help make life easier for people using inference pipeline code so the edge cases I'm hitting where I happen to be pedantic about not downloading files are hitting issues. 




One of the possible uses that I personally might have for this format is, I can create an empty "template" model that houses common model implementations in say: `rl337/template` and reference the implementation `rl337/template--SomeAwesomeModel` instead of including SomeAwesomeModel with the model's itself. So in that kind of auto_map, it might look like:

```
 "auto_map": {
    "AutoConfig": "rl337/UsecaseA--model_config.UseCaseAForSomeAwesomeModel",
    "AutoModel": "rl337/template--model_impl.SomeAwesomeModel"
  },
```

This would give me the benefit of making updates to only the python class in rl337/template and it taking effect in any model such as UsecaseA ... UsecaseN without having to update them all.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.