Custom pipeline in remote repo cannot load custom model from remote repo.
Author: ljw20180420
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.44.2
- Platform: Linux-6.8.0-41-generic-x86_64-with-glibc2.39
- Python version: 3.11.9
- Huggingface_hub version: 0.24.6
- Safetensors version: 0.4.4
- Accelerate version: 0.34.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.4.0+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA GeForce GTX 1080

### Who can help?

@Narsil
I define a custom pipeline ljw20180420/inDelphi_pipeline in the huggingface repository. It use a custom model from ljw20180420/SX_spcas9_inDelphi. I load the pipeline by
```python
from transformers import pipeline

pipe = pipeline(model="ljw20180420/inDelphi_pipeline", trust_remote_code=True)
```
It reports the following error:
```python
Traceback (most recent call last):
  File "/home/ljw/new_fold/old_desktop/CRISPR_AI/inDelphi/test.py", line 3, in <module>
    pipe = pipeline(model="ljw20180420/inDelphi_pipeline", trust_remote_code=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ljw/new_fold/old_desktop/CRISPR_AI/.conda/lib/python3.11/site-packages/transformers/pipelines/__init__.py", line 833, in pipeline
    targeted_task, task_options = clean_custom_task(custom_tasks[task])
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ljw/new_fold/old_desktop/CRISPR_AI/.conda/lib/python3.11/site-packages/transformers/pipelines/__init__.py", line 544, in clean_custom_task
    task_info["pt"] = tuple(getattr(transformers, c) for c in pt_class_names)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ljw/new_fold/old_desktop/CRISPR_AI/.conda/lib/python3.11/site-packages/transformers/pipelines/__init__.py", line 544, in <genexpr>
    task_info["pt"] = tuple(getattr(transformers, c) for c in pt_class_names)
                            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ljw/new_fold/old_desktop/CRISPR_AI/.conda/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1596, in __getattr__
    raise AttributeError(f"module {self.__name__} has no attribute {name}")
AttributeError: module transformers has no attribute inDelphiModel
```
It seems that pipeline() cannot find my custom model in transformers library. Is it possible to load custom model by custom pipeline?

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import pipeline

pipe = pipeline(model="ljw20180420/inDelphi_pipeline", trust_remote_code=True)
```

### Expected behavior

Load custom model "ljw20180420/SX_spcas9_inDelphi" successfully.

Comments:
cc @Rocketknight1 :)

Hi @ljw20180420, I'm not sure that a custom pipeline with a custom model in a totally different repo is supported. In particular, doing things this way makes it very hard to know what the `model` kwarg to `pipeline()` is supposed to mean - is it the repo you're loading the pipeline from or the repo you're loading the model from?

I think a better approach would be:

1) Make sure the repos you're interested in, like `ljw20180420/SX_spcas9_inDelphi` are loadable with the relevant AutoModel classes
2) Add the pipeline code to each repo
3) Check if you can load them with `pipe = pipeline(model="ljw20180420/SX_spcas9_inDelphi", trust_remote_code=True)`

If that workflow creates problems, please let us know here!

Thank you @Rocketknight1. I now push the pipeline to the same repository as the model at "ljw20180420/SX_spcas9_inDelphi".
```python
from transformers import pipeline
from transformers import AutoModel

inDelphi_model = AutoModel.from_pretrained("ljw20180420/SX_spcas9_inDelphi", trust_remote_code=True)
pipe = pipeline(model="ljw20180420/SX_spcas9_inDelphi", trust_remote_code=True)
```
AutoModel can load the model. But pipeline cannot be loaded.
```python
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ljw/new_fold/old_desktop/CRISPR_AI/.conda/lib/python3.11/site-packages/transformers/pipelines/__init__.py", line 833, in pipeline
    targeted_task, task_options = clean_custom_task(custom_tasks[task])
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ljw/new_fold/old_desktop/CRISPR_AI/.conda/lib/python3.11/site-packages/transformers/pipelines/__init__.py", line 544, in clean_custom_task
    task_info["pt"] = tuple(getattr(transformers, c) for c in pt_class_names)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ljw/new_fold/old_desktop/CRISPR_AI/.conda/lib/python3.11/site-packages/transformers/pipelines/__init__.py", line 544, in <genexpr>
    task_info["pt"] = tuple(getattr(transformers, c) for c in pt_class_names)
                            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ljw/new_fold/old_desktop/CRISPR_AI/.conda/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1596, in __getattr__
    raise AttributeError(f"module {self.__name__} has no attribute {name}")
AttributeError: module transformers has no attribute inDelphiModel
```

Hi @ljw20180420, okay! I took a quick look - I think one issue is that in `config.json`, you probably want to replace `"inDelphiModel"` in the pipeline `impl` key with `"AutoModel"`. The reason is that most pipelines are intended to work with multiple models, and so an `AutoModel` category is expected here, rather than a single specific model.

If that `config.json` was generated automatically by `transformers`, let me know - that might be something we need to fix!

Thank you @Rocketknight1. I generate `config.json` by `Trainer.push_to_hub()` and `Pipeline.push_to_hub()`.
My source codes are at git@github.com:ljw20180420/CRISPR_AI.git. In the folder CRISPR_AI, execute
```sh
inDelphi/report_bug.py
```
The source of report_bug.py is
```python
from inDelphi.configuration_inDelphi import inDelphiConfig
from inDelphi.modeling_inDelphi import inDelphiModel
from inDelphi.pipeline_inDelphi import inDelphiPipeline
from transformers import Trainer
from transformers.pipelines import PIPELINE_REGISTRY
from transformers import pipeline

inDelphiConfig.register_for_auto_class()
inDelphiModel.register_for_auto_class()
inDelphi_model = inDelphiModel(inDelphiConfig())

trainer = Trainer(model = inDelphi_model)
trainer.push_to_hub()

from huggingface_hub import HfApi
api = HfApi()
while True:
    try:
        api.upload_file(
            path_or_fileobj = "inDelphi/inDelphi_inertion_model.pkl",
            path_in_repo = "inDelphi_inertion_model.pkl",
            repo_id = "ljw20180420/tmp_trainer",
            repo_type = "model"
        )
        break
    except:
        print("upload failed, retry")

PIPELINE_REGISTRY.register_pipeline(
    task="predict CRISPR indel",
    pipeline_class=inDelphiPipeline,
    pt_model=inDelphiModel
)
pipe = pipeline(
    task="predict CRISPR indel",
    model="ljw20180420/tmp_trainer",
    trust_remote_code=True
)

pipe.push_to_hub("tmp_trainer")
```
After `trainer.push_to_hub()`, I upload an auxiliary file "inDelphi_inertion_model.pkl" (this should be irrelevant). The `config.json` is
```json
{
  "DELLEN_LIMIT": 60,
  "architectures": [
    "inDelphiModel"
  ],
  "auto_map": {
    "AutoConfig": "configuration_inDelphi.inDelphiConfig",
    "AutoModel": "modeling_inDelphi.inDelphiModel"
  },
  "mid_dim": 16,
  "model_type": "inDelphi",
  "seed": 63036,
  "torch_dtype": "float32",
  "transformers_version": "4.44.2"
}
```
After register the pipeline, `pipe` can load the remote model correctly because `config.json` have no attribute "custom_pipelines". Nevertheless, after `pipe.push_to_hub("tmp_trainer")`, it will modify `config.json` to
```json
{
  "DELLEN_LIMIT": 60,
  "_name_or_path": "ljw20180420/tmp_trainer",
  "architectures": [
    "inDelphiModel"
  ],
  "auto_map": {
    "AutoConfig": "configuration_inDelphi.inDelphiConfig",
    "AutoModel": "modeling_inDelphi.inDelphiModel"
  },
  "custom_pipelines": {
    "predict CRISPR indel": {
      "impl": "pipeline_inDelphi.inDelphiPipeline",
      "pt": [
        "inDelphiModel"
      ],
      "tf": []
    }
  },
  "mid_dim": 16,
  "model_type": "inDelphi",
  "seed": 63036,
  "torch_dtype": "float32",
  "transformers_version": "4.44.2"
}
```
The new attribute causes the bug.

Hi @ljw20180420, I think I have an idea about this

1. you have to register your model after it was initialized, I have highlighted your code in here
```python
inDelphiModel.register_for_auto_class()
inDelphi_model = inDelphiModel(inDelphiConfig())
```
2. you need to also use a class that already exists in the transformers library and not something that you made unless you want to add a `__init__.py` file , something like this for example is what i recommend `inDelphi_model.register_for_auto_class("AutoModelForImageClassification")` or any other thing **that is supported by the transformers library**

4. huggingface does indeed support custom pipelines along with custom models, but you have to follow certain steps to this, I wrote a blogpost about this in here https://huggingface.co/blog/not-lain/custom-architectures-with-huggingface feel free to check it whenever you are free.
I would also like to share with you my custom model, checkout the `config.json` file as well as `configuration_inDelphi.py` file too and change what needs to be changed in your model accordingly, link :  https://huggingface.co/not-lain/MyRepo

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.