noamaanMulla-03/transformers-issues huggingface dataset:
transformers-like library for Prompt or Agent library?
Author: ghosthamlet
Pull request: no

### Feature request

Add advanced Prompt or Agent methods as models to transformers library, or build a new advanced Prompt or Agent library use ARCH like transformers library.

### Motivation

There are many Prompt/Agent libraries like: https://langchain.com/  https://github.com/yoheinakajima/babyagi  https://github.com/Significant-Gravitas/Auto-GPT, but they are all have too many abstracts in code or just a framework for automatic tasks. 

transformers/diffusers library has simple abstracts, If there is advanced Prompt or Agent library like transformers/diffusers, it will be much better for researchers to do research work. and i think huggingface is the best to build this library.

### Your contribution

Currently no.

Comments:
Hi @ghosthamlet, 

Have you checked out Agents and Tools from Hugging Face?  https://huggingface.co/docs/transformers/v4.30.0/en/transformers_agents

@amyeroberts Thanks, it looks great. 
I found the code is all in src/transformers/tools, seems like it is not as flexible as transformers.
if someone wants to integrate a new agent like `Reflexion: Language Agents with Verbal Reinforcement Learning` https://arxiv.org/abs/2303.11366, can it be as easy as integrate a transformers model?

@ghosthamlet Easiness is subjective, so it would depend on what you do and don't find easy with the transformers library :) 

This is a question best placed in our [forums](https://discuss.huggingface.co/), as we try to reserve the github issues for feature requests and bug reports.

However, if it's a request for the model to be implemented, could you open a separate issue with the feature request?

@amyeroberts Thanks, I understand now. 
I will create a separate issue.
[LEDModel, Longformer] Make_fx compatibility
Author: Giuseppe5
Pull request: no

### System Info

- transformers version: 4.29.2
- Platform: Linux
- Python version: 3.8.16
- PyTorch version: 2.0.1

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Related to #23907

To reproduce:
```python
from torch.fx.experimental.proxy_tensor import make_fx
from transformers import LEDModel

model = LEDModel.from_pretrained('allenai/led-base-16384', torchscript=True)
inp = model.dummy_inputs['input_ids']
model.eval()
fx_g = make_fx(model)(inp)
```

The presence of `.item()` and `torch.div` cause make_fx to fail for LEDModel and Longformer with the following error:
```
RuntimeError: It appears that you're trying to get value out of a tracing tensor with aten._local_scalar_dense.default - erroring out! It's likely that this is caused by data-dependent control flow or similar.  It may be possible to trace this with dynamic shapes; try setting tracing_mode='symbolic' in your make_fx call.
```

The calls to `item()` could be removed without side effects, and I believe the same is true for replacing `torch.div` with regular python divisions.

Even with these adjustments, make_fx seems to fail in both models because of `is_global_attn`:
https://github.com/huggingface/transformers/blob/f1660d7e23d4432513fe060bde4f9b7b29f05204/src/transformers/models/led/modeling_led.py#L233
https://github.com/huggingface/transformers/blob/f1660d7e23d4432513fe060bde4f9b7b29f05204/src/transformers/models/longformer/modeling_longformer.py#L601

I am not sure if it would be possible to have a workaround for that.

One thing to note is that the presence of `.item()` and `torch.div` also causes graph breaks when using torch dynamo to get the FX representations of these models. It seems that `is_global_attn` is not an issue in that case.

### Expected behavior

The full FX representation of LEDModel/Longformer using make_fx

Comments:
@Giuseppe5 As with #23907, happy to look at a PR with a fix! 

cc @ArthurZucker @younesbelkada 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Convert PyTorch checkpoint for more recent models
Author: d-ataman
Pull request: no

### Feature request

Is it possible to add more examples for model conversion between PyTorch models e.g. convert_bart_original_pytorch_checkpoint_to_pytorch.py in more recent and popular architectures? (e.g. mt5)

### Motivation

It would really accelerate setting up new feature/model development using pretained models on the database.

### Your contribution

If anyone can give feedback I am happy to share the resulting conversion script.

Comments:
For each of the models added to the transformers repo, conversion scripts are created to port the weights from the original weights to the library format. You can find these under [models' respective folders](https://github.com/huggingface/transformers/tree/main/src/transformers/models) e.g. for [longformer](https://github.com/huggingface/transformers/blob/main/src/transformers/models/longformer/convert_longformer_original_pytorch_lightning_to_pytorch.py), [longt5](https://github.com/huggingface/transformers/blob/main/src/transformers/models/longt5/convert_longt5x_checkpoint_to_flax.py), or the [t5 script used for mt5](https://github.com/huggingface/transformers/blob/main/src/transformers/models/t5/convert_t5_original_tf_checkpoint_to_pytorch.py).

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
添加peft模型的resume_from_checkpoint实现
Author: shell-nlp
Pull request: no

### Feature request

现阶段
trainer.train(resume_from_checkpoint=resume_from_checkpoint)
不能够加载adapter 权重进行 resume

### Motivation

现阶段
trainer.train(resume_from_checkpoint=resume_from_checkpoint)
不能够加载adapter 权重进行 resume

### Your contribution

没有

Comments:
Hi @506610466, thanks for raising an issue. 

Could you please follow the issue template and provide all the information requested, including the running environment, a minimal code snippet to reproduce the error, full traceback and expected behaviour?
🌐 [i18n-KO] Translated `custom_tools.mdx` to Korean 
Author: sim-so
Pull request: yes

# What does this PR do?

Translated the `custom_tools.mdx` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

Team PseudoLab, may you please review this PR? 
@0525hhgus, @KIHOON71, @sim-so, @gabrielwithappy, @HanNayeoniee, @wonhyeongseo, @jungnerd

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)
@sgugger, @ArthurZucker, @eunseojo 
May you please review this PR?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

LGTM👍

Thanks for the PR, @sim-so! 

There's since been an update on main to all of our documentation files. Could you update the extension from `.mdx` to `.md` to match please?

Thanks for letting me know about the change! @amyeroberts! 
I updated the extention to `.md`.

Could you review this PR?
@sgugger, @ArthurZucker, @eunseojo
Extra else in GPTNeo prepare_inputs_for_generation
Author: namespace-Pt
Pull request: no

https://github.com/huggingface/transformers/blob/fe861e578f50dc9c06de33cd361d2f625017e624/src/transformers/models/gpt_neo/modeling_gpt_neo.py#L704

Here shouldn't be an extra `else`. Sometimes users want to input customized position_ids and attention_masks, however this `else` eliminates such practice.

Comments:
My bad. I think one should override the `prepare_inputs_for_generation` method if he wants to customize position_ids. The current code is okay for general use cases. But according to recent code of LLaMA https://github.com/huggingface/transformers/blob/fe861e578f50dc9c06de33cd361d2f625017e624/src/transformers/models/llama/modeling_llama.py#L740, the `else` should be deleted anyway.

Hi @namespace-Pt, thanks for raising this issue. 

Yes, the great thing about open source code is anyone can build upon and adapt it to their needs! 

Wrt comparing to the code with Llama, these are different models, and so there might be different assumptions about the inputs for generation; sometimes, even if the assumptions are the same, one model was implemented at a different time and the logic differs for backwards compatibility reasons; and sometimes it's just an oversight on our part :) 

cc @gante for reference :) 

@namespace-Pt 👋 

I think the `else` is indeed not needed. Would you like to open a PR to fix it? 🤗 

Create pull request in https://github.com/huggingface/transformers/pull/24826
Remove unused code in GPT-Neo
Author: namespace-Pt
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fix [#24820](https://github.com/huggingface/transformers/issues/24820#issuecomment-1635744675) by removing `else` statement in `GPTNeoForCausalLM`.

## Who can review?
@gante 

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
@sgugger context: this change was discussed in https://github.com/huggingface/transformers/issues/24820 -- this model is the only one that deletes `position_ids` in this function, and there is no apparent reason for it. This is a mostly unused code path and has been part of the original GPT-Neo commit.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_24826). All of your documentation changes will be reflected on that endpoint.
Ra
Author: jamesthesnake
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
is_vision_available() fails when using pillow-simd
Author: davidas1
Pull request: no

### System Info

- `transformers` version: 4.30.2
- Platform: Linux-5.11.0-1022-aws-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.0+cu118 (True)
- Tensorflow version (GPU?): 2.8.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@sgugger @apbard - additional checks introduced in https://github.com/huggingface/transformers/pull/23163 are failing on my environment.
This is because I'm using [pillow-simd](https://github.com/uploadcare/pillow-simd) and not the vanilla pillow package.
Is this drop-in pillow replacement not supported or is it possible to introduce additional logic to support this?

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

First install pillow-simd (`pip install pillow-simd`)
Run the following code:
```
import PIL
print(PIL.__version__)

import transformers
print(transformers.utils.import_utils.is_vision_available())
```

### Expected behavior

Previous versions (before 4.30.*) return `True`

Comments:
Thanks for reporting! This might be because of the new way we test if packages are available. @ydshieh would you have some bandwidth to take a look at this?

Sure, will take a look today!
Loading model microsoft/tapex-base-finetuned-wtq failed with error No module named 'transformers.models.tapex'
Author: WeichenXu123
Pull request: no

### System Info

OS: Ubuntu 22.04

Python 3.9

Packages installed:
```
pip install git+https://github.com/huggingface/transformers
pip install datasets huggingface_hub torch torchvision tensorflow accelerate librosa ffmpeg
```

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Code

```
import transformers

transformers.pipeline(
    task="table-question-answering", model="microsoft/tapex-base-finetuned-wtq"
)
```

raises error:

```
ModuleNotFoundError: No module named 'transformers.models.tapex'
```


### Expected behavior

Returns `transformers.pipelines.table_question_answering.TableQuestionAnsweringPipeline` object

Comments:
I'm unable to reproduce this on my side. Are you sure you are using the latest commit from main?

@sgugger 

Yes. I installed it  by:
`pip install git+https://github.com/huggingface/transformers`

and then in python REPL, run:
```
import transformers

transformers.pipeline(
    task="table-question-answering", model="microsoft/tapex-base-finetuned-wtq"
)
```


Our MLflow CI (run against transformer master) has the same error:
https://github.com/mlflow/mlflow/actions/runs/5567846085/jobs/10170056430#step:15:526


@sgugger Did you run it on ubuntu system ? 

I had some `__pycache__` remaining in `models/tapex` so it didn't error, but after cleaning that folder I can reproduce. Having a look, the fix should come this afternoon.

It was actually fairly easy to fix. Could you quickly check the PR above solves the issue for you too?
Add TAPEX to the list of deprecated models
Author: sgugger
Pull request: yes

# What does this PR do?

TAPEX was not in the list of deprecated models, leading to importing it with the auto API not working. I'll make a script to check the content of that constant is in sync with the content of the deprecated folder so this doesn't happen again.

Fixes #24852

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_24859). All of your documentation changes will be reflected on that endpoint.
Fix token pass
Author: sgugger
Pull request: yes

# What does this PR do?

The `token` passed along in `PreTrainedTokenizerBase.from_pretrained` is passed along twice at the end: one time in the kwargs and one time as `use_auth_token`. This caused the speech examples to fail.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_24862). All of your documentation changes will be reflected on that endpoint.
Fix the fetch of all example tests
Author: sgugger
Pull request: yes

# What does this PR do?

I noticed on recent PRs that when all tests are fetched, the example tests are not run. Upon closer inspection, it's because the test for `"all"` in the `test_fetcher` compares a list instead of a string. This PR addresses that.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_24864). All of your documentation changes will be reflected on that endpoint.
Replace assert statements with exceptions
Author: syedsalman137
Pull request: yes

# What does this PR do?

I have replaced the assert statements with appropriate exceptions in the directory `src/transformers/models/` with all models beginning with `a` and `b` letters.

Also, I have corrected error handling at places where except statements were handling AssertionError, even thought it was never to be raised. Here is an example:

```
try:
    if pointer.shape != array.shape:
        raise ValueError(f"Pointer shape {pointer.shape} and array shape {array.shape} mismatched")
except AssertionError as e:    # Incorrect line
    e.args += (pointer.shape, array.shape)
    raise
```

I changed the above to:

```
try:
    if pointer.shape != array.shape:
        raise ValueError(f"Pointer shape {pointer.shape} and array shape {array.shape} mismatched")
except ValueError as e:    # Corrected the line
    e.args += (pointer.shape, array.shape)
    raise
```

Fixes #12789 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@ArthurZucker @sgugger 


Comments:
_The documentation is not available anymore as the PR was closed or merged._