noamaanMulla-03/transformers-issues huggingface dataset:
Fix typo in Llama docstrings
Author: Kh4L
Pull request: yes

# What does this PR do?
Fix typos in docs.

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).


Comments:
@amyeroberts yes, I ran it locally, got the error and fixed it,

here are the types in my code
```
type(tokenizer)
<class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>

type(inputs)
<class 'torch.Tensor'
``

@Kh4L Out of interest - could you share the checkpoint being used? Could you also run this snippet with the checkpoint and share the output? 

```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(MODEL_CHECKPOINT)
prompt = "Hey, are you conscious? Can you talk to me?"
inputs = tokenizer(prompt, return_tensors="pt")
print(inputs)
print(type(inputs))
print(type(tokenizer))
```

The current changes need to be checked with a standard checkpoint for all the models affected here. For instance, if I run the snippet with the OPT checkpoint in the example

`MODEL_CHECKPOINT = "facebook/opt-350m"`

I get the following output:
```
{'input_ids': tensor([[    2, 13368,     6,    32,    47, 13316,   116,  2615,    47,  1067,
             7,   162,   116]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}
<class 'transformers.tokenization_utils_base.BatchEncoding'>
<class 'transformers.models.gpt2.tokenization_gpt2_fast.GPT2TokenizerFast'>
```




@amyeroberts  Checkpoint is the 7B LLama converted to HF, I get the same output!
Sorry for the confusion, I was using  `LlamaTokenizer`  and not `AutoTokenizer` in my code

Thanks for the detailed review!
I am a bit confused as I can't see the latest commit  https://github.com/Kh4L/pytorch-transformers/commit/62ea9f244b70ae190b20c69742027e277a241f2e  in this PR, even though I pushed it on my branch https://github.com/Kh4L/pytorch-transformers/tree/fix_conscious_typo 🤔 

_The documentation is not available anymore as the PR was closed or merged._

@Kh4L Github PRs were down for part of yesterday - I think it was just that. I can see the commit now and all tests are passing :)
Avoid `GPT-2` daily CI job OOM (in TF tests)
Author: ydshieh
Pull request: yes

# What does this PR do?

Clear (as much as possible) GPU memory usage allocated by torch, so the TF tests (GPT-2) get more room and make @Rocketknight1 's life easier 😆.

Some (TF) tests get OOM after #23234.

Note, the changes in this PR are in torch test files instead of TF test files! This is similar to #16881, which has more details mentioned.

Running manually and all gpt2 tests pass now.


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Exception when saving weights from QLORA due to UnboundLocalError
Author: emmatyping
Pull request: no

### System Info

- `transformers` version: 4.30.0.dev0
- Platform: Linux-6.1.0-9-amd64-x86_64-with-glibc2.36
- Python version: 3.10.10
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@sgugger you reviewed the PR so it looks like your eyes were on this most recently. Relevant commit: https://github.com/huggingface/transformers/commit/357f281ba24af8d49afd84c7628329d99868f411

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Run the following notebook: https://colab.research.google.com/github/utensil/llm-playground/blob/main/notebooks/axolotl/colab/axolotl_falcon_1b_qlora_gsm8k.ipynb.

I get the following error: 
```
Traceback (most recent call last):
  File "/home/e/et/ethanhs/axolotl/axolotl/scripts/finetune.py", line 295, in <module>
    fire.Fire(train)
  File "/home/e/et/ethanhs/miniconda3/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/e/et/ethanhs/miniconda3/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/e/et/ethanhs/miniconda3/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/e/et/ethanhs/axolotl/axolotl/scripts/finetune.py", line 282, in train
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/home/e/et/ethanhs/miniconda3/lib/python3.10/site-packages/transformers/trainer.py", line 1661, in train
    return inner_training_loop(
  File "/home/e/et/ethanhs/miniconda3/lib/python3.10/site-packages/transformers/trainer.py", line 2070, in _inner_training_loop
    self._load_best_model()
  File "/home/e/et/ethanhs/miniconda3/lib/python3.10/site-packages/transformers/trainer.py", line 2258, in _load_best_model
    self._issue_warnings_after_load(load_result)
UnboundLocalError: local variable 'load_result' referenced before assignment
```

### Expected behavior

I expect for transformers not to raise an exception.

Comments:
cc @younesbelkada 

Hi @ethanhs 
Thanks for reporting #24103 should solve the issue

Thanks for the quick response and quick fix!
Add option for `trust_remote_code=True` on transformers-cli download 
Author: radames
Pull request: no
Labels: name: Feature request

### Feature request

Currently is very convenient to download models using  `transformers-cli download`, however some models need an extra argument for `trust_remote_code=True`

for example 

`transformers-cli download "tiiuae/falcon-40b"`


### Motivation

Would it make sense to add

`transformers-cli download "tiiuae/falcon-40b" --trust_remote_code`

### Your contribution

PR

Comments:
Sounds like a good addition to me! 

cc @sgugger who's been doing a lot of the work enabling remote code integration. 

Yes, happy to review a PR!

thanks @amyeroberts and @sgugger , is there any other argument that worth adding and loading a model?
Nah
Author: jamesthesnake
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
doc issue from docs/source/en/model_doc/open-llama.mdx
Author: xingener
Pull request: no

In overview, 2nd paragraph：

The model is mainly based on LLaMA with some modifications, incorporating memory-efficient attention from Xformers, stable embedding from Bloom, and shared input-output embedding from PLAM. And the model is pre-trained on both Chinese and English, which gives it better performance on Chinese language tasks.

Shared input-output embedding should be from PaLM?

Comments:
@xingener Good spot - would you like to open a PR to fix it? 

> @xingener Good spot - would you like to open a PR to fix it?

Sure.
[Time-Series] Autoformer - Transformer For Time-Series Forecasting
Author: elisim
Pull request: no
Labels: name: New model

# Model Description

Following #20903 and #21099, Autoformer is the next Transformer in the series, published in NIPS 21.
* Paper: [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting
](https://arxiv.org/abs/2106.13008)
* Model implementation: https://github.com/thuml/Autoformer

I would like to implement the model :)

Thank you,
Eli

### Open source status

- [X] The model implementation is available
- [ ] The model weights are available

### Provide useful links for the implementation

@wuhaixu2016 - repository creator

@NielsRogge @kashif 

Comments:
Support "OptimizedModule" models obtained from torch.compile() in inference pipelines
Author: jorirsan
Pull request: no

### Feature request

Currently models obtained by the [torch.compile()](https://pytorch.org/docs/stable/generated/torch.compile.html) feature introduced in Pytorch 2.0 are not supported in inference pipelines from :hugs: Transformers. 

### Motivation

The same way pipelines support optimization methods such as the use of [accelerate](https://github.com/huggingface/accelerate) with "device_map" and [bitsandbytes](https://github.com/TimDettmers/bitsandbytes) with "load_in_8bit",  torch.compile() support would make a great addition for better inference performance in pipelines.

### Your contribution

Opening this feature request :P. Unfortunately I'm not familiar enough with the :hugs: Transformers codebase and the quirks that one could find with such a new feature such as torch.compile() to feel as I can tackle his task.

Comments:
cc @Narsil 

Hi @iranz15 can you share an example of code of failure  ?

I tried to do :

```python
pipe.model = torch.compile(pipe.model)
```

And everything seems to work ok after removing `inference_mode` and just using `torch.no_grad`.
@sgugger any preference between compilation and inference mode ?

I haven't played enough with either but I remember `inference_mode` benefits being minor, maybe `torch.compile` benefits are currently much higher.

I don't know `inference_mode` very much, so not sure. It also seems like it should work with compiled model so might be worth reporting to the PyTorch team.

Hey @Narsil. A error/warning message seems to show when using `torch.compile()` on the model before adding to the pipeline, that is, for the following case
```python
model = AutoModelForCausalLM.from_pretrained("bigscience/bloomz-560m")
tokenizer = AutoTokenizer.from_pretrained("bigscience/bloomz-560m")
model = torch.compile(model)
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
```
it will complain that `The model 'OptimizedModule' is not supported for $TASK_NAME ...`, but first loading the pipeline and then accessing the model and compiling it like you did seems to work fine. Surprisingly, after doing some additional tests, even when it complains about 'OptimizedModule' it seems that you can still use the pipeline without any problem. The additional errors that I was having where due to some oversights in my code that wrongly pointed me to blame `torch.compile()`.



> The additional errors that I was having where due to some oversights in my code that wrongly pointed me to blame torch.compile().

OK thanks, a warning will trigger when sending arbitrary objects (because it's likely an issue in user code) but still will use the model pass through so you can use any class you want so you can run onnx, optimum etc.. too. (as long as they support the same API as the original model)
[Lllama] Update tokenization code to ensure parsing of the special tokens [core]
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Adresses the issues with the fast tokenizer of LLama. Namely: 
- nit making it return token type ids. 
- the added tokens are not correctly encoded. 

There seems to be an issue with the conversion: before the python layer, just loading the tokenizer_config.json file with the rust backend still produced: `tokenizer.encode("this is not
").tokens`, ` ['
', '▁This', '▁is', '▁not', '</', 's', '>']`

Comments:
Ok, narrowed it down to this line: 
```python 
        # Check all our special tokens are registered as "no split" token (we don't cut them) and are in the vocab
        added_tokens = tokenizer.sanitize_special_tokens()
```
When converting the model from a slow one, the tokenizer correctly processes the inputs up until this point. Meaning that before, the special tokens where already registered as special tokens, but adding them once more most probably breaks the internal regex. Still checking but should be this. 

_The documentation is not available anymore as the PR was closed or merged._

After debugging with @Narsil it seems that the special tokens have to be not normalised, otherwise the normalizer prepends a space when adding it, which is why the token is not recognized. I suspect that there is another bug, as I tried with special tokens set to normalized = True (when calling `from_slow=True`+commenting `self._sanitize_special_tokens`) but the current should fix the conversion. 

A big discrepancy is that the default `AddedTokens` imported from `tokenizers` will set `normalized` to `!special`, so if you add tokens as special tokens, `normalized` will be False. But in `transformers` this is not the case, which explains why the call to sanitize is a source of problem.

We have to update the online models to change the `tokenizer.json`, (people might be confused because the `normalized` param is also in the slow files but always ignored)
Remove decoder_input_ids from RAG dummy inputs
Author: Rocketknight1
Pull request: yes

cc @ydshieh 

The old RAG dummy inputs didn't have `decoder_input_ids` but the new ones do - this seems like the most likely cause of the memory blowup, because RAG probably does a lot of weird retrieval stuff.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

unfortunately, the problematic situation (GPU usage) I described on Slack channel is still the same. It still takes 5 - 6 G extra memory at some point, compared to the commit before `814de8fa`.

Closing this because the dummies turned out not to be the problem after all!
Correctly build models and import call_context for older TF versions
Author: Rocketknight1
Pull request: yes

Our import for `call_context` was wrong for older TF versions - this unfortunately makes it quite hard to load models on older TF versions! This PR fixes it, and sorry about the issue!

Fixes #24133

Comments:
@amyeroberts The code is in a conditional block that checks TF versions. I'm currently testing all TF versions since 2.4 to make sure this works for all of them - give me a few minutes to finish that before we merge the PR!

Version testing looks good!

_The documentation is not available anymore as the PR was closed or merged._

Merging this now and will begin discussions about a patch release
LlamaForCausalLM generate() runtime error when top_p=0
Author: tranhungnghiep
Pull request: no

### System Info

- `transformers` version: 4.29.2
- Platform: Linux-4.4.0-210-generic-x86_64-with-debian-stretch-sid
- Python version: 3.7.3
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- Tensorflow version (GPU?): 2.5.0 (True)

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

This example follows and modifies the LLaMA documentation at: https://huggingface.co/docs/transformers/v4.29.1/model_doc/llama.

```python
from transformers import AutoTokenizer, LlamaForCausalLM

model = LlamaForCausalLM.from_pretrained(PATH_TO_CONVERTED_WEIGHTS).cuda()
tokenizer = AutoTokenizer.from_pretrained(PATH_TO_CONVERTED_TOKENIZER)

prompt = "Hey, are you conscious? Can you talk to me?"
inputs = tokenizer(prompt, return_tensors="pt").cuda()

# Generate
generate_ids = model.generate(
    inputs.input_ids, max_length=1024,
    do_sample=True,
    temperature=0.6,
    top_k=1000,
    top_p=0.0,  # cause error
    repetition_penalty=(1.0 / 0.85),
    )
tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
```

>RuntimeError: probability tensor contains either `inf`, `nan` or element < 0

### Expected behavior

`generate()` should check if `top_p==0` and disables it to avoid numerical error. This behavior is the same as when `top_p==1` and consistent with the documentation.

Comments:
cc @gante 

Hi @tranhungnghiep -- there is indeed an issue, but not on allowing p=0 :) I will open a PR to fix it (feel free to check the solution there, after it gets linked on this issue)

Please note that setting `top_p=0` is the effectively the same as doing `do_sample=False` ⚠️
fix bugs with trainer
Author: pacman100
Pull request: yes

# What does this PR do?
Context:
1. Issue 1 - Currently, when the lr_scheduler is specified in the deepspeed config file, we leverage a DummyScheduler to pass to the `accelerator.prepare` to get the correct scheduler post prepare call. A prior PR removed preparation of the lr_scheduler leading to a lot of DeepSpeed tests failing.
2. Issue 2 - when using apex we shouldn't be preparing optimizer else we get `AttributeError: 'AcceleratedOptimizer' object has no attribute '_amp_stash'`
3. Issue 3 - FSDP ckpt logic should create ckpt dir if not present. Fixes https://github.com/huggingface/transformers/issues/24130

This PR fixes the above issues. 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[`SAM`] Fix sam slow test
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes SAM slow test, link to failing job: https://github.com/huggingface/transformers/actions/runs/5206799863

* Why this fix is relevant? 

Before the PR, it seems I was initalizing the pipeline in the wrong way. Passing a string to `device` argument of pipeline leads to an error. To reproduce:
```python
from transformers import pipeline

pipe = pipeline("text-generation", device="cuda")
pipe("Hello")
>>> ValueError: Expected a torch.device with a specified index or an integer, but got:cuda
```
That is yield here: https://github.com/huggingface/transformers/blob/847b47c0eed4e6ab904f584fb415e3d3a397867f/src/transformers/pipelines/base.py#L905 

Whereas the typehint of pipeline says: https://github.com/huggingface/transformers/blob/847b47c0eed4e6ab904f584fb415e3d3a397867f/src/transformers/pipelines/base.py#L763

The fix seems to be to pass an int for the device if cuda is available and -1 if not

cc @ydshieh 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@younesbelkada Could you update the type hint for the pipeline too? 

nb: from pytorch it seems [we shouldn't use torch.cude.set_device()](https://pytorch.org/docs/stable/generated/torch.cuda.set_device.html) cc @Narsil 

Sure yes just updated it, we probably need also to address a proper fix for that in a separate PR, not sure though 

Code is from 3 years ago (and I'm pretty sure I just moved it from somewhere else).

Happy to refactor to something more up-to-date. Accepting `strings` as device should be supported imo.

@Narsil I also agree we should support str for pipeline, let me know if you want me to work on this, I am happy to have a look and ping you once something is ready
[lamaTokenizerFast] Update documentation
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Updates the documentation for llamaFast. 
I think that long term-wise it would make more sense that the rust tokenizer updates its internals if the specials tokens are updated as well, this would allow us to remove the python layer that takes care of it. 

Related to #23889 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[BlenderBotSmall] Update doc example
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Blenderbot small seems to be using `__start__` and `__end__`, updated the doc to reflect that. 
Super hard to fin original source, but based on the tokenizer this is what is used. 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix Pipeline CI OOM issue
Author: ydshieh
Pull request: yes

# What does this PR do?



Comments:
_The documentation is not available anymore as the PR was closed or merged._

>     1. Do you think this memory handling is something we should extend to most model tests by default?  Is there too much of an overhead repeatedly clearing the cache / GC or other reasons it's not suitable?

This cleanup is only necessary for integration tests, but yes, I think it's best to apply this to all model (integration) tests.

>     2. Could we create a general utility in testing utils to avoid some of the repeated code e.g.

Yes! I am also thinking if we should define a subclass of unittest.TestCase and have a common `def tearDown`.
Let's talk this and/or your suggestion above later.