noamaanMulla-03/transformers-issues huggingface dataset:
Many things are not importable in an environment with flash attention v1 installed
Author: dakinggg
Pull request: no

### System Info

- `transformers` version: 4.34.0
- Platform: Linux-5.4.0-148-generic-x86_64-with-glibc2.31
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

and

```
Name: flash-attn
Version: 1.0.9
Summary: Flash Attention: Fast and Memory-Efficient Exact Attention
Home-page: https://github.com/HazyResearch/flash-attention
Author: Tri Dao
Author-email: trid@stanford.edu
License: UNKNOWN
Location: /mnt/workdisk/danielking/miniconda3/envs/foundry-3.10/lib/python3.10/site-packages
Requires: einops, ninja, packaging, torch
Required-by: 
```

### Who can help?

@younesbelkada

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
In [1]: from transformers.models.auto.modeling_auto import MODEL_FOR_CAUSAL_LM_MAPPING                                                                                                                                                                                                                          
                                                             
In [3]: lll = list(MODEL_FOR_CAUSAL_LM_MAPPING.values())
```

or 

```
mistral = transformers.AutoModelForCausalLM.from_pretrained('mistralai/Mistral-7B-v0.1')
```

result in

```
File /mnt/workdisk/danielking/miniconda3/envs/foundry-3.10/lib/python3.10/site-packages/transformers/utils/import_utils.py:1284, in _LazyModule._get_module(self, module_name)
   1282     return importlib.import_module("." + module_name, self.__name__)
   1283 except Exception as e:
-> 1284     raise RuntimeError(
   1285         f"Failed to import {self.__name__}.{module_name} because of the following error (look up to see its"
   1286         f" traceback):\n{e}"
   1287     ) from e

RuntimeError: Failed to import transformers.models.mistral.modeling_mistral because of the following error (look up to see its traceback):
cannot import name 'flash_attn_func' from 'flash_attn' (/mnt/workdisk/danielking/miniconda3/envs/foundry-3.10/lib/python3.10/site-packages/flash_attn/__init__.py)
```

### Expected behavior

I should still be able to import things when i have flash attention v1 installed, even if i can't make use of it directly in transformers.

Comments:
Hi @dakinggg 
Thanks for reporting, https://github.com/huggingface/transformers/pull/26785 should hopefully resolve the issue
[`core`] Fix fa-2 import
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes: https://github.com/huggingface/transformers/issues/26778 

For users that have FA-1 installed in their environment importing some modules will lead to errors, making `transformers` unusable. This PR fixes this issue by changing `is_flash_attn_available()` to `is_flash_attn_2_available()`.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Can confirm on my end that installing an old version of FA now does not lead to an error! Merging !
🌐 [i18n-KO] Translated `big_models.md` to Korean
Author: wonhyeongseo
Pull request: yes

# What does this PR do?

@bolizabeth translated the `big_models.md` file of the documentation to Korean. I made sure hanging suggestions were resolved.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179
Moved from https://github.com/huggingface/transformers/pull/24985

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에, 이 아래에 리뷰를 요청할 팀원들을 멘션해주세요! -->
<!-- Team OSSCA, may you please review this PR? @bolizabeth, @nuatmochoi, @heuristicwave, @mjk0618, @keonju2, @harheem, @HongB1, @junejae, @54data, @Sunmin0520, @seank021, @augustinLib, @sronger, @TaeYupNoh, @kj021, @eenzeenee -->
Team PseudoLab, may you please review this PR? @0525hhgus, @KIHOON71, @sim-so, @gabrielwithappy, @HanNayeoniee, @wonhyeongseo, @jungnerd

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
<!-- May you please review this PR? @sgugger, @ArthurZucker, @stevhliu -->

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26245). All of your documentation changes will be reflected on that endpoint.
Disable default system prompt for LLaMA
Author: Rocketknight1
Pull request: yes

Disable the system prompt by default in LLaMA, as requested by Meta.

Note that I've already pushed chat templates to the repos, so this change should mostly already be in effect! I'm just changing it in the library too for consistency.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Just opened #26766, I see this is intentional at request of Meta. @Rocketknight1 was any reason given for removing the system prompt? Should no system prompt be used? Can it be replaced with a different system prompt if one still needs to be used?

Hi @AjayP13, the default system prompt was generally felt to cause the model to be very conservative, and to refuse to answer lots of queries, even valid ones. You can try running LLaMA-2 either without a system prompt at all, or with any prompt of your choice - either of those is a valid option!
Fix Falcon generation test
Author: Rocketknight1
Pull request: yes

One of the Falcon tests was broken by the shift to using in-library checkpoints. The reason is that the Falcon tokenizer now correctly doesn't return `token_type_ids`, since Falcon doesn't use that input. The test was discarding that key, but this is no longer necessary.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[i18n-<kan>] Translating docs to <Kannada>
Author: praj-pawar
Pull request: no
Labels: name: WIP

<!--
Note: Please search to see if an issue already exists for the language you are trying to translate.
-->

Hi!

Let's bring the documentation to all the <Kannada>-speaking community 🌐 (currently 0 out of 267 complete)

Who would want to translate? Please follow the 🤗 [TRANSLATING guide](https://github.com/huggingface/transformers/blob/main/docs/TRANSLATING.md). Here is a list of the files ready for translation. Let us know in this issue if you'd like to translate any, and we'll add your name to the list.

Some notes:

* Please translate using an informal tone (imagine you are talking with a friend about transformers 🤗).
* Please translate in a gender-neutral way.
* Add your translations to the folder called `<kan>` inside the [source folder](https://github.com/huggingface/transformers/tree/main/docs/source).
* Register your translation in `<kan>/_toctree.yml`; please follow the order of the [English version](https://github.com/huggingface/transformers/blob/main/docs/source/en/_toctree.yml).
* Once you're finished, open a pull request and tag this issue by including #issue-number in the description, where issue-number is the number of this issue. Please ping @stevhliu and @MKhalusova for review.
* 🙋 If you'd like others to help you with the translation, you can also post in the 🤗 [forums](https://discuss.huggingface.co/).

## Get Started section

- [ ] [index.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/index.md) https://github.com/huggingface/transformers/pull/20180
- [ ] [quicktour.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/quicktour.md) (waiting for initial PR to go through)
- [ ] [installation.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/installation.md).

## Tutorial section
- [ ] [pipeline_tutorial.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/pipeline_tutorial.md)
- [ ]  [autoclass_tutorial.md](https://github.com/huggingface/transformers/blob/master/docs/source/autoclass_tutorial.md)
- [ ]  [preprocessing.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/preprocessing.md)
- [ ]  [training.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/training.md)
- [ ]  [accelerate.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/accelerate.md)
- [ ]  [model_sharing.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/model_sharing.md)
- [ ]  [multilingual.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/multilingual.md)

<!--
Keep on adding more as you go 🔥
-->


Comments:
Hey! I'd like to translate the index.md file first.
broken FLAVA example code
Author: morrisalp
Pull request: no

### System Info

`transformers==4.33.1`

### Who can help?

@ArthurZucker @younesbelkada @amyeroberts 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Error is thrown when running the following [sample code from the FLAVA documentation](https://huggingface.co/docs/transformers/model_doc/flava#transformers.FlavaModel):
```
from PIL import Image
import requests
from transformers import AutoProcessor, FlavaModel

model = FlavaModel.from_pretrained("facebook/flava-full")
processor = AutoProcessor.from_pretrained("facebook/flava-full")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(text=["a photo of a cat"], images=image, return_tensors="pt", padding=True)

outputs = model(**inputs)
logits_per_image = outputs.contrastive_logits_per_image  # this is the image-text similarity score
probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities
```

Error:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[9], line 1
----> 1 logits_per_image = outputs.contrastive_logits_per_image  # this is the image-text similarity score
      2 probs = logits_per_image.softmax(dim=1)

AttributeError: 'FlavaModelOutput' object has no attribute 'contrastive_logits_per_image'
```

### Expected behavior

Add `contrastive_logits_per_image` to output or fix example in documentation.

Comments:
Hi @morrisalp 
Thanks for the issue, in fact, the documentation snippets looks wrong indeed. To retrieve `contrastive_logits_per_image `. one needs to use `FlavaForPreTraining` as the `global_contrastive_head` that is used to compute the contrastive logits seems to be available only for that class. 

This snippet seems to work fine for me:

```python
from PIL import Image
import requests
from transformers import AutoProcessor, FlavaForPreTraining

model = FlavaForPreTraining.from_pretrained("facebook/flava-full")
processor = AutoProcessor.from_pretrained("facebook/flava-full")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(text=["a photo of a cat"], images=image, return_tensors="pt", padding=True, return_codebook_pixels=True, return_image_mask=True)

outputs = model(**inputs)
logits_per_image = outputs.contrastive_logits_per_image  # this is the image-text similarity score
probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities
```

Would you be happy to open a PR to update the docs? Otherwise happy to do it!
[`Flava`] Fix flava doc
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes: https://github.com/huggingface/transformers/issues/26078

In fact the current flava docstring fails, running:

```bash
pytest --doctest-modules src/transformers/models/flava/modeling_flava.py::transformers.models.flava.modeling_flava.FlavaModel.forward
```

Leads to a failure. This PR fixes it

`contrastive_logits_per_image` cannot be retrieved from `FlavaModel` as detailed here: https://github.com/huggingface/transformers/issues/26078#issuecomment-1713369568 , therefore I have decided to modify the docstring to reflect the correct way of retrieving outputs from `FlavaModel`




Comments:
_The documentation is not available anymore as the PR was closed or merged._
translation brazilian portuguese
Author: alvarorichard
Pull request: yes

Hello, I would like to add the Brazilian Portuguese translation to the README.md. I translated it as faithfully and comprehensibly as possible to the original. I only did the translation, without adding anything else besides that. I kindly ask you to consider this contribution and review it for approval.
@stevhliu and @MKhalusova 

Comments:
I have already applied the suggested changes

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26769). All of your documentation changes will be reflected on that endpoint.
Bug: metrics inside on_evalute callback is passed wrongly 
Author: ghost
Pull request: no

Hi
this is very helpful to save all metrics every eval_step with evaluation_stategy = steps, for this I wrote a callback as follows to access metrics inside this callback:

```
class EvaluationCallback(TrainerCallback):
    def  on_evaluate(self, args, state, control, **kwargs):
        print("### kwargs ", kwargs['metrics']) #
```

{'eval_loss': 972.89990234375, 'eval_acc': 0.0}

I pass this callback to the trainer.py

from what I see this metrics does not match the output of evaluate function, for instance in my case this is 
`{'boolq_eval_loss': 525.3097534179688, 'boolq_eval_acc': 60.6, 'rte_eval_loss': 972.89990234375, 'rte_eval_acc': 0.0}
`
could you tell me how I can access output of evaluate() inside this callback as this is? thanks 


Comments:
Issue resolved with adding this line in evaluate function:

 self.control = self.callback_handler.on_evaluate(self.args, self.state, self.control, results)


reopened, since this is not solved after this line, indeed this looks like a bug, could you have a look please?

Hello, could you please put all of your environment information as asked in the template, as well as the command you used to launch the script? We need this in order to help you. Thank you.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

In my case, I found that trainer does not do the evaluation event at the end of each epoch even I set the trainer.args eval_stategy = epoch. So maybe we find the log info such as {eval_loss: xxx, eval_acc: xxx, **epoch: 1.98**}. And finally I prefer to use callback which is defined by myself to log metric.
RuntimeError: result type Float can't be cast to the desired output type Byte
Author: Vectorrent
Pull request: no

### System Info

- `transformers` version: 4.33.2
- Platform: Linux-6.5.3-arch1-1-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.17.2
- Safetensors version: 0.3.3
- Accelerate version: 0.23.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@gante @SunMarc 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

This problem occurs when trying to use RWKV with the "bnb_4bit_use_double_quant" argument in a BitsAndBytesConfig. You can fully reproduce the error with the following code:
```
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

model_name = "RWKV/rwkv-4-169m-pile"

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    llm_int8_threshold=6.0,
    llm_int8_has_fp16_weight=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
)

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name, torch_dtype=torch.bfloat16, quantization_config=quantization_config
)

inputs = tokenizer("A cat sat", return_tensors="pt")["input_ids"]
outputs = model.generate(inputs, max_new_tokens=5)
print(tokenizer.decode(outputs[0]))
```

### Expected behavior

I would expect this to work, fail gracefully, or perhaps revert to a supported setting. For now, simply disabling "bnb_4bit_use_double_quant" resolves the issue with RWKV, and I've not seen it happen elsewhere.

Comments:
cc @younesbelkada as well

Hi @LuciferianInk 
Thanks for the issue, I recently made https://github.com/huggingface/transformers/pull/26134 that should fix all issues related with RWKV and 4bit, please install transformers from source `pip install -U git+https://github.com/huggingface/transformers.git` and let me know if this fixes your issue

Thanks for the update! While this build does appear to fix my earlier problem, it breaks the custom CUDA kernel that's supposed to ship with RWKV. Because this results in 10x slower computations in RWKV, I'll have to revert back to 4.32.X for now.

For reference, I am installing from pip inside of the `nvcr.io/nvidia/cuda:12.2.0-devel-ubuntu22.04` container. The kernel works great, on the previous build. Feel free to close this issue if you'll track that one elsewhere.

As of 4.34.0, this does appear to be resolved. Thanks for the update!
Unclear documentation regarding sequence length for all-MiniLM-L6-v2
Author: sanderland
Pull request: no

On [the model card for MiniLM-L6](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) it says

> By default, input text longer than 256 word pieces is truncated.

However, looking at the model and tokenizer
```model = AutoModel.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")```

It seems to have a sequence length of 512 in its embeddings, and also have this set in the tokenizer as the max length.
The huggingface tokenizer picks up on this higher max length, and does not truncate to 256, although using the SentenceTransformer package does this in its tokenization step.

I am not sure whether this is merely a documentation oversight, or a sign of some deeper problem.

Version:  transformers  4.15.0


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

There is a related discussion here - https://stackoverflow.com/questions/75901231/max-seq-length-for-transformer-sentence-bert
[docstring] Fix docstring for bert generation tokenizer
Author: przemL
Pull request: yes

# What does this PR do?

It fixes docstring for bert generation tokenizer.

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@ydshieh


Comments:
RuntimeError: Failed to import transformers.data.data_collator
Author: xoumyax
Pull request: no

### System Info

![Screenshot 2023-10-15 at 5 15 57 PM](https://github.com/huggingface/transformers/assets/47978486/5a4c0c83-f283-4713-b990-48e413a3e850)


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Create New Conda Environment (Ubuntu 20.04 LTS)
2. Install PyTorch, Transformers, Datasets, Evaluate
3. Run Script that Contains **from transformers import DataCollatorForSeq2Seq**

### Expected behavior

RuntimeError: Failed to import transformers.data.data_collator

Comments: