noamaanMulla-03/transformers-issues huggingface dataset:
Image Classification Pipeline returns score= 1.0
Author: guillaumeguy
Pull request: no

### System Info

Vision Transformers Documentation mentions that they do support regression when num_labels == 1. However, it seems incompatible with Pipeline.

In this code, the logits are normalized into scores. However, when num_labels = 1, it effectively turns the score to `1`. 

https://github.com/huggingface/transformers/blob/63b204eadd9829985ba13e7e4d51f905adfc2d5e/src/transformers/pipelines/image_classification.py#L116

### Who can help?

@amyeroberts @nielsr

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Train a ViT on a regression  (num_labels = 1)
2. Use pipeline

![image](https://user-images.githubusercontent.com/5290678/214615264-13b5e355-a30b-4f6a-9360-41341054229a.png)


### Expected behavior

The model should returns predictions

Comments:
There is no pipeline available for regression tasks, you need to use the model directly and takes its outputs.

Thanks @sgugger! Super fast answer! 

As I found the pipelines to be very helpful, I'm sharing my solution below for folks that want to still use them. 

One can just rewrite the function in the `postprocess` [function](https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/image_classification.py#L116):

```
def postprocess(self, model_outputs, top_k=5):
        if top_k > self.model.config.num_labels:
            top_k = self.model.config.num_labels

        if self.framework == "pt":
            pred = model_outputs.logits
        else:
            raise ValueError(f"Unsupported framework: {self.framework}")

        scores = pred.tolist()
        return scores
```

You can then instantiate the pipeline of the overwritten class: 
`pipe = ImageClassificationPipeline(model=model,feature_extractor=extractor,device='cuda:0')`

And run your inference:
```
def data():
    for path in paths:
        yield PILImage.open(path)


from tqdm import tqdm
scores = []
for out in tqdm(pipe(data())):
    scores.append(out)
```

Yes that's why the pipeline is called classification, rather than regression. We would need an `ImageRegressionPipeline` for this use case ;)

Closing this issue as it seems resolved.
Template for framework-agnostic tests
Author: gante
Pull request: yes

# What does this PR do?

There are a few cross-framework pain points I often encounter: 
1. Ensuring the interface of `.generate()` or models stay consistent across frameworks;
2. Ensuring that TF doesn't get neglected and satisfies 1., when contributors add features/fixes on the PT side;
3. Blocking cases where the interface is the same, but there are numerical differences.

After a brief chat with @sgugger, I thought inheritable framework-agnostic tests could be nice to help with these problems. This week I'll have to write a few TF `.generate()` tests that already exist on the PT side, so this could be a great chance to kill 2 birds with one stone. 

However, I'd like to get the pattern right, hence this small PR -- replaces a pair of framework-specific tests with a framework-agnostic test. `Flax` is intentionally left out, as it is missing many testable features and is not being maintained, but it can easily be added in the future.

Let me know what you think of it!

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> I might be missing something, but not clear to me how/where this is tested here.

@amyeroberts I see, this example does not test numerical differences :) 

However, consider the tests [here](https://github.com/huggingface/transformers/blob/main/tests/generation/test_logits_process.py), which also exist for TF. They are primarily numerical tests, they test the resulting vector against a constant. It is not uncommon for a user to find an edge case on these processors, fix the PT side of it, and add a few more numerical checks. These new numerical checks would fail in TF, but I often can't convince the users to make the change on the TF side (and then fail to follow up myself after it is merged). Result: a bug that is trivial to fix in that moment gets lost 🙈 This unified framework would prevent it -- the fix would need to include the corresponding TF change to pass the tests (either fixed by the user or by one of us). PT == expected values == TF
Cleanup the usage of `layer_norm_eps` in some models
Author: ydshieh
Pull request: yes

# What does this PR do?

**(No breaking change in this PR)**
**(So far I only change `OneFormerConfig`, but I will update other config classes whose default `layer_norm_eps = 1e-5`)**

Fix the missing usage of `config.layer_norm_eps` in some pytorch models: **if the default value in the config class is `1e-05`** (i.e. the same as the default value in `nn.LayerNorm`. In this case, we don't break the current behavior, and **make the (WIP) test dealing with much fewer edge cases** (which is always a bit of burden).

**In #20699, my claim regarding the breaking change was a bit misleading**: only those config classes having default `layer_norm_eps` not being `1e-5` (for example, `LxmertConfig`) will have breaking change - which I don't do anything in this PR.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger @NielsRogge  Just want to hear from you to make sure we are good for this change 🙏 before I continue.

I can definitely just add these config classes to a list of edge cases in my (WIP) tests. But I feel it's better to clean them up, so future models won't copy/paste the same code, and we accumulate more and more edge cases to skip in the tests.
[Docs] Minor fixes
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR:
- moves TAPAS and LayoutLM to the "multimodal" section rather than "text", as these models leverage more modalities than just text (TAPAS leverages row and column information, LayoutLM leverages 2D coordinates).
- adds a figure for DETA and fixes the one of UPerNet

Comments:
_The documentation is not available anymore as the PR was closed or merged._
eval OOM when loading a pretrained model with output_hidden_states set to True for BertForSequenceClassification
Author: yunjiangster
Pull request: no

### System Info

- `transformers` version: 4.23.1
- Platform: Linux-4.15.0-166-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.12
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.10.0+cu102 (True)
- Tensorflow version (GPU?): 2.7.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@sgugger

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Run the following snippet

```
model = BertForSequenceClassification.from_pretrained('snunlp/KR-BERT-char16424', output_hidden_states=True)

# Now create any training_args, tokenized_train, tokenized_valid, and compute_metrics function (such as those in the offical tutorial).

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_train,
    eval_dataset=tokenized_valid,
    compute_metrics=compute_metrics,
)
trainer.train()
```

Eval will experience CUDA OOM on a gpu with 24Gb, after about 100 examples.



### Expected behavior

I tried warmstarting a BERT classification model from a pretrained embedding model, which sets output_hidden_states to True in config.json. But eval runs into OOM issue. 


Comments:
You will need to use the [eval_accumulation_steps](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.eval_accumulation_steps) argument in your `TrainingArguments` as it's not possible to accumulate all those tensors coming from the hidden states on the GPU.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Remove more unused attributes in config classes
Author: ydshieh
Pull request: yes

# What does this PR do?

Remove more unused attributes in config classes.

There are more changes needed than I expected previously. I have to adopt the new test (now only in my branch) in a progressive way and make the changes to pass that test in the same time.

I tried to change more places in single PR to avoid too many PRs related to this topic. But there will still be a few PRs in the future 🙏 .

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Generate: fix TF XLA tests on models with `max_position_embeddings` or `max_target_positions`
Author: gante
Pull request: yes

# What does this PR do?

Extracted from #20901 -- the lines added in this PR were incorrectly removed [here](https://github.com/huggingface/transformers/commit/0f78529f982eceb79c5855d0466c287ec8a18df1), causing some XLA tests to fail.

These changes fixes 8 slow TF tests on `test_xla_generate_slow`. They were also approved in the PR linked above, which was redone as part of the discussion (and closed).

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Simplify column_names in run_clm/mlm
Author: lhoestq
Pull request: yes

Following https://github.com/huggingface/transformers/pull/21343

Just a minor change to simplify the code, and fix a small bug (`column_names` needs to be a list to be able to call `column_names[0]`)

cc @stas00 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

This is much better - thank you, @lhoestq!
UnboundLocalError: local variable 'seq_length' referenced before assignment
Author: zhuzihan728
Pull request: no

### System Info

Hi, I am using the `ConvBertForTokenClassification` model in models.convbert and encountered the bug when passing only `input_embeds` to `forward()`.
The traceback says that at line 833 in modeling_convbert.py
```
if token_type_ids is None:
    if hasattr(self.embeddings, "token_type_ids"):
        buffered_token_type_ids = self.embeddings.token_type_ids[:, :seq_length]
```
The seq_length is unassigned.

I noticed just above this piece of code in
```
elif input_ids is not None:
    input_shape = input_ids.size()
    batch_size, seq_length = input_shape
elif inputs_embeds is not None:
    input_shape = inputs_embeds.size()[:-1]
```
seq_length is not assigned if the program enters `elif inputs_embeds is not None`.

Not sure if it is the `batch_size, seq_length = input_shape` missing for `inputs_embeds` or I am not using the model correctly?

### Who can help?

text models: @ArthurZucker and @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

passing only `inputs_embeds` and `attention_mask` to `ConvBertForTokenClassification` model.

### Expected behavior

There should be no error.

Comments:
I think it might be a bug, request you to post a sample script of your usage to reproduce the error.

> I think it might be a bug, request you to post a sample script of your usage to reproduce the error.

```
import torch
from transformers import ConvBertConfig, ConvBertForTokenClassification
embeddings = torch.tensor([1])
mask = torch.tensor([1])
convbert_model_config = ConvBertConfig()
convbert_model = ConvBertForTokenClassification(convbert_model_config)
outputs = convbert_model(inputs_embeds=embeddings, attention_mask=mask)
```

Thanks for providing a reproducing script and thanks @raghavanone for jumping on this. Your fix looks  good!
Update Hebrew language code to he per IANA registry
Author: altryne
Pull request: yes

Here's my original PR into whisper that changes the same:  https://github.com/openai/whisper/pull/401

# What does this PR do?

Changes the language code for the Hebrew language from `iw` to `he` 
Per [IANA registry](https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry), `iw` was deprecated as the code for Hebrew in 1989 and the preferred code is `he`

The correct subtag: 
```
%%
Type: language
Subtag: he
Description: Hebrew
Added: 2005-10-16
Suppress-Script: Hebr
%%
``` 
And the deprecation
```
%%
Type: language
Subtag: iw
Description: Hebrew
Added: 2005-10-16
Deprecated: 1989-01-01
Preferred-Value: he
Suppress-Script: Hebr
%%
```


Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@ArthurZucker @sanchit-gandhi


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Cool! thanks for this, it would be best to also update the models online wdyt @sanchit-gandhi 

Funny bug that happens now, I cannot process hebrew. 

I use original whisper to detect language (because #21311 is not fixed yet)  and it returns "he" 

Then I use that to send to transformers whisper, and it fails as the token is not recognized (it expects iw)  🤭 

Added PRs for models (this.. was not easy 😅 ) 
Base PR - https://huggingface.co/openai/whisper-base/discussions/8
Tiny PR - https://huggingface.co/openai/whisper-tiny/discussions/6
Small PR - https://huggingface.co/openai/whisper-small/discussions/13
Medium PR - https://huggingface.co/openai/whisper-medium/discussions/8
Large PR - https://huggingface.co/openai/whisper-large/discussions/21
Large V2 PR - https://huggingface.co/openai/whisper-large-v2/discussions/18

@sgugger @ArthurZucker thanks for merging this in! 
The model PRs need to be merged in for this to work, correct? 
Otherwise there's a mismatch between this repo and loaded models in token names

Yes, they'll need to be merged.

They are all merged 😉
How to sending my request about parameters in inference API?
Author: Fangyinfff
Pull request: no
Labels: name: New model

### Model description

<img width="999" alt="image" src="https://user-images.githubusercontent.com/61076726/216067290-7969e29b-8c83-41fa-80a6-6c71bc4bac16.png">

I can't find where I can modify, and, does generating inference not include beam search?

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
Hi there, this is the Transformers repository. You can address your questions about the inference API [here](https://github.com/huggingface/api-inference-community)

Can I modify parameters of Hosted inference API?
    For example, I want the multiple output of this Summarization task rather than one:
<img width="459" alt="image" src="https://user-images.githubusercontent.com/61076726/216101165-e26760d7-2163-42b3-bb53-44136ab032da.png">

How can I achieve this? Many Thanks!

> Hi there, this is the Transformers repository. You can address your questions about the inference API [here](https://github.com/huggingface/api-inference-community)

Thanks a lot! I want to know if I can modify the parameters of **Hosted inference API** so that the demo of the webpage outputs multiple results?
Generate: decoder-only models can generate with `inputs_embeds`
Author: gante
Pull request: yes

# What does this PR do?

2-in-1 PR 🔥 

### 1 - Experimenting with input embeddings
Accepting `.generate()` calls with `inputs_embeds` on decoder-only models is a long-standing request (#6535) -- see [this recent comment ](https://github.com/huggingface/transformers/issues/6535#issuecomment-1353658984) in particular and its reacts.

It has to be added on a per-model basis, and this PR adds the necessary changes for GPT2. Other models will throw an informative exception if the user passes `inputs_embeds`, asking them to check this PR and implement the same pattern on the model they want to use it with 🤗 

Please note that it is still expected that the user passes `input_ids`, i.e.
```
outputs = model.generate(input_ids, inputs_embeds=inputs_embeds)
```
This is because decoder-only models expect the prompt to be present in the output, and this is the only way to preserve it! `input_ids` can also be omitted and, in that case, the output won't contain the prompt.

### 2 - BLIP 2 (cc @NielsRogge)

This change is a soft-requirement for BLIP 2. The alternatives to add BLIP 2 are:
1. Support `.generate()` with `inputs_embeds` on decoder-only models
2. Change OPT's reference implementation to accept a new kwarg (`query_embeds`, that gets appended to the embeddings of the `input_ids`)

Option 1, this PR, seems like a better option according to our library philosophy :)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fixed num_channels!=3 normalization training [#20630]
Author: ydshieh
Pull request: yes

# What does this PR do?

Fork from #20630 (which is to fix https://github.com/huggingface/transformers/issues/20580 and https://github.com/huggingface/transformers/issues/19913)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@NielsRogge CI is green, but I don't review (yet) the content of PR though

@NielsRogge 

I think it is already there: when I clicked `Squash and merge`, it shows at the end

```
Co-authored-by: Lay Jain <layjain@basil.csail.mit.edu>
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
```

cc @layjain 

The CI got fixed on the other PR so I merged it. Is there still a need for this one?

@sgugger OK, this morning it was not running even I pushed a commit. No more need of this PR - despite I have a question regarding the logic. We can fix it if @NielsRogge think what I pointed is indeed a bug.
Fix `test_run_seq2seq_bnb` CI
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix `test_run_seq2seq_bnb` CI. See [failed job run](https://github.com/huggingface/transformers/actions/runs/3834635537/jobs/6527258225)

It seems the expected reduced GPU memory usage only happens when running the test on multi GPU env.
I simply add `require_torch_multi_gpu` without trying to understand why it fails with single GPU env.

I can try to figure it out, but the probability that @stas00 knows the reason > 1.0, so I would like to see if he has any comment first.

Error
```
tests/extended/test_trainer_ext.py::TestTrainerExt::test_run_seq2seq_bnb
(line 256)  AssertionError: -0.0023021928988980356 not greater than 10 : should use very little peak gpu memory with BNB, compared to without itbut got gpu_peak_mem_orig=509447168 and gpu_peak_mem_bnb=510622720
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._

cc @manuelciosici

hmm, I didn't write this test and don't know why these numbers were set, so perhaps it's easier to ask the one who wrote it?

If it's not possible please let me know I will be able to study it later, as I'm off to the airport now.

but yes, such tests should do the measurements per number of gpus usually. i.e. 1 gpu - measurement 1, 2 gpus - measurement 2, etc.

the easiest fix is to ensure you run on exactly that many gpus always and then you need only one "truth" to measure against. 

(we can investigate later, not urgent)

@stas00 The first error occurred is that we expect the GPU memory usage will be larger when not using BNB (i.e. `gpu_peak_mem_orig `) than when using BNB (i.e. `gpu_peak_mem_bnb`). The test assert 10% difference. However, on our single GPU runner, 

```bash
gpu_peak_mem_orig=509447168
gpu_peak_mem_bnb=510622720
```
which is quite weird (intuitively).

- We can definitely adjust the values for different environment, but probably it's a good idea to understand what's going on here if possible.
- We left comment in the original PR page, but haven't heard from the PR author. But I cc them in a comment in this PR.

 

some quick thoughts:
- do we clear cuda cache there between the measurements? and first call `gc.collect()` (then cache clear) 
- using a larger model should make the difference (savings) more distinct

and of course the test might be failing if bnb is broken - recent update? try earlier version?

I can reproduce the failure, looking

Fixed here https://github.com/huggingface/transformers/pull/21030


Close in favor of #21030 21030
Fix TF generation (especially for `TFMarian`)
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix TF generation (especially for the `TFMarian` generation issue in #18149)

Fix #18149

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hey @ydshieh 👋 

Thank you for opening this PR, it made me realize a detail that is wrong in *both* frameworks 👀 

We know that logprobs is a negative value, and we want to maximize it in beam search (i.e. make it as close to 0 as possible). Since logprobs is always negative, and the final score is the sum of the logprobs, we can anticipate the best possible score and use it to end beam search with no drawback. Well, it turns out that the method to compute the best possible score depends on `length_penalty`, and we are not accounting for that!

- Scenario 1, length_penalty > 0.0: In this case, as the sentence grows, the denominator grows as well. This means the score can get closer to 0 (i.e. higher) as the sentence grows, and longer sentences are promoted. In this case, the best possible score can be determined from the maximum sequence length (TF implementation).
- Scenario 2, length_penalty < 0.0: In this case, as the sentence grows, the denominator gets smaller. This means the score will get farther away to 0 (i.e. lower) as the sentence grows, and shorter sentences are promoted. In this case, the best possible score can be determined from the current sequence length (PT implementation).

On top of this incomplete best score computation on both ends, your PR made me realize that the stopping condition for TF also had a problem (after factoring in the correct length penalty computation, a few tests failed).

I'm opening a PR to compare against this one with what I think is the correct solution to this bug 🐛 



Close in favor of #20901