noamaanMulla-03/transformers-issues huggingface dataset:
Update setup.py to be compatible with pipenv
Author: georgiemathews
Pull request: yes

# What does this PR do?
This enables installing transformers from source using pipenv.
Currently installing transformers through pipenv via a git source is blocked by this issue:  https://github.com/pypa/pipenv/issues/5167#issuecomment-1349316531

Installation will fail with:
```
AttributeError: 'Subscript' object has no attribute 's'
```
## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger 

Comments:
> I'm confused. `install_requires` is already a list as seen on line 415.

Thanks for taking a look @sgugger!

It seems to be a requirementslib bug that occurs when one of the items in the list of dependencies is declared with a string interpolation..

I'm not sure why using the list constructor fixes the bug. This is the behavior when installing transformers through pipenv without the bugfix:

```
$ pipenv install git+https://github.com/huggingface/transformers#egg=transformers
Installing git+https://github.com/huggingface/transformers@v4.30.0#egg=transformers...
Resolving
git+https://github.com/huggingface/transformers@v4.30.0#egg=transformers...
✘ Locking Failed!
Traceback (most recent call last):
  ...
  File "/home/gmathews/.local/lib/python3.8/site-packages/pipenv/vendor/requirementslib/models/setup_info.py", line 659, in _find_install_requires
    return [el.s for el in variable.elts]
  File "/home/gmathews/.local/lib/python3.8/site-packages/pipenv/vendor/requirementslib/models/setup_info.py", line 659, in <listcomp>
    return [el.s for el in variable.elts]
AttributeError: 'Subscript' object has no attribute 's'
```

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_24789). All of your documentation changes will be reflected on that endpoint.
Run hub tests
Author: sgugger
Pull request: yes

# What does this PR do?

It looks like the hub tests are currently not running :grimacing: 
This is because the env variable for staging test is not set to `True`, due to me when we moved the circle CI to a dynamic config. Hopefully nothing is broken...

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Fixed the tests that did not work, so can merge :-)
Revert "Unpin protobuf in docker file (for daily CI)"
Author: ydshieh
Pull request: yes

Reverts huggingface/transformers#24761

ONNX unfortunately doesn't support using protobuf v4, and our daily CI have many ONNX tests broken after #24761.

See [failing jobs](https://github.com/huggingface/transformers/actions/runs/5526985620)



Comments:
Merge directly so we can have a better CI report in the next run.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_24800). All of your documentation changes will be reflected on that endpoint.

@sgugger Just to let you know I revert a merged PR #24761: There is no super easy way to move (all) the ONNX-related tests to another job. We get stuck with protobuf v3 on daily CI.

Shouldn't all the ONNX tests be done on the optimum side now?

Yes. I know you once mentioned to me that we will ignore failing ONNX tests (`tests/onnx`) - but I don't know if you are OK for us to completely remove it (as all of them are failing with protobuf v4).

But the story is longer: we have tests  like `TFGPT2ModelTest::test_onnx_runtime_optimize` that are defined in individual model test file, and not in `tests/onnx`. Of course, I am happy if the Optimum can take care of this on their side.

cc @michaelbenayoun Can you confirm it's okay for us to remove all ONNX tests? They all test the deprecated way of using ONNX as far as I know.

The issue here is that these tests rely on a pretty old release (v1.12.0) of `onnx`: https://github.com/huggingface/transformers/actions/runs/5526985620/jobs/10082374086#step:8:159

The last release should be compatible with `protobuf` v4.

Other than this, it sounds good to me to remove all ONNX tests in Transformers :+1: 
@michaelbenayoun and @fxmarty will know better though.

@regisss Thank you for the heads up ❤️  - but yes it would be great if we can delegate the ONNX testing to Optimum CI.

@ydshieh @sgugger Yes I believe it is fine to remove the ONNX tests from transformers, as the export in Optimum is now mature, extended and well tested!

Thanks!
[DOC] Clarify relationshi load_best_model_at_end and save_total_limit
Author: BramVanroy
Pull request: yes

Clarify the relationship between `load_best_model_at_end` and `save_total_limit`. Hope this is clear.

As discussed on Slack @sgugger 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Can you just do a quick rebase/merge on the main branch? I think the issue in the tests (due to a release of Pillow apparently) is fixed on main.
Copy code when using local trust remote code
Author: sgugger
Pull request: yes

# What does this PR do?

When using the `trust_remote_code=True` feature with local models (for instance using a clone of a repo with custom code) the custom code files are not copied over if the user does `save_pretrained` (as a result of #22814)  but they should in this specific case. This PR fixes that.

Fixes #24737 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[`bnb`] Fix bnb config json serialization
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes #24131 
Fixes https://github.com/huggingface/peft/issues/558

Replaces the PR: https://github.com/huggingface/transformers/pull/24094

To reproduce:

```python
import torch
from transformers import BitsAndBytesConfig, AutoModelForVision2Seq

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)
model = AutoModelForVision2Seq.from_pretrained("Salesforce/blip2-opt-2.7b", quantization_config=bnb_config, device_map='auto')
print(model.config.to_json_string())
```
(or use any causal LM/ text models)

Adds also a nice test

cc @amyeroberts 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Is there a way to use Blip2Model for Zero-Shot Classification?
Author: danielamassiceti
Pull request: no

Hi there

I am attempting to adapt the Blip2Model for a zero-shot classification task as follows:
- N text sentences/classes --> x = N text embeddings
- 1 test image -> y = 1 image embedding
- soft-max(dot-product(x, y)) to get the probabilities over classes

This is my solution so far:
```
    def get_img_embedding(images]):
        """
        Turn a list of image inputs into tensor of embedding vectors
        images should be of shape (batch_size, channels, height, width)
        """
        image_tensors = blip2model.preproc([
            Image.open(i.path) # type: ignore
        for i in images], return_tensors='pt') # Dict with 'pixel_values' entry of size batch_size, C, H, W

        image_tensors = image_tensors.to(self.device, torch.float16) # type: ignore

        # pass images through the vision model and then the qformer to get query-conditional image features
        query_outputs = blip2model.get_qformer_features(**image_tensors) # tuple (last_hidden_state, pooler_output)
        query_output = query_outputs['pooler_output'] # (batch_size, hidden_size)
        # project query-conditional image features into language space
        image_features = blip2model.language_projection(query_output) # shape (batch_size, hidden_size)
        image_features /= image_features.norm(dim=-1, keepdim=True)

        return image_features

    def get_text_embedding(texts):
        """
        Turn a list of text inputs into tensor of embedding vectors.
        texts is a list of strings to embed.
        """

        text_tokens = blip2model.text_tokenizer(texts, padding=True, return_tensors='pt')
        text_tokens = text_tokens.to(self.device) 

        text_outputs = blip2model.get_text_features(**text_tokens, output_hidden_states=True) # type: ignore
        text_features = text_outputs['hidden_states'][-1][:, 0, :] # extract [CLS] embedding from last hidden state, shape (batch_size, hidden_size)
        text_features /= text_features.norm(dim=-1, keepdim=True)
        return text_features
```
Then I would take the dot product between the two. Am I on the right track?

Thanks


Comments:
Hi @danielamassiceti, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

Thanks, will repost it there!
Adding support for scaling rotary position embeddings
Author: kaiokendev
Pull request: no

### Feature request

Hello,

I would like if possible for Rotary Position Embedding scaling factors to be usable in the library. Currently this can only be done by monkey-patching the library.

Namely, it requires modifying the:
- `max_position_embeddings`: This can already be done via the model's config class or `config.json`
- `position_scale`: This variable doesn't exist currently, and there is no way to incorporate this effect at the moment without monkey-patching the existing `LlamaRotaryEmbeddings` class. (I'd also like to not step over toes of a possible future XPos implementation which also uses it's own scale for different purposes)

### Motivation

Recently I demonstrated it is possible to drastically reduce training compute when fine-tuning pre-trained RoPE models with an adjusted scaling factor for the purpose of extending the context length of the model. This has the effect of interpolating the position embeddings making it easier to fine-tune the model using in-distribution positions as opposed to out-of-distribution positions typically used via pure extrapolation. There is an extended write-up with motivations here https://kaiokendev.github.io/context as well as the code I used (for the 8K example) can be found here https://huggingface.co/kaiokendev/superhot-13b-8k-no-rlhf-test/blob/main/llama_rope_scaled_monkey_patch.py

Some existing discussions and benchmarks can be found here: https://github.com/ggerganov/llama.cpp/discussions/1965

Several models currently use this scaling feature, but they will not produce coherent output unless the scale is applied correctly during inference (scale is a hyperparameter):
- https://huggingface.co/kaiokendev/superhot-30b-8k-no-rlhf-test
- https://huggingface.co/Peeepy/Airoboros-13b-SuperHOT-8k
- https://huggingface.co/emozilla/open_llama_7b-scaled

EDIT: Meta has recently written a paper about it: https://arxiv.org/abs/2306.15595

### Your contribution

I would love to help in any way possible. While the basic implementation would be easy, I'm not sure what the best way could be for adding this modification (such as if users want to used a fixed scale versus having it dynamically applied based on the input sequence length)

Comments:
any updates on this?

@lucasjinreal Yes, it is now an official part of the library
https://github.com/huggingface/transformers/pull/24653#issuecomment-1635324005

So I will close this issue

Here are the docs btw
https://huggingface.co/docs/transformers/main/en/model_doc/llama#transformers.LlamaConfig.rope_scaling

@kaiokendev So it actually support just same thing like longchat?
BTW, how to adopt it to Baichuan model properly?

Yes, for LongChat specifically, you would use "linear" method with factor of 8.

For Baichuan model you would not use this, as Baichuan uses ALiBi, not RoPE

@kaiokendev I was not aware of this issue, my bad 🙈 

Suggestion: tag someone when opening an issue; sometimes things fly under our radar
Trainer.hyperparameter_search() should give the option to reload the model from the best run before completing
Author: fantauzzi
Pull request: no

### Feature request

After calling `Trainer.hyperparameter_search()`, the instance of `Trainer` contains the last trained model, among the multiple models trained for optimization of the hyperparameters. There should be an option, perhaps among those of `TrainingArguments`, to have the instance of `Trainer` reload the model from the best run (the best model) before `Trainer.hyperparameter_search()` returns.

### Motivation

After optimizing hyperparameters we are typically interested in the trained model that optimizes them, not the model that, accidentally, was trained in the last run of optimization. We are interested in the *best* model, not the *last* model. 

Currently, accessing the best model is tricky, as one has to reload it from checkpoints on disk, after figuring out what the path to the wanted checkpoint is. The latter is further complicated by the fact that there may be multiple checkpoints on disk for the run that produced the best model, and we specifically want the checkpoint at the end of the epoch where the objective function was maximized (or minimized), which is not necessarily the last epoch of the run.

### Your contribution

N/A

Comments:
I have found the path to the saved checkpoint with the best model, after hyperparameter optimization has completed, is in `Trainer.state.best_model_checkpoint`, the model can then be easily loaded from that checkpoint.
I leave this here in case it may help someone else find where that information is tucked away.

Hi, I have found that the proposed solution does not work. Seems like `Trainer.state.best_model_checkpoint` will always have the best checkpoint for the latest Trial run.

I went around this by creating my own `TrainerCallback` that tracks best and last checkpoints for each all trials in a sweep. In the end I just need to get it from it.
How to use LogitsWarper within .generate()?
Author: JellePiepenbrock
Pull request: no

### System Info

- `transformers` version: 4.16.2
- Platform: Linux-5.4.0-100-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1+cu101 (True)

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Using a GPT2 Model, I want to affect the logits, as they are used in the generate function. 

To do this, I created a LogitsWarper, which is the only member of the LogitsProcessorList:
```

logits_processor_list = LogitsProcessorList([
            TemperatureLogitsWarper(10.0)
                ])
```
This is then given as an argument to the generate() function:
```

beam_output = model.generate(
            sample,
            ...
            logits_processor_list=logits_processor_list,
            early_stopping=True,
            num_return_sequences=k,
            ...           
            do_sample=False,
            output_scores=True,
            return_dict_in_generate=True,
            length_penalty=0,

        )

```

### Expected behavior

When changing the temperature, I expect the output sequence probabilities to be different, but they do not differ between a temperature of 1.0 and 10.0.

My understanding is that logits_processor_list will be propagated to the specific search function that will be called (beam search, in this case). Should I do this differently, or is there an easier way to affect the temperature for all the search procedures? I know that .generate has a _temperature_ parameter, but this seems to only be used automatically when do_sample=True (https://github.com/huggingface/transformers/issues/22405) . How can I change the temperature when that is not the case?

Comments:
Hi,

See this for a great blog post on that: https://towardsdatascience.com/the-power-of-constrained-language-models-cf63b65a035d.

The blog post includes a Colab notebook that showcases creating a custom `LogitsProcessor`

cc @gante 

@JellePiepenbrock 👋 

Temperature only has an effect on sample-based text generation method. When no sampling is done (`do_sample=False`), the most likely token(s) will be selected at each token selection step, so temperature scaling doesn't change the output at all.

Have a look at this blog post: https://huggingface.co/blog/how-to-generate

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Parameter: encoder_no_repeat_ngram_size or something that makes model not repeat input tokens in the output.
Author: Oxi84
Pull request: no

### Feature request

Could you please add more explanations in the docs about  what encoder_no_repeat_ngram_size practically does.

It says in the docs that it makes sure that ngram sequence of specified length that is in the encoder input ids does not repeat in the decoder input ids, but I have no idea how this parameter changes decoder outputs ids.

I use it with t5


  

### Motivation

 When i set encoder_no_repeat_ngram_size=4 it does not repeat even ngram 2-3 in the output mostly.

### Your contribution

        with torch.no_grad():
                beam_outputs = model1a.generate(
                    input_ids=input_ids, attention_mask=attention_masks,                
                    encoder_no_repeat_ngram_size = 4,
                    do_sample = False,
                    num_return_sequences=4,
                    num_beams=4,
                    max_length=128
                    )  

Comments:
cc @gante 

Hey @Oxi84 👋 

Without a stand-alone short script to reproduce the issue (as well as the desired output), it is hard for me to help :)

Nevertheless, I suspect the keyword argument you want to use is `no_repeat_ngram_size`, and not `encoder_no_repeat_ngram_size`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
set correct model input names for gptsw3tokenizer
Author: DarioSucic
Pull request: yes

# What does this PR do?

Makes it so the tokenizer doesn't output `token_type_ids`, as these break generation.

Seems like a harmless change, but I'm not sure what these are used for so let me know if this is the wrong approach!

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @amyeroberts @ekgren @Apsod


Comments:
_The documentation is not available anymore as the PR was closed or merged._

LGTM
We're not using token type ids during training, so there is no reason for the tokenizer to output them, and doing so just leads to unintended behaviour.
Check models used for common tests are small
Author: sgugger
Pull request: yes

# What does this PR do?

This is the first PR in a series that will aim at reducing the time spent on the tests (and the corresponding cost 😅 ). A first analysis of the slowest tests show that the common tests sometimes use real-life models instead of tiny ones. This PR adds a check that the size of the model for common tests is not bigger than 1M parameters (which is a wide bar, BERT has a version with 55k parameters for its common tests for instance).

To avoid making the PR too long, the new test is skipped in most of the failures, only the DETR variants, BridgeTower, Canine and CLAP are treated in this PR. For the DETR variants, a full pretrained ResNet-50 was used as the backbone which is replaced by a tiny random ResNet.

The following models will be treated in follow-up PRs:
* CTRL
* CVT
* DETA
* DPT
* DPT Hybrid
* EfficientNet
* Encodec
* ESM
* Flava
* Git
* GPTSan-Japanese
* Graphormer
* LayoutLM
* LayoutLMv2
* LeViT
* Mask2Former
* Maskformer
* MobileViT
* MobileVit2
* OneFormer
* Perceiver
* SegFormer
* SpeechT5
* SwiftFormer
* TableTransformer
* TimmBackbone
* TVLT
* UperNet
* VideoMAE
* ViT-MAE
* ViViT

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[🔗 Docs] Fixed Incorrect Migration Link
Author: kadirnar
Pull request: yes

I couldn't find it in transformers files. Can you check? Is it true?

Comments:
Hi @amyeroberts , I am new to transformes library. I wanted to fix this error that I saw in the documentation. Because the link is broken.


The documentation page MIGRATION doesn't exist in v4.30.0, but exists on the main version. Click here to redirect to the main version of the documentation.

I looked in transformers documentation(https://huggingface.co/docs) and couldn't find it. Can you help?

Thank you for the help💖