noamaanMulla-03/transformers-issues huggingface dataset:
[FuturWarning] Add futur warning for LEDForSequenceClassification
Author: ArthurZucker
Pull request: yes

# What does this PR do?
fixes #19019 by replacing the construction of the `eos_mask` in the `SequenceClassification`. 
Also adds a test to make sure that long sequence are properly processed



Comments:
_The documentation is not available anymore as the PR was closed or merged._

I think deleting the class would be less confusing, we discussed this with Patrick, the original paper does not use the encoder decoder model for sequence classification. WDYT @LysandreJik 

Let's please do a deprecation cycle :pray: 

Would love to learn about, what can I do? 

I think there is still work to do here?

@ArthurZucker, instead of deleting the class, you would first start by adding a `FutureWarning` when it is instantiated or called, mentioning that it is deprecated and what is recommended instead. You would mention that the class will be deleted in version 5, and that such a code will error out then.

Oh it ! Thanks for the pointers
Can't load too big text file for dataset (RAM exhausts), HELP PLEASE !!!!!
Author: mv96
Pull request: no
Labels: name: bug

### System Info

MacBook Pro (13-inch, M1, 2020)
Chip Apple M1
Memory 16 GB

(Also tried with Google Colab)


### Who can help?

copy of #19161

Hello All,

A bit new to HuggingFace environment.

I am trying to load a big text file for pertaining a BERT model from scratch the size of the txt file is about 11GB and trying to load the model for pertaining is exhausting all the RAM on the system.

Is it possible to load all the data in batches and then perform training.

I am a bit new to hugging face ecosystem so I would request you all to help me around if you have any clue about this.
I am using Google Colab for the purpose.

Please share code snippets, If possible.

Cheers !

`#construct dataset
from transformers import LineByLineTextDataset

file_path="/content/drive/MyDrive/full_text_data.txt"
dataset = LineByLineTextDataset(
tokenizer=tokenizer,
file_path=file_path,
block_size=32)`

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

try to load the data with a big text file 

### Expected behavior

The data should be read in the batches to combat OOM.

Comments:
try using iterable dataset and decrease the batch_size

@Kunlun-Zhu 

I am a bit new to HF environment, can you please provide an example code snippet to do so ?

Thanks !!

> @Kunlun-Zhu
> 
> I am a bit new to HF environment, can you please provide an example code snippet to do so ?
> 
> Thanks !!

https://pytorch.org/docs/stable/data.html
more examples could be found on stackoverflow and many blogs

Hi @mv96 👋 

As per our [issues guidelines](https://github.com/huggingface/transformers/blob/main/ISSUES.md), we reserve GitHub issues for bugs in the repository and/or feature requests. For any other matters, we'd like to invite you to use our [forum](https://discuss.huggingface.co/) 🤗

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Trainer().train() just idles, neccesitates kernel restart
Author: Liudvikas-vinted
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.22.1
- Platform: Linux-4.19.0-21-cloud-amd64-x86_64-with-debian-10.13
- Python version: 3.7.12
- Huggingface_hub version: 0.9.1
- PyTorch version (GPU?): 1.11.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@sgugger because I think you're the original author of the example

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Run https://github.com/huggingface/notebooks/blob/main/examples/language_modeling.ipynb, Section *Fine-tuning a language model*. (change `push_to_hub=False).
2. trainer.train() prints epoch and step counts, then doesn't seem to do anything useful (python process cpu usage stays at +-0, so does GPU utilization), and also process interrupt doesn't work


### Expected behavior

Either training starts or some kind of error is shown

Comments:
How are you running the notebook? It works fine on my side.

I'm unable to upload it here, but it's literally the original example, sans the `notebook_login` and `push_to_hub=False`. 
And it *works* on my local machine (installed the same libraries as listed above, except my kernel is `5.19.8-200.fc36.x86_64`)


As reported, it fails when I'm running on a Vertex AI instance. That's what the instance says about itself: 

```
Environment
    PyTorch 1.11 (with Intel® MKL-DNN/MKL)
Environment version
    M94
Machine type
    n1-standard-8 (8 vCPUs, 30 GB RAM)
GPU
    NVIDIA Tesla V100 x 1
```
I also tried running it (on the instance) with `os.environ['CUDA_VISIBLE_DEVICES'] = '-1'`, but that didn't help.


This Vertex AI instance type *works*: 
```
Environment
    Python 3 (with Intel® MKL and CUDA 11.0)
Environment version
    M97
Machine type
    n1-standard-4 (4 vCPUs, 15 GB RAM)
GPU
    NVIDIA Tesla T4 x 1
```
So I can now get my work done, but I think the reported issue is still valid

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Only resize embeddings when necessary
Author: sgugger
Pull request: yes

# What does this PR do?

As seen in #19959, using our examples with models where the embedding size is larger than the tokenizer for padding reasons (to get embedding dim a multiple of a give number like 8 or 128), the fine-tuned models become incompatible with the original model.

This has confused a lot of users, just to support a full pretraining example. This is why this PR proposes to only resize the embeddings when the tokenizer has more tokens. This might in turn confuse users that do a full pretraining on a small vocab and expect the model to have a smaller embedding size, which is why a comment is added.

Fixes #19959

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Speed up TF token classification postprocessing by converting complete tensors to numpy
Author: deutschmn
Pull request: yes

# What does this PR do?

The postprocessing of the token classification when using TensorFlow is not as fast as it could be. We discovered that with some experiments and profiling that showed that, in some settings, most time of the pipeline is spent in `gather_pre_entities`:

Before:
<img width="500" alt="before" src="https://user-images.githubusercontent.com/37573274/198983453-8a34e7a8-6f67-4010-9509-359b4fb9bdf7.png">

After:
<img width="500" alt="after" src="https://user-images.githubusercontent.com/37573274/198983760-1f27b4b3-6d55-4137-a22c-707992a11637.png">

This PR speeds it up by converting `input_ids` and `offset_mapping` to numpy before passing them to `gather_pre_entities`. Thereby, the tensor is moved to the appropriate device only once. Besides, it's also in line with the type annotation of `gather_pre_entities`.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case: **n/a**
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation): **n/a**
- [ ] Did you write any new necessary tests? **n/a**


## Who can review?

Could you please review, @LysandreJik 😊


Comments:
_The documentation is not available anymore as the PR was closed or merged._

cc @Rocketknight1 

This looks like a great improvement, thank you! I didn't realize how inefficient the postprocessing was there.

The PR is failing style checks, but I can fix that here, and will merge once that's done. Thank you!

Update: I believe the failing checks are caused by issues unrelated to this PR - you just happened to fork at a bad time. I'll merge and watch tests to make sure nothing goes too terribly wrong. Thanks again!

Great! Thanks for the quick review and merge 😊
Type annotation for `pipeline()`s
Author: davidgilbertson
Pull request: no

### Feature request

Currently, the return type of many functions are not defined.
```py
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english")
tokenizer( # no type checking/help here
```

It would be nice if the return type was set, the equivalent of me doing it manually with:
```py
tokenizer: DistilBertTokenizerFast = AutoTokenizer.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english")
tokenizer( # now I get checking
```

### Motivation

The main benefits are probably pretty well understood...
* preventing typos in inputs that take strings (tasks, checkpoints, etc)
* autocomplete for methods/parameters in IDEs
* documentation tooltips in IDEs that support them.

### Your contribution

As you may already be aware, the pattern for enabling this is to use `Literal` and `@overload` from the `typing` module, to return a particular class based on the value of a string passed in.

A quick mock up:
```py
from typing import overload, Literal

class Model1:
    def model_1_thing(self):
        pass


class Model2:
    def model_2_thing(self):
        pass


@overload
def get_pipe(model: Literal["model_1"]) -> Model1:
    pass


@overload
def get_pipe(model: Literal["model_2"]) -> Model2:
    pass


def get_pipe(model):
    if model == "model_1":
        return Model1()
    return Model2()


mod = get_pipe("model_1")
# `mod` is correctly identified as an instance of `Model1`
```

Now I not only get auto-complete when typing in the string:
![image](https://user-images.githubusercontent.com/4443482/199659651-f3203eae-4034-48f3-bb15-6af44f833116.png)

But of course get the usual goodies when accessing attributes of the returned object:
![image](https://user-images.githubusercontent.com/4443482/199659741-da90c4f0-4459-411e-872d-93581c64d32d.png)

To support older Python, I think [typing-extensions](https://pypi.org/project/typing-extensions/) would work, although I'm not certain.

The big question of course is whether the "developer experience" gains are worth the effort/complexity this would add to the codebase.

Apologies if this has already been discussed and decided on, I couldn't see an existing issue, but did sense a willingness to get types right from other issues.

Comments:
Thanks for opening the issue. As of now, we have decided not to add any type annotations that render the code less readable. So whenever they can be added with no cost in readability, we welcome PRs, but for something more complex like you are suggesting, we would probably be less interested.

That seems reasonable :)

For completeness, I'll note that the `@overload`s can go in a separate `.pyi` file, leaving the main application logic clean. But the complexity would still be there, and you'd have `.pyi` files everywhere, so this is not a great solution.
Request for Examples on Correct Use the CLIPText Model (transformers.CLIPTextModel)
Author: mbdzi
Pull request: no

From the documentation on the [Hugging Face Hub](https://huggingface.co/docs/transformers/model_doc/clip#transformers.CLIPTextModel), it is not clear:

1. How the CLIPTextModel can be used.
2. If it can be used to conduct Zero Shot Classification of textual inputs using a predefined list of tokens as classes.  To be precise, let us say I have a list dataset called x with two strings such that x = ["father holding a baby", "man at war"], and; a list of classes or image generation tokens; such that y = ["family time", "next generation weapons"]: Can I use the CLIPTextModel to class x using y?

In short, I am asking for an example on:

1. The correct usage of the CLIPTextModel.
2. The use-cases for the model.



Comments:
I'm transferring this to transformers repo as this is more related to the docs there

cc @stevhliu 

Hi,

The docs is actually correct, CLIPTextModel can be used to encode text into a vector representation (embedding).

```
from transformers import CLIPTokenizer, CLIPTextModel

model = CLIPTextModel.from_pretrained("openai/clip-vit-base-patch32")
tokenizer = CLIPTokenizer.from_pretrained("openai/clip-vit-base-patch32")

inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")

outputs = model(**inputs)
last_hidden_state = outputs.last_hidden_state
pooled_output = outputs.pooler_output  # pooled (EOS token) states
````

So you have a set of texts (here "a photo of a cat" and "a photo of a dog") which you first prepare for the model using the tokenizer. Next, you forward them through CLIP's text encoder to get an embedding (here called "pooled output") out, which is of shape (batch_size, hidden_size), which in this case will be (2, 512).

That's all the CLIP text encoder does! Turn text into embedding vectors.

So no you can't use only the CLIP text encoder to perform zero-shot classification of images, for that you need both the image and text encoders (which is what `CLIPModel` is; it consists of both `CLIPTextModel` and `CLIPVisionModel`).
How to annotate these type of data for custom OCR training
Author: mohit-217
Pull request: no

@NielsRogge @gante can you please explain how to annotate the below files for custom handwritten mathematical equation training. More importantly s^2 
![FCA(1)-1-0tensor(0 89785)720](https://user-images.githubusercontent.com/51528367/198884771-5b7bf056-ae81-43f7-8ce9-a187cf371602.png)
![FCA(1)-4-3tensor(0 89384)888](https://user-images.githubusercontent.com/51528367/198884777-95d35008-c665-460d-b996-d0835e7f5a0e.png)
![FCA6-1-0tensor(0 83108)1827](https://user-images.githubusercontent.com/51528367/198884814-8cbfb1e8-66e0-4956-bb44-f9c912bc35f3.png)
![FCA6-3-1tensor(0 88415)2013](https://user-images.githubusercontent.com/51528367/198884835-19619bf3-2a8c-4113-b8fb-e28614cd44d2.png)

![FCA6-4-1tensor(0 82750)1928](https://user-images.githubusercontent.com/51528367/198884849-267d3228-2253-4eb1-a005-23626760c6a5.png)

_Originally posted by @mohit-217 in https://github.com/huggingface/transformers/issues/16007#issuecomment-1296276393_
      

Comments:
@NielsRogge @gante Please review
How I need to annotate for these type of data.
1.
- 3 ( 7s+ 8 )
- 5 (6s + 7)
- s^2
- s^2 + 3s + 1

2.
- 3 ( 7 s + 8 )
- 5 ( 6 s + 7 )
- s 2
- s 2 + 3 s + 1

which one is correct ?

Hi,

Could you please ask this question on our [forum](https://discuss.huggingface.co/), rather than here?

Github issues are meant for bugs or feature requests.

Thanks!
Fix ESM LM head test
Author: Rocketknight1
Pull request: yes

The original ESM-2 checkpoints had a bug that meant the LM head bias was not saved correctly. Now this has been fixed, we need to update our LM test as well.

cc @ydshieh

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@ydshieh Unfortunately I deleted and remade the repos, so there's no commit to point at! I'll try to do that in future though.
[Swin] Add Swin SimMIM checkpoints
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds 2 checkpoints for Swin Transformer pre-trained using the SimMIM objective (taken from [here](https://github.com/microsoft/Swin-Transformer/blob/main/MODELHUB.md#simmim-pretrained-swin-v1-models)).

They are on the hub: https://huggingface.co/models?other=simmim

It also fixes an important bug in `modeling_swin.py` regarding the window size not being set properly.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Update READMEs for ESMFold and add notebooks
Author: Rocketknight1
Pull request: yes

This PR adds ESMFold to the main README and adds links to the protein LM and protein folding notebooks.

Comments:
Woah, PyCharm murdered the formatting on that table. One sec!

Formatting fixed now, sorry about that!

_The documentation is not available anymore as the PR was closed or merged._
Update documentation on absolute position embed seq2seq models
Author: jordiclive
Pull request: yes

Update documentation on seq2seq models with absolute positional embeddings to be in line with BERT and GPT2. 
Issue #19581. For models with absolute positional embeddings, it is not usually a good idea to left-pad as if positional embeddings are not shifted the right amount for each element in the batch the results will not be correct.

Further work may be required to incorporate `position_ids` kwargs (but possibly for both encoder/decoder) for these models similar to BERT and GPT2. However, at the very least the documentation should be updated to be consistent with BERT/GPT2 to provide a warning.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
PoolformerImageProcessor defaults to match previous FE
Author: amyeroberts
Pull request: yes

# What does this PR do?

Output of PoolformerImageProcessor didn't exactly match previous feature extractor. Updates defaults and size calculation logic to match outputs. 

Running:
```
import torch
from transformers import PoolFormerFeatureExtractor, PoolFormerModel
from datasets import load_dataset
dataset = load_dataset("huggingface/cats-image")
image = dataset["test"]["image"][0]

feature_extractor = PoolFormerFeatureExtractor.from_pretrained("sail/poolformer_s12")
model = PoolFormerModel.from_pretrained("sail/poolformer_s12")

inputs = feature_extractor(image, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)

last_hidden_states = outputs.last_hidden_state
r = list(last_hidden_states.shape)
print(r)
```

Now outputs images of size `[1, 512, 7, 7]` - matching the output of the images before #19796 was merged in. 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Allow passing arguments to model testers for CLIP-like models
Author: ydshieh
Pull request: yes

# What does this PR do?

This is a continuation of PR #19954, but for model likes `CLIP`. Currently for such models, we have 
```python
class CLIPModelTester:
    def __init__(self, parent, is_training=True):
        self.parent = parent
        self.text_model_tester = CLIPTextModelTester(parent)
        self.vision_model_tester = CLIPVisionModelTester(parent)
```
and there is no way to pass any argument to the 2 component testers.

If this POC is approved, I will work on other models like `GroupViT`, `OwlViT`, `XCLIP` etc.



Comments:
_The documentation is not available anymore as the PR was closed or merged._

`GroupViT`, `OwlViT`, `XCLIP`.

`Flava` too, but with name `image_model_tester` instead of `vision_model_tester`.
Update defaults and logic to match old FE
Author: amyeroberts
Pull request: yes

# What does this PR do?

Updates defaults and logic in image processors to match the previous feature extractors. Fixes some broken inference tests. 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Unable to load CodeGenTokenizer
Author: PtrMan
Pull request: no

### System Info

I did `pip install git+https://github.com/huggingface/transformers.git`

try to load the class with `from transformers import CodeGenTokenizer`

results in

```
ImportError: cannot import name 'CodeGenTokenizer' from 'transformers' (/usr/local/lib/python3.9/dist-packages/transformers/__init__.py)
```

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

from transformers import CodeGenTokenizer

### Expected behavior

load class

Comments:
You should double-check that the version of Transformers seen when executing your code is indeed a source install (you can print `transformers.__version__` after importing transformers), as it looks like an issue within your env.

Thx!