noamaanMulla-03/transformers-issues huggingface dataset:
[Flax] Add logging steps, eval steps, and save steps for hybrid CLIP example
Author: galuhsahid
Pull request: yes

# What does this PR do?

This PR enables users to set `logging_steps`, `eval_steps`, and `save_steps` when training a model using the Hybrid CLIP example. `logging_steps` helps to keep the train_metrics small so that we can avoid fragmentation errors. `eval_steps` and `save_steps` enables users to save evaluation results and model checkpoints based on steps instead of epochs which may run for days especially when using large datasets.

Discussed in #13095

## Notes
I'd like to have input on the following:
- I've tested the script using the same dataset as the one described in the readme. The run can be found on [tensorboard.dev](https://tensorboard.dev/experiment/WH8xEX25RVavnizS4VaU8Q/#scalars). I'm not sure if I should update the tensorboard in the readme or not.
- I'm also not sure if we should save the final model once the training is done, or only save the model based on the steps only. Right now the script also saves the final model after the whole training is done.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. Yes, discussed in #13095
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@patil-suraj


Comments:
@galuhsahid could you run `make style` that will fix the failing test. Thanks :) 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

gently pinging @galuhsahid :) 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
AttributeError: type object 'EnglishDefaults' has no attribute 'create_tokenizer'
Author: Tortoise17
Pull request: no

I am using OpenAI gpt2. and I am facing this error. 
```

File "/home/anaconda/envs/nlp/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 1871, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/anaconda/envs/nlp/lib/python3.7/site-packages/transformers/models/openai/tokenization_openai.py", line 107, in __init__
    self.nlp = _nlp.Defaults.create_tokenizer(_nlp)
AttributeError: type object 'EnglishDefaults' has no attribute 'create_tokenizer'
```

I have 
```
tokenizers                    0.10.3
transformers                  4.11.2
```

Can you guide me why is this.? If you can guide to fix it. 

Comments:
I think that this functionality requires spaCy v2 and you have spaCy v3 installed (or the other way around). Try installing the opposite version.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Fix SpeechEncoderDecoderModel
Author: NielsRogge
Pull request: yes

# What does this PR do?

The current `SpeechEncoderDecoderModel` doesn't pass the `labels` to the decoder, hence no loss can be calculated. Not sure why it was not included. 

This PR fixes this.

Fixes #13716 



Comments:
Hey Niels, 

yeah I let this out on purpose because I have to think a bit about the design of the loss function to make it work nicely with the trainer...it's not as straight-forward. Would it be ok to wait with this 1,2 weeks ? :-) 

Hi there! Would the `Seq2SeqTrainer` also need to provide for an `input_values` key [over here](https://github.com/huggingface/transformers/blob/11c69b80452fae4b13c6d8bc22bdc19f3a752199/src/transformers/trainer_seq2seq.py#L168) for it to be compatible when using speech models?

Hey, 

I will take care of training SpeechEncoderDecoder models next week :-)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Fixed by #14139
Feature: Tail Free Sampling
Author: mrseeker
Pull request: no
Labels: name: WIP

# 🚀 Feature request

I would like to have Huggingface implement Tail Free Sampling, or TFS for short, to the official repo. The original paper for this can be found here: https://trentbrick.github.io/Tail-Free-Sampling/#tail-free-sampling-algorithm

Implementation of this code can be found here: https://github.com/finetuneanon/transformers/blob/gpt-neo-localattention3-rp-b/src/transformers/generation_logits_process.py#L243-L284

## Motivation

At the moment KoboldAI uses Finetuneanon's version of Transformers for creating stories, however, this version has several changes made to the repository that make it impossible for me to use (eg: I use an aarch64 architecture with an integrated Nvidia chip, which is only supported by Huggingface's main branch). There are calls from the KoboldAI devs to migrate due to the integrated support of GPT-J in Huggingface, however, there are some features in Finetuneanon's Transformer which are not found in the main branch. Tail Free Sampling is one such feature.

## Your contribution

I would like to file a PR with the code, however, I do lack the knowledge on how to implement the required tests and checks. An example is available on Finetuneanon's branch but it will require cherry-picking the commit to the main branch.

Comments:
cc @patrickvonplaten 

@mrseeker - would you like to open a PR for it :-) ?

Bumping to prevent getting closed, PR is WIP.
Add more missing models to models/__init__.py
Author: ydshieh
Pull request: yes

# What does this PR do?

After #14151, I just checked if there are other missing models - and added them in this PR.
There will be some issues showing up that are previously undetected.

## Who can review?

@sgugger  @LysandreJik 

Comments:
Could you fix the issues it generates? You should see them locally with `make quality`

> Could you fix the issues it generates? You should see them locally with `make quality`

Sure, will do later. 

The issues are fixed - I leave Hugging Face to double check with the original model authors if the reason `# Building part of bigger (tested) model.` are valid here.

For `UniSpeechSatForPreTraining`, it needs a reason for it to be in 

https://github.com/huggingface/transformers/blob/96d1b3c76224e0f923c6e8c8257df95f8156b81a/utils/check_repo.py#L75-L80

In general, it would be great if this situation can be avoided (i.e. check if the models in `models/__init__.py` corresponds to `src/models/`), so `make quality` will fail if the necessary tests are not provided or exceptions not specified in special lists.


I add a new check `check_model_list` in `check_repo.py` in order to avoid the missing models.

For example, if `vision_encoder_decoder` is not included in `src/transformers/models/__init__.py`, the test would fail with the following message
```
Checking all models are included.
Traceback (most recent call last):
  File "C:\Users\33611\Desktop\Projects\transformers-dev-2\transformers\utils\check_repo.py", line 600, in <module>
    check_repo_quality()
  File "C:\Users\33611\Desktop\Projects\transformers-dev-2\transformers\utils\check_repo.py", line 587, in check_repo_quality
    check_model_list()
  File "C:\Users\33611\Desktop\Projects\transformers-dev-2\transformers\utils\check_repo.py", line 168, in check_model_list
    raise Exception(
Exception: The following models should be included in src/transformers\models/__init__.py: vision_encoder_decoder.
make[1]: *** [Makefile:40: extra_quality_checks] Error 1
make[1]: Leaving directory '/c/Users/33611/Desktop/Projects/transformers-dev-2/transformers'
make: *** [Makefile:50: quality] Error 2
```

> As an aside, is there a reason `bort` is excluded?

Glad it will be helpful.

`bort` only contains a checkpoint conversion script.

If I include it, then we also need to have `bort` included in `models/__init__.py` (otherwise the test will fail).
```
from . import (
    albert,
    ...
    bort
    ...
```
but this gives a warning `unused import statement from...` in PyCharm IDE. (I don't have a clear idea about why).
I can add it though.

**Update**

Well, I just realized that it is fine to remove `_ignore_models = ["bort"]`, because of having a condition `"__init__.py" in os.listdir(model_dir)`:

https://github.com/huggingface/transformers/blob/f55ee867107d2b69f2be0b06b1b6c281acb2bfe7/utils/check_repo.py#L160-L161
Fixing `image-segmentation` tests.
Author: Narsil
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@LysandreJik

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

Comments:
[T5v1_1] Add lm model pretrained models as well
Author: patrickvonplaten
Pull request: no

Reminder to @patrickvonplaten to add all [LM-Adapted: t5.1.1.lm100k checkpoitns](https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#lm-adapted-t511lm100k) to the hub.

Comments:
FYI @TevenLeScao @craffel @VictorSanh 

https://huggingface.co/models?other=t5-lm-adapt
PIL and soundfile shouldn't be required to run `transformers-cli env`
Author: merleyc
Pull request: no

PIL and soundfile shouldn't be required to run `transformers-cli env`. 
Considering that, can someone fix this issue please? Please let me know if need more information from me.

_Originally posted by @LysandreJik in https://github.com/huggingface/transformers/pull/13588#pullrequestreview-757529042_

Thank you!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Wav2vec2 with different tokenizer ?
Author: Shiro-LK
Pull request: no

Hello,


thank you for having make wav2vec compatible with the transformer package.
I have a question regarding the tokenizer part. By default, the tokenizer is doing prediction at character level. I wanted to know if it is possible to use similar tokenizer than the one use in NLP such as the RobertaTokenizer ? (Or other one based on sentence piece/word piece)

I tried to use the roberta tokenizer for example but got an error with the processor : 

> ValueError: `tokenizer` has to be of type <class 'type'>, but is <class 'transformers.models.roberta.tokenization_roberta.RobertaTokenizer'>

Is it possible to use a different tokenizer than char level with this wav2vec implementation ? 

Comments:
[This](https://huggingface.co/transformers/master/model_doc/speechencoderdecoder.html) could be of interest to you. Notice the examples given, you can use the [Speech2Text2Processor](https://huggingface.co/transformers/master/model_doc/speech_to_text_2.html#speech2text2processor) to wrap the feature extractor and the tokenizer you want to use.

Hey @Shiro-LK,

When using `Wav2Vec2ForCTC` then it is necessary to use the `Wav2Vec2Tokenizer` which operates on the char level. In order to use a Roberta-like tokenizer, one should probably use the `SpeechEncoderDecoder` framework with `Wav2Vec2` being the encoder and roberta the decoder.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Deprecate Mirror
Author: JetRunner
Pull request: yes

The mirror option for downloading is now deprecated due to the switching to Git LFS and removal of Tuna.

https://github.com/tuna/issues/issues/937

Comments:
(i am not a maintainer, please wait for maintainer review/approval before merging!) @sgugger @LysandreJik 

> (i am not a maintainer, please wait for maintainer review/approval before merging!) @sgugger @LysandreJik 

Oh I'll be more cautious next time (though I'm sure this one is okay)

> Thanks for the PR and the concept of deprecating this `mirror` argument sounds good but:
> 
> - we use proper deprecation warnings for this
> 
> - we never do breaking changes between minor releases, so the argument should still behave in the same way for now with the deprecation warning, and we should only change its behavior or remove it for the next major version.
> 
> 
> 
> I think the easiest is to revert this merge and start fresh on a new PR.

Hi @sgugger 

I was in a hurry (gonna be on a trip soon). Sorry for rushing for it!

It's not an actual breaking change because the HF mirrors have already been removed from these servers. They are already broken now. The warning is a way to explain why it breaks.

I'll fix it in a new PR. Gonna throw a ValueError with no deprecation warning.

I am interested in mirroring for another reason, beyond speed, which is security. To use these datasets/models (and wrapper code) in an enterprise setting, one would need to ensure they are properly scanned for security vulnerabilities and malicious code.

Can anyone update me on the status and best method to mirror these (git lfs is fine if necessary, s3 too)?

Hi @nreith, We have an Enterprise solution for licensing-compatible on-prem syncing of public models/datasets.

Please get in touch with `expert-acceleration@huggingface.co` if you'd like to know more, cc @jeffboudier

Thanks @julien-c !
I just sent an email. Can you clarify how this would differ from the self-hosted option above, and when the self-hosted option would be available?
How can i add a Bi-LSTM layer on top of some model?like xlm_roberta
Author: bubbleqi
Pull request: no
Labels: name: Migration

I'm using pytorch and I'm using the base pretrained bert to classify sentences for hate speech. I want to implement a Bi-LSTM layer that takes as an input all outputs of the latest transformer encoder from the bert model as a new model (class that implements nn.Module), and i got confused with the nn.LSTM parameters. 

Comments:
Hi,

Please ask training-related questions on our [forum](https://discuss.huggingface.co/) instead of here, as we'd like to keep Github issues for bugs/feature requests.

Thanks!
GPT-J Models Cannot Load If Tokens Have Been Resized Using resize_token_embeddings Method
Author: alexorona
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` versions: 4.11.1, 4.11.3
- Platform: Ubuntu 18.04, Windows
- Python version: 3.7.5

### Who can help
@patil-suraj @patrickvonplaten, @LysandreJik @stas00

## Information

Something seems to be wrong in the serialization of GPTJ models. They are not behaving like other models in the library. After loading a GPTJ model using `.from_pretrained`, calling `model.resize_token_embeddings` and saving the model using `.save_pretrained`, it is not possible to load a GPTJ model again using `.from_pretrained`. Is there a workaround that's suggested? 

This error is thrown:

```
RuntimeError: Error(s) in loading state_dict for GPTJForCausalLM:
	size mismatch for lm_head.weight: copying a param with shape torch.Size([50400, 4096]) from checkpoint, the shape in current model is torch.Size([54001, 4096]).
	size mismatch for lm_head.bias: copying a param with shape torch.Size([50400]) from checkpoint, the shape in current model is torch.Size([54001]).
```

## To reproduce

```
from transformers import GPTJForCausalLM
model = GPTJForCausalLM.from_pretrained("EleutherAI/gpt-j-6B")
model.resize_token_embeddings(54001)
model.save_pretrained("model_dir")
model = GPTJForCausalLM.from_pretrained("model_dir")
```

## Expected behavior

The model should be loaded correctly if the token embeddings have been changed in the standard way.


Comments:
@patil-suraj Changing the two methods below in modeling_gptj.py fixes the problem, but it needs to fine-tune longer than other gpt-style models (2 epochs vs. 1), so I'm not sure that this is correct. Please advise.

Methods updated:

```
def get_output_embeddings(self):
    return self.lm_head

def set_output_embeddings(self, new_embeddings):
    self.lm_head = new_embeddings
```

@alexorona We also need to set `tie_word_embeddings=False` in `config`, because GPT-J does not share word embeddings with `lm_head`. It was not set by default, so when we resize the embeddings, the weights get tied, maybe this could be the issue.
[GPTJ] enable common tests and few fixes
Author: patil-suraj
Pull request: yes

# What does this PR do?

Currently, GPTJ does not run common tests because its testes class does not subclass `ModelTesterMixin, GenerationTesterMixin`.  This PR enables common tests for GPTJ and fixes a few things along the way.
I've run the slow tests manually and verified that they pass.

Thanks a lot, @sgugger  for spotting this!

cc @StellaAthena 

Fixes #14107




Comments:
@patil-suraj For the specific issue of `resize_token_embeddings` on gptj (#14107), I got it to work by changing two methods in modeling_gptj.py below. I'm not sure this is right because the model needs to train 2 epochs to get a good result whereas I almost always need just 1 epoch with other model types (gpt-2, gpt-neo, etc.). Will this PR cover the `resize_token_embeddings` issue? It doesn't seem to make changes to these methods.

```
def get_output_embeddings(self):
    return self.lm_head

def set_output_embeddings(self, new_embeddings):
    self.lm_head = new_embeddings
```

> @patil-suraj For the specific issue of `resize_token_embeddings` on gptj (#14107), I got it to work by changing two methods in modeling_gptj.py below. I'm not sure this is right because the model needs to train 2 epochs to get a good result whereas I almost always need just 1 epoch with other model types (gpt-2, gpt-neo, etc.). Will this PR cover the `resize_token_embeddings` issue? It doesn't seem to make changes to these methods.
> 
> ```
> def get_output_embeddings(self):
>     return self.lm_head
> 
> def set_output_embeddings(self, new_embeddings):
>     self.lm_head = new_embeddings
> ```

@patil-suraj Nevermind! It looks like you caught this and also made additional changes!

Looks good to me too!