noamaanMulla-03/transformers-issues huggingface dataset:
Small bug while converting wav2vec2 model trained using fairseq to huggingface
Author: harveenchadha
Pull request: no

Hi,

I was trying to convert a wav2vec2 model trained using fairseq to have support with HuggingFace but there is a small error in inference.

When I use the code below:


```
import soundfile as sf
import torch
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor

# load pretrained model
processor = Wav2Vec2Processor.from_pretrained('hf/output')
model = Wav2Vec2ForCTC.from_pretrained("hf/output")


# load audio
audio_input, sample_rate = sf.read('004-M-23_001.wav')

# pad input values and return pt tensor
input_values = processor(audio_input, sampling_rate=sample_rate, return_tensors="pt").input_values

# INFERENCE

# retrieve logits & take argmax
logits = model(input_values).logits
predicted_ids = torch.argmax(logits, dim=-1)

# transcribe
transcription = processor.decode(predicted_ids[0])

```

I get this as the output:

```

ह
ॉ
ट
ल
 
र
ॉ
य
ल
 
ह
े
र
ि
ट
े
ज
 क
े
 
च
ी
ज
 
क
े
 
ए
क
 
ब
ह
ु
त
 
अ
च्
छ
ा
 
ह
ै
 
क
्य
ा
 

```

but, I should ideally get this:

```
हॉटल रॉयल हेरिटेज के चीज के एक बहुत अच्छा है क्या
```

I can easily solve this by using:

```
print(transcription.replace('
', ''))
```

But if I deploy the model, the output of inference will contain ```
``` as I cannot change the output of deployed model.

Can you please let me know if I am doing any mistake in the conversion process.

My vocab.json looks like this:

```
{"
": 0, "<pad>": 1, "
": 2, "<unk>": 3, "|": 4, "0": 5, "1": 6, "2": 7, "3": 8, "4": 9, "5": 10, "6": 11, "7": 12, "8": 13, "9": 14, "ँ": 15, "ं": 16, "ः": 17, "अ": 18, "आ": 19, "इ": 20, "ई": 21, "उ": 22, "ऊ": 23, "ऋ": 24, "ए": 25, "ऐ": 26, "ऑ": 27, "ओ": 28, "औ": 29, "क": 30, "ख": 31, "ग": 32, "घ": 33, "ङ": 34, "च": 35, "छ": 36, "ज": 37, "झ": 38, "ञ": 39, "ट": 40, "ठ": 41, "ड": 42, "ढ": 43, "ण": 44, "त": 45, "थ": 46, "द": 47, "ध": 48, "न": 49, "प": 50, "फ": 51, "ब": 52, "भ": 53, "म": 54, "य": 55, "र": 56, "ल": 57, "व": 58, "श": 59, "ष": 60, "स": 61, "ह": 62, "ा": 63, "ि": 64, "ी": 65, "ु": 66, "ू": 67, "ृ": 68, "ॅ": 69, "े": 70, "ै": 71, "ॉ": 72, "ो": 73, "ौ": 74, "्": 75, "क़": 76, "ख़": 77, "ग़": 78, "ज़": 79, "ड़": 80, "ढ़": 81, "फ़": 82, "य़": 83}
```

Comments:
Hi @harveenchadha 

if you pass `skip_special_tokens=True` to `decode` method, it will skip all the special tokens. 

Hi Suraj,

Thanks! That works. But in the inference service that is deployed to test in browser, how will this change?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[Deepspeed] various fixes
Author: stas00
Pull request: yes
Labels: name: DeepSpeed

This PR includes a few small fixes in config files, tests and docs:

- replace deprecated config `cpu_offload` with `offload_optimizer`
- `sub_group_size` setting was too big - needing too much GPU RAM

@sgugger 

Comments:
That's a good call, Sylvain. I think the deprecation warning has been showing up long enough that I didn't bother checking. But I did check now and all is good it was done before deepspeed==0.3.16 was released ([commit](https://github.com/microsoft/DeepSpeed/commit/0d4a54a04d658db40a120bc10c6f1f1a4478f6f1)) and I retested with 0.3.16 just to be sure.
Cannot unzip the XNLI-MT 1.0 zip file.
Author: minstar
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

Is there anyone who succeeding the unzip of XNLI-MT 1.0 zip file?

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I am able to unzip the archive form this link: https://dl.fbaipublicfiles.com/XNLI/XNLI-MT-1.0.zip
[testing] set tests to not rebuild datasets
Author: stas00
Pull request: no

recently `datasets` created in memory datasets enabled by default - which is great for those who wants it but is a terrible idea for tests and those who need to develop things constantly restarting the scripts as datasets aren't being cached and rebuilt on every run.

So we should turn this feature off in `*/conftest.py` by setting:
```
HF_MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES=0 
```

But it's going to be renamed shortly to `HF_DATASETS_IN_MEMORY_MAX_SIZE`
https://github.com/huggingface/datasets/pull/2409#issuecomment-850549742
https://github.com/huggingface/datasets/pull/2454

So for now this issue is tracking this change and then will add it to the tests.



Comments:
`datasets` has just made this feature disabled by default: https://github.com/huggingface/datasets/pull/2460

So nothing needs to be done.
[training] SAVE_STATE_WARNING was removed in pytorch
Author: stas00
Pull request: yes

`SAVE_STATE_WARNING` has been removed from pytorch 3 days ago: pytorch/pytorch#46813

I had to add redundant ()'s to avoid a terrible auto-formatter outcome.

Fixes: #8232

@sgugger, @LysandreJik 

Comments:
Thank God they removed that horrible thing! Instead of the complex parsing, maybe a `try`/`except` would be cleaner?

 no horrible formatting that way! good idea - done.

@stas00, could it be possible to apply your fix to the code of version `3.5.X` and release the next **minor** version (like v3.5.2)?

@LysandreJik, your help is needed here. I don't know anything about how old branches maintenance is done. 

This PR was merged in 4x series and @vyshkant is requesting this fix applied to 3.5.x for the next release.

Thank you.

No we won't do fixes for old versions, either upgrade to v4 or use PyTorch < 1.8 if you want to stick to v3.5.
Constrained decoding?
Author: kailashkarthiks
Pull request: no

Is it possible to implement constraints on the beam during decoding using a seq2seq model? [NeuroLogic Decoding](https://arxiv.org/abs/2010.12884), [Constrained Abstractive Summarization](https://arxiv.org/abs/2010.12723)

I see that there is a Callback feature in the library - but AFAIK it only lets modify the training state and not the beam. 

Comments:
I think you should take a look at `prefix_allowed_tokens_fn` on the [`generate` method](https://huggingface.co/transformers/main_classes/model.html?highlight=generate#transformers.generation_utils.GenerationMixin.generate). 

It let's you create a function to constrain the generation based on previously generated tokens. It is inspired by this paper:  [Autoregressive Entity Retrieval](https://arxiv.org/abs/2010.00904).

thanks! Is there a way to have more generic callbacks that can possibly access the beams (for instance, if I want to do custom scoring/tracking/modification to the entries in the beam)?

If not, would that be a useful PR to submit?

So you mean to access the different beams so that generations is conditioned to not just the current beam so far but also the others? 
You probably should look at transformers.LogitsProcessor in:
https://github.com/huggingface/transformers/blob/1750e629006bb6989aef5b4e141f3477f891a098/src/transformers/generation_utils.py#L555-L629

 which deals with the constraints and scoring of tokens at generation. Perhaps what you described could be introduced in a similar fashion as `prefix_allowed_tokens_fn`.

Regarding a PR I am not the best to say, I would first make sure if what you aim for can be done within the existing functionality. 

I did take a look at `LogitsProcessor` and might be able to work with that - thanks!

@kailashkarthiks Hi, I am also interested in performing constrained decoding with seq2seq language models like BART. I was wondering if you made any progress on this, and if yes, would you be able to share your findings or code? Thank you!
updated the original RAG implementation to be compatible with latest Pytorch-Lightning
Author: shamanez
Pull request: yes

The original RAG version was not working with PL>=1.3, specially due to the fact that DDPAccelerator class has removed (Retriever Initialization of RAG).  The new version of PL library advises us to use DDP pluggings as a replacement.  

I also updated lightning_base.py regarding the new PL version. Now RAG works with the latest libraries. 


@patrickvonplaten  @lhoestq 


Comments:
Hey @shamanez,

Could you run `make style`? 

@lhoestq - could you take a look as well?

Hey @patrickvonplaten 

I did run the "make style" and it changed following files and working alright.


Thanks @patrickvonplaten  :)
Update run_ner.py with id2label config
Author: KoichiYasuoka
Pull request: yes

# What does this PR do?

Enhancement for `run_ner.py` to produce more meaningful `id2label`.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

@sgugger 


Comments:
Sorry but I've rarely used `run_ner_no_trainer.py`. And, well, since `run_ner_no_trainer.py` does not change `config.json`, thus I want to keep this PR as is.

Will take care of it then, thanks!
Fix missing id2label and label2id in run_ner.py
Author: lohjine
Pull request: yes

This is to retain the NER labels when training, so it can be used to map the labels during later prediction.

This functionality is present in the old version [https://github.com/huggingface/transformers/blob/master/examples/legacy/token-classification/run_ner.py#L170](https://github.com/huggingface/transformers/blob/master/examples/legacy/token-classification/run_ner.py#L170), but missing in the current one.

@sgugger 

Comments:
This has just been done in #12001 :-)
Thanks for the PR!
[wav2vec2 / Deepspeed] sync LayerDrop for Wav2Vec2Encoder + tests
Author: stas00
Pull request: yes

This PR continues https://github.com/huggingface/transformers/pull/11638 and:
- adds the same gpu syncing for `Wav2Vec2Encoder` LayerDrop as there is for ` Wav2Vec2EncoderStableLayerNorm`
- double the tests to test `"patrickvonplaten/wav2vec2_tiny_random"` to exercise `Wav2Vec2Encoder` too

@patrickvonplaten 


Comments:
Wav2Vec2/XLRS-Wav2Vec2 Pre-Training
Author: xaiguy
Pull request: no

Dear 🤗-team,

I'd like to do pre-training with your implementation of Wav2Vec2 and/or XLRS-Wav2Vec2. I was wondering if there are any plans to add such scripts (or even a demo) to the repository?

PS: I already did pre-training in NVIDIA NeMo, but I'm having problems with porting my checkpoints. Being able to do everything within the Huggingface framework would be great.


Comments:
Wav2Vec2 Pre-Training is more important.

I would also like to be able to pre-train a Wav2Vec2 model using my own raw audio files in a self-supervised way. It would be even better if I could use a pre-trained model as a starting point. Is there any way to do this currently?

@czonios Yes, you can fine-tune a Wav2Vec2 model! Please check [this blogpost](https://huggingface.co/blog/fine-tune-xlsr-wav2vec2) by @patrickvonplaten. 

Pre-training is not available as of now.

Hey, 

We should have Wav2Vec2 Pretraining added in ~2 weeks.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Unstale

#11306 is under way I think

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
examples requirements isn't in sync with `require_version_examples`
Author: stas00
Pull request: no

We have `require_version_examples`
https://github.com/huggingface/transformers/blob/b1a8aa94f0a2ccea7c68b79066141aa822b96e42/src/transformers/utils/versions.py#L123-L126

but it looks like it wasn't updated in the last reshuffle and requirements got split and it suggests incorrect solution.

I tried to offer to use it here: https://github.com/huggingface/transformers/pull/11927
but since requirements are now scattered over multiple files we probably should remove it and its usage in legacy scripts, since it gives wrong info where it's currently still used.

It's just one usage in several legacy scripts:
```
require_version_examples("pytorch_lightning>=1.0.4")
```
so we can just replace it with:
```
require_version("pytorch_lightning>=1.0.4")
```
which would do the trick.

@sgugger 

Comments:
Yes, that works.
[versions] rm require_version_examples
Author: stas00
Pull request: yes

As explained in https://github.com/huggingface/transformers/issues/12086 `require_version_examples` wrapper is no longer useful since examples' requirements are now scattered across multiple files, so removing it as it can't be used because of that.

Fixes: https://github.com/huggingface/transformers/issues/12086

@sgugger  

Comments:
'EncoderDecoderModel' object has no attribute '_init_weights' after `model.resize_token_embeddings(len(tokenizer))`
Author: XinXia2019
Pull request: no

## Details
Version transformers==3.4.0
torch==1.6.0
torchvision==0.7.0

>>> tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
>>>tokenizer.add_special_tokens({"additional_special_tokens": ['extra1', 'extra2']})
1
>>> model = EncoderDecoderModel.from_encoder_decoder_pretrained('bert-base-uncased', 'bert-base-uncased')
>>> model.resize_token_embeddings(len(tokenizer))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/xxia/anaconda3/lib/python3.8/site-packages/transformers/modeling_utils.py", line 607, in resize_token_embeddings
    model_embeds = base_model._resize_token_embeddings(new_num_tokens)
  File "/Users/xxia/anaconda3/lib/python3.8/site-packages/transformers/modeling_utils.py", line 622, in _resize_token_embeddings
    new_embeddings = self._get_resized_embeddings(old_embeddings, new_num_tokens)
  File "/Users/xxia/anaconda3/lib/python3.8/site-packages/transformers/modeling_utils.py", line 659, in _get_resized_embeddings
    self._init_weights(new_embeddings)
  File "/Users/xxia/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 771, in __getattr__
    raise ModuleAttributeError("'{}' object has no attribute '{}'".format(
torch.nn.modules.module.ModuleAttributeError: 'EncoderDecoderModel' object has no attribute '_init_weights'

Thanks for help!

Comments:
Hello @XinXia2019, sadly `resize_token_embeddings` is not supported yet for `EncoderDecoderModel`. Instead you could just manually instantiate the encoder and decoder and apply `resize_token_embeddings` on each part before wrapping them into the `EncoderDecoderModel` framework.

Thanks!

(for anyone that might stumble upon this)

I believe you can access the encoder and decoder models from the EncoderDecoderModel instance and resize their corresponding token embeddings, e.g.:

```
...
tokenizer_length = len(tokenizer)
model.encoder.resize_token_embeddings(tokenizer_length)
model.decoder.resize_token_embeddings(tokenizer_length)
...

```
Fine tuning with transformer models for Regression tasks
Author: zyberg2091
Pull request: no

- `transformers` version: Bert, Albert, openai-gpt2
- Tensorflow version (GPU?): 2.5.0


## Information

Model I am using :  Bert, Albert, openai-gpt2

The problem arises when using:
* [x] my own modified scripts: (give details below) <br>
- performed fine tuning


The tasks I am working on is:

* [x] my own task or dataset: (give details below)<br>
- I have been trying to use BertModel, albert and GPT2 models for fine tuning on my regression task and i was able to produce unwanted results . i will mention it below: <br>

- I tried it two times: <br>

1. I used CLS token embeddings and fine tuned over my entire custom model but it produced some random number repeating over and over in my output matrix space.<br>

2. I simply passed CLS token embeddings to the feed forward NN. In this case also it produced some random number and no learning is seen here.<br>

<br>

**what can be the solution to this problem? is there any issues with transformers with respect to regression?**

Comments:
I think it's better to ask this question on the [forum](https://discuss.huggingface.co/) rather than here. For example, all questions related to training BERT for regression can be found [here](https://discuss.huggingface.co/search?q=bert%20regression).
feature-extraction pipeline is second last layer better representation than the last hidden layer
Author: kaushaltrivedi
Pull request: no

# 🐛 Bug

I see that for feature-extraction pipeline, you output the last hidden layer of the transformer.  Another approach I have seen is people using second to last layer as the output (Bert-as-a-service).  The rationale being that the last layer is too close to the model task such as masked language model objective. I am a bit confused as to what would be an ideal representation layer for sentence embedding.

Model I am using (Bert, XLNet ...): Bert


## Environment info

     
- `transformers` version: 2.8.0
- Platform: CPU
- Python version: 3.7
- PyTorch version (GPU?): 1.5.0
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


Comments:
This is quite a general question, and I fear there is no one right answer. You can see the "best layer to use" as another hyperparameter to tune in your task: the right value will depend on your task, dataset, and query. Oftentimes paper do some kind of evaluation of this. For instance, in the original [BERT paper](https://arxiv.org/pdf/1810.04805.pdf) they use BERT for feature extraction and evaluate which layer combinations worked best (Table 7). They find that the second-to-last layer performs better than the last layer, but that a concatenation of the last four layers is best. But again, that is using BERT, on a specific task, with a specific dataset. Your results may differ, so it's best to test and tune to your problem.

@BramVanroy @kaushaltrivedi How do you customize what layers go into the feature representation (via the pipeline)?
Continuous training on Fine-tuned Model
Author: noskidr
Pull request: no

# 🚀 Feature request

How can I continue training on a Fine-tuned Model?
I have a fine tuned model from OpenSLR data. And I want to continue training on an model as I continue to gain transcribed audio data over time. Can I do like making the fine-tuned model as a checkpoint?

## Motivation

I am aiming to make a model for Nepali Language. I have a way to collect data over time and it is continuous. So, I want to find a way I can train the model continuously as I gain data over time



Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!

Sure, thank you.
GPT2 Flax "TypeError: JAX only supports number and bool dtypes, got dtype object in array"
Author: s4sarath
Pull request: no

On GPU

```
>>> from transformers import AutoTokenizer, FlaxAutoModelForCausalLM

>>> tokenizer = AutoTokenizer.from_pretrained("gpt2-medium")
>>> model = FlaxAutoModelForCausalLM.from_pretrained("gpt2-medium")
>>> input_context = "The dog"
>>> # encode input context
>>> input_ids = tokenizer(input_context, return_tensors="jax").input_ids
>>> # generate candidates using sampling
>>> outputs = model.generate(input_ids=input_ids, max_length=20, top_k=30, do_sample=True)

TypeError: JAX only supports number and bool dtypes, got dtype object in array
```

@patrickvonplaten @patil-suraj 

Comments:
I see where this is coming from. For now, could you try initializing model and tokenizer like this
```
tokenizer = GPT2TokenizerFast.from_pretrained(model_id, padding_side="left", pad_token="
")
model = FlaxGPT2LMHeadModel.from_pretrained(model_id, pad_token_id=50256,)
```

We'll soon publish a detailed colab about Flax generate 

Done buddy. Worked. Thanks a lot. 
Does Flax ```model.generate``` makes use of caching ``` Query and Value ``` in attention layers ?

I have too run a benchmark for generation. Its fair to compare only if caching is supported. 

yeah it only works with caching 

Thanks . Closing the issue.