noamaanMulla-03/transformers-issues huggingface dataset:
Fix regression in regression
Author: sgugger
Pull request: yes

# What does this PR do?

This PR fixes the regression introduced in #11012 for regression problems with only one label (like STS-B), see discussion on #11780. I checked both `run_glue` and `run_glue_no_trainer` on this branch and get the proper results for this task now.

Fixes #11780
Fixes #11583

Comments:
Thank you for fixing the issue!
A cleaner and more scalable implementation of symbolic tracing
Author: michaelbenayoun
Pull request: yes

# What does this PR do?

This PR provides a much cleaner and less hacky implementation of symbolic tracing for models of the library.
It also provides support for more architectures:

- ALBERT
- DistilBERT
- MobileBERT
- MegatronBERT
- GPT2
- GPT Neo


Comments:
What do you think about `dtype` being hardcoded?

While this is OK for now hardcoding dtype might be an issue down the road. For most NLP models inputs are int, but for example for wav2vec2 they are floats. 

And would this have an impact if the final usage is in fp16 for where you used `float`.

We can't derive the dtype from the model in this context. 

Thoughts?

This is not a showstopper to merge this, but just something to consider - I'm sure we will cross the bridge if we encounter it.
Fix failing test on Windows Platform
Author: kere-nel
Pull request: yes

- test_tokenization_wav2vec2.py failure on Windows - fixed by changing directory separator when running on Windows. 
- tests/test_utils_check_copies.py failure on Windows due to splitting text with "\n" on Windows, instead of "\r\n" - fixed by using universal newlines. 
- tests/extended/test_trainer_ext.py failure on Windows due to "\r" characters encoding issue - fixed by specifying utf-8 encoding. 

Fix for tests in Windows #11586 


Comments:
Hey @Lynx1820,

Thanks a lot for the PR! Could you run `make style` to make the tests pass? :-)
CamemBert Tokenizer AttributeError: 'NoneType' object has no attribute 'tokenize'
Author: Quang-Vinh
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.5.1
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.8
- PyTorch version (GPU?): 1.8.1+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: No

### Who can help

Library:
- tokenizers: @LysandreJik


## Information

Model I am using camemBert https://huggingface.co/camembert-base.

The problem arises when using:
* [x ] the official example scripts: 

``` python
from transformers import CamembertModel, CamembertTokenizer

# You can replace "camembert-base" with any other model from the table, e.g. "camembert/camembert-large".
tokenizer = CamembertTokenizer.from_pretrained("camembert-base")
camembert = CamembertModel.from_pretrained("camembert-base")

camembert.eval()  # disable dropout (or leave in train mode to finetune)

import torch
# Tokenize in sub-words with SentencePiece
tokenized_sentence = tokenizer.tokenize("J'aime le camembert !")
```

## To reproduce

Steps to reproduce the behavior:

1. Install transformers
2. Run code 

I get a AttributeError: 'NoneType' object has no attribute 'tokenize' Error as the tokenizer is None when I load from pre trained.


Comments:
Hi! Could you try installing `sentencepiece` to see if that solves the problem?

I got an error when sentencepiece wasn't installed and after installing it returned None. Trying it again now I don't see the error anymore though so I'll close the issue 🙂

If this was on colab it's possible that you needed the runtime to restart!
Question about validation_set
Author: Shengyu-Liu558
Pull request: no

I want to ask a simple question. The parameters of the model have been set before model training. What is the purpose of the validation set in model training? Thank you！

Comments:
Hi,

You can use Stackoverflow for that: https://stats.stackexchange.com/questions/19048/what-is-the-difference-between-test-set-and-validation-set

We like to keep Github issues for feature requests/bug reports. There's also the [forum](https://discuss.huggingface.co/) where you can ask training-related questions.

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
It don't find simple logic sequences 
Author: seb16120
Pull request: no

![Screenshot_2021-04-13-01-15-24-79](https://user-images.githubusercontent.com/45233573/114474232-0ef30600-9bf6-11eb-8ffc-62f6bee83def.jpg)


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Beam search on BART seq2seq
Author: ashleylew
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.5.0.dev0
- Platform: Linux-3.10.0-1160.15.2.el7.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.8.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: ?

### Who can help

@patrickvonplaten, @patil-suraj

## Information

Model I am using (Bert, XLNet ...): BART seq2seq

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.  I'm attempting to use beam search and have the model output the 10 best possible predictions for each test item.
2. I found the parameter `num_beams`, which I am using though it does not appear to work by itself. No error occurs, but only 1 output per test item is produced.
3. I thought I should use the parameter `num_return_sequences` as well, but it does not appear to be a possible argument for this model and I have not been able to find anything comparable. 

Here is my command:
```
python transformers/examples/seq2seq/run_translation.py \
    --model_name_or_path facebook/bart-base \
    --do_train \
    --do_predict \
    --source_lang en \
    --target_lang lf \
    --source_prefix "translate English to Logical Forms: " \
    --train_file folds/0_train.json \
    --test_file folds/0_val.json \
    --num_train_epochs=5 \
    --num_beams=10 \
    --output_dir ./test_results_beam \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate
```

## Expected behavior

Outputting the 10 best predictions per test item. 


Comments:
Hi @ashleylew 

The `run_translation.py` uses the `Seq2SeqTrainer` which does not pass the `num_return_sequences` argument to `generate`, this is because if multiple sequences are returned then its not clear what sequence should be used to compute the metrics. 

you could generate test set predictions by using the `generate` method and passing the `num_return_sequences` argument. But if you want to do this using `Seq2SeqTrainer` then you would need to modify it.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Documentation enhancement - model_type
Author: gwc4github
Pull request: no

# 🚀 Feature request


Please provide a clear explanation of what the valid values would be for "model_type".  I think that the answer is any model name you would use in from_pretrained() but I am not sure.

## Motivation

Clarity of the parameter and saving time on trial and error and guesswork.

## Your contribution

If the assumption above is correct, I am willing to write up the answer if it will help.  If you have a HTML page with the list of valid values (maybe on a model page) we can just add a link to that.



Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[run_clm.py] restore caching
Author: stas00
Pull request: yes

`datasets==0.1.6` introduced in-memory datasets, which unfortunately has no caching which makes it very slow to develop with as the dataset gets reprocessed on every run. Supposedly this should make things faster overall, but at this huge cost to us developers.

It's also inconsistent where some datasets behave in one way, others in another way. This is too magical, IMHO.

This PR adds `keep_in_memory=False`, to disable in-memory cache, but restores normal caching.

Perhaps adding a note in the example that the user can change it to `True` if they don't care for the slow startup?

Alternatively, if you believe that the new behavior is good, let's create an env var at `datasets` that will control that, so that we can turn off this painful behavior w/o needing to manually modify the code.

Fixes: https://github.com/huggingface/transformers/issues/11801

p.s. working on this one script on many fronts - and then will sync other scripts at once.

@sgugger, @VictorSanh, @lhoestq 

Comments:
No we can't just add the new argument without checking the version, as it's probably not going to work anymore for earlier versions of datasets (that's why it's bad to do breaking changes :-P).
It seems like it's the way the Datasets library wants to be used, so I would leave the default behavior here and you can change the script locally for your use case. If the defaults of the Datasets library are not satisfactory, then maybe those defaults should be changed.

Makes sense, @sgugger - thank you - back to `datasets`
Training transformer XL from scratch with my own dataset
Author: Syrup274
Pull request: no

## ❓ Questions & Help
Is it possible to train a transformer XL model from scratch with my own dataset? 
Just initialize the model with default params and compile & fit the model?
<!-- A clear and concise description of the question. -->


Comments:
I think yes, 
Just load the model (class) and then start training. As it is mentioned [here](https://huggingface.co/transformers/model_doc/transformerxl.html#transformers.TransfoXLModel), model classes are just a PyTorch torch.nn.Module sub-class.

> This model is a PyTorch torch.nn.Module sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

Is it also possible to train tensorflow model?

If we think about [this](https://huggingface.co/transformers/model_doc/transformerxl.html#tftransfoxlmodel) expression, yes. 

> This model is a tf.keras.Model tf.keras.Model sub-class. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and behavior.

Also, check [this](https://github.com/huggingface/transformers#quick-tour-tf-20-training-and-pytorch-interoperability) out.

Thanks very much!
get_length_grouped_indices() uses slow list concat
Author: ctheodoris
Pull request: no

Hi,

get_length_grouped_indices() in LengthGroupedSampler and DistributedLengthGroupedSampler is prohibitively slow for large number of megabatches (in my case takes hours for ~270k megabatches with 100 items each) due to slow list concatenation with sum(megabatches, []). 

Concatenating the lists with sum() may be repeatedly reallocating memory with each successive concatenation (similar to performance issues with string concatenation).

[item for sublist in megabatches for item in sublist] approach appears to significantly improve speed for large megabatch number, especially for megabatches with larger number of items.

For example:

    # 50,000 megabatches with 3 items each:
    megabatches = [[1,2,3] for _ in range(50_000)]

    %timeit [item for sublist in megabatches for item in sublist];
    3.72 ms ± 75.5 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

    %timeit sum(megabatches, []);
    7.66 s ± 31.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

------------------------------------------
    # 100,000 megabatches with 3 items each:
    megabatches = [[1,2,3] for _ in range(100_000)]

    %timeit [item for sublist in megabatches for item in sublist];
    8.03 ms ± 14.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

    %timeit sum(megabatches, []);
    29.6 s ± 36.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

------------------------------------------
    # 100,000 megabatches with 100 items each:
    megabatches = [list(range(100)) for _ in range(100_000)]

    %timeit [item for sublist in megabatches for item in sublist];
    208 ms ± 44.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

    %timeit -r1 -n1 sum(megabatches, []);
    44min 3s ± 0 ns per loop (mean ± std. dev. of 1 run, 1 loop each)


Thank you for your wonderful work and consideration of this edit. @sgugger

Comments:
Thanks a lot for looking at this optimization. It does look like a nice speedup! Do you want to open a PR with the suggested changes since you're the one who designed it?
Megatron fused CUDA kernels to improve Hugging Face model classes' scalability
Author: g-karthik
Pull request: no
Labels: name: Performance

name: WIP

# 🚀 Feature request

Support for custom fused CUDA kernels with HF model classes.

## Motivation

It appears that Hugging Face model classes do not scale very well as-is unlike Megatron-LM, even when the latter is configured with a degree of model-parallelization = 1 for a "fair" performance comparison.

One of the presumed reasons for this is that Megatron-LM leverages custom fused CUDA kernels written by NVIDIA, specifically [these](https://github.com/NVIDIA/Megatron-LM/blob/aed2f75e209e525c842aec7c044af7acae2a4614/megatron/model/transformer.py#L26L27).

Could we get variants of existing HF classes (perhaps for `GPT2Model`, `GPT2LMHeadModel`, etc.) such that the variants leverage some/all of these fused CUDA kernels? All this while still ensuring that one can load the original pre-trained weights into these variant classes.

Any guidance/low-level thoughts towards making this happen would also be greatly useful!

@thomwolf @patrickvonplaten @LysandreJik @stas00 

Comments:
I think the biggest barrier to using custom CUDA kernel is that it'd require `transformers` to move from a python-only package, to a compilation-required type of package (even if JIT), which in my experience is the type of a package that is far from trivial to use and often raises a barrier to entry.

If I'm not mistaken some fused kernels have been pushed upstream into the pytorch-core, so if you know of any that we could receive precompiled via pytorch, then we can definitely use those.

And if they aren't and you have some resources to initiate the conversation - it'd definitely help to request that such kernels will be added to pytorch-core. Definitely tag me if I do start such a thread at pytorch Issues.

-----

I love your spirit of proposing various performance optimizations, @g-karthik and I'd love to work on all of those you have been proposing here and at Deepspeed issues, but so far I find no free resources to do so and all my time is spent on making things work.
Delete key or set to `None` in __getstate__ impl.
Author: PhilipMay
Pull request: no

Hi,
there are some places that implement `__getstate__` because an object has a reference to an other object that is not pickable.

`__getstate__` then "deletes" the reference by setting it to `None`. Just a few examples:

https://github.com/huggingface/transformers/blob/73fde1defe9be259a47b9024525882f3ec420994/src/transformers/models/m2m_100/tokenization_m2m_100.py#L272

https://github.com/huggingface/transformers/blob/73fde1defe9be259a47b9024525882f3ec420994/src/transformers/models/marian/tokenization_marian.py#L299

IMO it would be better to delete the keys instead of setting them to `None`. Like this: `del state["sp_model"]`

What do you think @sgugger @LysandreJik ? I can provide a PR if wanted.

Comments:
Hmmm - thinking about it - the value is also set to None by default in the constructor. I will close this...
chunk of words for input token
Author: jihwanp
Pull request: no

Hi I have some questions about using pretrained bert.

Can I put a chunk of words into one input token? For example, split "hi my name is Linda and today i will~" as "hi my name is Linda" and "and today i will" and make each split as one embedding vector (i.e using average word2vec) and treat each split vector as one input token. Is it okay to apply it to the existing pre-trained models?

Actually i'm forced to use phrase wise token in my task so the  models for long sequences are not the option.

Thanks

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect
Author: deadsoul44
Pull request: no

Probably due to assertion, traceback is lost and I cannot debug the code. 

C:\Users\m00596504\.virtualenvs\porn_tr\lib\site-packages\transformers\modeling_utils.py:1759: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert all(

Process finished with exit code -1073741819 (0xC0000005)

Comments:
Hi, please provide all the information required in the template so that we may help you. Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
About pre-trained model : facebook/wav2vec2-large-xlsr-53 & facebook/wav2vec2-base
Author: LifaSun
Pull request: no

Hi,

I am trying to load the pre-trained 'no fine-tuning' model of wav2vec2 to exact some features . The models, 'wav2vec2-base' and 'wav2vec2-large-xlsr-53', are not fine-tuned. Why are these files not exactly the same? 

https://huggingface.co/facebook/wav2vec2-base/tree/main
https://huggingface.co/facebook/wav2vec2-large-xlsr-53/tree/main

 'wav2vec2-base'  can be loaded smoothly, but it doesn't work for 'wav2vec2-large-xlsr-53'
processor = Wav2Vec2Processor.from_pretrained("facebook/wav2vec2-base")
model = Wav2Vec2Model.from_pretrained("facebook/wav2vec2-base").to('cuda')


@patrickvonplaten 
Thank you very much! 




Comments:
The xlsr model has no vocab, you need to build the processor yourself

https://huggingface.co/blog/fine-tune-xlsr-wav2vec2
You could check out this blogpost, there is it explained very well

@flozi00 Thank you very much! Processor is built. 

Actually, I want to visualize the output of hidden layer in the wav2vec model before fine-funing. It seems the output of wav2vec2-base is normal, but the output of wav2vec2-large-xlsr-53 is not. The results are attached (x axis: time, y axis: hidden units).  

The output of hidden layer using wav2vec2-base
![base](https://user-images.githubusercontent.com/6188893/115206484-3b92aa80-a12d-11eb-95ea-13d109250c17.png)

The output of hidden layer using wav2vec2-large-xlsr-53
![xlsr](https://user-images.githubusercontent.com/6188893/115206466-37668d00-a12d-11eb-9c8f-aa11965b2291.png)

Could you explain it? Thank you!



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[Benchmark]
Author: juhyeok123
Pull request: no

# 🖥 Benchmarking `transformers`

## Benchmark

Which part of `transformers` did you benchmark?

## Set-up

What did you run your benchmarks on? Please include details, such as: CPU, GPU? If using multiple GPUs, which parallelization did you use?

## Results

Put your results here!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Batching in NER pipeline
Author: parakalan
Pull request: no

# 🚀 Feature request

Currently, the NER pipeline in transformers iterates through the list of input sentences and processes them, sequentially.
It would we beneficial to add batching support in the pipeline to decrease latency and use GPU more efficiently.

## Motivation

Batching will help use the GPU more efficiently and reduce latency by a lot. The NER pipeline is amazing with its post processing, could be a production ready construct if batching is added.

This issue has been discussed here - https://github.com/huggingface/transformers/issues/8942 at the end, but looks like no one is working on it actively.

## Your contribution

Working on this issue in this PR - https://github.com/huggingface/transformers/pull/11251

Please raise if this is on the radar already, will close the issue.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.