noamaanMulla-03/transformers-issues huggingface dataset:
Scoring each word from the sentence using Pretrained LM
Author: thak123
Pull request: no

# ❓ Questions & Help


## Details
I am interested in scoring each word  from the sentence using the LM. Detect how likely possible is the word in the given context by scoring it using the LM.

Is there a way out for this ?


Comments:
Something like [this](https://github.com/huggingface/transformers/issues/5000#issuecomment-647560847)?

You mean to say that use soft-max on the the hidden states of every word to get the score ?

I think that should work.

Great!

Since the code referenced tries to get the next prediction and I want to score existing sentence I modified to code to take the soft-max on the 2nd axis which is the word. Here is output. But I am unable to score the sentence or find the perplexity.

```
from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch

model = GPT2LMHeadModel.from_pretrained("gpt2")
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")

inputs = tokenizer.encode("This is just now", return_tensors="pt")
output = model(inputs)

model_output = output[0]
last_token_softmax = torch.softmax(model_output, dim=-1).squeeze()
last_token_softmax.shape
top_n_values = last_token_softmax.topk(1)
for index, value in zip(top_n_values.indices, top_n_values.values):
    print("Score: ", value.tolist())
    print("This is just" + tokenizer.decode(index.tolist()))
```

```
Score:  [0.04588045924901962]
This is just is
Score:  [0.1578938215970993]
This is just a
Score:  [0.17527997493743896]
This is just a
Score:  [0.13700434565544128]
This is just,

```

Hi, welcome back 🤗!

We now have a document for perplexity especially: https://huggingface.co/transformers/perplexity.html

Could you take a look and let me know if it's helpful for your use-case?
[Good first issue] BERT Generation PyTorch Integration tests
Author: LysandreJik
Pull request: no
Labels: name: PyTorch

name: Tests

name: Good First Issue

The PyTorch implementation of the BERT for generation model currently has no integration tests. This is problematic as the behavior can diverge without being noticed.

The [test_modeling_bert_generation.py](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_bert_generation.py) file should be updated to include integration testing.

An example of a good modeling integration test is visible in the [test_modeling_bert.py#L552-L565](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_bert.py#L552-L565) file:

https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_bert.py#L552-L565

Some additional tips:
- The test must be marked as slow using the `@slow` decorator, so as to be run *daily*, and not on every commit of every branch/pull request of this repository.
- The test must be decorated with the `@require_torch` decorator so as to only be run in environments using PyTorch.
- A single test is necessary. If you feel like implementing multiple of these, then sharing the same checkpoint would be ideal so as to reduce download time.

Comments:
@LysandreJik  I think this should be closed by now!

You're correct! Thanks again @sadakmed!
Line endings should be LF across repo and not CRLF
Author: LysandreJik
Pull request: yes

Up to now we've had quite a few issues with users on Windows having their line endings be "CRLF" by default, while Linux users have "LF" line endings by default.

### Problem

This can be problematic in the following scenarios where no handling of the issue has been done on the user's side:
- When a user runs `make style`, their line endings will switch from LF to CRLF in *all* files, essentially rewriting the entire file
- When a user adds a new file to the repository, it will be in the "CRLF" format and will be committed as such.

### Resolution

The resolution is either to have the user handle that, or to handle that ourselves. Handling it ourselves is simple as it only requires adding a `.gitattributes` file at the root of the repository which will specify the line endings we're looking for, thus this is what this PR is proposing. On the other hand, we had issues handling it on the user side with the proposed `git core.autocrlf` as it seemed to have different results according to the setup.

Additionally, if users already have files in `CRLF` mode, then an additional command is required to convert these files to `LF`: `git add --renormalize .`. I believe this only impacts users that created files previous to this PR, as newly created files will already benefit from the `.gitattributes` file.

---

This PR completely reformats two files: `examples/research_projects/bertology/run_prune_gpt.py` and `tests/test_modeling_deberta.py`. These files had CRLF line endings, and will now have LF line endings.

---

Further readings:
- [🙏 Please Add .gitattributes To Your Git Repository](https://dev.to/deadlybyte/please-add-gitattributes-to-your-git-repository-1jld)
- [Why should I use core.autocrlf=true in Git?](https://stackoverflow.com/questions/2825428/why-should-i-use-core-autocrlf-true-in-git)
- [git replacing LF with CRLF](https://stackoverflow.com/questions/1967370/git-replacing-lf-with-crlf?noredirect=1&lq=1)

cc @NielsRogge 

Comments:
Thank you @LysandreJik, this will alleviate much of the pain Windows users had with git in the past!

To hack my way up the contributors list, can I change all line endings to CRLF then revert? 😂
[DeepSpeed] restore memory for evaluation
Author: stas00
Pull request: yes
Labels: name: DeepSpeed

I spent some time trying to see if we could gain from DeepSpeed during inference - and while in the future there will be goodies to make it useful at the moment we don't need it, so let's make DeepSpeed cleanly contained to `train` for now.
 
This PR has a few small tweaks:
- frees up all the memory used by DeepSpeed at the end of training
- makes a clean way of not switching `model.to()` - only for when `--do_train` is used with deepspeed (so this is the case where you @sgugger were concerned about eval before train - no problem now)
- adds a warning if a user tries to use `--deepspeed` without `--do_train`
- re-works the test suite
- applies consistent json config formatting

@sgugger, @LysandreJik 

Comments:
[License info] Longformer SQuAD finetuned model
Author: guillaume-be
Pull request: no
Labels: name: New model

Hello @patil-suraj ,

would it be possible to provide licensing information for the pretrained model weights shared at:
https://huggingface.co/valhalla/longformer-base-4096-finetuned-squadv1

I would be interested in offering a Rust implementation for this model, but would like to know under which license this model was shared so that I can document my codebase accordingly.

Thank you!

Comments:
Hey @guillaume-be, glad to know that you are offering Rust implementation of this model :)

There's no license currently, but I'll add MIT license to this model. 

Hello @patil-suraj ,
Could you please share an update on this issue? 
Thank you!

Hi @guillaume-be 
I just added an MIT license  https://huggingface.co/valhalla/longformer-base-4096-finetuned-squadv1/blob/main/LICENSE

@patil-suraj also referenced it from your model card's YAML: https://huggingface.co/valhalla/longformer-base-4096-finetuned-squadv1/commit/1ad74ed17896eb4d3a314b1acedefbfc184cc582 so that it's reflected in the model tags etc.

Thanks Julien ! 

This is great thank you!
Improve the `run_xlni` example to use the Datasets library
Author: sgugger
Pull request: no
Labels: name: Good Second Issue

The [`run_xlni`](https://github.com/huggingface/transformers/blob/master/examples/text-classification/run_xnli.py) example should be improved (following the model of [`run_glue`](https://github.com/huggingface/transformers/blob/master/examples/text-classification/run_glue.py)) to use the Datasets library to download and preprocess the datasets.

Ideally, copying the `run_glue` example and adapting the relevant parts should be the way to go.

Comments:
Really good idea! I was looking at the `run_xnli` script the last days - because I wanted to test the "Language-Agnostic" models from [here](https://github.com/AIPHES/Language-Agnostic-Contextualized-Encoders) - and datasets integration would make experiments a lot more easier.
Add new community notebook - Blenderbot
Author: lordtt13
Pull request: yes

Updated community.md file to add new notebook, How to fine tune T5 for summarization using the Trainer API

Comments:
Thank you for your work @lordtt13 

Actually, there are already plenty of notebooks about how to fine-tune T5. It would be great if could add a notebook for missing/new models/task. Below are some of the new models which aren't used much

- T5_v1_1, mT5
- ProphetNet, xlm-prophetnet
- Blenderbot
- mBART, mBART-50

Using languages other than English would be even better, we now have so many languages in the `datasets` library after the sprint. So it's a good opportunity to use those datasets to fine-tune/evaluate multi-lingual models on them (mT5, mBART,  xlm-prophetnet) 

cc @patrickvonplaten 

I agree with @patil-suraj that notebooks on multilingual T5 would be super useful as well! 

But nevertheless, I think we can merge this notebook :-) 

Thank you for the suggestions, and yes maybe T5 has been trained on too much, I will change the notebook to have it train a different model and then request for merge.

Have trained now on BlenderBotSmall, will add multilingual model training tutorial in next PR! 
Please check @patil-suraj @patrickvonplaten
ValueError: `Checkpoint` was expecting a trackable object (an object derived from `TrackableBase`), got GPT2LMHeadModel
Author: George-Ogden
Pull request: no

I'm having this issue, and I think it's my fault, but can someone, please, advise me in case this is a bug rather than a mistake:
```
from transformers import TFTrainer, TFTrainingArguments, GPT2Tokenizer, GPT2LMHeadModel
training_args = TFTrainingArguments(
    do_train=True,
    output_dir="results",
    overwrite_output_dir=True,
    num_train_epochs=4,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=64,
    logging_dir="logs",
)
model = GPT2LMHeadModel.from_pretrained("distilgpt2")
trainer = TFTrainer(model=model,args=training_args,train_dataset=data)
trainer.train()
```
which returns the error:
```
Traceback (most recent call last):
  File ".\train-tf.py", line 52, in <module>
    trainer.train()
  File "C:\Anaconda3\lib\site-packages\transformers\trainer_tf.py", line 492, in train
    ckpt = tf.train.Checkpoint(optimizer=self.optimizer, model=self.model)
  File "C:\Anaconda3\lib\site-packages\tensorflow\python\training\tracking\util.py", line 1929, in __init__
    _assert_trackable(converted_v)
  File "C:\Anaconda3\lib\site-packages\tensorflow\python\training\tracking\util.py", line 1410, in _assert_trackable
    raise ValueError(
ValueError: `Checkpoint` was expecting a trackable object (an object derived from `TrackableBase`), got GPT2LMHeadModel(
```
and then it displays the model data.
Thanks for anything that you can add.

Comments:
Oh no! I was using the PyTorch model with the TF trainer! I have fixed it now.
Where the helsinki models downloaded to? when using the pretrained models
Author: vishnu3741
Pull request: no

src_text=['No, los préstamos existentes continuarán por debajo de la tasa de referencia existente.']

model_name='Helsinki-NLP/opus-mt-es-en'
tokenizer=MarianTokenizer.from_pretrained(model_name)
model=MarianMTModel.from_pretrained(model_name)
translated=model.generate(**tokenizer.prepare_seq2seq_batch(src_text, return_tensors="pt"))
tgt_text=[tokenizer.decode(t, skip_special_tokens=True) for t in translated]

when calling the MarianTokenizer and MarianMTModel, the package is automatically downloading the pre_trained model, is there a way to download and integrate it manually from https://huggingface.co/Helsinki-NLP/opus-mt-es-en.

Thank you.

Comments:
You can also just clone the repo: 

```
git clone https://huggingface.co/Helsinki-NLP/opus-mt-es-en
```

and then load the model and tokenizer locally from the cloned repo:

```python
model = MarianMTModel.from_pretrained("/path/to/cloned/repo")
tokenizer = MarianTokenizer.from_pretrained("/path/to/cloned/repo")
```

OSError: Can't load tokenizer for './model/opus-mt-en-es/'. Make sure that:

- './model/opus-mt-en-es/' is a correct model identifier listed on 'https://huggingface.co/models'

- or './model/opus-mt-en-es/' is the correct path to a directory containing relevant tokenizer files

I am getting this error, even though I am passing the correct path.

Thanks, it is working.
[Wav2Vec2] Improve Tokenizer & Model for batched inference
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR improves batched inference for Wav2Vec2 models by:

- adding an `attention_mask`
- adding zero-mean unit-variance normalization to the tokenizer
- correctly setting returning `attention_mask` and doing normalization depending on which architecture is used

## Background

Some of Fairseq's Wav2Vec2 models apply Group Normalization over the time axis in the feature extractor. This means that the convolutional layers in the feature extractor can not 100% correctly treat padded input resulting in those models giving different results depending on whether the input is padded or not. See https://github.com/pytorch/fairseq/issues/3227 . Those models should never make use of `attention_mask` which is made sure by setting `return_attention_mask=False` in their corresponding tokenizer configs: https://huggingface.co/facebook/wav2vec2-base-960h/blob/main/tokenizer_config.json . Also some explicit warnigs have been added to both the tokenizer and model.

For the "newer" models however that have the improved layer norm architecture in the feature extraction: https://huggingface.co/facebook/wav2vec2-large-960h-lv60-self , normalization and correct padding via `attention_mask` gives some nice performance improvements and works correctly.

## Performance Evaluation

 I've evaluated both `wav2vec2-large-960h-lv60-self` and `wav2vec2-large-960h-lv60` on the test set of librispeech and got some nice improvements:

- `wav2vec2-large-960h-lv60-self`: 2.2 WER -> 1.8 WER
- `wav2vec2-large-960h-lv60`: 3.4 WER -> 2.2 WER 

So that the results now seem to match the paper's results very nicely.

Also, I checked that `wav2vec2-base-960h` should **not** use an `attention_mask` as the performance on librispeech test then drop heavily from ~4 WER to ~20 WER.

## TODO

Once this PR is merged, I can fully focus on adding the fine-tuning functionality and will also update the model cards with the new evaluation code & results.

Comments:
Go Patrick!!!!! YES! someone who cares!

Merging since @LysandreJik is off today and this is blocking me a bit
Seeking clarification on T5  prefix for summarization
Author: ari9dam
Pull request: no

In the paper, I see the the prefix for summarization is "TL;DR:" . If I look into the model [config.json](https://huggingface.co/t5-base/blob/main/config.json)  of T5-Base, I see it is "summarization:".

"task_specific_params": {
    "summarization": {
      "early_stopping": true,
      "length_penalty": 2.0,
      "max_length": 200,
      "min_length": 30,
      "no_repeat_ngram_size": 3,
      "num_beams": 4,
      "prefix": "summarize: "
    },
  .....
 If I want to finetune Huggingface T5 for summarization, which prefix should I use?

Thank you


Comments:
Hi @ari9dam 
Please use the [forum](https://discuss.huggingface.co/) for such questions, and there's a discussion about this in the post 
https://discuss.huggingface.co/t/t5-finetuning-tips/684
Supporting truncation from both ends of the sequence in BertTokenizerFast
Author: shangw-nvidia
Pull request: no
Labels: name: Feature request

# 🚀 Feature request

For `BertTokenizerFast` (inherited from `PreTrainedTokenizerFast`), it seems like `__call__` only supports truncating from the end of the sequences if we set `truncation` to be `longest_first`, `only_first` or `only_second`. For example, assuming `max_length` is 6 and `truncation` is `longest_first`:

(`I have a pen`, `I have an apple`) --> truncation --> (`I have a`, `I have an`)

However, if we take a closer look at [Google's original data-preprocessing script for BERT](https://github.com/google-research/bert/blob/master/create_pretraining_data.py#L430), truncation can happen at both ends of the sequence randomly:

(`I have a pen`, `I have an apple`) --> truncation --> (`I have a`, `have an apple`) or (`have a pen`, `I have an`) or (`I have a`, `I have an`) or (`have a pen`, `have an apple`)

For `BertTokenizer`, perhaps I could reassigned its `truncate_sequences` member function (https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils_base.py#L2887) with a new function that implements Google's truncation scheme; however, for `BertTokenizerFast`, truncation is handled completely in Rust, about which I can't do anything.

An alternative is to call `tokenize` first, then truncate the sequence using Google's scheme, ~~then call `__call__` and passing `is_split_into_words` as `True`~~. However, this approach has significant performance impact comparing to calling `__call__` on a batch of sequences directly (the average total tokenization latency doubled in our experiments).

> PS: Turned out `is_split_into_words` doesn't work this way (since when it sees a subword `##abc`, `__call__` would further tokenize it into `#` `#` `abc` even if `is_split_into_words==True`). Thus, the actual (but slow) alternative is to 1) call `tokenize` 2) implement the truncation scheme and making sure a subword starting with `##` won't be at the boundary 3) call `convert_tokens_to_string` 4) call `__call__`. Effectively, this alternative tokenizes the same sequence twice.

I'm wondering if's possible to add official support for random truncation from both ends of the sequence?

## Motivation

To match Google's truncation scheme exactly and minimizing artificial impacts on pretraining convergence.

## Your contribution

Unfortunately I'm not very familiar with Rust (I can read it, but I neve learned/wrote Rust before), thus I can't help much.


Comments:
Hi, thanks for opening an issue! We have the `padding_side` tokenizer attribute, but it doesn't work for truncation unfortunately.
@n1t0, what do you think?

@LysandreJik Thanks a lot for your response! @n1t0  I'm wondering what your thoughts are on this feature?
model.generate needs BART config update
Author: swethmandava
Pull request: no

### Who can help
@patrickvonplaten @patil-suraj 



```
model = BartForConditionalGeneration.from_pretrained('facebook/bart-large-cnn')

tokenizer = BartTokenizer.from_pretrained('facebook/bart-large-cnn')

ARTICLE_TO_SUMMARIZE = "My friends are cool but they eat too many carbs."
inputs = tokenizer([ARTICLE_TO_SUMMARIZE], max_length=1024, return_tensors='pt')

# Generate Summary
summary_ids = model.generate(inputs['input_ids'], num_beams=4, max_length=5, early_stopping=True)
```

model.generate runs into errors. num_beam_groups, return_dict_in_generate and encoder_no_repeat_ngram_size are not defined in BART config. Should they be added?

Comments:
Hey @swethmandava,

Sorry I don't understand the issue here - what do you mean by `model.generate` runs into errors? Your above code snippet works fine for me. Could you clarify the issue? Thank you!

`summary_ids = model.generate(inputs['input_ids'], num_beams=4, max_length=5, early_stopping=True, num_beam_groups=1, output_scores=False, return_dict_in_generate=False, encoder_no_repeat_ngram_size=0, diversity_penalty=0.0)
`

works for me. I have to define the following defaults (num_beam_groups, output_scores, return_dict_in_generate, encoder_no_repeat_ngram_size, diversity_penalty) explicitly since they are not in BARTConfig and default to None

Hey @swethmandava 

You shouldn't need to define these param. All these config params have default values defined in the `PretrainedConfig` class from which all other configs inherit. 

Could you try again with the newest transformers version?
Tenosrflow Loading the saved Model For GPT2
Author: santhoshkolloju
Pull request: no
Labels: name: wontfix

```py
from time import time
from transformers import TFGPT2LMHeadModel, GPT2Tokenizer
import tensorflow as tf

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = TFGPT2LMHeadModel.from_pretrained('gpt2')

text = "What are you doing after you have finished working?"
generated = tokenizer.encode(text)
context = tf.constant([generated])
past = None
start = time()
for i in range(100):
    output, past = model([context, past])
    logits = output[0, -1, :]
    tok = tf.argmax(logits)
    generated.append(tok.numpy())
    context = tf.expand_dims(tf.expand_dims(tok, 0), 0)
sequence = tokenizer.decode(generated)
print(time() - start, sequence)

#save the model 
tf.saved_model.save(model,"temp")

#loading back the model
nm = tf.saved_model.load("temp")
infer = nm.signatures['serving_default']
```
Not able to call the model as shown below

```py
text = "What are you doing after you have finished working?"
generated = tokenizer.encode(text)
context = tf.constant([generated])
past = None
start = time()
for i in range(100):
    output, past = **infer**([context, past])
    logits = output[0, -1, :]
    tok = tf.argmax(logits)
    generated.append(tok.numpy())
    context = tf.expand_dims(tf.expand_dims(tok, 0), 0)
sequence = tokenizer.decode(generated)
print(time() - start, sequence)
```

Comments:
~Hi! What's the problem?~ Edited your message so that we can read it.



Could you put the error you had, as well as the environment? i.e., everything asked in the issue template.

I am running it in colab CPU

`TypeError`                                 Traceback (most recent call last)
<ipython-input-5-0c3c85adea42> in <module>()
      5 start = time()
      6 for i in range(100):
----> 7     output, past = infer([context, past])
      8     logits = output[0, -1, :]
      9     tok = tf.argmax(logits)

2 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in _call_with_flat_signature(self, args, kwargs, cancellation_manager)
   1719         raise TypeError("{}: expected argument #{}(zero-based) to be a Tensor; "
   1720                         "got {} ({})".format(self._flat_signature_summary(), i,
-> 1721                                              type(arg).__name__, str(arg)))
   1722     return self._call_flat(args, self.captured_inputs, cancellation_manager)
   1723 

TypeError: signature_wrapper(input_ids): expected argument #0(zero-based) to be a Tensor; got list ([<tf.Tensor: shape=(1, 10), dtype=int32, numpy=
array([[2061,  389,  345, 1804,  706,  345,  423, 5201, 1762,   30]],
      dtype=int32)>, None])``

> `

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


What is the resolution for this. Having the same issue.

If you can open a new issue with the issue template filled out (environment information, code that fails, expected behavior), then we can help you! Thank you.