noamaanMulla-03/transformers-issues huggingface dataset:
regarding the max token length of longformer
Author: rkoystart
Pull request: no
Labels: name: wontfix

In the encode_plus function  of the Tokenizer , there is a argument called max_length whose default value is 4096.
So is it possible to increase the max token length  beyond 4096 or the maximum value of max_length argument is 4096
Can anyone clear my doubt

Thanks in advance !!!!

Comments:
Hi, @rkoystart 
I think this notebook will [help](https://github.com/allenai/longformer/blob/master/scripts/convert_model_to_long.ipynb)

@patil-suraj  so it means by default  the longformers model provided by huggingface supports maximum tokens of 4096 right ?
if suppose we want to pretrained model to support for even more longer sentences than 4096 we have to follow the instructions in the notebook you have mentioned above


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
QA Loss Cleanup
Author: sshleifer
Pull request: no
Labels: name: wontfix

name: cleanup

This snippet appears a lot of places and could be factored out into a `calc_qa_loss(logits)`

Requires some care, because I'm not sure how good the test coverage is, and if it doesn't improve readability we shouldn't do it.


```python
        logits = self.qa_outputs(sequence_output)
        start_logits, end_logits = logits.split(1, dim=-1)
        start_logits = start_logits.squeeze(-1)
        end_logits = end_logits.squeeze(-1)

        outputs = (start_logits, end_logits,) + outputs[2:]
        if start_positions is not None and end_positions is not None:
            # If we are on multi-GPU, split add a dimension
            if len(start_positions.size()) > 1:
                start_positions = start_positions.squeeze(-1)
            if len(end_positions.size()) > 1:
                end_positions = end_positions.squeeze(-1)
            # sometimes the start/end positions are outside our model inputs, we ignore these terms
            ignored_index = start_logits.size(1)
            start_positions.clamp_(0, ignored_index)
            end_positions.clamp_(0, ignored_index)

            loss_fct = CrossEntropyLoss(ignore_index=ignored_index)
            start_loss = loss_fct(start_logits, start_positions)
            end_loss = loss_fct(end_logits, end_positions)
            total_loss = (start_loss + end_loss) / 2
```

@stas00 

Comments:
I will experiment. Thank you!


See: https://github.com/huggingface/transformers/pull/6430

So I guess we can close this one, since the experiment wasn't accepted.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Get Scores for each NE Label
Author: Stimmot
Pull request: no

I'm running the run_ner.py script to use the bert-base-german-cased transformer model in the token classification task to train it on custom NE labels and make predictions for German documents. I have 11 labels in total.

I wondered if there is any way to get prediction results (meaning loss, accuracy, precision, etc.) not only for the whole task, but for each label individually. This would make it easier to compare the real perfomance of the model. So that each label has results like:

```
eval_loss = 0.07476427406072617
eval_accuracy_score = 0.9818217086485438
eval_precision = 0.6756756756756757
eval_recall = 0.676378772112383
eval_f1 = 0.6760270410816434
```

Reason I'm asking: the O-Labels (the only thing I kept from BIO tagging) are of course the majority of all labels, therefore the accuracy is quite high as the model correctly predicts most of them, but as a consequence the scores for the real labels get lost in the statistical noise. 

Is there any way to achieve this? 
Thanks in advance.


Comments:
Maybe @sgugger or @jplu have an idea

`seqeval` can return you more information, for instance the version in the Datasets library returns all the metrics per label IIRC. The code is [here](https://github.com/huggingface/datasets/blob/8005fed0887236804a07bfdc7dc69298e15dac7c/metrics/seqeval/seqeval.py#L96), you just need to adapt what's inside the `compute_metrics` function to fit your needs.

Hi @sgugger - that did the trick, thank you!
Readme for News Headline Generation (bert2bert)
Author: m3hrdadfi
Pull request: yes
Labels: name: model card

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSTM: @stas00
 -->


Comments:
Really cool!
Readme for Wiki Summary [Persian] bert2bert
Author: m3hrdadfi
Pull request: yes
Labels: name: model card

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSTM: @stas00
 -->


Comments:
XLM-RoBERTa tokenizer changes characters during tokenization
Author: konstantinmiller
Pull request: no

I'm using `transformers` 3.5.0.

Whenever the XLM-RoBERTa tokenizer encounters some characters such as '²' (super-script 2, `ord('²')` is 178), it converts them to other characters (in this example, to the plain '2', with `ord('2')` being 50).

That is, with `tokenizer = AutoTokenizer.from_pretrained('xlm-roberta-large')`, `tokenizer.tokenize('²')` or, altnernatively, `tokenizer.tokenize(chr(178))` returns `['▁2']`.


Comments:
Hi, this is unfortunately a limit of this tokenizer. We try to stay as close as possible to the original implementation, so we will not be able to change this behavior.

I see. Well, it's not a big deal, as the characters it replaces are quite rare and not really important, at least not for my application. So, I just replace them upfront in the text to make sure that the output of the tokenizer still matches the input.

Thanks for the awesome work you are doing by providing this and other libraries!

Glad you like it :)
"setup.py" does not seem to have been updated for v3.5.1
Author: forest1988
Pull request: no

## Environment info

I try 
```
!git clone https://github.com/huggingface/transformers.git
%cd transformers/
!pip install -e .
```
on Colaboratory.
     
- `transformers` version: 3.5.0   <- this seems strange.
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
 
 documentation: @sgugger

## Information

"setup.py" does not seem to have been updated for v3.5.1.

When I install transformers by `pip install -e .`, the version of transformers is shown as v3.5.0.

## To reproduce

Steps to reproduce the behavior:

I try 
```
!git clone https://github.com/huggingface/transformers.git
%cd transformers/
!pip install -e .
```
on Colaboratory after v3.5.1 release.

Then,
```
import transformers
transformers.__version__
```
returns 
```
'3.5.0'
```

## Expected behavior

The return of `transformers.__version__` is expected to be '3.5.1' now, if my understanding is not wrong.

Maybe, in  https://github.com/huggingface/transformers/blob/afb50c663a5d5623906ead1e87481926467d59fa/setup.py#L120 

'3.5.0' should be changed to '3.5.1'.

Is my understanding correct? Sorry if I misunderstand your intension.


Comments:
I found https://github.com/huggingface/transformers/commit/d5b3e56de5376aa85ef46e7f0325139d9e299a41 and it seems the related files are updated there.
If there is a reason (or rules for releases) not to merge the change into the master branch, I'm sorry for opening this issue that I haven't fully considered.

This was a hotfix on a branch which is why we didn't update master (mostly because we forgot).
To do things right, we'll actually put v4.0.0-dev since that's what master is right now :-)

@sgugger 
Thank you for your quick and detailed response!
Now I understood what is the reason.
I appreciate your creating a new PR for this issue.
Here maybe a bug, when we load staged checkpoint
Author: hengchao0248
Pull request: no
Labels: name: wontfix

https://github.com/huggingface/transformers/blob/40d98ebf50c4662bcd6dce6395bbed0b2142ea52/src/transformers/trainer.py#L458

I met this bug when I used the setting below:

global_steps = 2748
len(train_dataloader) = 27484
gradient_accumulation_steps = 4

In the original code, "steps_trained_in_current_epoch" will be 2748 !  BUT this variable should be  2748*4 = 10,992

the code I suggested is below:

```
epochs_trained = (self.global_step * self.args.gradient_accumulation_steps) // len(train_dataloader)
steps_trained_in_current_epoch = (self.global_step * self.args.gradient_accumulation_steps) % len(train_dataloader)
```

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I'm also puzzled by this. The calculations here seems incorrect.

To me these calculations are not incorrect if we take `step` as optimization steps, however `steps_trained_in_current_epoch` is wrongly used to skip training batches without considering gradient accumulation. 

+1 for the proposed calculation for `steps_trained_in_current_epoch` as the number of batches to be skipped.

@sgugger might be interested in this.

There is indeed a problem, but only with `steps_trained_in_current_epoch`. The `global_step` variable represents the number of optimization steps, not the number of batches seen. The variable `num_update_steps_per_epoch` take this into account so `epochs_trained` is correct. `steps_trained_in_current_epoch` represents the number of update steps to skip but is used as the number of batches to skip, so either need to multiply it by the `gradient_accumulation_steps` (and rename it for clarity) or skip  `gradient_accumulation_steps` batches before subtracting 1 to it later in the loop.

This also shows that we direly miss a test to check resuming training works with gradient accumulation. I can look into this when I have a bit of time, but will be fairly busy with the preparation for v4.
[doc] typo fix
Author: stas00
Pull request: yes

s/e.g./i.e./ as what follows is not an example, but a "that is" statement + fix language

@sgugger


Comments:
Apparently, there is no agreement on the comma: https://www.dailywritingtips.com/comma-after-i-e-and-e-g/, especially after i.e. https://english.stackexchange.com/questions/16172/should-i-always-use-a-comma-after-e-g-or-i-e. 

Therefore it's probably the best to stick to whatever manual of style this projects prefers. And, perhaps, even starting a brief guide for the `transformers` manual of style, where you also include all those stylistic recommendations, such as, not documenting `None` as a default for optional objects, the item in question, etc.

My recommendation is that every time a style change is introduced, to change all docs at once, so that the diverse developers will most likely copy/observe how the text is written and follow suite when they create new docs. It's not hard to do:

```
find . -type d -name ".git" -prune -o -type f -exec perl -pi -e "s|e\.g\. |e.g., |g; s|i\.e\. |i.e., |g;" {} \;
```
this one fixes a missing comma.

Thanks @stas00 !

I had missed your comment. We can do the perl magic thing (though check me with privately before applying it as we are scheduling a few big changes today and tomorrow in preparation for the v4.0 so you don't want to do this in the middle of one ;-) ).

As for the documentation, we can add it to our [documentation guide](https://github.com/huggingface/transformers/tree/master/docs#writing-documentation---specification) (which already contains the not defaults to `None`).

> I had missed your comment. We can do the perl magic thing (though check me with privately before applying it as we are scheduling a few big changes today and tomorrow in preparation for the v4.0 so you don't want to do this in the middle of one ;-) ).

You can run it any time you know it's a good time. All credits go to perl ;)
 
> As for the documentation, we can add it to our [documentation guide](https://github.com/huggingface/transformers/tree/master/docs#writing-documentation---specification) (which already contains the not defaults to `None`).

That works. Probably start a "grammar style" section.
`log_history` does not contain metrics anymore
Author: PhilipMay
Pull request: no

Since version 3.5.0 the `log_history` of the trainer does not contain the metrics anymore. Version 3.4.0 works...

My trainer uses a `compute_metrics` callback. It avaluates after each epoch. At version 3.4.0 after the training I am extracting the last epoch results: `trainer.state.log_history[-1]` to log the metrics.

At version 3.5.0 the dict only contains loss and epoch number but not the computed metrics. 

I think anything was changed that broke the metric logging. I can not provide example code. Sorry...



Comments:
`evaluate` calls `log` which appends the results to `log_history`. So the code is there. Without a reproducer to investigate, there is nothing we can do to help.

Here is the demo code that shows the bug: https://colab.research.google.com/drive/1dEzkDoMampL-VVrQeO924HQmHXffya0Z?usp=sharing

The last line should print all metrics and does that with version 3.4.0 but not with 3.5.0

Output of 3.4.0 (which is correct):
```
{'eval_loss': 0.5401068925857544, 'eval_f1_OTHER': 0.8642232403165347, 'eval_f1_OFFENSE': 0.6730190571715146, 'eval_recall_OTHER': 0.9230427046263345, 'eval_recall_OFFENSE': 0.5834782608695652, 'eval_acc': 0.8081224249558564, 'eval_bac': 0.7532604827479499, 'eval_mcc': 0.5547059570919702, 'eval_f1_macro': 0.7686211487440247, 'epoch': 2.0, 'total_flos': 668448673730400, 'step': 628}
```


Bug in 3.5.0:
```
{'total_flos': 668448673730400, 'epoch': 2.0, 'step': 628}
```

Ah it's not a bug. In 3.5.0 there is one final log entry for the total_flos (instead of logging them during training as it's only useful at the end). So you can still access all your metrics but with the second-to-last entry (`trainer.state.log_history[-2]`).

Hi @sgugger ok thanks for the info.
It might be no bug but honestly. This logging "API" is very fragile. What happens if I do a `[-2]` now and in the next release the final log entry for the total_flos is moved to an other list. Then I am getting the result of the 2nd last epoch instead of the last one.

IMO this logging "API" needs a clean and better redesign. Or do I just use it in a wrong way?

There are plenty of things that could log more info: a callback, some other tweak in training performed at the end. IMO you shouldn't rely on a hard-coded index but loop from the end of the `log_history` until you find a dict with the metric values.

Ok. Closing this.
RagSequenceForGeneration how to get document texts retrieved  in response to a query
Author: mchari
Pull request: no

When I run the retriever separately, how can I find out the text of the documents (from the doc_ids ?) that are retrieved ?

I created the retriever using:
retriever = RagRetriever.from_pretrained(rag_example_args.rag_model_name,index_name="custom",passages_path=passages_path,index_path=index_path,n_docs=8)
tokenizer = RagTokenizer.from_pretrained(rag_example_args.rag_model_name)

model = RagSequenceForGeneration.from_pretrained(rag_example_args.rag_model_name,index_name="custom",indexed_dataset=dataset)

question_hidden_states = model.question_encoder(input_ids)[0]
docs_dict = retriever(input_ids.numpy(), question_hidden_states.detach().numpy(), return_tensors="pt")


Comments:
retriever.index.get_doc_dicts(docs_dict["doc_ids"])[0]['text']  gets me the text of the retrieved documents.