noamaanMulla-03/transformers-issues huggingface dataset:
[file_utils] do not gobble certain kinds of requests.ConnectionError
Author: julien-c
Pull request: yes

Backport from https://github.com/huggingface/huggingface_hub/pull/14/commits/34b7b70d07ab1c9fc2f7da603d47cb344e256af6

might close (or at the very least provide more transparency into) #8690, #10067, and others

Comments:
@sgugger Definitely on my radar at some point.

For now though it's useful for me to have a more experimental codebase where the API can change/break :)

> LGTM but I don't have sufficient `requests` knowledge to be sure this catches all exceptions that we want to catch

we're in the same boat, sailor

In order to validate your strategy, I took the list of every exception in the public API of requests: https://2.python-requests.org/en/master/api/#exceptions

I built the inheritance tree between exceptions: https://requests.readthedocs.io/en/master/_modules/requests/exceptions/

Here's a readable version:

```
IOError
+-- RequestException 
    +-- HTTPError
    +-- ConnectionError
    |  +-- ProxyError
    |  +-- SSLError
    |  +-- ConnectTimeout (also inherits Timeout)
    +-- Timeout
    |  +-- ConnectTimeout (also inherits ConnectionError)
    |  +-- ReadTimeout
    +-- URLRequired
    +-- TooManyRedirects
    +-- MissingSchema (also inherits ValueError)
    +-- InvalidSchema (also inherits ValueError)
    +-- InvalidURL (also inherits ValueError)
    |  +-- InvalidProxyURL
    +-- InvalidHeader (also inherits ValueError)
    +-- ChunkedEncodingError
    +-- ContentDecodingError (also inherits urllib3.exceptions.HTTPError)
    +-- StreamConsumedError (also inherits TypeError)
    +-- RetryError
    +-- UnrewindableBodyError
```

Multiple inheritance is most likely for backwards-compatibility when changing exception tyes. For example, if you want to raise the more accurate `ContentDecodingError` instead of a generic `urllib3.exceptions.HTTPError`, making `ContentDecodingError` inherit `urllib3.exceptions.HTTPError` ensures you don't break the code of users who used to catch `urllib3.exceptions.HTTPError`.

I assume you want to tell apart situations where it's worth retrying from situations it isn't worth retrying, because there's a configuration issue that won't solve itself by retrying.

Based on the above, on the documentation, and on a quick search in the source code of requests, what you're doing looks correct to me, modulo my suggestion on the style.

Applied most of your comments @aaugustin, merging after internal review from @julien-c

👍
fix backend tokenizer args override: key mismatch
Author: theo-m
Pull request: yes

# What does this PR do?

Related to #10390

Turns out it was a simple key mismatch - leaving as draft for now just to see the results of the full test suites, but hopeful this will fix the main problem for the related issue.

Comments:
I'm talking with @n1t0 soon to get a better sense of where it might make sense as well yes! Will update then

went looking for similar case with this regex `(\[|\()"do_lower_case`, found only one more: 7e42461

I think this is it, will move onto #10121 once this is merged

There does seem to be an issue remaining though, as the tests in the suite currently fail with:
```
=========================== short test summary info ============================
FAILED tests/test_tokenization_auto.py::AutoTokenizerTest::test_do_lower_case
=== 1 failed, 5143 passed, 2103 skipped, 2207 warnings in 266.40s (0:04:26) ====
```
Truncated words on GPT-2 output
Author: Vova-B
Pull request: no

Hi! I use the GPT-2 model for the seq2seq task, but unfortunately, at the output of the model, words are cut off and sentences are not added, how can I make the model end sentences at the output and not cut off words? (increasing the maximum length does not correct the situation)
P.S.
I'm sorry, this question is probably very stupid, but I just can't figure it out.

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!
Expand a bit the presentation of examples
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds a bit more information to the examples README (main and specific per example), copying some information from the main philosophy and expanding a bit, to make sure all users know what we want for the examples.

Comments:
I'd be super-handy to link directly to suitable datasets and models for each example as in 
- https://huggingface.co/datasets?search=squad
- https://huggingface.co/models?filter=squad

may be this could be an easy first good issue.

Some of the keywords and whether to use `?filter=` or `?search=` will require some investigation since the former is hidden and packs some power missing from the latter.

The first may be helpful, but the second is not necessarily: it shows the models that have been fine-tuned on a squad dataset, not the models that can be fine-tuned on it. There is no way to filter all the models that have an architecture containing a question-answering head as far as I know, which is what we would want to show.

Would this be at least in the right direction? https://huggingface.co/models?pipeline_tag=question-answering 


Mmm, those seem to be models fine-tuned on a question-answering task, not all models with a QuestionAnswering arch available (for instance, you should see all BERT checkpoints, all distilBERT checkpoints etc).

OK, then it won't work.

It'd be really awesome if in the future we had a filter to filter models by architecture - and sub-architecture in this case - that is without the model-specific part of the class name.
How to get a probability for the result of t5_tokenizer.decode(output,...)?
Author: LianaN
Pull request: no

Hello,

I am using `t5-base` to map phrases into categories, for example: "I want to eat" -> "hunger".
Is there any way to get the probability for `result` values?

For example, if the input is "He is hungry", the model returns 5 labels. These results seem to be ordered by some relevance rank, so that the most relevant label is always first in `outputs`. So, my question is how can I retrieve these probabilities?

My final goal is to set a threshold on the probability, so that `outputs` would only include results that pass this threshold, or it can be empty if nothing relevant found. 

```
t5_tokenizer = T5Tokenizer.from_pretrained('t5-base')
t5_model = T5ForConditionalGeneration.from_pretrained('t5-base')
...
model.model.eval()
outputs = model.model.generate(
    input_ids=test_input_ids,attention_mask=test_attention_mask,
    max_length=64,
    early_stopping=True,
    num_beams=10,
    num_return_sequences=5,
    no_repeat_ngram_size=2
)

for output in outputs:
    result = t5_tokenizer.decode(output, skip_special_tokens=True, clean_up_tokenization_spaces=True)
    print(result)
```

Thanks.

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!
Unable to reduce time in summarization!
Author: divyanshugit
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:  `t5-large`
- Platform:  Amazon Sagemaker
- Python version: 3.7
- Tensorflow version (GPU?):2.3
- Using distributed or parallel set-up in script?:  Unable to implement it properly.

### Who can help
@LysandreJik, @patil-suraj, @jplu

### Problem:
Unable to reduce the summarization time.

The tasks I am working on is:

I am using pretrained transformer of T5(`TFT5ForConditionalGeneration`)  for text summarization.
Brief script:
```
inputs = tokenizer("summarize: " + text, return_tensors="tf").input_ids

outputs = model.generate(
        inputs, 
        max_length=200, 
        min_length=5,  
        num_beams=5,)
```
I tried to use distributed strategy of tensorflow. But it doesn't made any improvement.
```
strategy = tf.distribute.MirroredStrategy()
  
strategy = tf.distribute.MirroredStrategy(devices=["/gpu:0", "/gpu:1"])

```

## Expected behavior
I am hoping that if we increase the number of **GPU**, time must be reduced. But It is not happening in this case.


Comments:
Hello!

The generation part of TensorFlow can only be run on eagermode, hence doesn't matter how you execute it, you will not be able to run it "fast". It is planned to bring a graph execution for the generation, but no ETA yet. Sorry for the inconvenience.
[Example] Updating Question Answering examples for Predict Stage
Author: bhadreshpsavani
Pull request: yes

# What does this PR do?

Fixes #10482
1. It fixes the error that comes while using SQAUD_V2 on question-answering task using `max_val_sample_***` 
2. Adds predict method for question-answering examples

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@stas00 @sgugger 

Comments:
Sort init import
Author: sgugger
Pull request: yes

# What does this PR do?

Not a high-priority item but I get bored at nights and I like writing those kinds of scripts 😅

So this PR adds a script to properly sort the import inside `_import_structure` because people have been absolutely ruthless and putting their objects in any kind of random order. That's not very feng-shui so I'm bringing back harmony by having the same sort as isort applied to all `__init__` that contain an `_import_structure`.


Comments:
To address your comments on the `MakeFile`, I have removed some checks from `extra_quality_checks` because they are checks that modify content and `make quality` is only supposed to check, not change.

To have `make fixup` still work as intended, I put the checks that change content in `extra_style_checks` that is called both by `make fixup` and `make style`. Could you double-check it looks okay @LysandreJik and @stas00 ? Thanks!
Running "convert_graph_to_onnx.py" doesn't work.
Author: Jorghi12
Pull request: no

## Environment info
- `transformers` version: 4.5.0.dev0
- Platform: Darwin-19.6.0-x86_64-i386-64bit
- Python version: 3.7.2
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@patrickvonplaten, @patil-suraj

Models:
Bart (https://huggingface.co/transformers/model_doc/marian.html)

 -->

## Information

A problem arises when I run "python3 convert_graph_to_onnx.py" where I receive the following error message
```
Traceback (most recent call last):
  File "convert_graph_to_onnx.py", line 22, in <module>
    from .file_utils import ModelOutput, is_tf_available, is_torch_available
ModuleNotFoundError: No module named '__main__.file_utils'; '__main__' is not a package
```

## To reproduce
Run "python3 convert_graph_to_onnx.py" inside the following directory transformers/src/transformers.

Steps to reproduce the behavior:

1. cd "transformers/src/transformers/"
2. python3 convert_graph_to_onnx.py

## Expected behavior

I expect convert_graph_to_onnx.py to begin running.

Comments:
This appears to be the proper way to run it.

`python3 -m transformers.convert_graph_to_onnx --framework pt --model bert-base-cased bert-base-cased.onnx`
Stacked Roberta run_mlm.py
Author: matteomedioli
Pull request: no

# 🖥 Benchmarking `transformers`

## Benchmark

I try to run [transformers/experiments/language_modeling/run_mlm.py](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py) in order to train Roberta from scratch on Wikipedia dataset.

## Set-up
Transformers version: 4.4.0.dev0
4 GPUs: NVIDIA Tesla V100 16 GB; 4096 Bit, PCI Express 3.0 x16.
Used bash run script:
```bash
#!/bin/bash

export CUDA_LAUNCH_BLOCKING=1

source /data/env/bin/activate

nohup python3 transformers/examples/language-modeling/run_mlm.py \
--dataset_name wikipedia \
--tokenizer_name roberta-base \
--model_type roberta \
--dataset_config_name 20200501.en \
--do_train \
--do_eval \
--learning_rate 1e-5 \
--num_train_epochs 5 \
--save_steps 5000 \
--warmup_steps=10000 \
--output_dir /data/models/wikipedia_roberta \
&
```
Tested with the script and also directly with python command (without nohup)

## Results

Code seems stacked in `trainer.py`, at the first compute_loss step, when performing inference:
```python
    def compute_loss(self, model, inputs, return_outputs=False):
        """
        How the loss is computed by Trainer. By default, all models return the loss in the first element.

        Subclass and override for custom behavior.
        """
        if self.label_smoother is not None and "labels" in inputs:
            labels = inputs.pop("labels")
        else:
            labels = None
        print("STACKED HERE")
        outputs = model(**inputs)
...
```
I can't understand which running parameters are wrong. Could inference for a single batch take more than 30 mins?
Thanks in advance!

UPDATE: Without --warmup_steps is working.

Comments:
Improve PyTorch examples for FP16
Author: sgugger
Pull request: no
Labels: name: Good First Issue

To get the full speed-up of FP16 training, every tensor passed through the model should have all its dimensions be a multiple of 8. In the new PyTorch examples, when using dynamic padding, the tensors are padded to the length of the biggest sentence of the batch, but that number is not necessarily a multiple of 8.

The examples should be improved to pass along the option `pad_to_multiple_of=8` when `fp16` is True, if using a data collator that applies padding (or replace the `None` passed along to `Trainer` for `data_collator` by a `DataCollatorWithPadding(tokenizer, pad_to_multiple_of=8)`).

Comments:
Hi @sgugger 
This is not done, although closed, please have a look into https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py 
for line-by-line padding is not multiple of 8, thanks
Sagemaker Model Parallel tensoboard writing fix
Author: mansimane
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->


Fixes # 10402

https://github.com/huggingface/transformers/issues/10402

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@LysandreJik @sgugger 





Comments:
Option to change loss function for fine tuning
Author: frankhart2018
Pull request: no

# 🚀 Feature request

## Motivation

I was working in a multi class text classification problem for which I was using `DistilBertForSequenceClassification` and I found out that there is no way for me to change the loss function from CrossEntropyLoss. 

## Your contribution

I can submit a PR, if this feature request is approved. 


Comments:
You can change the loss function to anything you want. Here's an example:

```
from transformers import BertModel
from transformers.modeling_outputs import SequenceClassifierOutput
import torch.nn as nn

class FancyBertModelWithCustomLossFunction(nn.Module):
    def __init__(self):
        super(FancyBertModelWithCustomLossFunction, self).__init__()
        self.bert = BertModel.from_pretrained("bert-base-uncased")
        self.dropout = nn.Dropout(0.3)
        self.classifier = nn.Linear(768, 1)

    def forward(self, ids, mask, token_type_ids, labels=None):
        outputs = self.bert(ids, attention_mask=mask, token_type_ids=token_type_ids)

        output = self.dropout(outputs.pooler_output)
        logits = self.classifier(output)
        
        loss = None
        if labels is not None:
            if self.num_labels == 1:
                #  We are doing regression
                loss_fct = MSELoss()
                loss = loss_fct(logits.view(-1), labels.view(-1))
            else:
                # you can define any loss function here yourself
                # see https://pytorch.org/docs/stable/nn.html#loss-functions for an overview
                loss_fct = nn.BinaryCrossEntropyLoss()
                # next, compute the loss based on logits + ground-truth labels
                loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))

        return SequenceClassifierOutput(
            loss=loss,
            logits=logits,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )

```

@NeilsRogge I am aware of this, what I was referring to is an option that can be passed directly to let's say `DistilBertForSequenceClassification` or any other model class without having to write a pytorch model like this. 

This has already been asked before, but we are not planning to do this.

See also [this comment](https://github.com/huggingface/transformers/issues/9625#issuecomment-762167788) in #9625

Oh ok, got it. Thanks @NeilsRogge. Closing this.
Domain adaptation 
Author: lematmat
Pull request: no

Hi all,

I'm just wondering how to do model adaptation of pre-trained Camembert model on my custom dataset ?

I haven't found any information in Transformers documentation.

Best regards,

Lematmat
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:4.4.0
- Platform:Jupyter Notebook
- Python version:3.7
- PyTorch version (GPU?):1.8, no GPU
- Tensorflow version (GPU?):
- Using GPU in script?:no
- Using distributed or parallel set-up in script?:no


 -->

## Information

Model I am using (Bert, XLNet ...): Camembert


Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!

Thank you very much.
I will contact the forum.
Regards,
lematmat
[vulnerability] in example deps fix
Author: stas00
Pull request: yes

Takes care of:
https://github.com/huggingface/transformers/security/dependabot/examples/research_projects/lxmert/requirements.txt/jinja2/open

@LysandreJik


Comments:
Ah actually the dependabot PR was earlier in my notifications so I merged it without seeing you had opened a PR here. Sorry about that, closing as already taken care of in https://github.com/huggingface/transformers/commit/dbfe3795147e1360b3afac53a9ee0e14374d2ea6

Actually, the proposed `>=` is probably better, so fixing conflicts and merging this. Thanks @stas00!
Fix on_step_begin and on_step_end Callback Sequencing
Author: siddk
Pull request: yes

# What does this PR do?

Currently, the Trainer exhibits the following behavior (simplified):
 
```
for step, input in epoch_iterator:
     if (step + 1) % self.args.gradient_accumulation_steps == 0:
          callback_handler.on_step_begin()
     
     ...
     if (step + 1) % self.args.gradient_accumulation_steps == 0:
          # Apply Gradient Update (Finished accumulating)
          optimizer.step()
          callback_handler.on_step_end()
```

Unfortunately, this means that `on_step_begin()` gets called during the same iteration, *before* `on_step_end()` which is incorrect, and confuses folks implementing custom callbacks for timing individual iterations (like my team!).

Instead, this fix starts by calling `on_step_begin()` at steps = 0 (iteration 0) and will only be called on the next step after `on_step_end()`. 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

Code updates part of the Trainer, so tagging @sgugger.


Comments: