noamaanMulla-03/transformers-issues huggingface dataset:
[CI] Better reports #2
Author: stas00
Pull request: yes

As discussed at https://github.com/huggingface/transformers/pull/8110 this PR:

* [x] - generates 3 types of failure reports - long, short and one-per-line
* [x] - fixes multiple test suite tasks in a single job to allow them all to run regardless of the outcome of the previous test suites (using `if: always()`
* [x] - adds a workaround for the cumbersome way github makes the artifacts available by printing the short failure report in its own tab, so getting to errors should be very easy now.

Once we perfect this hack to our liking, I intend to submit this to `pytest` and see if perhaps they would consider accepting it as a feature.

@sshleifer 

Comments:
Thanks!

OK, everything seems to be working well. Let me know if you have any comments/suggestions/recommendations before replicating this to the rest of the jobs.

See: https://github.com/huggingface/transformers/runs/1329578690?check_suite_focus=true 

I will wait for https://github.com/huggingface/transformers/pull/8007 to be merged before spreading the love to the rest of the jobs, so that they won't need to deal with a lot of conflicts.

I also proposed this a `pytest` feature: https://github.com/pytest-dev/pytest/issues/7972 - probably others would benefit from it.
Fixing some warnings in DeBerta
Author: Narsil
Pull request: yes

# What does this PR do?

Just fixes some simple warning coming from python from incorrect escapes in docstrings + `collections.abc` import.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@sgugger

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

Comments:
TF: Faster to way to set one column/all but one column of a tensor to -inf
Author: sshleifer
Pull request: no
Labels: name: TensorFlow

in `_force_token_id_to_be_generated` we have much simpler torch code:
```python
scores[:, [x for if x != token_id]] = -float("inf")
```
Is it possible to make the TF Code simpler? TF doesn't support assignment, but maybe to and from `numpy` could be faster. Would definitely be simpler.

```python
    @staticmethod
    def _force_token_id_to_be_generated(scores, token_id) -> None:
        """force one of token_ids to be generated by setting prob of all other tokens to 0 (logprob=-float("inf"))"""
        output_list = []

        # Is there a better way to do  in TF?
        bs, vocab_size = scores.shape
        inf_tensor = tf.convert_to_tensor([-float("inf")] * bs, dtype=scores.dtype)
        for x in range(vocab_size):
            if x != token_id:
                output_list.append(inf_tensor)
            else:
                output_list.append(scores[:, x])
        scores = tf.stack(output_list, axis=1, name="scores")
        assert scores.shape == (bs, vocab_size)
        return scores
```






Comments:
Solution: https://stackoverflow.com/questions/64575346/tensorflow-set-column-of-tensor-to-infinity
TFMarian, TFMbart, TFPegasus, TFBlenderbot
Author: sshleifer
Pull request: yes

### Notes:
- add `TFSinusoidalPositionalEmbeddings`.
- Code structure identical to the corresponding pytorch code -- same classes, implementations differ only slightly.
- Integration tests and common tests, rst updates, for all 4 children. All 4 children run the same common tests as TFBart and at least 1 integration test.
- For pegasus, generations are not identical to PT because Linear layers are slightly different in tf/pt. For Marian, generations are identical.
- Loading will generate 0 warnings.


Comments:
@patrickvonplaten 
+ I deleted the `_force_token_id` function, replaced with faster `tf.where` one-liner. (+ added regression test). 
+ Replaced unneeded `TFSharedEmbedding` with `tf.keras.layers.Embedding`
+ switched all `.shape` to `shape_list`

WDYT?
raining loss is not decreasing when using the Roberta pre-trained model from the transformers library
Author: ZahraAbbasiantaeb
Pull request: no

I load the Roberta pre-trained model from the transformers library and use it for the sentence-pair classification task. The loss function used to decrease during the training per epoch until the last week, but now even though all of the parameters, including the batch size and the learning rate have the same value, when I fit my model the value of the loss function is not decreasing. I am a little bit confused and I have trained my model using various parameters and also I utilized another code in PyTorch, but still, the loss function is not decreasing. Can anyone help me to figure out the problem?

here is the link to my code:
https://colab.research.google.com/drive/1CFg41KDHJSJNkehJOHbp3gfXRdva60oW?usp=sharing

and the dataset:
https://drive.google.com/drive/folders/1CUH_z_HI31-yfj8hOmRfJBKRKe_BNkku


Comments:
Hello! Could you open a post on the [forum](https://discuss.huggingface.co) instead? We try to keep issues for bugs only.

Hi, sure.
Finalize lm examples
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds a `run_mlm_wwm` script for an example of MLM with whole word masking, which was the last kind of examples supporter by `run_language_modeling`.

As a result, it moves this script to `contrib/legacy/` and udpates the README to document how to use all the new example scripts.

I also reworked a tiny bit the table of tasks in the main README to include whether or not the examples leverage the Datasets library.

Comments:
Replace swish with silu
Author: TFUsers
Pull request: yes

This pull request replaces the swish with the silu. Note "silu" still maps to tf.keras.activations.swish not tf.keras.activations.silu for tensorflow since the silu is in the tensorflow nightlies, but not in the stable version of tensorflow.

This fixes https://github.com/huggingface/transformers/issues/8100

@LysandreJik

Comments:
Thanks @TFUsers for this important PR.

As far as I know the activation names are also directly inside the the `config.json` files in the model hub. @sgugger @LysandreJik  Do we plan to update all of them?

Very good point @jplu. We can't change those names in the config hosted online because it wouldn't be backward-compatible, so we need to still accept the old names (without documenting the behavior). So we should leave the old `swish` in the dictionaries `ACT2FN`.

It looks like everything passes (ignoring "src/transformers/activations.py:52:5: F811 redefinition of unused 'silu' from line 40").

> I guess the cleanest approach in that regard would be to remove the definition of ACT2FN in these files, and instead import the centralized ACT2FN from the activations files

Line 30 of `modeling_bert.py` is
`from .activations import ACT2FN`

Are the main concerns resolved?

You're right, I was checking in the wrong file. Could you fix the code quality issue related to the redefinition of `silu`? You can follow what's done with the `gelu` method, by renaming the `silu` method to `_silu_python` and doing an if/else statement according to the torch version.

Also that version check (same with the `gelu`) doesn't seem robust at all. Could we use the `packaging` util to do something better? Something like:

```py
from packaging import version

if version.parse(torch.__version__) < version.parse("1.4"):
    ...
```

Thanks!
Rename swish to silu to give appropriate credit
Author: TFUsers
Pull request: no

The swish was originally coined the "SiLU" in https://arxiv.org/pdf/1606.08415.pdf and https://arxiv.org/abs/1702.03118 long before the swish paper. Renaming other peoples' exact same ideas is unacceptable and huggingface's naming convention implicitly erases the research and work of people outside of Google.
This request inspired by a [discussion](https://www.reddit.com/r/MachineLearning/comments/hkiyir/r_google_has_a_credit_assignment_problem_in/) and a recent [tensorflow issue](https://github.com/tensorflow/tensorflow/issues/41066), but this problem has been brought up every few months for the past few years. In light of recent efforts to make the ML community more equitable and _fair_, this is a no-brainer and long overdue.

**Will this change the current api? How?**
The API would replace the "swish" argument with the "silu" argument and deprecate the swish.
[PyTorch 1.7](https://pytorch.org/docs/1.7.0/generated/torch.nn.SiLU.html?highlight=silu) added the SiLU. Tensorflow added the [SiLU](https://github.com/tensorflow/tensorflow/blob/27d26a8d86bceda282ad9ba3e3116a00759d4ebc/tensorflow/python/ops/nn_impl.py#L517) and should be in the next version. Jax has already added the SiLU;
jax.nn.swish will eventually be deprecated and jax.nn.silu will be added and both of the aforementioned papers will be cited in the documentation.

Comments:
It is worth mentioning that PyTorch's SiLU op is an optimized implementation: https://github.com/pytorch/pytorch/pull/42976

You are correct, we should update and deprecate. Do you want to take a stab at a PR?
Fix two bugs with --logging_first_step
Author: abisee
Pull request: yes

# What does this PR do?
This PR fixes two bugs relating to the `--logging_first_step` flag:
1. Though the description for `--logging_first_step` says `"Log and eval the first global_step"`, the flag doesn't actually eval (it only logs). This PR makes sure that eval happens on the first step.
2. When `--logging_first_step` is on, the logged training loss for the first step is miscalculated in `Trainer._maybe_log_save_evaluate`:
```python
logs["loss"] = (tr_loss_scalar - self._logging_loss_scalar) / self.args.logging_steps
```
This divides the loss by `logging_steps` (which is typically large, e.g. 500), when it should be divided by 1. This PR makes sure that the loss is divided by the correct number of steps.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?
@sgugger

Comments:
Thanks for your PR! For the first point, I think we should fix the docs, not add an evaluation. It doesn't make any sense to evaluate at step 1 (one could call `trainer.evaluate()` before training if they really wanted to).
For the second point, good catch, this is certainly useful!

OK, both the description and the behavior is now logging only (no eval).

Perfect, thanks!
New model addition
Author: klaudia122195
Pull request: no
Labels: name: New model

# 🌟 New model addition

## Model description

<!-- Important information -->

## Open source status

* [x] the model implementation is available: (give details)
* [ ] the model weights are available: (give details)
* [x] who are the authors: (mention them, if possible by @gh-username)


Comments:
##

P#86

user blocked
Mmmmianam
Author: klaudia122195
Pull request: no

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Comments:
😁

#876
longformer padding logging
Author: yuvalkirstain
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

The longformer model logs the extra padding length that it adds (when the padding length is bigger than zero). Can you please add a verbose or non-verbose option to control this behavior? It basically creates a log every forward pass. 
the code is in: 
module - transformers/modeling_longformer.py 
function - _pad_to_window_size
line - 555

Thank you so much !

Comments:
Hi! This should be manageable with the logger:

```py
import transformers
import logging

logger = logging.getLogger("transformers")
logger.setLevel(logging.ERROR)
```

thank you! sounds great. However, I do think that you can perhaps just notify this behavior at the start of training rather than logging every step.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Documentation detail in (TF)RobertaForSequenceClassification 
Author: DiegoKaram
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help
The documetation for TFRobertaForSequenceClassification lacks a description for the `labels` parameter. That is not the case for RobertaForSequenceClassification. The parameters actually exists for the TF version and the documentation causes a little confusion. Not sure where to post this in the forum.

## Details
<!-- Description of your issue -->
I see that the  docstring for these classes comes from the variable `ROBERTA_INPUTS_DOCSTRING` in the [source](https://huggingface.co/transformers/_modules/transformers/modeling_roberta.html#RobertaForSequenceClassification) and the doc for labels is added after that on the pytorch model, but not on the tf model.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Image GPT
Author: minimaxir
Pull request: no
Labels: name: wontfix

name: New model

# 🌟 New model addition

## Model description

OpenAI just announced Image GPT: https://openai.com/blog/image-gpt/

Although image rendering would be out of scope for Transformers, the RGB generation would still be in scope and it would be best to port the weights to a `GPT2LMModel`.

However, it's not immediately clear here how the tokenization is implemented in the downloaded model. (no separate `vocab.json`)

## Open source status

* [ ] the model implementation is available: https://github.com/openai/image-gpt
* [ ] the model weights are available: see README above
* [ ] who are the authors: @openai


Comments:
I'd like a google colab of it 

Hey @minimaxir! Here's a [colab](https://colab.research.google.com/github/apeguero1/image-gpt/blob/master/Transformers_Image_GPT.ipynb) which loads the weights into a subclass of `GPT2LMHeadModel` and demonstrates unconditional image generation and conditional image completion. 

Some differences I've found between Image-GPT and GPT2 which are reflected in the subclass. 

1) Image-GPT layer normalization doesn't subtract off the mean
2) different activations used in the MLP
3) In Image-GPT, the input and output embeddings are not tied
4) Image-GPT has an extra learned "sos" token embedding which is concatenated at the beginning of the sequence
5) The GPT2 `[n_embd, 3*n_embd]`  dimensional linear layer, `c_attn`, which produces queries, keys, and values is instead split into 3 separate linear layers each with dimension `[n_head, n_embd/n_head, n_embd]` in Image-GPT (this only affects how to load the weights and not the actual model).
6) In Image-GPT, the `conv1d` module doesn't have a bias term

So what's our next step to add this to the repo?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@apeguero1 we have an "Adding a new model" checklist at https://github.com/huggingface/transformers/tree/master/templates/adding_a_new_model

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
missing reference `from model_bertabs import BertAbsSummarizer`
Author: MarcoGorelli
Pull request: no
Labels: name: wontfix

[This line](https://github.com/huggingface/transformers/blob/1a779ad7ecb9e5215b6bd1cfa0153469d37e4274/examples/seq2seq/bertabs/convert_bertabs_original_pytorch_checkpoint.py#L28) reads

> from model_bertabs import BertAbsSummarizer

yet neither `model_bertabs` not `BertAbsSummarizer` can be found in the repository

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
words of overflowing_tokens in function truncate_sequences is not in right order
Author: cin-levi
Pull request: no
Labels: name: wontfix

https://github.com/huggingface/transformers/blob/6b4c617666fd26646d44d54f0c45dfe1332b12ca/src/transformers/tokenization_utils_base.py#L2570
```
                   if not overflowing_tokens:
                        window_len = min(len(pair_ids), stride + 1)
                    else:
                        window_len = 1
                    overflowing_tokens.extend(pair_ids[-window_len:])
                    pair_ids = pair_ids[:-1]
```
In my understanding, the overflowing_tokens is a sub sequences of the second sequence (pair_ids). But in this code, the order of words in overflowing_tokens is not same as the pair_ids (and also different with the reversed sequence of pair_ids if window_len != 1).

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Questions on the date of Wikipedia dumps for pretrained checkpoints (BERT and RoBERTa).
Author: woojeongjin
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

Hi, I really appreciate the Huggingface and the fantastic code for pretrained LMs.
I am trying to figure out the date of Wikipedia dumps used for pretrained checkpoints. (BERT and RoBERTa)



<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->
I'd like to know the date (month, year) of Wikipedia dumps that were used for the current pretrained checkpoints of BERT-base uncased and RoBERTa-base and large.
I am looking for an older version of pretrained checkpoints that were trained on a Wikipedia dump before 2019.
If available, is there a way to get the older version of pretrained checkpoints (before 2019)?
Thanks!

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Value Error & dev file parameter: run_squad.py BERT QA finetuning
Author: haenvely
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

Hello, I have a question about implementing run_squad.py finetuning.
I have 2 questions.

**1. --dev_file parameter**
I have 3 datasets : train, dev, and predict file.
However, I discovered that run_squad.py finetuning seems it does not supply `--dev_file` parameter, so I can use only 2 dataset(train and predict).
Is there any way to use dev_file for evaluation? Or how can I check the accuracy score?

**2. Value Error**
I just implemented the run_squad.py script using train and predict file, and I got several errors while implementing the script.
Due to CUDA out of memory error, I set the 'max_seq_length' as 64 and per_gpu_train_batch_size' as 2, and it didn't bring CUDA out of memory error, but instead, it brought 'Valueerror expected sequence of length 64 at dim 1 (got 878) '.
Could you give some idea to fix this error?

Thank you,

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.