noamaanMulla-03/transformers-issues huggingface dataset:
You must specify an aggregation method to update a MirroredVariable in Replica Context.
Author: iamneerajverma
Pull request: no
Labels: name: wontfix

name: TensorFlow

name: Usage

# You must specify an aggregation method to update a MirroredVariable in Replica Context.

##  
   <ipython-input-28-7cf32baaf070>:52 step_fn  *
        gradient_accumulator(grads)
    /tensorflow-2.1.0/python3.6/tensorflow_core/python/distribute/distribute_lib.py:763 experimental_run_v2
        return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
    /usr/local/lib/python3.6/dist-packages/transformers/optimization_tf.py:229 __call__  *
        accum_gradient.assign_add(gradient)
    /tensorflow-2.1.0/python3.6/tensorflow_core/python/distribute/values.py:1124 assign_add
        return self._assign_func(f=assign_add_fn, *args, **kwargs)
    /tensorflow-2.1.0/python3.6/tensorflow_core/python/distribute/values.py:1108 _assign_func
        variable_type="MirroredVariable"))

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

using GPU: Yes 

The problem arises when using:
 Training on multiple gpus and accumulating gradient as given in run_tf_ner.py



Comments:
Tried executing the test case: **test_optimization_tf.py**
The test case also fails when on GPU.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Support DeepSpeed for language modeling finetuning
Author: minimaxir
Pull request: no
Labels: name: wontfix

name: Ex: LM (Finetuning)

name: Ex: LM (Pretraining)

name: External

# 🚀 Feature request

https://github.com/microsoft/DeepSpeed

This was just released, and given the code flow in `run_language_modeling.py` it seems like it would not be too difficult to drop-in, and it has a permissible license (MIT).

However, given the dependencies and difficulty installing them, it would likely have to be done in a separate file.

## Motivation

![](https://i.kym-cdn.com/entries/icons/mobile/000/006/360/gottago.jpg)

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
output padding different to zero in hidden layers with attention mask
Author: ShiroKL
Pull request: no
Labels: name: wontfix

name: PyTorch

name: Core: Modeling

name: Usage

# 🐛 Bug
On the last layer, the token corresponding to the padding does not return 0, even if attention masking is used.
## Information

Model I am using (Bert, XLNet ...):
roberta and roberta XLM

Language I am using the model on (English, Chinese ...):
english

The problem arises when using:
* [x] the official example scripts: (give details below)


The tasks I am working on is:
looking at the output of the last layer of RobertaModel

## To reproduce

Steps to reproduce the behavior:

1. use some padding in your inputs data
2. create accordingly the attention mask
3.  

example of code :
```

def tokenize_sentences_Bert(sentences, tokenizer, maxlen):
    tokens = []
    lengths = []
    for s in sentences:
        token = tokenizer.encode(s, add_special_tokens=True, max_length=maxlen)
        lengths.append(len(token))
        token = token + [tokenizer.pad_token_id]  * (maxlen - len(token) )

        tokens.append(token)

    return tokens, lengths
tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
model = RobertaModel.from_pretrained('roberta-base',
                                          output_hidden_states=False,
                                          output_attentions=True)
max_length = 10
sequence = ["I eat a green apple", "I am playing football tomorrow"]
tokens, lengths = tokenize_sentences_Bert(sequence, tokenizer, maxlen=max_length)

lengths = torch.tensor(lengths)
tokens = torch.tensor(tokens)
attention_mask = (torch.arange(max_length).expand(len(lengths), max_length) < lengths.unsqueeze(1)).float()
print(attention_mask)
outputs = model(tokens, attention_mask=attention_mask)
print(outputs[0][:, :, :2]) # last step should return 0 last hidden layers
```



- `transformers` version:
- Platform: ubuntu
- Python version: 3.6
- PyTorch version (GPU?): 1.4
- Tensorflow version (GPU?):
- Using GPU in script? no:
- Using distributed or parallel set-up in script?: no


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
TFAlbertModelTest::test_pt_tf_model_equivalence  -> Fatal Python Error on Mac
Author: sshleifer
Pull request: no
Labels: name: wontfix

name: PyTorch

name: TensorFlow

name: Tests

Running the unit tests locally on mac, I get "Fatal Python error: Aborted"

To reproduce, try `pytest tests/test_modeling_tf_albert.py::TFAlbertModelTest::test_pt_tf_model_equivalence ` 


### Environment Info

- `transformers` version: 2.4.1
- Platform: Darwin-19.0.0-x86_64-i386-64bit
- Python version: 3.7.5
- PyTorch version (GPU?): 1.3.1 (False)
- Tensorflow version (GPU?): 2.0.0 (False)

### Traceback

```
tests/test_modeling_tf_albert.py Fatal Python error: Aborted

Current thread 0x0000000110d2adc0 (most recent call first):
  File "/Users/shleifer/miniconda3/envs/nb/lib/python3.7/site-packages/torch/nn/functional.py", line 1372 in linear
  File "/Users/shleifer/miniconda3/envs/nb/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87 in forward
  File "/Users/shleifer/miniconda3/envs/nb/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541 in __call__
  File "/Users/shleifer/transformers_fork/src/transformers/modeling_albert.py", line 321 in forward
  File "/Users/shleifer/miniconda3/envs/nb/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541 in __call__
  File "/Users/shleifer/transformers_fork/src/transformers/modeling_albert.py", line 566 in forward
  File "/Users/shleifer/miniconda3/envs/nb/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541 in __call__
  File "/Users/shleifer/transformers_fork/tests/test_modeling_tf_common.py", line 111 in test_pt_tf_model_equivalence

```

https://github.com/huggingface/transformers/issues/2240 has a different error message from a similar test.


Thanks!

Comments:
#2240 was an error with DistilBERT and was fixed with https://github.com/huggingface/transformers/commit/ea2600bd5f1d36f2fb61958be21db5b901e33884 

Does this error happen every time you run the test suite?

yes!

I'm running on Darwin 19.2, Python 3.7.5, torch 1.3.1, tensorflow 2.0.0 and transformers from source and I can't replicate this bug 😕

I'm thinking this may be due to a memory issue but it's hard to say given the cryptic error message

I bumped tensorflow to 2.1 and cant replicate this failure **or** the flaky CircleCI test #2781
- `transformers` version: 2.4.1
- Platform: Darwin-19.0.0-x86_64-i386-64bit
- Python version: 3.7.5
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): 2.1.0 (False)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

I also just tried to use python 3.5 and can't replicate.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
error while training distilbert multilingual model
Author: divyag11
Pull request: no
Labels: name: wontfix

hi,
I am trying to finetune distilbert multilingual cased model, but i am getting error while training the model:
while with the same code using distilbert uncased , there is no such error.
Can you please check if there is some problem with distilbert multilingual cased model?
error is:


ValueError: Error when checking model target: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 8 array(s), for inputs ['output_1', 'output_2', 'output_3', 'output_4', 'output_5', 'output_6', 'output_7', 'output_8'] but instead got the following list of 1 arrays: [<tf.Tensor 'ExpandDims:0' shape=(None, 1) dtype=int64>]



Comments:
pls reply to above

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Multiple Choice BERT, SWAG task, failure to test 
Author: PhaelIshall
Pull request: no
Labels: name: wontfix

name: PyTorch

name: Ex: Multiple Choice

name: Usage

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ x] the official example scripts: (give details below)
* [ x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x ] an official GLUE/SQUaD task: (give the name)
* [x ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Download SWAG dataset and put it in some directory and set the path by `export SWAG_DIR=path/to/swag/dir`
2. Copy `run_multiple_choice.py` and `utils_multiple_choice.py`
3. Run the code only for testing with the following command 
`./run_multiple_choice.py --model_type bert --task_name swag --model_name_or_path bert-base-uncased --do_lower_case --max_seq_length 80 --output_dir models_bert/swag_testing --data_dir $SWAG_DIR --do_test`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
```
Traceback (most recent call last):
  File "./run_multiple_choice.py", line 678, in <module>
    main()
  File "./run_multiple_choice.py", line 669, in main
    result = evaluate(args, model, tokenizer, prefix=prefix, test=True)
  File "./run_multiple_choice.py", line 248, in evaluate
    eval_dataset = load_and_cache_examples(args, eval_task, tokenizer, evaluate=not test, test=test)
  File "./run_multiple_choice.py", line 354, in load_and_cache_examples
    examples = processor.get_test_examples(args.data_dir)
  File "utils_multiple_choice.py", line 168, in get_test_examples
    "For swag testing, the input file does not contain a label column. It can not be tested in current code"
ValueError: For swag testing, the input file does not contain a label column. It can not be tested in current codesetting! 
```

In the code it says for testing there is no need for label column, but it doesn't work with or without it. It does not work with the default `test.csv` file (it will be called by default for testing if it is in the directory), but it also does not work with `val.csv` (has label column).

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `Transformers` version: current 
- Platform: Linux
- Python version: 3.7.5
- Using GPU in script?:  yes
- Using distributed or parallel set-up in script?:


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to get longer output for summary?
Author: GraphGrailAi
Pull request: no
Labels: name: wontfix

name: Usage

name: Summarization

#Question
https://stackoverflow.com/questions/60157959/transformers-summarization-with-python-pytorch-how-to-get-longer-output

Should i train it myself to get summary output longer than used in original training script?
:

    python run_summarization.py \
        --documents_dir $DATA_PATH \
        --summaries_output_dir $SUMMARIES_PATH \ # optional
        --no_cuda false \
        --batch_size 4 \
        --min_length 50 \
        --max_length 200 \
        --beam_size 5 \
        --alpha 0.95 \
        --block_trigram true \
        --compute_rouge true

When i do inference with 

    --min_length 500 \
    --max_length 600 \
    
I got a good output for 200 tokens, but the rest of the text is 

    . . . [unused7] [unused7] [unused7] [unused8] [unused4] [unused7] [unused7]  [unused4] [unused7] [unused8]. [unused4] [unused7] . [unused4] [unused8] [unused4] [unused8].  [unused4]  [unused4] [unused8]  [unused4] . .  [unused4] [unused6] [unused4] [unused7] [unused6] [unused4] [unused8] [unused5] [unused4] [unused7] [unused4] [unused4] [unused7]. [unused4] [unused6]. [unused4] [unused4] [unused4] [unused8]  [unused4] [unused7]  [unused4] [unused8] [unused6] [unused4]   [unused4] [unused4]. [unused4].  [unused5] [unused4] [unused8] [unused7] [unused4] [unused7] [unused9] [unused4] [unused7]  [unused4] [unused7] [unused5] [unused4]  [unused5] [unused4] [unused6]  [unused4]. .  . [unused5]. [unused4]  [unused4]   [unused4] [unused6] [unused5] [unused4] [unused4]  [unused6] [unused4] [unused6]  [unused4] [unused4] [unused5] [unused4]. [unused5]  [unused4] . [unused4]  [unused4] [unused8] [unused8] [unused4]  [unused7] [unused4] [unused8]  [unused4] [unused7]  [unused4] [unused8]  [unused4]  [unused8] [unused4] [unused6] 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Why only use the hidden state of last token of last layer is used for predicting the next word?
Author: mainulquraishi
Pull request: no
Labels: name: wontfix

I was trying to generate text and also reading the code to understand how it is working. 
I found that, after providing some text as context (first iteration), it goes through the transformer and the output of the transformer (`output[0] `of `GPT2Model` ) is for each token position there is a vector. To my understanding, these vectors are the context-aware representation of each token position. 
Now for generating the next word, the representation of the last token of last layers is being used.
This is the case for the first iteration.  
Then for each subsequent iteration, only the representation of the last predicted word is used to predict the next word. 

My question is, why the only the representation of the last word is used to predict the next word?
 
This raises another question, is the last token-position representation hold the context of the whole sequence (like LSTM)?   

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Albert multilingual
Author: nimning
Pull request: no
Labels: name: Need more information

name: wontfix

name: New model

# 🚀 Feature request

Provide multilingual pre-trained albert model.

## Motivation
Albert is a light weighted bert. It would be nice if it has the multilingual version.


## Your contribution



Comments:
As far as I know (from following https://github.com/google-research/ALBERT/issues/5 and https://github.com/google-research/ALBERT/issues/91), ALBERT multilingual is not yet released.

We'll make sure to support it once it's released.

https://github.com/google-research/ALBERT/pull/152/files

😱😱😱

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
BERT generating prediction in 120sec approx using squad 2.0 in prediction.json
Author: tusharsh23
Pull request: no
Labels: name: wontfix

I am using the below command to predict the question answer using BERT with squad but it taking too long to generate the prediction.json approx 120 sec. I wanna reduce this time to 10secs.

run_squad.py --vocab_file=uncased_L-12_H-768_A-12/vocab.txt --bert_config_file=uncased_L-12_H-768_A-12/bert_config.json --init_checkpoint=model.ckpt-21899 --do_train=False --train_file=train-v1.1.json --do_predict=True --train_batch_size=32 --learning_rate=5e-5 --num_train_epochs=3.0 --max_seq_length=384 --doc_stride=128 --version_2_with_negative=True --output_dir=/ --predict_file=input.json --use_tpu=False

Please suggest me some solution.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Distillation code loss functions
Author: snaik2016
Pull request: no
Labels: name: wontfix

name: PyTorch

name: Core: Modeling

name: Usage

name: Distillation

# ❓ Questions & Help
Why compute cross entropy loss from the hard labels in distillation code?

if self.alpha_clm > 0.0:
            shift_logits = s_logits[..., :-1, :].contiguous()
            shift_labels = lm_labels[..., 1:].contiguous()
            loss_clm = self.lm_loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
            loss += self.alpha_clm * loss_clm

The model outputs loss when passed with the labels.


<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 

Comments:
Hello @snaik2016,
The part of code you're referring to is not a distillation loss. It's the "classic" causal language modeling loss.
Victor

Not referring to the "distillation loss" just the part of the code where loss is computed in distillation code. The exact same quantity is return by model output when labels are passed.

Oh yes, you are right, this could be factorized in.
Just note that you have to be careful with the `ignore_index` and make sure it's coherent with your processing (if I remember correctly, at one point, not all the models were using the same `ignore_index` in the loss computation).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
sequence labeling for sentences and not tokens
Author: antgr
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->
I have sentences that belong to a paragraph. Each sentence has a label.
[s1,s2,s3,..], [l1,l2,l3,...]
I understand that I have to encode each sentence using an encoder e.g bert, and then use sequence labeling. Could you guide me on how I could do that, combining them?

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 
https://stackoverflow.com/questions/60048900/sequence-labeling-for-sentences-and-not-tokens

Comments:
Hi! You could leverage one of the `XXXForSequenceClassification` models for this. Their purpose is to classify sequences into a given number of labels. You would need to initialize a model from a pre-trained checkpoint:

```py
from transformers import BertForSequenceClassification

model = BertForSequenceClassification.from_pretrained("bert-base-cased")
```

This instantiates the base transformer model, but doesn't instantiate the classifier layer on top, you would need to train that with a fine-tuning on your own specific task. 

Does the fact that I want to classify entire sentences and not words, makes any difference? And if yes what is this difference? Is there any example with this specific use case?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Update README.md
Author: ahotrod
Pull request: yes
Labels: name: model card

Improved results from new hardware

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3860?src=pr&el=h1) Report
> Merging [#3860](https://codecov.io/gh/huggingface/transformers/pull/3860?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a21d4fa410dc3b4c62f93aa0e6bbe4b75a101ee9&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3860/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3860?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #3860   +/-   ##
=======================================
  Coverage   78.61%   78.61%           
=======================================
  Files         106      106           
  Lines       17953    17953           
=======================================
  Hits        14114    14114           
  Misses       3839     3839           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3860?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3860?src=pr&el=footer). Last update [a21d4fa...9e4fe33](https://codecov.io/gh/huggingface/transformers/pull/3860?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Bug in run_glue 
Author: rabeehk
Pull request: no

Hi
I am getting this error when running run_glue.py 

ImportError: cannot import name 'TrainingArguments' from 'transformers' (/idiap/user/rkarimi/libs/anaconda3/envs/iborn/lib/python3.7/site-packages/transformers/__init__.py)

Traceback (most recent call last):
  File "run_glue.py", line 34, in <module>
    from transformers import (
ImportError: cannot import name 'HfArgumentParser' from 'transformers' (/idiap/user/rkarimi/libs/anaconda3/envs/iborn/lib/python3.7/site-packages/transformers/__init__.py)

To fix I searched in the repo on how you use the hf_argparser, I modified it as below:
from transformers.hf_argparser import HfArgumentParser

again, getting  the error: 
Traceback (most recent call last):
  File "run_glue.py", line 44, in <module>
    from transformers.hf_argparser import HfArgumentParser
ModuleNotFoundError: No module named 'transformers.hf_argparser'

however, this is how you call it in the tests but this does not work. Seems to me that things have been changed but the codes are not updated.  thanks 

Comments:
You need to install from source as specified in the README.