noamaanMulla-03/transformers-issues huggingface dataset:
How can I use the this result?
Author: bboyxu5928
Pull request: no
Labels: name: Usage

I wan to call the next sentence prediction function on new data. 
 and this webpage tell it 
https://stackoverflow.com/questions/55111360/using-bert-for-next-sentence-prediction

 input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]])
    input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]])
    token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]])
    config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768,
        num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072)
    model = BertForNextSentencePrediction(config)
    seq_relationship_logits = model(input_ids, token_type_ids, input_mask)

but when I run this demo and get the result ( seq_relationship_logits) ,
(tensor([[-0.0728,  0.1863],
        [ 0.3190, -0.1528]], grad_fn=<AddmmBackward>),)
how to use it as the predict result （like if sentence B follows sentence A, so the predict label is 0,else the label is 1）

Comments:
That answer links to an old version of the codebase. Here is an example using the current-day implementation. The documentation can be improved for this particular task, though, since as currently written the given example doesn't seem to do any NSP.

```python
from torch.nn.functional import softmax
from transformers import BertForNextSentencePrediction, BertTokenizer


seq_A = 'I like cookies !'
seq_B = 'Do you like them ?'

# load pretrained model and a pretrained tokenizer
model = BertForNextSentencePrediction.from_pretrained('bert-base-cased')
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')

# encode the two sequences. Particularly, make clear that they must be 
# encoded as "one" input to the model by using 'seq_B' as the 'text_pair'
encoded = tokenizer.encode_plus(seq_A, text_pair=seq_B, return_tensors='pt')
print(encoded)
# {'input_ids': tensor([[  101,   146,  1176, 18621,   106,   102,  2091,  1128,  1176,  1172, 136,   102]]),
#  'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]]),
#  'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}
# NOTE how the token_type_ids are 0 for all tokens in seq_A and 1 for seq_B, 
# this way the model knows which token belongs to which sequence

# a model's output is a tuple, we only need the output tensor containing
# the relationships which is the first item in the tuple
seq_relationship_logits = model(**encoded)[0]

# we still need softmax to convert the logits into probabilities
# index 0: sequence B is a continuation of sequence A
# index 1: sequence B is a random sequence
probs = softmax(seq_relationship_logits, dim=1)

print(seq_relationship_logits)
print(probs)
# tensor([[9.9993e-01, 6.7607e-05]], grad_fn=<SoftmaxBackward>)
# very high value for index 0: high probability of seq_B being a continuation of seq_A
# which is what we expect!
```

Got it,thanks a lot!
Dropout rates to be updated in all ALBERT v2 configs
Author: matteodelv
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): ALBERT v2

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [x] the official example scripts: run_squad.py
* [ ] my own modified scripts: 

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: SQuAD v2.0
* [ ] my own task or dataset: 

## To Reproduce

Steps to reproduce the behavior:

1. Just choose one of the following default ALBERT v2 configs: base, large, xlarge

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

As stated in the updated ALBERT Tensorflow repo from Google Research, model v2 introduces no dropout at all for downstream tasks, like SQuAD, MRPC or COLA.
Following the discussing from this [issue](https://github.com/google-research/ALBERT/issues/23), model configurations on TF-Hub were wrong and thus, the ones used by transformers (loaded on Amazon S3).


While configs on TF-Hub will be updated in the near future
 As configs on TF-Hub have already been updated, transformers' ones should be updated too: parameters `attention_probs_dropout_prob` and `hidden_dropout_prob` should be both 0 for all v2 configs.

## Environment

* OS: Platform Linux-4.14.152-98.182.amzn1.x86_64-x86_64-with-glibc2.9
* Python version: Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56)
* PyTorch version: PyTorch 1.3.1
* PyTorch Transformers version (or branch):
* Using GPU ? YES, 4x NVIDIA V100 16GB
* Distributed of parallel setup ? No
* Any other relevant information: Using Amazon AWS Deep Learning Linux AMI 26.0

## Additional context

Hope this will be useful!


Comments:
I found it already been updated: https://s3.amazonaws.com/models.huggingface.co/bert/albert-xxlarge-v2-config.json

Did I miss something?

Base, large and xlarge v2 configs have to be updated too, as confirmed by this [issue](https://github.com/google-research/ALBERT/issues/23) in the official Google Research repository. 

Thanks. Will pay attention to this one. Hope it will be fix soon.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Model2Model inference
Author: berzentine
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
I am trying to implement a simple Model2Model question-answer task, where the input is question and the answer needs to be generated. At inference time I feed in the "[CLS]" token to let it generate but it only generates a single token instead of the entire sentence. The ppl at which I do the inference is ~5 ppl on valid set. 

Is there a fundamental issue with my model?

Training time 
`outputs = model(input_ids,
                        batch['speakableAnswer'],
                        decoder_lm_labels=batch['speakableAnswer'])`
print(outputs.size() -> batch x seq_len x vocab_size)

Test time
`outputs = model(input_ids,
                        batch['speakableAnswer'])`
print(outputs.size() -> batch x 1 x vocab_size)
<!-- A clear and concise description of the question. -->


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Why are you getting just the last encoder states in the summarization code?
Author: ohmeow
Pull request: no
Labels: name: wontfix

The line is here:

https://github.com/huggingface/transformers/blob/v2.3.0/examples/summarization/modeling_bertabs.py#L142

By changing the line to `encoder_hidden_states = encoder_output` I was able to fine-tune the model successfully, as well as, run the inference code from the `run_summarization.py` script.

So just wondering why you're indexing into the the encoder output rather than passing all of it along to the decoder?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
What's the exact name of BERT large in results ( GermEval 2014)?
Author: zhipengChen
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
I use BERT large cased model downloaded by run_ner.py script. But I can't get the result in the table below.  
<!-- A clear and concise description of the question. -->
![image](https://user-images.githubusercontent.com/13817269/71427541-cb207880-26f4-11ea-9b15-5f6f13b6f31f.png)


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Forward pass with GPT2 using both past and attention_mask as an input leads to dimension error
Author: patrickvonplaten
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): GPT2

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ x ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ x ] my own task or dataset: (give details below)

## To reproduce run the following code:

```
from transformers import GPT2LMHeadModel
import torch

model = GPT2LMHeadModel.from_pretrained('gpt2')

input_ids = torch.tensor([8, 8, 0, 50256, 50256]).unsqueeze(0)
attn_mask = torch.tensor([1, 1, 1, 0, 0]).unsqueeze(0)

# first step there is no past so lets get it from the model and append new embedding id to inputs and extend the attn_mask
logits_output, past = model(input_ids, attention_mask=attn_mask)
next_token = torch.argmax(logits_output[:, -1, :]).unsqueeze(0)
input_ids = torch.cat([input_ids, next_token.unsqueeze(-1)], dim=-1) 
attn_mask = torch.cat([attn_mask, torch.ones((attn_mask.shape[0], 1)).long()], dim=1)


# now we have a past so we can use it to speed up training
model_inputs = model.prepare_inputs_for_generation(input_ids=input_ids, past=past)
logits_output, past = model(**model_inputs, attention_mask=attn_mask)  # this leads to an error which it should not
```

## Expected behavior

No error should be thrown and the forward pass to get `logits_output` and `past` should work

## Environment info
     
- `transformers` version: 2.5.1
- Platform: Linux-5.3.0-40-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No



Comments:
FYI - I created a stackoverflow question about this here: https://stackoverflow.com/questions/60459292/using-past-and-attention-mask-at-the-same-time-for-gpt2

@Damiox - answered you stackoverflow question :-)
No Causal Attention Masking in GPT-2 LM Finetuning Script
Author: alvinchangw
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): GPT-2

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: (give the name)
running run_language_modeling.py on WikiText-2 dataset

* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. attention_mask is None at each forward step of the GPT-2 model (GPT2LMHeadModel)


## Expected behavior
attention_mask should reflect causal attention masking for the LM objective in finetuning GPT-2 so that outputs (t) only attend to inputs at previous time steps (1,..,t-1) instead of relying on input at the same time-step of output (t) where GPT-2 can simply learn to copy the input as output to optimize the LM objective.



## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.5.1
- Platform: Linux-4.15.0-76-generic-x86_64-with-glibc2.10
- Python version: 3.8.1
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


Comments:
Hi @alvinchangw, 

GPT2 always uses causal masking no matter what kind of attention_mask you give it. 
It's easy to see when you print out the computed attentions for each layer (by setting `output_attentions=True`) => see for this also #2975. 

In the code this is done in this line:
https://github.com/huggingface/transformers/blob/298bed16a841fae3608d334441ccae4d9043611f/src/transformers/modeling_gpt2.py#L146

I admit it is very cryptic and probably should have better naming. Essentially what happens here is the following:
`self.bias` is defined as a lower triangular mask (see torch function [here](https://pytorch.org/docs/stable/torch.html?highlight=tril#torch.tril) ). according to the sequence length (params `nd` and `ns`), we derive `b`. `b` is then a lower triangular mask of shape sequence length x sequence length. Using this mask, we substract 10^4 from all values in `w` which should be masked, which sets their attention to 0.


Hi @patrickvonplaten,

Thank you for pointing this out and for the detailed explanation!
how to do a simple multi-classifier by bert 2.0,training set ,and label set all lines
Author: tongbc
Pull request: no
Labels: name: wontfix

how to do a simple multi-classifier by bert 2.0,training set ,and label set all lines

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Predict the next word in sentence context from the list of possible words in Russian
Author: svk-man
Pull request: no
Labels: name: Usage

Hello from Russia. I have a task of predicting the next word from the list of possible words in Russian. How can I do this?


Comments:
Example
My sentence: Мой кот ... (My cat ...)
3 word is ест (eat)
List of possible words: ест (eat), поглощает (absorb), глотает (swallow), кушают (eats) etc.
I need to determine the probabilities of each word from the given list in the context of the phrase and make the most correct sentence.
Output: Мой кот ест (My cat eat).

This is a very general question. Please use [Stack Overflow](https://stackoverflow.com/) for this.

Note that you'll need to use a model that is pretrained on Russian.
Chinese BERT model can be used represented by words instead of character
Author: WenTingTseng
Pull request: no

# ❓ Questions & Help

I want to ask about the Chinese BERT model can be used represented by words instead of character?
Because when I do BERT visual for Chinese can only see its attention from character to character.
I want to see attention from words to words. Can I change this?
Thanks a lot for your help

Comments:
I am not sure which Chinese BERT you are referring to, but the original multilingual BERT has trained Chinese on the character-level. From their [README](https://github.com/google-research/bert/blob/master/multilingual.md):

> Because Chinese (and Japanese Kanji and Korean Hanja) does not have whitespace characters, we add spaces around every character in the CJK Unicode range before applying WordPiece. This means that Chinese is effectively character-tokenized. Note that the CJK Unicode block only includes Chinese-origin characters and does not include Hangul Korean or Katakana/Hiragana Japanese, which are tokenized with whitespace+WordPiece like all other languages.

OK, Thanks for your help 
But I mean I want Chinese BERT for word-level not for character-level.

Yes, I understand that, and as I said the default multilingual BERT does **not** support that. You'll have to find another implementation, perhaps https://arxiv.org/abs/1906.08101
Delete Model2Model
Author: sshleifer
Pull request: yes

- the quickstart code doesn't work
- the tests don't test a forward pass

if you need it `git checkout e8ce63ff`

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3019?src=pr&el=h1) Report
> Merging [#3019](https://codecov.io/gh/huggingface/transformers/pull/3019?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/e8ce63ff2163259276fc0a4a2f35b836fe9f4aa0?src=pr&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3019/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/3019?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3019      +/-   ##
==========================================
+ Coverage   77.25%   77.27%   +0.01%     
==========================================
  Files          98       98              
  Lines       16040    16030      -10     
==========================================
- Hits        12392    12387       -5     
+ Misses       3648     3643       -5
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3019?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_encoder\_decoder.py](https://codecov.io/gh/huggingface/transformers/pull/3019/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19lbmNvZGVyX2RlY29kZXIucHk=) | `21.05% <ø> (-4.33%)` | :arrow_down: |
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/3019/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `98.91% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3019?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3019?src=pr&el=footer). Last update [e8ce63f...835a807](https://codecov.io/gh/huggingface/transformers/pull/3019?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Hi, why is the Model2Model deleted? I am using it and it works great for me. 

It was not super well documented or tested and we didn't want to maintain it. If you're interested in sending a PR with working quickstart code and tests (could start by reverting this) we would definitely be happy to add it back! Sorry!
load_and_cache_examples crashes on windows
Author: yugant-git
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using ALBERT:

Language I am using the model on (English):

The problem arise when using:
[examples/run_squad.py] the official example scripts: (run evaluation for offline model)

It crashes in "load_and_cache_examples" for paths with windows format. It's due to split being done using ('/'). For windows, it needs to be ('\\\\')

The tasks I am working on is:
SQUaD

## To Reproduce

Steps to reproduce the behavior:

1. Get one model cached into a local folder by running Evaluation for SQUAD 2.0 using run_squad.py. This will load online model to the system
2. Run evaluation again with model_name_or_path needs to be a local relative path with "..\\" in it.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Windows
* Python version: Python 3.6
* PyTorch version: torch 1.3.1
* PyTorch Transformers version (or branch): Latest

## Additional context

Not needed


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[CLS] token /  is used as the aggregate sequence representation for classification tasks
Author: cherepanovic
Pull request: no
Labels: name: wontfix

[CLS]  is  fed into an output layer for classification. How will be built this token. Is there something special done for it during training? in which way does happen this aggregations of sequences?

Thanks for response

Comments:
[CLS], [SEP] are "special token" of bert. Its included in the vocabulary of size 30522. Its starts with like 101, 103 something.

of course these two tokens are special tokens. 

 **is  used  as  the  aggregate sequence representation for classification tasks.**

in which way does happen this aggregations?

>>The first token of every sequence is always a special classification  token  ([CLS]).  The  final  hidden  state corresponding  to  this  token  is  used  as  the  aggregate sequence representation for classification tasks.

from papert https://arxiv.org/pdf/1810.04805.pdf

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
fix n_gpu count when no_cuda flag is activated
Author: VictorSanh
Pull request: yes

As I understand it, `no_cuda` should prevent the use of GPU in the `run_*` example scripts. However, `n_gpu` doesn't take it into account and count the numbers of GPUs available on the machine. It sends the model on the GPUs while the tensors are still on CPUs...

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3077?src=pr&el=h1) Report
> Merging [#3077](https://codecov.io/gh/huggingface/transformers/pull/3077?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/298bed16a841fae3608d334441ccae4d9043611f?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3077/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/3077?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3077      +/-   ##
==========================================
+ Coverage   77.18%   77.19%   +<.01%     
==========================================
  Files          98       98              
  Lines       16063    16063              
==========================================
+ Hits        12399    12400       +1     
+ Misses       3664     3663       -1
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3077?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3077/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `92.38% <0%> (+0.16%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3077?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3077?src=pr&el=footer). Last update [298bed1...69041c1](https://codecov.io/gh/huggingface/transformers/pull/3077?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).