noamaanMulla-03/transformers-issues huggingface dataset:
Update README.md
Author: mar-muel
Pull request: yes
Labels: name: model card

Created PyTorch version of model. Minor update on README.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5789?src=pr&el=h1) Report
> Merging [#5789](https://codecov.io/gh/huggingface/transformers/pull/5789?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/3b924fabeef717be8399f1888280c29c69e9ab00&el=desc) will **increase** coverage by `0.11%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5789/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5789?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5789      +/-   ##
==========================================
+ Coverage   78.13%   78.25%   +0.11%     
==========================================
  Files         146      146              
  Lines       26047    26047              
==========================================
+ Hits        20352    20383      +31     
+ Misses       5695     5664      -31     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5789?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_electra.py](https://codecov.io/gh/huggingface/transformers/pull/5789/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9lbGVjdHJhLnB5) | `26.02% <0.00%> (-69.52%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5789/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `85.21% <0.00%> (-1.01%)` | :arrow_down: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/5789/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `82.31% <0.00%> (+1.28%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/5789/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9vcGVuYWkucHk=) | `95.18% <0.00%> (+74.91%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5789?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5789?src=pr&el=footer). Last update [3b924fa...f284d8c](https://codecov.io/gh/huggingface/transformers/pull/5789?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Update README.md
Author: mrm8488
Pull request: yes
Labels: name: model card

Fix missig "-" in meta data

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5812?src=pr&el=h1) Report
> Merging [#5812](https://codecov.io/gh/huggingface/transformers/pull/5812?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/89a78be51f1c2afd263da66ec76d3297432f0c2a&el=desc) will **increase** coverage by `0.31%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5812/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5812?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5812      +/-   ##
==========================================
+ Coverage   78.19%   78.50%   +0.31%     
==========================================
  Files         146      146              
  Lines       26047    26047              
==========================================
+ Hits        20367    20448      +81     
+ Misses       5680     5599      -81     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5812?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/5812/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `64.90% <0.00%> (-33.90%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5812/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.21% <0.00%> (ø)` | |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/5812/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `82.31% <0.00%> (+1.28%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/5812/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9vcGVuYWkucHk=) | `95.18% <0.00%> (+74.91%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5812?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5812?src=pr&el=footer). Last update [89a78be...a5f39ab](https://codecov.io/gh/huggingface/transformers/pull/5812?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
TypeError: forward() got an unexpected keyword argument 'head_mask'
Author: gandharvsuri
Pull request: no

I'm getting the above error while training for EncoderDecoder Model for longformer.

Following is the code snippet from the training loop, followed by the model definition.

`    output = model(input_ids = b_input_ids, attention_mask = b_input_masks, decoder_input_ids = b_decoder_input_ids, decoder_attention_mask = b_decoder_input_masks )
`
`model = EncoderDecoderModel.from_encoder_decoder_pretrained('allenai/longformer-base-4096','allenai/longformer-base-4096')
`
The crazy thing is I've not even defined 'head_mask' and left it to take its default value of 'none'.

Following the complete error.
`---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-18-68f3d2859ad0> in <module>()
     32     optimizer.zero_grad()
     33 
---> 34     output = model(input_ids = b_input_ids, attention_mask = b_input_masks, head_mask = None,decoder_input_ids = b_decoder_input_ids, decoder_attention_mask = b_decoder_input_masks )
     35     loss = output[0]
     36 

2 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

TypeError: forward() got an unexpected keyword argument 'head_mask'`

Comments:
The `EncoderDecoderModel` does not work with longformer yet. Closing this in favor of https://github.com/huggingface/transformers/issues/4225 .
GPT2 weights don't initialize from checkpoint
Author: alma-lindborg
Pull request: no

OS: OSX 10.15.5 (Catalina)
Transformers version 3.0.2

I'm running into this warning when I'm trying to initialize a pre-trained GPT2 model. 
This looks a bit worrying to me because it looks like like it ignores all the pre-trained attention heads, or am I missing something here? 

<img width="1087" alt="Screenshot 2020-07-16 at 11 19 10" src="https://user-images.githubusercontent.com/8694790/87654081-b837c900-c756-11ea-925e-106314ad9942.png">

Any idea of what's gone wrong? I've been running the same code on another computer earlier without encountering this problem, but on my current setup I haven't been able to get around it. I also tried deleting the downloaded files from the cache and re-loading the models with no luck.

Any help is very appreciated!

Comments:
Hey @almaLindborg,

This is a known warning. We should probably disable this...a couple of other models have it as well. But the model works as intended, so no worries about the message.

Also pinging @sshleifer for notification.
LongFormerAttention For AutoRegressive Models
Author: santhoshkolloju
Pull request: no

Longformer Currently supports only Bidirectional Attention,It would be a great feature to finetune current language models like GPT-2 on longer sequences .

Comments:
Yes! This will be added when starting the `Longformer` Encoder framework :-) Closing this in favor of https://github.com/huggingface/transformers/issues/5170 and https://github.com/huggingface/transformers/issues/4225
How to use pytorch_model.bin to classify a single sentence?
Author: HiroshigeAoki
Pull request: no

Hi!

I fine-turned BERT on my own datasets by using run_glue.py. And I got pytorch_model.bin as output. I want to use pytorch_model.bin in another system to classify a single sentence from web browser. 
I would be grateful if you could teach me about usage.

Comments:
Which task did you fine tune the model on? For single sentence it is probably Cola or SST-2  task right? 
You must use predict function by specifying do_predict in the input parameters.

Thank you for answering my question!
I fine-tuned my model on a original task for binary classifications of Japanese sentences. The processor for original task is below.

in transformers/data/processors/glue.py
```
class OriginalProcessor(DataProcessor):
    """Processor for the original data set."""
    def get_example_from_tensor_dict(self, tensor_dict):
        """See base class."""
        return InputExample(
            tensor_dict["idx"].numpy(),
            tensor_dict["sentence"].numpy().decode("utf-8"),
            None,
            str(tensor_dict["label"].numpy()),
        )

    def get_train_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")

    def get_dev_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "dev.tsv")), "dev")

    def get_labels(self):
        """See base class."""
        return ["0", "1"]

    def _create_examples(self, lines, set_type):
        """Creates examples for the training and dev sets."""
        examples = []
        for (i, line) in enumerate(lines):
            # if tsv files have a header, remove #
            # if i == 0:
            #     continue
            guid = "%s-%s" % (set_type, i)
            text_a = line[0]
            label = line[1]
            examples.append(InputExample(guid=guid, text_a=text_a, text_b=None, label=label))
        return examples
```

And please let me ask two questions.
1. Which is better to execute run_glue.py as an external process or to use my own script which mimic run_glue.py's predict function?
2. Can I load my fine-tuned model by specifying the directory which have 'pytorch_model.bin' in parameters or my script as written below? The directory includes output of fine-turning.

```
#parameter
--model_name_or_path="the path for the directory"

#script
model = BertForSequenceClassification.from_pretrained('the path for the directory')
```

I made it with first one. Thank you!

Both are doable. I would turn off training option and just use prediction option. 
Good to know you did it.
Using segments ids in encoder-decoder model in generate function
Author: mmsamiei
Pull request: no

# 🐛 Bug

## Information

I have implemented an encoderdecoder model where both of them are bert, in encoder module i'm using segment ids (token_type_ids) and give it to model easily in this way and works correctly:
```
kwargs = {'token_type_ids':segments_tensors}
outputs = model(input_ids=encoder_input, decoder_input_ids=decoder_input, **kwargs)[0]
```
but in generate function i use below code to give segment ids:
```
kwargs = {'token_type_ids':segments_tensors}
generated = model.generate(encoder_input, decoder_start_token_id=101
num_return_sequences=16,num_beams=32, **kwargs)
```
I thought this should work but commenting or uncommenting kwargs shows me that no changes in output have happened. Is there another correct approach to give arguments of encoder module and token_type_ids through generate function or not?

Comments:
`segments_tensors` are currently not implemented for `generate`. Why do you want to pass them to `generate`?

I'm working on a conditional dialogue task (knowledge grounded dialogue) in which I give the conversation history and fact sentence as inputs to the model. I want to show the model that fact sentence is different from conversation history, so I decided to use segment embedding to do this and learn different segment embeddings for history and fact sentence. in training I have no problem and pass token_type_ids as kwargs, but in inferencing phase, I have this mentioned problem. 

To use `token_type_ids`, you would actually need to change the line in the generate function where the encoder is called to something like this:
`encoder_outputs: tuple = encoder(input_ids, attention_mask=attention_mask, token_type_ids=model_specific_kwargs['token_type_ids'])`

or maybe better:
`encoder_outputs: tuple = encoder(input_ids, attention_mask=attention_mask, token_type_ids=model_specific_kwargs.pop('token_type_ids', None))`
In case no `token_type_ids` are passed.


https://github.com/huggingface/transformers/blob/master/src/transformers/generation_utils.py#L385



> To use `token_type_ids`, you would actually need to change the line in the generate function where the encoder is called to something like this:
> `encoder_outputs: tuple = encoder(input_ids, attention_mask=attention_mask, token_type_ids=model_specific_kwargs['token_type_ids'])`
> 
> or maybe better:
> `encoder_outputs: tuple = encoder(input_ids, attention_mask=attention_mask, token_type_ids=model_specific_kwargs.pop('token_type_ids', None))`
> In case no `token_type_ids` are passed.
> 
> https://github.com/huggingface/transformers/blob/master/src/transformers/generation_utils.py#L385

Thank you very much for your guidance! 🙏
Fix bug in sample code
Author: joehalliwell
Pull request: yes
Labels: name: wontfix


Comments:
It seems this example was not updated after #3734. This is not the correct way to do it, as it now requires the user to specify the entire context all the time. Will open a PR now.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Can I use the pretrained BERT-Base model directly for predict isNextSentence task?
Author: nomadlx
Pull request: no

# ❓ Can I use the pretrained BERT-Base model directly for predict isNextSentence task?

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->
I has a document-level corpus, but it don't have document boundaries.
I want to confirm that whether can I predict isNextSentence task by pretrained BERT-Base model? The model does not use any data finetune and don't mask any token when i used it. 
Is prediction reliable by this way?

Comments:
I have find anwser in [code](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L1138).
language tag addition on albert-mongolian
Author: bayartsogt-ya
Pull request: yes
Labels: name: model card

hello!
I just added the language tag to model card. Can you guys check for me?

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5828?src=pr&el=h1) Report
> Merging [#5828](https://codecov.io/gh/huggingface/transformers/pull/5828?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/d088d744adb4e5aa45262a34acab3ae9e81de169&el=desc) will **decrease** coverage by `0.83%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5828/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5828?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5828      +/-   ##
==========================================
- Coverage   78.10%   77.26%   -0.84%     
==========================================
  Files         146      146              
  Lines       26047    26047              
==========================================
- Hits        20344    20126     -218     
- Misses       5703     5921     +218     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5828?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_mobilebert.py](https://codecov.io/gh/huggingface/transformers/pull/5828/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9tb2JpbGViZXJ0LnB5) | `23.38% <0.00%> (-73.39%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5828/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `83.95% <0.00%> (-2.01%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5828/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `81.49% <0.00%> (+0.29%)` | :arrow_up: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/5828/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `82.31% <0.00%> (+1.28%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/5828/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9vcGVuYWkucHk=) | `95.18% <0.00%> (+74.91%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5828?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5828?src=pr&el=footer). Last update [d088d74...fa415dc](https://codecov.io/gh/huggingface/transformers/pull/5828?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks!
Can we have a way for a tokenizer to transform word level or character level annotations?
Author: brian8128
Pull request: no
Labels: name: Core: Tokenization

# 🚀 Feature request

If I have a string with some annotations and I want to tokenize it I'd like the tokenizer to be able to transform the annotations as well. For example suppose I have `s = "therein lies the problem."` and I'm interested in the substring `"the problem"`. So I have a string `s` and I know that the substring I'm interested in is at index 13:24. But then I tokenize `s` so that I can put it into a huggingface model and get out `['there', '##in', 'lies', 'the', 'problem', '.']` and it doesn't match up with my annotation anymore. Could we add an annotation as an additional argument to the tokenizer.tokenize function so that I could get something like the following:

```
tokenizer.tokenize("therein lies the problem.", selection=(13,24))
> ['there', '##in', 'lies', 'the', 'problem', '.'], [0, 0, 0, 1, 1, 0]
```

Substrings of interest are almost always going to line up with token boundaries so it doesn't matter too much what happens to a token that is partially in and partially outside of the selected region. 

Is there a way of doing something like this now?


Comments:
I just found out about the `return_offsets_mapping` functionality in the `PreTrainedTokenizerFast` tokenizers. I think I can use that functionality to solve my problem. 

@brian8128 could you share a code example of how you do that?

Hey Avijit - I didn't test this because it's sort of copied from various places in my codebase but here's the general idea. There may be multiple different char level labels within one token so here I take the most common one using the scipy.stats.mode function. 

```
from scipy.stats import mode
from transformers import BertTokenizerFast

texts = ... # list of strings
char_level_labels = ... # list of 1-d numpy arrays corresponding in length to texts

tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')
batch = tokenizer.batch_encode_plus(texts, return_offsets_mapping=True)

offset_mappings = batch['offset_mapping']

token_level_labels = []
for offset_mapping, char_level_label in zip(offset_mappings, char_level_labels):
    token_level_label = []
    for so, eo in offset_mapping:
        # Huggingface adds a start and end token that don't correspond to any
        # chars, we'll label these tokens with -1
        label = mode(char_level_label[so:eo]).mode[0] if eo - so > 0 else -1
        token_level_label.append(label)

    token_level_labels.append(np.array(token_level_labels))
```