noamaanMulla-03/transformers-issues huggingface dataset:
load_tf_weights_in_bert : 'BertModel' object has no attribute 'bias'
Author: luyi6111
Pull request: no
Labels: name: Core: Modeling

```
AttributeError                            Traceback (most recent call last)
<ipython-input-14-0d66155b396d> in <module>
     12 
     13         K.clear_session()
---> 14         model = create_model()
     15         optimizer = tf.keras.optimizers.Adam(learning_rate=2e-5)
     16         model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['acc', 'mae'])

<ipython-input-13-4f429fe61419> in create_model()
      5     config = BertConfig.from_pretrained(BERT_PATH + 'bert_config.json')
      6     config.output_hidden_states = False
----> 7     bert_model = BertModel.from_pretrained(BERT_PATH + 'bert_model.ckpt.index', from_tf=True, config=config)
      8     # if config.output_hidden_states = True, obtain hidden states via bert_model(...)[-1]
      9     embedding = bert_model(input_id, attention_mask=input_mask, token_type_ids=input_atn)[0]

~/anaconda3/envs/fasterai/lib/python3.7/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    482             if resolved_archive_file.endswith(".index"):
    483                 # Load from a TensorFlow 1.X checkpoint - provided by original authors
--> 484                 model = cls.load_tf_weights(model, config, resolved_archive_file[:-6])  # Remove the '.index'
    485             else:
    486                 # Load from our TensorFlow 2.0 checkpoints

~/anaconda3/envs/fasterai/lib/python3.7/site-packages/transformers/modeling_bert.py in load_tf_weights_in_bert(model, config, tf_checkpoint_path)
    103                 pointer = getattr(pointer, "weight")
    104             elif scope_names[0] == "output_bias" or scope_names[0] == "beta":
--> 105                 pointer = getattr(pointer, "bias")
    106             elif scope_names[0] == "output_weights":
    107                 pointer = getattr(pointer, "weight")

~/anaconda3/envs/fasterai/lib/python3.7/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    574                 return modules[name]
    575         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 576             type(self).__name__, name))
    577 
    578     def __setattr__(self, name, value):

AttributeError: 'BertModel' object has no attribute 'bias'
```

related libs&version:
transformers 2.5.1
tensorflow 2.1.0

environment:
NVIDIA-SMI 440.59       Driver Version: 440.59       CUDA Version: 10.2

Comments:
change 
`BertModel.form_pretrained`
to
`BertForPreTraining.from_pretrained`
it seems to work

Glad you could get it to work! Indeed, `BertForPreTraining` should be used to convert from official BERT models.
Fix spelling of strictly in error messages
Author: sshleifer
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3042?src=pr&el=h1) Report
> Merging [#3042](https://codecov.io/gh/huggingface/transformers/pull/3042?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/b370cc7e99c5b8c7436154d4694c33b461ea0f08?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3042/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/3042?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3042      +/-   ##
==========================================
- Coverage   77.28%   77.27%   -0.01%     
==========================================
  Files          98       98              
  Lines       16038    16038              
==========================================
- Hits        12395    12394       -1     
- Misses       3643     3644       +1
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3042?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3042/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `92.22% <100%> (-0.17%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3042?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3042?src=pr&el=footer). Last update [b370cc7...5bdaba9](https://codecov.io/gh/huggingface/transformers/pull/3042?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
how to do next word prediction in xlnet?
Author: MuruganR96
Pull request: no

##  how to do next word prediction in xlnet?

First of all thanks for **huggingface - transformers**
community.
I am actually beginner for XLnet. I want to do Next word prediction by using XLnet. How can i do this?
and I have my own domain-specific datasets(1000 lines), finetune this dataset in xlnet.
is this dataset enoughfor us to get good results?


Comments:
Take a look at the example code [here](https://github.com/huggingface/transformers/blob/master/examples/run_generation.py). 1000 lines of text for fine-tuning shouldn't be an issue I think, since you're just fine-tuning. As always, try it out and you'll see.

Thank you so much for your reply

@BramVanroy But how to fine-tune xlnet for next-word-prediction ?
Is it correct to use perm_mask and target_mapping to simulate left-to-rigth ?

Can you direct me to a code on how to predict next word in tf 2.0?

You can see the [causal language modeling example in usage](https://huggingface.co/transformers/usage.html#causal-language-modeling). There's a TensorFlow toggle, and it showcases gpt-2.
Reusing states for sequential decoding in BERTForMaskedLM
Author: da03
Pull request: no
Labels: name: PyTorch

name: Core: Encoder-Decoder

name: seq2seq

# 🚀 Feature request

I am using Bert as a decoder (by setting is_decoder=True). However, during sequential decoding, there is no way of reusing the hidden states, so for every word to be generated we need to rerun the model on the ENTIRE decoded sequence, which renders decoding inefficient. Can you add something similar to the keyword `past=` in GPT2 model to BERT's forward function (https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L938)?

## Generalization of my Issue
More generally, to the best of my knowledge, there's no model in this library that simultaneously supports 1) cross attention (by feeding `encoder_hidden_states=` or `memory=`), and 2) reusing decoder states during sequential decoding (by feeding `past=`). 1) rules out models like GPT-2 and XLNet which only supports language modeling (although in theory we can just use a decoder to do translation, I want to use a separate encoder and decoder); and 2) rules out models like BERT and T5 which supports 1) but not 2). For example, the point of T5 is to use it for text-to-text translation problems, but since we cannot reuse hidden states, sequential decoding (beam search) would be extremely inefficient.

## Example
In the provided summarization example, both 1) and 2) are supported. However, the decoder is defined in its own code (examples/summarization/modeling_bertabs.py) and cannot be used directly in the library. Besides, supporting incremental state update is a basic function that every decoder shall support.

## Relevant Issues
I checked the suggested similar issues and did not find the same issue. Please let me know if my issue duplicates others'.

Comments:
That's a cool idea.

Closed by #3059
TF GPU CI
Author: julien-c
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3085?src=pr&el=h1) Report
> Merging [#3085](https://codecov.io/gh/huggingface/transformers/pull/3085?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/0e56b37e805279ecb61670159fa8c71487214e0a?src=pr&el=desc) will **decrease** coverage by `1.02%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3085/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/3085?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##           master   #3085      +/-   ##
=========================================
- Coverage   77.62%   76.6%   -1.03%     
=========================================
  Files          98      98              
  Lines       16230   16230              
=========================================
- Hits        12599   12433     -166     
- Misses       3631    3797     +166
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3085?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_pytorch\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3085/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9weXRvcmNoX3V0aWxzLnB5) | `8.72% <0%> (-81.21%)` | :arrow_down: |
| [src/transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/3085/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yb2JlcnRhLnB5) | `85.71% <0%> (-10%)` | :arrow_down: |
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/3085/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `73.48% <0%> (-2.3%)` | :arrow_down: |
| [src/transformers/modeling\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/3085/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19jdHJsLnB5) | `96.03% <0%> (-2.21%)` | :arrow_down: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/3085/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `80.2% <0%> (-1.35%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3085/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `93.02% <0%> (-0.16%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3085?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3085?src=pr&el=footer). Last update [0e56b37...83f65ff](https://codecov.io/gh/huggingface/transformers/pull/3085?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
XLNet and RoBERTa embeddings
Author: abhikjha
Pull request: no
Labels: name: wontfix

Referring to Jay Alammar's awesome blog post wherein he showed how to create sentence embeddings from BERT (DistilBert as well), can we use the workings he showed here for XLNet and RoBERTa models as well?

http://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/

I was thinking majorly to use everything same what he did for RoBERTa assuming `
` token in RoBERTa contains the classification output and  change following lines for XLNet since `<cls>` is at the end of the sequence length unlike in case of BERT where it is at the beginning

`features = last_hidden_states[0][:,-1,:].numpy()`

Any idea if my assumptions are correct?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
add models cards for camembert-base-fquad camembert-base-squad
Author: n0thingLLM
Pull request: yes
Labels: name: model card

Following #2893

Model links:

- [`fmikaelian/camembert-base-fquad`](https://huggingface.co/fmikaelian/camembert-base-fquad)
- [`fmikaelian/camembert-base-squad`](https://huggingface.co/fmikaelian/camembert-base-squad)

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3089?src=pr&el=h1) Report
> Merging [#3089](https://codecov.io/gh/huggingface/transformers/pull/3089?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/f169957d0cf17b110f27cacc1b1fb43efaa01218?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3089/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/3089?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3089      +/-   ##
==========================================
+ Coverage   77.59%   77.59%   +<.01%     
==========================================
  Files          98       98              
  Lines       16250    16250              
==========================================
+ Hits        12609    12610       +1     
+ Misses       3641     3640       -1
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3089?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3089/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `93.02% <0%> (+0.15%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3089?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3089?src=pr&el=footer). Last update [f169957...ac7be7c](https://codecov.io/gh/huggingface/transformers/pull/3089?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks for sharing!
Summarization ROGUE scores don't equal that of the paper ...
Author: ohmeow
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Just ran the `run_summarization.py` script, with the parameters specified [here](https://github.com/huggingface/transformers/tree/master/examples/summarization) and the ROGUE scores are far off from what is reported in the related paper.

The ROGUE scores reported in [PreSumm paper](https://github.com/nlpyang/PreSumm) (R1, R2, RL):

> BertSumExtAbs | 42.13 | 19.60 | 39.18





The ROGUE scores after running the HF script:

> ROGUE 1:
> F1 = .275
> Precision = .299
> Recall = .260
> 
> ROGUE 2:
> F1 = .161
> Precision = .184
> Recall = .149
> 
> ROGUE L:
> F1 = .305
> Precision = .326
> Recall = .290

The README file seems to suggest that running the script as is, with all the stories in a single directory, will give you ROGUE scores similar to that of the paper.  That doesn't seem the case.

***Any ideas why? Or what I may be doing wrong here?***

Thanks much!

FYI ... ran the script as in the README:

```
python run_summarization.py \
    --documents_dir $STORIES_DIR
    --summaries_output_dir $OUTPUT_SUM_DIR
    --no_cuda false \
    --batch_size 4 \
    --min_length 50 \
    --max_length 200 \
    --beam_size 5 \
    --alpha 0.95 \
    --block_trigram true \
    --compute_rouge true
```



Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Don't crash if fine-tuned model doesn't end with a number
Author: davidefiocco
Pull request: yes

That's the same fix applied in https://github.com/huggingface/transformers/issues/2258 , but for the GLUE example

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3099?src=pr&el=h1) Report
> Merging [#3099](https://codecov.io/gh/huggingface/transformers/pull/3099?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/eec5ec807135ae61fa2266f3c7ad947cc207abda?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3099/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/3099?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3099      +/-   ##
==========================================
- Coverage   77.59%   77.59%   -0.01%     
==========================================
  Files          98       98              
  Lines       16250    16250              
==========================================
- Hits        12610    12609       -1     
- Misses       3640     3641       +1
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3099?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3099/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `92.86% <0%> (-0.16%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3099?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3099?src=pr&el=footer). Last update [eec5ec8...de4dc15](https://codecov.io/gh/huggingface/transformers/pull/3099?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Pipeline doc
Author: LysandreJik
Pull request: yes

This PR adds documentation to the pipelines and slightly modifies their behavior:

- `modelcard` is no longer available when using the `pipeline` factory method. As discussed with @thomwolf and @mfuntowicz, it doesn't serve any purpose for the user.
- All task-specific pipelines can now be instantiated without any model/tokenizer, instead relying on the defaults defined for the `pipeline` factory method.

Comments:
Hmm, why do we want to be able to spawn a `NerPipeline` or a `QuestionAnsweringPipeline` without a model or a tokenizer?

Isn't this what `pipeline("ner")` or `pipeline("question-answering")` is for?

I think the question is rather why can't we spawn a `NerPipeline` or a `QuestionAnsweringPipeline` even though we have defined defaults for them?

What I see in the `pipeline` factory is the ability to simply specify a task and get the appropriate pipeline. I don't see a strong reason for the task-specific pipelines to not be able to load a default, but I may be missing part of the picture.

If you think this is unnecessary I can revert the code - we'll just need to make sure that the doc explains what the `pipeline` factory is for and how it handles defaults compared to task-specific pipelines, because I was misled.

I'll let @mfuntowicz and @thomwolf chime in, but for me, the subclasses of Pipeline are the actual implementations – preferably well-typed – that do not expose too much magic. 

I don't see the point of having two public APIs that do exactly the same thing.

E.g., the logic behind [get_defaults](https://github.com/huggingface/transformers/pull/3055/files#diff-1e87b75d7b313550a38be1daecd653f7R485-R504) is a duplication of what's already in `pipeline()`

In any cases, the subclasses of Pipeline dont accept a model/tokenizer of type `str`, in contradiction to the doc (it crashes), because the spawning of model/tokenizer with `from_pretrained()` is only inside the `pipeline` wrapper 

Ok, this makes sense. I'll revert that later today. Thanks @julien-c
bert-base-arabic model card
Author: alisafaya
Pull request: yes
Labels: name: model card

Please add model card file to the newly added asafaya/bert-base-arabic model

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3102?src=pr&el=h1) Report
> Merging [#3102](https://codecov.io/gh/huggingface/transformers/pull/3102?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/eec5ec807135ae61fa2266f3c7ad947cc207abda?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3102/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/3102?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3102      +/-   ##
==========================================
- Coverage   77.59%   77.59%   -0.01%     
==========================================
  Files          98       98              
  Lines       16250    16250              
==========================================
- Hits        12610    12609       -1     
- Misses       3640     3641       +1
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3102?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3102/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `92.86% <0%> (-0.16%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3102?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3102?src=pr&el=footer). Last update [eec5ec8...2538e35](https://codecov.io/gh/huggingface/transformers/pull/3102?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks for sharing this is awesome!