noamaanMulla-03/transformers-issues huggingface dataset:
Regarding generate method used in BART
Author: kunalpagarey
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarily intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiasts can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you can always open a question on GitHub. You should then link to the SO question that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if you didn't get an answer ask it here on GitHub. -->

I am new to NLP and text generation so please help me understand some basic things.
I fine-tuned BART on CNN/DM dataset using provide scripts in the examples section and it works fine.
I have some understanding of how model.generate() method works. But need to clarify some basic questions.
Is the generate() method used at the time of training/validation?
If yes, then how loss is computed if the beam size is greater than 1?
Or do we even use the parameters like in generate() method during training?

Please help. Thank you.


Comments:
Hey @kunalpagarey, 

The `.generate()` method cannot be used for training. It can be used for validation and testing. 
To create a good model for summarization the following components are important:

**Train**:
1. What model do you want to use? => Bart for example
2. What loss function do you want to use? => Usually people do pretraining and then finetuning on summarization using standard maximum likelihood. For some detail on how to fine-tune Bart with `transformers` check out: https://github.com/ohmeow/ohmeow_website/blob/master/_notebooks/2020-05-23-text-generation-with-blurr.ipynb

** Decoding method** 
After having fine-tuned a model on summarization, the decoding method to apply is another whole question and is often done independently of training. Now, you can decide how to use the `generate()` method. You could for example try out a bunch of different hyperparameters for `.generate()` on your validation set and then decide on one setting for want to use for your test set. 
For more details on how to choose hyperparameters for `.generate()` check out: 
https://huggingface.co/blog/how-to-generate
Reformer hidden_size of output is doubled.
Author: h324yang
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

The default hidden_size is 256 but the output dim is 512. Similarly, when I change the config and set hidden_size to 512, the output dim is 1024. 

```python

enc_config = {
    "attention_head_size": 64,
    "attn_layers": ["local", "lsh", "local", "lsh", "local", "lsh"],
    "axial_pos_embds": True,
    "sinusoidal_pos_embds": False,
    "axial_pos_embds_dim": [256, 256],
    "axial_pos_shape": [64, 64],
    "lsh_attn_chunk_length": 64,
    "local_attn_chunk_length": 64,
    "feed_forward_size": 256,
    "hidden_act": "relu",
    "hidden_size": 512,
    "is_decoder": False,
    "max_position_embeddings": 4096,
    "num_attention_heads": 12,
    "num_buckets": [64, 64],
    "num_hashes": 4,
    "lsh_attention_probs_dropout_prob": 0.0,
    "lsh_num_chunks_before": 1,
    "lsh_num_chunks_after": 0,
    "local_num_chunks_before": 1,
    "local_num_chunks_after": 0,
    "local_attention_probs_dropout_prob": 0.025,
    "hidden_dropout_prob": 0.025,
    "pad_token_id": tokenizer.pad_token_id,
    "eos_token_id": tokenizer.eos_token_id, 
    "vocab_size": tokenizer.vocab_size,
}

reformer = ReformerModel(ReformerConfig(**enc_config))
input_ids = tokenizer.encode(doc, max_length=4096, pad_to_max_length=True, return_tensors='pt')
out = reformer.forward(input_ids)
out[0].shape
```
```
torch.Size([1, 4096, 1024])
```


<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Comments:
Hey @h324yang, 

Good observation! The reason for this is that Reformer uses Reversible Residual Layers and thus always has two input streams (two hidden states inputs) instead of one. Both of these streams have to be concatenated after running through the model which leads to double the size of `hidden_states`. I will soon ~1 week publish a notebook explaining this in more detail :-)
Add model cards for FinBERT.
Author: haamis
Pull request: yes
Labels: name: model card

These are a copy of https://github.com/TurkuNLP/FinBERT/blob/master/README.md.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3331?src=pr&el=h1) Report
> Merging [#3331](https://codecov.io/gh/huggingface/transformers/pull/3331?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/20139b7c8d88f380f1e4e0ae2baf0b0ac9351039&el=desc) will **decrease** coverage by `0.06%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3331/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3331?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3331      +/-   ##
==========================================
- Coverage   77.63%   77.57%   -0.07%     
==========================================
  Files         100      100              
  Lines       16943    16943              
==========================================
- Hits        13154    13143      -11     
- Misses       3789     3800      +11     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3331?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3331/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `86.40% <0.00%> (-1.97%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3331?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3331?src=pr&el=footer). Last update [20139b7...d354a22](https://codecov.io/gh/huggingface/transformers/pull/3331?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Great. Could you add a metadata block at the top of the file with:
```
---
language: finnish
# optional thumbnail: ...
---
```

Thanks!

@haamis I've one question regarding to the FinBERT training corpus: would it be possible to obtain the final pre-processed data that you've used for training the BERT model 🤔

I would really like to train an ELECTRA model and release it to the community :)

@stefan-it We can't publish the corpus due to licensing/copyright issues, but since we are also interested in training a Finnish ELECTRA maybe we could collaborate on this? Please send me an email sajvir(at)utu.fi.
Add .vs to gitignore
Author: sgugger
Pull request: yes

My VS code writes stuff there so added it to the .gitignore.

I also have other untracked files:
- in tests/fixtures/ I have some cached_lm_\*Tokenizer_\*.txt and .txt.lock after running the tests
- in docs/ I have the simlink to examples.md as indicated in [here](https://github.com/huggingface/transformers/tree/master/docs#building-the-documentation)

Should I add stuff in .gitignore to ignore those as well?

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4774?src=pr&el=h1) Report
> Merging [#4774](https://codecov.io/gh/huggingface/transformers/pull/4774?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/cd4e07a85e6161111016ca6d811d97e59368971a&el=desc) will **increase** coverage by `0.22%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4774/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4774?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4774      +/-   ##
==========================================
+ Coverage   77.09%   77.32%   +0.22%     
==========================================
  Files         128      128              
  Lines       21059    21059              
==========================================
+ Hits        16235    16283      +48     
+ Misses       4824     4776      -48     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4774?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4774/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.15% <0.00%> (+0.12%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4774/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.96% <0.00%> (+0.16%)` | :arrow_up: |
| [src/transformers/modeling\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/4774/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19ncHQyLnB5) | `86.39% <0.00%> (+14.55%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4774?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4774?src=pr&el=footer). Last update [cd4e07a...e51cd58](https://codecov.io/gh/huggingface/transformers/pull/4774?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Add note about doc generation
Author: sgugger
Pull request: yes

Just make it explicit that doc generation is only for local inspection.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4770?src=pr&el=h1) Report
> Merging [#4770](https://codecov.io/gh/huggingface/transformers/pull/4770?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/e645b9ab9407e1c1b2c168317dc79fe13fc6e0b4&el=desc) will **decrease** coverage by `0.04%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4770/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4770?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4770      +/-   ##
==========================================
- Coverage   77.31%   77.26%   -0.05%     
==========================================
  Files         128      128              
  Lines       21059    21059              
==========================================
- Hits        16281    16271      -10     
- Misses       4778     4788      +10     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4770?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_transfo\_xl.py](https://codecov.io/gh/huggingface/transformers/pull/4770/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdHJhbnNmb194bC5weQ==) | `36.80% <0.00%> (-3.88%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4770/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.79% <0.00%> (+0.16%)` | :arrow_up: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4770/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `74.27% <0.00%> (+0.41%)` | :arrow_up: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/4770/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `82.03% <0.00%> (+1.35%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4770?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4770?src=pr&el=footer). Last update [e645b9a...39dc245](https://codecov.io/gh/huggingface/transformers/pull/4770?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Loading config file bug
Author: abrhaleitela
Pull request: no

Hi guys.

I have released a new (XLNet based) transformer model for low-resource language Tigrinya 
 [(TigXLNet)](https://github.com/abryeemessi/Transferring-Monolingual-Model-to-Low-Resource-Language)  and found a bug when loading a pre-trained config file: 

My config file looks like:
https://s3.amazonaws.com/models.huggingface.co/bert/abryee/TigXLNet/config.json


       config = AutoConfig.from_pretrained("abryee/TigXLNet")
       print(config.d_head) #prints 48 even though d_head in the given config file is 64.



Comments:
Hello! The `d_head` is actually computed in the configuration: `self.d_head = d_model // n_head`. 

It would probably be better to handle `d_head` directly, but currently the `d_model` and `d_head` are linked to each other. 

This would crash in your case as  `d_model // n_head != d_head`

@LysandreJik Thanks a lot. Yes, I am setting d_head directly before loading my model. But it would be nice to see the model load its configuration from the given config file. Just my opinion though :)

You're right! Raising an error if the `d_head` is wrong in #4747
No silent error when d_head already in the configuration
Author: LysandreJik
Pull request: yes

closes #4696

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4747?src=pr&el=h1) Report
> Merging [#4747](https://codecov.io/gh/huggingface/transformers/pull/4747?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ed4df85572924871758ca32133b46116121c706f&el=desc) will **increase** coverage by `0.09%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4747/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4747?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4747      +/-   ##
==========================================
+ Coverage   77.12%   77.22%   +0.09%     
==========================================
  Files         128      128              
  Lines       21061    21063       +2     
==========================================
+ Hits        16243    16265      +22     
+ Misses       4818     4798      -20     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4747?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/configuration\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/4747/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX3hsbmV0LnB5) | `94.00% <100.00%> (+0.25%)` | :arrow_up: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4747/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.85% <0.00%> (-0.42%)` | :arrow_down: |
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4747/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `89.59% <0.00%> (+0.11%)` | :arrow_up: |
| [src/transformers/data/processors/squad.py](https://codecov.io/gh/huggingface/transformers/pull/4747/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvc3F1YWQucHk=) | `35.03% <0.00%> (+6.36%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4747?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4747?src=pr&el=footer). Last update [ed4df85...fe85f3e](https://codecov.io/gh/huggingface/transformers/pull/4747?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Add model summary
Author: sgugger
Pull request: yes

This PR adds a high-level summary of all the models in the documentation.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4789?src=pr&el=h1) Report
> Merging [#4789](https://codecov.io/gh/huggingface/transformers/pull/4789?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/b9109f2de1c4f52967976dc840074a9d62713498&el=desc) will **increase** coverage by `0.40%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4789/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4789?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4789      +/-   ##
==========================================
+ Coverage   76.06%   76.46%   +0.40%     
==========================================
  Files         128      128              
  Lines       21498    21498              
==========================================
+ Hits        16353    16439      +86     
+ Misses       5145     5059      -86     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4789?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4789/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.85% <0.00%> (-0.42%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4789/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.20% <0.00%> (-0.16%)` | :arrow_down: |
| [src/transformers/data/processors/squad.py](https://codecov.io/gh/huggingface/transformers/pull/4789/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvc3F1YWQucHk=) | `56.68% <0.00%> (+28.02%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4789?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4789?src=pr&el=footer). Last update [b9109f2...0e3789d](https://codecov.io/gh/huggingface/transformers/pull/4789?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Looks great :-) 

Eh, forgot to finish my sentence linking to the pretrained models doc page at the beginning. No sure if I can link to part of the tables however, does restructured text can support that?

I doubt it. One link is OK I think
Question Answering Modeling through Hugging Face Models
Author: AishwaryaVerma
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->
Hello everyone, I have large number of documents and I need to extract specific information through Hugging Face Question Answering Model. First issue I faced was the document size was very large so it gave me token error and afterwards, I divided the data into small paragraphs, then I applied the given model. But this time, answer was not accurate. So, I just want to know, is there any alternative method or model to do this.

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Comments:
How long is your document, you may wanna try longformer model which can handle sequences upto 4096 tokens. Here's a longformer model trained for QA  https://huggingface.co/valhalla/longformer-base-4096-finetuned-squadv1

Also, take a loot at this https://github.com/deepset-ai/haystack. This might help you a lot

> How long is your document, you may wanna try longformer model which can handle sequences upto 4096 tokens. Here's a longformer model trained for QA https://huggingface.co/valhalla/longformer-base-4096-finetuned-squadv1
> 
> Also, take a loot at this https://github.com/deepset-ai/haystack. This might help you a lot

I looked into it. That is a great help. Thanks. Can we also decide the output length with these type of pretrained models?

Theses QA models aren't generative. So there's no output length constraint

@AishwaryaVerma - For QA the output length is usually very small (only a couple of words). It is very rare that the answer of `AutoModelForQuestionAnswering` is longer than 3,4 words.

You might also want to take a look at: https://huggingface.co/allenai/longformer-large-4096-finetuned-triviaqa

And this notebook: https://colab.research.google.com/drive/1m7eTGlPmLRgoPkkA7rkhQdZ9ydpmsdLE?usp=sharing
Question Answering Pipeline with big texts.
Author: thiagomoeng
Pull request: no

# ❓ Questions & Help

I am searching for some ideas of how to use QA pipeline with big data texts and get a good time response. (I am already using GPU)

Comments:
Longformer would be a good choice, but it is currently not implemented in the pipelines. This issue might help :-) : https://github.com/huggingface/transformers/issues/4762

and #4615

This notebook might help :-) 
https://colab.research.google.com/drive/1m7eTGlPmLRgoPkkA7rkhQdZ9ydpmsdLE?usp=sharing