noamaanMulla-03/transformers-issues huggingface dataset:
LongformerTokenizerFast
Author: patil-suraj
Pull request: yes

This PR adds `LongformerTokenizerFast` by sub-classing `RobertaTokenizerFast`.

@patrickvonplaten 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4547?src=pr&el=h1) Report
> Merging [#4547](https://codecov.io/gh/huggingface/transformers/pull/4547?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a34a9896ac2a4a33ff9cd805c76eed914c8d8965&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4547/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4547?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4547   +/-   ##
=======================================
  Coverage   77.87%   77.87%           
=======================================
  Files         123      123           
  Lines       20566    20569    +3     
=======================================
+ Hits        16016    16019    +3     
  Misses       4550     4550           
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4547?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/4547/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `99.13% <100.00%> (ø)` | |
| [src/transformers/tokenization\_longformer.py](https://codecov.io/gh/huggingface/transformers/pull/4547/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fbG9uZ2Zvcm1lci5weQ==) | `100.00% <100.00%> (ø)` | |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4547/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.66% <0.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4547?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4547?src=pr&el=footer). Last update [a34a989...453d496](https://codecov.io/gh/huggingface/transformers/pull/4547?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


LGTM, but adding @mfuntowicz here, since I'm not very familiar with the FastTokenizers. @mfuntowicz - do we also have to apply changes on the Rust side for this? 

LGTM if the tokenizer doesn't have any different pre-processing / post-processing than the current Roberta Tokenizer 👍 

Great I think it's good to merge then :-)
[Reformer] fix reformer num buckets
Author: patrickvonplaten
Pull request: yes

Fix automatic setting of `num_buckets` by making sure `num_buckets` is always a power of 2 and set as default.

The idea behind this whole function is that `num_buckets` should not be set by the user, but calculated on the fly to a good value before training (`num_buckets` ~ 2 * sequence length / chunk length as recommend in the paper). This value will then be saved in the config and can be reapplied for inference.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4564?src=pr&el=h1) Report
> Merging [#4564](https://codecov.io/gh/huggingface/transformers/pull/4564?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a34a9896ac2a4a33ff9cd805c76eed914c8d8965&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `14.28%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4564/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4564?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4564      +/-   ##
==========================================
- Coverage   77.87%   77.86%   -0.02%     
==========================================
  Files         123      123              
  Lines       20566    20569       +3     
==========================================
  Hits        16016    16016              
- Misses       4550     4553       +3     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4564?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/configuration\_reformer.py](https://codecov.io/gh/huggingface/transformers/pull/4564/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX3JlZm9ybWVyLnB5) | `100.00% <ø> (ø)` | |
| [src/transformers/modeling\_reformer.py](https://codecov.io/gh/huggingface/transformers/pull/4564/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yZWZvcm1lci5weQ==) | `87.94% <14.28%> (-0.24%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4564/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.41% <0.00%> (-0.12%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4564/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.66% <0.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4564?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4564?src=pr&el=footer). Last update [a34a989...7b40493](https://codecov.io/gh/huggingface/transformers/pull/4564?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Automatically setting number of LSH buckets in Reformer may give invalid value
Author: erickrf
Pull request: no
Labels: name: reformer

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Reformer
transformers version: 2.9.0

When using a Reformer model such that `config.num_buckets` is set to `None` (as recommended), the model automatically determines the number of necessary buckets. However, depending on some hyperparameters, it may compute an odd number of buckets, which is invalid.

It happens at this line, because of the +1 in the second element:
https://github.com/huggingface/transformers/blob/a34a9896ac2a4a33ff9cd805c76eed914c8d8965/src/transformers/modeling_reformer.py#L541

This triggers the assertion in `_hash_vectors`: https://github.com/huggingface/transformers/blob/a34a9896ac2a4a33ff9cd805c76eed914c8d8965/src/transformers/modeling_reformer.py#L454


I think a simple fix is just to check if the number is odd, and add one in that case.

Comments:
Hi @erickrf, 

Thanks a lot for catching the error. The linked PR should solve it by making sure `num_buckets` is always a power of 2.
Return token span from NerPipeline
Author: EmilStenstrom
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

I would like to suggest that NerPipeline should return the span in the original text where the matched entity exists. 

Instead of:
```python
[
    {"word": "New", "score": 0.9995751976966858, "entity": "LOC"}, 
    {"word": "York", "score": 0.9996403455734253, "entity": "LOC"}
]
```

I would like to see this:
```python
[
    {"word": "New", "score": 0.9995751976966858, "entity": "LOC", "span": (0, 3)}, 
    {"word": "York", "score": 0.9996403455734253, "entity": "LOC", "span": (4, 8)}
]
```

## Motivation

I'm trying to use transformers for NER, and I specifically want to return multi word entities as one phrase. 

With the above example, I would like to return "New York". With spans added, I would be able to merge nearby tokens into one. 

This makes it possible to differentiate the result of "A place called New, and a place called York" from "A place called New York". With the current scheme, they both return the same thing.

## Your contribution

I think I understand NerPipeline enough to make a PR, if this is something you would be open to.

Comments:
Oh, I guess a workaround is to pass in `ignore_labels=[]` when creating the pipeline. This makes the nlp call return all tokens, including the ones that are not part of NER. Then I can just chunk two tokens together if their label is the same and they are nearby. Does this make sense, or am I missing something fundamental?

Hi again, would you be open to a PR fixing this?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
❓ [BART] Why using bias for LM head if not trained ? 
Author: astariul
Pull request: no

# ❓ Questions & Help

As I understood, BART is not using a regular Linear layer as LM head, but instead reuse the weights of the shared embeddings.

As show here, biases are added :
https://github.com/huggingface/transformers/blob/a34a9896ac2a4a33ff9cd805c76eed914c8d8965/src/transformers/modeling_bart.py#L876

But these biases are registered as buffer, not as parameter. **Since they are not trained, they will always stay 0 ?**

If they stay 0, what's the point of having bias at all ?

@sshleifer 

Comments:
Great Q! The point of using them is that `MarianMTModel`, which inherits from `BartForConditionalGeneration` uses them. You're correct for the bart checkpoints they stay 0. If you think there is a comment or different approach that would be clearer, I'm very open to PR/other ideas.
How to tokenize word to characeter
Author: ynebula
Pull request: no
Labels: name: wontfix

name: Core: Tokenization

I am studying machine reading comprehension on xlmroberta.
My data is korquad.

I need to tokenize all word to character.
e.g. by english
This is a dog 
-> _T h i s _i s _a _d o g

please let me know.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
transformers.PreTrainedTokenizer.tokenize does lower case work all the time and discards space and tab. Want this changed.
Author: ChineseYjh
Pull request: no
Labels: name: wontfix

name: Core: Tokenization

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):'albert_xxlarge_zh'

Language I am using the model on (English, Chinese ...):Chinese

Two problems are all related to method `transformers.PreTrainedTokenizer.tokenize`:

1. How can I not let this method automatically lower the case of the English words in the input sentence? `tokenizer.init_kwargs["do_lower_case"]=True` doesn't work...
2. How can I not let this method discard '\t' and space in default? Or is there any method that can solve this problem?

## To reproduce

Steps to reproduce the behavior:

`
tokenizer=BertTokenizer.from_pretrained("./albert_pytorch/prev_trained_model/albert_xxlarge_zh/")`
`print(tokenizer.init_kwargs.get("do_lower_case")) #output None`
`tokenizer.init_kwargs["do_lower_case"]=True`
`print(tokenizer.init_kwargs.get("do_lower_case")) #output True`
`seq=tokenizer.tokenize("我喜欢\tAPP和WIFI。")`
`print(seq)
`

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Expected output is:
['我', '喜', '欢',**'\t'**, **'APP'**, '和', **'WIFI'**, '。']
While the actual output is:
['我', '喜', '欢', **'app'**, '和', **'wifi'**, '。']

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:2.5.1
- Platform:ubuntu
- Python version:3.6.1
- PyTorch version (GPU?):1.1.0 cuda9
- Using GPU in script?:nope
- Using distributed or parallel set-up in script?:nope


**BTW**, `python transformers-cli env` didn't work, the callback :

> python: can't open file 'transformers-cli': [Errno 2] No such file or directory

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[LongformerForQuestionAnswering] fix qa example in docstring
Author: patil-suraj
Pull request: yes

@patrickvonplaten

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4589?src=pr&el=h1) Report
> Merging [#4589](https://codecov.io/gh/huggingface/transformers/pull/4589?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/b86e42e0ac1b59f21f0eccf351d3346bbe3ed4eb&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4589/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4589?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4589   +/-   ##
=======================================
  Coverage   78.09%   78.09%           
=======================================
  Files         123      123           
  Lines       20624    20624           
=======================================
  Hits        16106    16106           
  Misses       4518     4518           
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4589?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_longformer.py](https://codecov.io/gh/huggingface/transformers/pull/4589/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19sb25nZm9ybWVyLnB5) | `97.40% <ø> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4589?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4589?src=pr&el=footer). Last update [b86e42e...368ce8e](https://codecov.io/gh/huggingface/transformers/pull/4589?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks for the PR! I actually just uploaded a pretrained question answering model from allen ai and changed the docs accordingly. So I think we don't need this PR anymore ;-). See #4593
Remove pytorch codes in Class TFXLNetMainLayer
Author: ZhuBaohe
Pull request: yes

This PR removes pytorch codes in Class TFXLNetMainLayer.

Comments:
Sorry I don't really understand how this PR removes pytorch code in `TFXLNetMainLayer` - can you explain a bit more in-detail?

@patrickvonplaten  

The codes in file modeling_tf_xlnet.py  is the XLNet Model of tensorflow implementation. So the parameter **head_mask**  should have a type of tf.Tensor or Numpy array (see Line 780 ).

But in  Lines 643-650, the parameter **head_mask** uses the methods of the pytorch, such as "expand" or "unsqueeze" which can't apply to tf.Tensor or Numpy array. 

Actually these codes are copied from modeling_xlnet.py by mistake  and should be removed.

Perfect thanks a lot! 

@LysandreJik  - the RUN_SLOW=1 tests all poss for TFXLNetMainLayer.

Good to merge for me!
[Longformer For Question Answering] Conversion script, doc, small fixes
Author: patrickvonplaten
Pull request: yes

This PR adds 
- Longformer For Question Answering doc
- Adds the link to the (official) uploaded model: https://huggingface.co/allenai/longformer-large-4096-finetuned-triviaqa
- Some minor refactoring 
- Conversion script

@ibeltagy @patil-suraj 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4593?src=pr&el=h1) Report
> Merging [#4593](https://codecov.io/gh/huggingface/transformers/pull/4593?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/b86e42e0ac1b59f21f0eccf351d3346bbe3ed4eb&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4593/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4593?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4593   +/-   ##
=======================================
  Coverage   78.09%   78.09%           
=======================================
  Files         123      123           
  Lines       20624    20625    +1     
=======================================
+ Hits        16106    16108    +2     
+ Misses       4518     4517    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4593?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/configuration\_longformer.py](https://codecov.io/gh/huggingface/transformers/pull/4593/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX2xvbmdmb3JtZXIucHk=) | `100.00% <ø> (ø)` | |
| [src/transformers/modeling\_longformer.py](https://codecov.io/gh/huggingface/transformers/pull/4593/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19sb25nZm9ybWVyLnB5) | `97.41% <100.00%> (+<0.01%)` | :arrow_up: |
| [src/transformers/tokenization\_longformer.py](https://codecov.io/gh/huggingface/transformers/pull/4593/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fbG9uZ2Zvcm1lci5weQ==) | `100.00% <100.00%> (ø)` | |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4593/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.53% <0.00%> (+0.11%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4593?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4593?src=pr&el=footer). Last update [b86e42e...d8d4187](https://codecov.io/gh/huggingface/transformers/pull/4593?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
[Reformer] automate axial_pos_shape
Author: flozi00
Pull request: yes

This PR automates the calculation of the axial_pos_shape.
checked that every combination of 2**n works using this sheet: https://docs.google.com/spreadsheets/d/19gnP1ve2fT2F59LNiky44SPpmtmOegfIlk5_3OFXjyU/edit?usp=sharing

@patrickvonplaten 


Comments:
Hi @flozi00,

Thanks for the PR! To be honest I don't think we should merge this. A couple of reasons:

1. In contrast to `num_buckets` which is said in the paper to be always around ~ `2 * sequence_length / chunk_length` `axial_pos_shape` can be freely set by the user. 
2. We are trying to add as little automatic settings that are not visible to the user as possible (also @thomwolf here). The reason is that it can later lead to errors that are hard to understand for the user. In this case, I don't think the user should use AxialPositionEmbeddings before having read the docs and understood how it works. Automatically setting the `num_buckets` is already suboptimal in this sense. 

BTW, I just answered your email - sorry I forgot about this
[Model hub web parsing MD code error]
Author: mrm8488
Pull request: no

# 🐛 Bug

## Information

Hi guys!
If you navigate to https://github.com/huggingface/transformers/blob/master/model_cards/mrm8488/bert-italian-finedtuned-squadv1-it-alfa/README.md

You will see the emojis without problem. But if you go to its HTML page on the model hub: https://huggingface.co/mrm8488/bert-italian-finedtuned-squadv1-it-alfa
Emojis are not shown

Comments:
Yes, that is a GitHub-only (GFM) feature, while we use marked.js (https://marked.js.org/) for markdown parsing.

You'll have to use the actual emojis for now
Add BART fine-tuning summarization community notebook
Author: ohmeow
Pull request: yes

An example of how to train, evaluate, deploy a BART summarization model with fastai using the blurr library

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4539?src=pr&el=h1) Report
> Merging [#4539](https://codecov.io/gh/huggingface/transformers/pull/4539?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a34a9896ac2a4a33ff9cd805c76eed914c8d8965&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4539/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4539?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4539      +/-   ##
==========================================
- Coverage   77.87%   77.87%   -0.01%     
==========================================
  Files         123      123              
  Lines       20566    20566              
==========================================
- Hits        16016    16015       -1     
- Misses       4550     4551       +1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4539?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4539/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.41% <0.00%> (-0.12%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4539/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.66% <0.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4539?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4539?src=pr&el=footer). Last update [a34a989...e8c7951](https://codecov.io/gh/huggingface/transformers/pull/4539?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Awesome thanks for the great notebook. Did a tiny change in the github link :-)