noamaanMulla-03/transformers-issues huggingface dataset:
Convert hans to Trainer
Author: sgugger
Pull request: yes

This follows up from #4854 (@julien-c I took all your comments into account) and will close #4742.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5025?src=pr&el=h1) Report
> Merging [#5025](https://codecov.io/gh/huggingface/transformers/pull/5025?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/1bf4098e03afaed2c6e3671c69fd57e9ac304752&el=desc) will **decrease** coverage by `0.09%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5025/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5025?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5025      +/-   ##
==========================================
- Coverage   77.18%   77.09%   -0.10%     
==========================================
  Files         128      128              
  Lines       21877    21877              
==========================================
- Hits        16886    16866      -20     
- Misses       4991     5011      +20     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5025?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/5025/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `76.35% <0.00%> (-2.30%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5025/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `85.86% <0.00%> (-1.40%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5025/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.73% <0.00%> (+0.23%)` | :arrow_up: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5025/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.79% <0.00%> (+0.40%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5025?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5025?src=pr&el=footer). Last update [1bf4098...201fae2](https://codecov.io/gh/huggingface/transformers/pull/5025?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
how can i download the model manually?
Author: shange1996
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I want to download the model manually because of my network. But now I can only find the download address of bert. Where is the address of all models? Such as XLNET。

Comments:
- xlnet-base-cased : https://s3.amazonaws.com/models.huggingface.co/bert/xlnet-base-cased-config.json
- xlnet-large-cased : https://s3.amazonaws.com/models.huggingface.co/bert/xlnet-large-cased-config.json

I got it from [here](https://huggingface.co/transformers/_modules/transformers/configuration_xlnet.html#XLNetConfig)  
Go to the source of that model in hugging face repo and there you can find the links set in config.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I used [this link ](https://huggingface.co/jplu/tf-xlm-roberta-large#tensorflow-xlm-roberta)to download the tf-xlm-roberta-base. But such information is not available for every model.

As @nauman-chaudhary indicated some are available [here](https://huggingface.co/transformers/_modules/transformers/configuration_xlnet.html#XLNetConfig)..
Pipelines: use tokenizer.max_len
Author: sshleifer
Pull request: yes

fix #4224 by passing tokenizer.max_len to `tokenizer.batch_encode_plus`


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4236?src=pr&el=h1) Report
> Merging [#4236](https://codecov.io/gh/huggingface/transformers/pull/4236?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/cf08830c2868c5335b1657f95779f370862e3a28&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4236/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4236?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4236      +/-   ##
==========================================
- Coverage   78.38%   78.37%   -0.02%     
==========================================
  Files         120      120              
  Lines       19918    19918              
==========================================
- Hits        15613    15611       -2     
- Misses       4305     4307       +2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4236?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/pipelines.py](https://codecov.io/gh/huggingface/transformers/pull/4236/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9waXBlbGluZXMucHk=) | `76.56% <ø> (+0.19%)` | :arrow_up: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4236/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.48% <0.00%> (-0.49%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4236/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `92.93% <0.00%> (+0.16%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4236?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4236?src=pr&el=footer). Last update [cf08830...158cd06](https://codecov.io/gh/huggingface/transformers/pull/4236?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


This PR is actually very related to this issue: #4501

Abandoning.
Fix marian tokenizer save pretrained
Author: sshleifer
Pull request: yes
Labels: name: marian


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5043?src=pr&el=h1) Report
> Merging [#5043](https://codecov.io/gh/huggingface/transformers/pull/5043?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/36434220fc807c5015bc8f0f1e50ab21f7d34914&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5043/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5043?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #5043   +/-   ##
=======================================
  Coverage   77.36%   77.37%           
=======================================
  Files         130      130           
  Lines       21989    21990    +1     
=======================================
+ Hits        17012    17014    +2     
+ Misses       4977     4976    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5043?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_marian.py](https://codecov.io/gh/huggingface/transformers/pull/5043/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fbWFyaWFuLnB5) | `92.85% <100.00%> (+0.96%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5043?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5043?src=pr&el=footer). Last update [3643422...5899f7a](https://codecov.io/gh/huggingface/transformers/pull/5043?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
NER: fix construction of input examples for RoBERTa
Author: stefan-it
Pull request: yes

Hi,

this PR avoids adding an extra `
` symbol, so that the final sequence ends with `
 
`.

The documentation clearly states, that ther's only one `
` expected at the end of a single sequence:

https://huggingface.co/transformers/model_doc/roberta.html#transformers.RobertaTokenizer.build_inputs_with_special_tokens

The `fairseq` reference implementation shows also only one `
` at the end:

https://github.com/pytorch/fairseq/tree/master/examples/roberta#apply-byte-pair-encoding-bpe-to-input-text

Technically, the `sep_token_extra` argument is set to `False` now - I didn't remove this parameter, so future models/tokenizers can use it (when they really need it).

This fixes #4755.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4943?src=pr&el=h1) Report
> Merging [#4943](https://codecov.io/gh/huggingface/transformers/pull/4943?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/473808da0d476792070f0e7dfebcf1121a12a34f&el=desc) will **increase** coverage by `0.06%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4943/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4943?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4943      +/-   ##
==========================================
+ Coverage   77.14%   77.20%   +0.06%     
==========================================
  Files         128      128              
  Lines       21745    21745              
==========================================
+ Hits        16775    16789      +14     
+ Misses       4970     4956      -14     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4943?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4943/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.42% <0.00%> (ø)` | |
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/4943/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `78.65% <0.00%> (+2.29%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4943?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4943?src=pr&el=footer). Last update [473808d...e9eb626](https://codecov.io/gh/huggingface/transformers/pull/4943?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


LGTM, thanks!
Update pipeline examples to doctest syntax
Author: sgugger
Pull request: yes

Also fix the values to what's returned. This way we can run
```
python -m doctest README.md
```
to test this codes produces the exact same results. Not sure if we want to include this in our CI in some way.


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5030?src=pr&el=h1) Report
> Merging [#5030](https://codecov.io/gh/huggingface/transformers/pull/5030?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/bbad4c6989d489097f42bbe38001a3f8ca1c5c11&el=desc) will **decrease** coverage by `0.07%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5030/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5030?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5030      +/-   ##
==========================================
- Coverage   77.19%   77.12%   -0.08%     
==========================================
  Files         128      128              
  Lines       21877    21877              
==========================================
- Hits        16888    16872      -16     
- Misses       4989     5005      +16     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5030?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/5030/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `76.35% <0.00%> (-2.30%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5030/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.38% <0.00%> (-0.41%)` | :arrow_down: |
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/5030/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `39.34% <0.00%> (-0.24%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5030?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5030?src=pr&el=footer). Last update [bbad4c6...db3cb29](https://codecov.io/gh/huggingface/transformers/pull/5030?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


I didn't know this feature, this is pretty cool
TypeError: function() argument 1 must be code, not str
Author: manhlab
Pull request: no

# 🐛 Bug
TypeError: function() argument 1 must be code, not str
## Information

@dataclass
class T2TDataCollator(DataCollator):



Comments:
I get this error when create datacollator class

Could you provide more information? It's a bit hard to help you here. What is the code you're using and what is the stacktrace?

thanks you! i have the same problem with this #5049
❓ [TFTrainer] How to run on 8 TPU cores ?
Author: astariul
Pull request: no
Labels: name: TensorFlow

# ❓ Questions & Help

I'm trying to run TFTrainer on 8 TPU cores, but I don't understand how to make it work.

I tried running my script with the flags `--tpu_num_cores 8 --per_device_train_batch_size 8`, expecting each core to handle a batch size of 1.

But when I print the shape of my inputs, I have `[8, x]` instead of `[1, x]`, which leads to memory error.

---

If I start the training with `--tpu_num_cores 8 --per_device_train_batch_size 1`, the shape of inputs is correct (`[1, x]`), but the number of optimization steps computed is not correct (If I have 8k samples, it says I have 8k optimization steps, but I expected 1k steps because I am using 8 TPU cores...).

---

Am I doing something wrong ? **How can I train on 8 TPU cores, with a batch size of 1 for each core ?**

Comments:
Hello !

This is because the `--tpu_num_cores` is not taken into account yet. If you want to use TPUs, just fill the TPU name with `--tpu_name` and it will detect automatically the number of cores. For now TPUs with TF Trainer is under development so some use cases might not work properly.

Thanks for the input @jplu 

So as you mentioned the number of TPU cores is automatically detected. It is accessible with `training_args.n_gpu`.

Also I didn't notice but here : https://github.com/huggingface/transformers/blob/e4aaa4580515446cd5a2972ab42fec0b95819c84/src/transformers/training_args.py#L150

The batch size is automatically adjusted to the number of cores.

So the behavior I observed is completely normal, as `--per_device_train_batch_size` is the batch size **per TPU cores**.
Fine Tuning Bert for Q&A
Author: priteshpatel15
Pull request: no
Labels: name: wontfix

This is more of a theoretical question.  I would like to use a Bert Model trained on SQUAD 2.0 then train it further on my domain's Q&A dataset.

How would I do that.  I've read through the code.  As I understand it, I see that the BertforQuestionAnswering would be what I would need to use loaded with a model that is fine tuned on Squad so the weights match the architecture.

But now I want to further fine tune this model to include Q&A training data from my target domain.  How would I do that?

Comments:
A quick workaround would be appending your data to the SQUAD training dataset and doing the fine-tuning as usual.


Yes thats an approach.  I've been reading a bit more about GPT and GPT-2 ... i'm wondering if I could use the generative approach with fine-tuning on a specific task that would help with SQUAD Q&A ability for my target domain?  What are people's thoughts.  Does the math work out?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> This is more of a theoretical question. I would like to use a Bert Model trained on SQUAD 2.0 then train it further on my domain's Q&A dataset.
> 
> How would I do that. I've read through the code. As I understand it, I see that the BertforQuestionAnswering would be what I would need to use loaded with a model that is fine tuned on Squad so the weights match the architecture.
> 
> But now I want to further fine tune this model to include Q&A training data from my target domain. How would I do that?

Hi @priteshpatel15 I am interested in this problem. Have you found a more elegant solution than what Matthew suggested above?
distil-finetuning in run_squad
Author: VictorSanh
Pull request: yes

- Add the option for double loss: fine-tuning + distillation from a larger squad-finetune model.
- Fix `inputs` for `DistilBERT` (also see fix in  `run_glue.py` 702f589848baba97ea4897aa3f0bb937e1ec3bcf)

Comments:
cf https://github.com/huggingface/transformers/issues/1193#issuecomment-534740929

Ok, as discussed let's copy this script to the `examples/distillation` folder and keep `run_squad` barebone for now as it's going to evolve in the short term.

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1361?src=pr&el=h1) Report
> Merging [#1361](https://codecov.io/gh/huggingface/transformers/pull/1361?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/2dc8cb87341223e86220516951bb4ad84f880b4a?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1361/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1361?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1361   +/-   ##
=======================================
  Coverage   84.69%   84.69%           
=======================================
  Files          84       84           
  Lines       12596    12596           
=======================================
  Hits        10668    10668           
  Misses       1928     1928
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1361?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1361?src=pr&el=footer). Last update [2dc8cb8...b4df865](https://codecov.io/gh/huggingface/transformers/pull/1361?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


no squash @VictorSanh? 😬
SummarizationPipeline crashes
Author: julien-c
Pull request: no
Labels: name: Core: Pipeline

name: Summarization

```python
summarize = pipeline("summarization")
summarize("Sam Shleifer writes the best docstring examples in the whole world.")
```
➡️
```
/usr/local/lib/python3.6/dist-packages/transformers/pipelines.py in _parse_and_tokenize(self, pad_to_max_length, *args, **kwargs)
    461         # Parse arguments
    462         inputs = self._args_parser(*args, **kwargs)
--> 463         inputs = self.tokenizer.batch_encode_plus(
    464             inputs, add_special_tokens=True, return_tensors=self.framework, pad_to_max_length=pad_to_max_length,
    465         )

AttributeError: 'dict' object has no attribute 'batch_encode_plus'
```

Comments:
Is this issue fixed in version 2.10.0?

@julien-c I still get the same error when doing

```
summarizer = pipeline('summarization')
```

and using it to summarize.

However the following explicitely works for me:

```
summarizer = pipeline('summarization', model='bart-large-cnn', tokenizer='bart-large-cnn')
```

Yeah that sounds like this issue. It will be fixed in the next release or you can build from source with
```bash
git clone [this repo]
pip install -e .
```

> Yeah that sounds like this issue. It will be fixed in the next release or you can build from source with
> 
> ```shell
> git clone [this repo]
> pip install -e .
> ```

I have installed the package from GitHub repo but still have the same issue right now.

@khalilRhouma: It works for me at commit d976ef262e0b2c52363d201b2e14e5ecc42abbb3 , so you may need to `git pull` or some such. If that doesn't work I would love to see the output of 
```bash
transformers-cli env
```




@sshleifer I get this error when I clone with that commit ID.
KeyError: "Unknown task summarization, available tasks are ['feature-extraction', 'sentiment-analysis', 'ner', 'question-answering', 'fill-mask']"
@dipanjanS  Would be great to know what configuration you used

current master should also work.

@sshleifer The kernel still crashes
Attaching the code.
```
from transformers import pipeline
import torch

!git clone https://github.com/huggingface/transformers.git
%cd transformers
`!pip` install -e ".[dev]"

#summarizer = pipeline("summarization")
summarizer = pipeline('summarization', model='facebook/bart-large-cnn', tokenizer='facebook/bart-large-cnn') ##Kernel dies after running this line
```

transformers version - 2.11.0
torch - 1.5.0

Can't replicate :(.
Can I see your `transformers-cli env` output?

How do I get that output? I'm running these on Jupyter without any virtual env

Got it.
- `transformers` version: 2.11.0
- Platform: Linux-4.14.181-108.257.amzn1.x86_64-x86_64-with-glibc2.9
- Python version: 3.6.10
- PyTorch version (GPU?): 1.5.0 (False)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: <No>
- Using distributed or parallel set-up in script?: <No>

@sshleifer It works finally. There was a problem with GPU allocation. Thanks for your response.
Question about whitespace filtering in squad data processor 
Author: Santosh-Gupta
Pull request: no
Labels: name: wontfix

Usually to detect for whitespaces in python, the `isspace()` built-in is used. But I noticed for the squad data processor, this is used instead


```
def _is_whitespace(c):
    if c == " " or c == "\t" or c == "\r" or c == "\n" or ord(c) == 0x202F:
        return True
    return False
```

https://github.com/huggingface/transformers/blob/master/src/transformers/data/processors/squad.py#L80

Is there any reason why this is used instead? My guess would be to deal with strings that were processed outside of python. 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.