noamaanMulla-03/transformers-issues huggingface dataset:
How to save tokenize data when training from scratch
Author: 008karan
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

I am training Allbert from scratch following the blog post by hugging face. As it mentions that :

> If your dataset is very large, you can opt to load and tokenize examples on the fly, rather than as a preprocessing step.

How this can be done any suggestions?
As of now , using the method given in the notebook:
```
from transformers import TextDataset

dataset = TextDataset(
    tokenizer=tokenizer,
    file_path="./oscar.eo.txt",
    block_size=128,
)
```
 there is no method to save tokenize data, can anyone suggest how to save that as its already taking long enough before starting training. 

     




Comments:
There is a method to save tokenizer. Check this notebook: https://github.com/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb



Thats what I am using.
its saving it in the `dataset` variable not in any file. ByTokenize data I mean pretraining data.

You can look at serialization practices, you should able to do with torch at least.
https://huggingface.co/transformers/serialization.html#serialization-best-practices

well thats for all required model files. I am not getting how to save pretraining data

Once you have your data you can pickle it or use `torch.save` to save it to your disk and reload it later.

That worked @LysandreJik thanks! 
I still not getting how you can prepare pretraining data on the fly while training. I got large training data and don't want to wait until it gets prepared for training.

Have you taken a look at PyTorch's Dataset/Dataloader utilities? I recommend taking a look at [loading hude data functionality](https://discuss.pytorch.org/t/loading-huge-data-functionality/346) or [how to use a dataset larger than memory](https://discuss.pytorch.org/t/how-to-use-dataset-larger-than-memory/37785/8).

I personnally prefer using IterableDatasets when loading large files, as I find the API easier to use to limit large memory usage. This [tutorial](https://medium.com/swlh/how-to-use-pytorch-dataloaders-to-work-with-enormously-large-text-files-bbd672e955a0) is interesting on that subject.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
customize special tokens
Author: XiaoLiuAI
Pull request: no

# ❓ Questions & Help

## Details

I am trying to add more special tokens in order to train new language models without modifying the model architecture but simply modifying the input data. For example, adding more separators to involving more complex structure information. 

Is there a simple, clear, consistent way to add/customize special tokens. 
1. The function `add_special_tokens` can only add token string to token attribute specified in `SPECIAL_TOKENS_ATTRIBUTES`
2. I tried to extend this list and `add_special_tokens` works, but calling 
`tokenizer.get_special_tokens_mask` report that _XXX does not exist, where XXX is the new customized special token attribute

finally I found a solution: modifying `tokenizer._additional_special_tokens.extend` directly makes `tokenizer.get_special_tokens_mask` work. But I don't know how could this work and if there is any other side effect. 

Properties like `special_tokens_map` is also confusing, it seems like a @property implementation, which is not modifiable. And I don't know where this could be used to have what effect. 

I cost much time to read the source code but still lack a overview about the special token architecture in tokenizers. 

Comments:
For special tokens that don't correspond to one of the attributes of tokenizers, you should pass them to `add_special_tokens` with the `additional_special_tokens` keyword arguments:
```
tokenizer.add_special_tokens(additional_special_tokens = ['TK1', 'TK2' ...])
```
You can then change `tokenizer.additional_special_tokens` directly if you need to add or remove some of those tokens.

@sgugger Thank you very much.
Codecov migration to marketplace app
Author: thomasrockhu
Pull request: no
Labels: name: wontfix

Hi, Tom from Codecov here.

We noticed that you are using our app with high frequency, and we’re so excited to see that! However, because you are not using our app, you may have experienced issues with uploading reports or viewing coverage information. This is due to rate-limiting from GitHub.

**In order to prevent any future outages, we ask that you move over to our GitHub marketplace app: https://github.com/marketplace/codecov.**

Let me know if you have any questions, or if I can help at all with this process.

Comments:
cc @LysandreJik @thomwolf 

Hi @thomasrockhu, indeed we've faced such issues before. We'll take a look, thanks!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Recommended preprocessing steps for english sentences in GPT2
Author: Damiox
Pull request: no

# ❓ Questions & Help

To run inferences in english sentences... I'm not really sure what preprocessing steps I need to do before sending the tokenized text to gpt-2 in order to get the predictions. Any advice?

How to handle non-english words that can be found in an english sentences? Extra punctuation, space, new lines, user's mentions, hashtags, urls, alternative apostrophes, etc...?

## Details
<!-- Description of your issue -->

**A link to original question on Stack Overflow**:  https://stackoverflow.com/questions/60807799/gpt2-huggingfaces-transformer-preprocessing

Comments:
Just pre-process them as you would for any other NLP task. This may include: normalisation of punctuation, dealing with HTML/XML, white-space normalisation, language verification. Something that people used to do in the age of word2vec is creating new entities for specific items. For instance, any number could be replaced with all 1s (1234 becomes 1111) to have relatively small vocabulary that still takes the number of characters into account. Same with users (e.g @user@) and mentions and urls (@URL@), and so on. This might still be worth the effort, but not necessarily. In such cases you may want to ensure that these tokens are not split in the tokenizer.

I hope that you understand that this is a very general question not related to this repository, so I am closing this.

@BramVanroy I think I forgot to mention that I'm using the gpt-2 training models from this repo. I'm not re-training gpt-2. I think I shouldn't create new entities for specific terms if that process has not happened during training step. Am I right? 

If you are not planning to pretraining the model (and generating a new vocab) then, indeed, you should not try to add new tokens. So in your case you would just need to do some basic normalisation of punctuation, HTML, etc.

@BramVanroy Thanks for the answer. I'm directly using gpt-2 pretrained models from https://huggingface.co/models . Specifically the ones that are created by Huggingface. I can't find the preprocessing code that was used when those models were trained... so I can replicate the same at inference time. I'm wondering if I should just assume things or it'd be safer to see how it was trained so I can make sure to prepare the sentences in a similar way.

Has there been any preprocessing during training phase @BramVanroy ? 

I don't know. Note that HuggingFace didn't train GPT2. They ported the weights to their own architecture. You can try to get into contact with the people that created it, OpenAI. https://github.com/openai/gpt-2

@Damiox have you found the original preprocessing code?

@don-prog no, I have not 😢 - I am doing some subset of the preprocessing heuristics that @BramVanroy detailed before when serving the model. But I still think it'd be really good to have a consistency between both preprocessing mechanisms: training (whatever it was) vs inference. I just haven't had the time to identify that original preprocessing code from OpenAI
How to deal with summarization task to long sequences input? 
Author: laetokang
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

I am going to carry out the sumarization task using the 'transformers' module you provided. But there's a problem. The sequence which i have is too long, so an error occurs when inserting it. Is there any way to summarize the entire document by sliding it?

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Comments:
Usually, the input is simply cut in this case. Bart cuts the input to 1024 tokens when training on CNN Daily Mail. T5 cuts the input to 512 tokens when training on CNN Daily Mail.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Better hyperparameter tensorboard logging in Trainer.
Author: jaymody
Pull request: no

# 🚀 Feature request

`Trainer` should write `train_batch_size` and `eval_batch_size` to tensorboard. Currently only `per_device_` batch sizes are logged as hyperparamters, which means unless you know how many gpus you trained on (which is also not logged), you can't know the actual batch sizes used in training.

## Your contribution
 
Submitted a PR: [Adds train_batch_size, eval_batch_size, and n_gpu to to_sanitized_dict output for logging.](https://github.com/huggingface/transformers/pull/5331#issue-440857548)


Comments:
Problems with TFT5
Author: Guillem96
Pull request: yes

As far as we are concerned, Tensorflow T5 has some problems:

- Shifting inputs right should add at the beginning of the sequences a *Start of Sequence token* (padding token in T5 case), but currently, it generates a useless sequence of zeros. This is due to this line of code:

```python
shifted_input_ids = tf.zeros_like(input_ids, dtype=tf.int32) # No matter what, this generates zeros
# ... carry this 0s all over the function
```

*Note*

We observe, that PyTorch's `T5ForConditionalGeneration` loss is a scalar. Contrary, with the `TFT5ForConditionalGeneration`, the loss is a tensor where each loss element corresponds to a *non-ignored* label. We think that the loss should be reduced. Is there any reason for keeping the loss *as is* instead of reducing it? 



Comments:
Closing in favor of #6214
s2s: fix LR logging, remove some dead code.
Author: sshleifer
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6205?src=pr&el=h1) Report
> Merging [#6205](https://codecov.io/gh/huggingface/transformers/pull/6205?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/82a0e2b67ec94d28b20e24b3393644002bbd0d4b&el=desc) will **increase** coverage by `0.07%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6205/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6205?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6205      +/-   ##
==========================================
+ Coverage   79.65%   79.73%   +0.07%     
==========================================
  Files         146      146              
  Lines       26607    26607              
==========================================
+ Hits        21194    21214      +20     
+ Misses       5413     5393      -20     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6205?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_xlm\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/6205/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25feGxtX3JvYmVydGEucHk=) | `84.52% <0.00%> (-10.72%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6205/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `81.70% <0.00%> (+1.00%)` | :arrow_up: |
| [src/transformers/tokenization\_bart.py](https://codecov.io/gh/huggingface/transformers/pull/6205/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYmFydC5weQ==) | `95.77% <0.00%> (+35.21%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6205?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6205?src=pr&el=footer). Last update [82a0e2b...f39a67f](https://codecov.io/gh/huggingface/transformers/pull/6205?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
add XLMRobertaForQuestionAnswering
Author: Liangtaiwan
Pull request: yes
Labels: name: wontfix

1. add XLMRobertaForQuestionAnswering in modeling_xlm_roberta.py
2. add XLMRobertaForQuestionAnswering in modeling_auto.py

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Closing since already implemented
Error when fine tuning GPT2 on GPU 
Author: cppntn
Pull request: no

An error occur when training with `run_language_modeling.py` file (Ubuntu 18, pytorch with cuda compiled), while the error does not occur on Macbook without the GPU

```
  File "train.py", line 283, in <module>
    main()
  File "train.py", line 247, in main
    trainer.train(model_path=model_path)
  File "/home/antonio/anaconda3/lib/python3.7/site-packages/transformers/trainer.py", line 518, in train
    for step, inputs in enumerate(epoch_iterator):
  File "/home/antonio/anaconda3/lib/python3.7/site-packages/tqdm/_tqdm.py", line 1017, in __iter__
    for obj in iterable:
  File "/home/antonio/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 346, in __next__
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/antonio/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/home/antonio/anaconda3/lib/python3.7/site-packages/transformers/data/data_collator.py", line 90, in __call__
    labels[labels == self.tokenizer.pad_token_id] = -100
TypeError: eq() received an invalid combination of arguments - got (NoneType), but expected one of:
 * (Tensor other)
      didn't match because some of the arguments have invalid types: (NoneType)
 * (Number other)
      didn't match because some of the arguments have invalid types: (NoneType)
```

Command and arguments used to run:
```bash
python train.py \
    --output_dir ckpts/prova \
    --model_type=gpt2 \
    --model_name_or_path=gpt2 \
    --do_train \
    --train_data_file data.train \
    --num_train_epochs 1 \
    --per_device_train_batch_size 1 \
    --gradient_accumulation_steps 1 \
    --save_steps 200
```

Any idea on how to solve this?

Comments:
Hi @antocapp  thank you for reporting this. ignore index (-100) shouldn't be set when `pad_token_id` is `None`, which is the case with GPT-2 

Hi @patil-suraj , thanks for the answer! Where should I set this parameter?

edit: Never mind, just saw the fix you merged. Thanks!
How to fine tune EncoderDecoder model for training a new corpus of data ?
Author: banunitte
Pull request: no
Labels: name: wontfix

name: Core: Encoder-Decoder

is there any documentation available for the same?

Comments:
We are currently working on implementing the encoder decoder framework. See PR: https://github.com/huggingface/transformers/pull/3383

I think in a week it should be ready :-) 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


See https://huggingface.co/patrickvonplaten/bert2bert-cnn_dailymail-fp16#bert2bert-summarization-with-%F0%9F%A4%97-encoderdecoder-framework

Thank u Patrick

On Mon, Aug 3, 2020 at 10:19 PM Patrick von Platen <notifications@github.com>
wrote:

> See
> https://huggingface.co/patrickvonplaten/bert2bert-cnn_dailymail-fp16#bert2bert-summarization-with-%F0%9F%A4%97-encoderdecoder-framework
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/3785#issuecomment-668127257>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABUHUMC742YUNXLDJDGVCGTR63TBPANCNFSM4MHODIYA>
> .
>
BertTokenizer custom UNK unexpected behavior
Author: idocal
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [X] my own modified scripts: (give details)
Importing transformers to my own project but using BertTokenizer and BertModel with pretrained weights, using 'bert-base-multilingual-cased' for both tokenizer and model.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details)
Fine tuning BERT for English NER on a custom news dataset.

## To Reproduce

Steps to reproduce the behavior:

1. Initialize a tokenizer with custom UNK
2. Try to convert the custom UNK to ID
3. Receive None as 

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
```
>>>tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased', do_lower_case=False, pad_token="<pad>", unk_token="<unk>")
>>> tokenizer.tokenize("<unk>")
['<unk>']
>>> tokenizer.convert_tokens_to_ids(["<unk>"])
[None]
```

## Expected behavior

My custom UNK should have an ID. (instead, I get None)

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Mac OSX
* Python version: 3.7.3
* PyTorch version: 1.1.0.post2
* PyTorch Transformers version (or branch): 2.1.1
* Using GPU ? No
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I actually have the same problem with GPT-2 tokenizer. Is this the expected behavior?
T5 Tensorflow: _shift_right returns wrong result
Author: maurice-g
Pull request: no

# 🐛 Bug

## Information

The [_shift_right](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_tf_t5.py#L777) method of the `TFT5PreTrainedModel` returns all zeros instead of shifting the `input_ids`.

## To reproduce

```
import tensorflow as tf

def shape_list(x):
    """Deal with dynamic shape in tensorflow cleanly."""
    static = x.shape.as_list()
    dynamic = tf.shape(x)
    return [dynamic[i] if s is None else s for i, s in enumerate(static)]

def _shift_right(input_ids):
    decoder_start_token_id = 0
    pad_token_id = 0

    assert (
        decoder_start_token_id is not None
    ), "self.model.config.decoder_start_token_id has to be defined. In TF T5 it is usually set to the pad_token_id. See T5 docs for more information"

    # shift inputs to the right
    shifted_input_ids = tf.zeros_like(input_ids, dtype=tf.int32)
    shifted_input_ids = tf.roll(shifted_input_ids, 1, axis=-1)
    start_tokens = tf.fill((shape_list(shifted_input_ids)[0], 1), decoder_start_token_id)
    shifted_input_ids = tf.concat([start_tokens, shifted_input_ids[:, 1:]], -1)

    assert pad_token_id is not None, "self.model.config.pad_token_id has to be defined."
    # replace possible -100 values in labels by `pad_token_id`
    shifted_input_ids = tf.where(
        shifted_input_ids == -100, tf.fill(shape_list(shifted_input_ids), pad_token_id), shifted_input_ids
    )

    assert tf.math.reduce_any(
        shifted_input_ids >= 0
    ).numpy(), "Verify that `labels` has only positive values and -100"

    return shifted_input_ids


input_ids = tf.convert_to_tensor([[32000, 1, 2, 3, 0, 0, 0]])
print(_shift_right(input_ids))
```

## Expected behavior

Should shift the tensor to the right.

## Suggested solution

Replace line `shifted_input_ids = tf.zeros_like(input_ids, dtype=tf.int32)` with `shifted_input_ids = tf.cast(input_ids, tf.int32)`.
Further I'd recommend removing the assertion for positive label values, as it depends on the `numpy()` method, which is not available in some cases (e.g. when using datasets loaded from tfrecord files) and will throw an error then.
Shall I open a PR directly?

## Environment info     
- `transformers` version: Master (file commit `4dc6559`)
- Tensorflow version: 2.1.0


Comments:
Hi, thanks for raising this issue! Indeed, this looks like an issue. Do you want to open a PR with the fix you propose?

Hey @maurice-g, 

Thanks a lot for the fix!