Pegasus Tokenizer is throwing away newline tokens
Author: njbrake
Pull request: no

### System Info

 it's important for my model to learn where the newlines should be placed in the output, and from my understanding, this information is being removed by the Pegasus tokenizer (applicable to the latest version of the Transformers/Tokenizer library):

For example, if my target output is

```
SECTION HEADING \n\nHere is the output for this section, cool!
```
If I encode and decode through the tokenizer, it becomes
```
SECTION HEADING Here is the output for this section, cool!
So I guess my question would be
```
Am I missing something, and is there some toggle I can enable that would allow for the tokenizer to preserve new lines?
If there is not a toggle, is there a reason that one shouldn't be added?
Of course I have the option of pre-processing my text to convert new lines to `<n>` and then post-processing to turn the `<n>` back to `\n`, but seems a little hacky for my liking 😅

### Who can help?

@ArthurZucker, @younesbelkada

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction


```
tokenizer = transformers.AutoTokenizer.from_pretrained("google/pegasus-x-base")
sample = "I am a section \n \n Now I should be a few lines below!"
inputs = tokenizer.encode(sample, return_tensors="pt")
out = tokenizer.decode(inputs[0])
```
Out is 
```
'I am a section Now I should be a few lines below!
'
```
So it is stripping out the newline characters


### Expected behavior

It should not strip out the newline characters, or I should have the option to tell the tokenizer not to remove newlines (This functionality may already exist and I'm just unaware of it)

Comments:
Even more concise:
```
>>> tokenizer.encode('\n')
[1]
>>> tokenizer.decode(1)
'
'
```

Hey! Thanks for posting this. 
The reason why the new lines are automatically removed is because this is the default behavior for `Pegasus` see [here](https://github.com/google-research/pegasus/blob/939830367bcf411193d2b5eca2f2f90f3f9260ca/pegasus/ops/sp_text_encoder.cc#L79), where they have a `preserve_new_line` variable. I don't really know why we don't, but you can add this token as part of the `added_vocab` in order to make sure that it is tready as a special token and not removed. That is the quickest fix  .

Otherwise, I can open a PR to add this argument (as it was in the original code I guess it makes sense?). I also need to check what it the common practice for that, cc @Narsil if you have an idea

The `\n` characters get removed by the normalization of this model.

It's both within the `setnencepiece.model` file and the equivalent fast tokenizer (PrecompiledCharMap).
The only way to get those back would be to modify the normalizer. But since the model vocab doesn't contain any such tokens, you're going to end up with only `unk` everywhere there's a newline.

Is it possible for you to use `return_offsets_mapping` to get the offsets and see where those missing values are ?

```python
encoded = tokenizer("This \n is", return_offsets_mapping=True)
encoded["offset_mapping"]
# [(0, 4), (7, 9), (0, 0)]
```
Those show the "hole"  within the original string. This can account for newlines as well as other regularized content. It's the only generally working approach for these kind of things.

Since there is no newline in the vocab, the model will never be able to output back any new lines so I'm not sure adding an option will be of any help here @ArthurZucker 

Those offsets show there's

Thanks for the feedback, @Narsil  and @ArthurZucker! Based on your feedback I did this:
```
>>> tokenizer = transformers.AutoTokenizer.from_pretrained("google/pegasus-x-base")
>>> token = tokenizers.AddedToken(content="\n", normalized=False)
>>> tokenizer.add_tokens(list([token])
>>> sample = "I am a section \n \n Now I should be a few lines below!"
>>> inputs = tokenizer.encode(sample, return_tensors="pt")
>>> inputs
tensor([[  125,   346,   114,  1201, 96103, 96103,  1032,   125,   246,   129,
           114,   324,  1569,   487,   147,     1]])
>>> out = tokenizer.decode(inputs[0])
>>> out
'I am a section\n\n Now I should be a few lines below!
'
```
So it appears that everything is ok now. Is there something wrong with my approach, since it's not exactly what you recommended?

It is exactly what I was recommending 👍🏻 also you can now push the tokenizer to the hub and that should be it! 
If you do not want the model to strip right and left for this token, you can also control that. Glad this solved your issue! 

Can't we just replace all the new line characters `\n` with `<n>`, which I believe is its equivalent in Pegasus?