noamaanMulla-03/transformers-issues huggingface dataset:
tokenizer.vocab has not changed after using add_tokens
Author: suyulan
Pull request: no
Labels: name: wontfix

name: Core: Tokenization

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->

When I use the add_tokens, I have the following problems
```python3
# len(tokenizer)  == 30522
 tokens_dict = ['[HL]']
 num_added_toks = tokenizer.add_tokens(tokens_dict)
# len(tokenizer)  == 30523
# But tokenizer.vocab_size == 30522
```
Should I change the dictionary by myself ?


Comments:
This is the expected behaviour: `len(tokenizer)` shows you the actual size (including the added tokens), whereas `.vocab_size` tells you the original size.

https://github.com/huggingface/transformers/blob/a08652772791fdaeed6f263b1a99926ca64be5dc/src/transformers/tokenization_utils.py#L2285-L2290

PS: don't forget to update your model's embeddings!

```python
model.resize_token_embeddings(len(tokenizer))
```

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I find it confusing that the vocab_size doesn't get modified. Also, the Hugging Face documentation describes `tokenizer.add_tokens` as follows: 

> Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices starting from length of the current vocabulary.
Feature Request: Flax Encoder-Decoder Model
Author: gchhablani
Pull request: no

# 🚀 Feature request
I am trying to combine ViT with BART/mBART in Flax and it would be great to have something similar to the [EncoderDecoderModel](https://github.com/huggingface/transformers/blob/master/src/transformers/models/encoder_decoder/modeling_encoder_decoder.py) in Flax. 

I am willing to look into it but it will probably take some time (I am not familiar with Flax).

Please let me know if anything can be done regarding this.

Thanks,
Gunjan

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Training RoBerta using transformers on masked language task giving weird results
Author: cabhijith
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

## Details
I trained a RoBERTa model following this colab - https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb#scrollTo=XaFAsB_fnU3K

Here is how my data looked:
```
Merkel bemoans lack of rain as Germany fears for its forests .\n
Germany’s forests, covering a third of its territory and as much a part of its cultural landscape as its physical one, are in danger.\n
An aerial view shows a forest near Gummersbach, Germany, April 24, 2020, following an unusually warm, dry winter after a summer of record temperatures leaving forests dried out.\n
Picture taken with a drone.\n
The last two exceptionally hot and dry summers have weakened millions of trees, undermining their defences against the bark beetle, which can be fatal to ancient woodlands.\n
And after an exceptionally dry April, with summer still two months away, a forest fire has already had to be put out near the town of Gummersbach in western Germany this week.\n
“We’re already noticing these days that it’s not raining enough in many areas. 
```
After training the model I used pipeline from the transforms library for the fill_mask task
```
from transformers import pipeline

fill_mask = pipeline(
    "fill-mask",
    model="./output",
    tokenizer="./output"
fill_mask("Merkel bemoans lack of rain as <mask> fears for its forests")
)
```
These are the results:
```
[{'sequence': '
 Merkel bemoans lack of rain as. fears for its forests
',
  'score': 0.040456026792526245,
  'token': 18},
 {'sequence': '
 Merkel bemoans lack of rain as, fears for its forests
',
  'score': 0.03502459451556206,
  'token': 16},
 {'sequence': '
 Merkel bemoans lack of rain as the fears for its forests
',
  'score': 0.03497963398694992,
  'token': 269},
 {'sequence': '
 Merkel bemoans lack of rain as\n fears for its forests
',
  'score': 0.03180328756570816,
  'token': 203},
 {'sequence': '
 Merkel bemoans lack of rain as to fears for its forests
',
  'score': 0.020796578377485275,
  'token': 288}]
```
As you can see there is no meaningful word(s) returned only punctuations and one other word (to) which doesn't make sense. What am i doing wrong here? Do I have to remove all punctuations?
<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->

**A link to original question on Stack Overflow**:
https://stackoverflow.com/questions/62276011/training-roberta-using-transformers-on-masked-language-task-giving-weird-results


Comments:
Still facing this problem. Has anyone else encountered something similar? 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I also face this problem. How to solve it?
How to make BART infill unmasked deletions (not masked tokens)?
Author: d4buss
Pull request: no

Hello,

I see that BART is able to infill <mask> tokens in the input and was able to run this example. https://huggingface.co/transformers/model_doc/bart.html#mask-filling

Since BART can also handle other type of pre-training noise such as token deletion, is there a way I could get BART to take inputs with deletions, and see the output of BARTs attempt to infill those deleted spans?

In the BART paper, 
`A _C. _E.   ->. A B C. D E. `    is an example of denoising Token Masking, and 
`A. C.  E.     ->  A B C. D E. `     is an example of denoising Token Deletion 

Since the best BART model in the paper is found to be pre-trained with "sentence shuffling" and "text-infilling,
where arbitrary length spans of text are replaced with a single mask token," and NOT with the token deletion strategy (where no <mask> token is left behind), would we expect to have to obtain a variant of BART that has been pre-trained on token deletion in order to use BART for infilling unmasked deletions?

I am asking because I would like to use BART to repair text where certain spans may be missing/ deleted. For example:
input: `"Wikipedia is a free online, created and edited by volunteers around the world"`
desired output:  `"Wikipedia is a free online encyclopedia, created and edited by volunteers around the world"`
(Note the generation of "encyclopedia" but no <mask> token included in the input.)

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Extend the testing of tokenizers that just have a legacy version
Author: SaulLu
Pull request: no

# 🚀 Feature request

For the moment, only tokenizers that have a rust version use the tests in `test_tokenization_common.py` file that use the `self.tokenizers_list` attribute that stores a (eventually some) model name stored on the hub.

As mentioned by @LysandreJik in this [discussion](https://github.com/huggingface/transformers/pull/11810#discussion_r643083834_), it might be interesting not to limit this tests for tokenizers that have a rust version. Indeed, some tests that use `self.tokenizers_list` do not compare the behavior of the legacy tokenizer and the rust tokenizer but only test their correct functioning. Moreover, other tests which use the `self.tokenizers_list` attribute, test a behavior of the rust version whereas one could expect the same behavior of the legacy version.

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation

This would allow to test rust only and legacy only tokenizers in an (almost) equivalent way.

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

I started working on it but some tests do not pass as is on legacy versions.

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->
     
cc @LysandreJik , @sgugger 


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Can hidden states be passed instead of input_ids or inputs_embeds in Transformers OpenAI GPT2?
Author: QMrpy
Pull request: no

I am working on an encoder decoder model which uses a fine tuned RoBERTa as the encoder and GPT2 as the decoder. Before passing the encoder context to the decoder, I am mixing it with some context from a different domain. This mixing module is a simple NN. Hence, I now want to pass these transformed hidden states to the GPT2 decoder to do decoding, and I will train the decoder and the mixer only, not the encoder. How can I pass these transformed hidden states to the GPT2 decoder instead of the `input_ids` or `inputs_embeds`? The shape of my transformed hidden states is `(n_layers, batch_size, 1, hidden_size)` and I am currently using `batch_size=1`.  Any help will be appreciated. 

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!

Thank you! I have asked my [question](https://discuss.huggingface.co/t/can-hidden-states-be-passed-instead-of-input-ids-or-inputs-embeds-in-transformers-openai-gpt2/8073) on the forum.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Rely on huggingface_hub for common tools
Author: sgugger
Pull request: yes

# What does this PR do?

This PR removes the `hf_api` module from Transformers to rely on the one in `huggingface_hub`. It also fully deprecates the `transformers-cli` command lines relying on it (such as `login`, `whoami`, `logout`).

In passing, when `model_list` was used, this PR switches to the new version, `list_models`.

cc @julien-c 

Comments:
Awesome, wanted to do this for quite some time, thanks!
[WIP] Add Few Shot Named Entity Recognition (FSNER) model
Author: sayef
Pull request: yes

# What does this PR do?

- This PR adds a new model FSNER (few shot named entity recognition) which has been implemented and trained based on the paper: [Example-Based Named Entity Recognition](https://arxiv.org/abs/2008.10570) from the researches in Microsoft Dynamics 365 AI.
- It includes only FSNERModel, not any other derivations, i.e., MaskedLM, ForQuestionAnswering etc.
- Doc strings are also updated, but not sure how it would appear visually.
- No additional tests are included.


<details><summary>Usage Example</summary>
<p>


```
from transformers import FSNERModel, FSNERTokenizerFast, FSNERTokenizer

device = 'cpu'
fsner_model = FSNERModel.from_pretrained("sayef/fsner-bert-base-uncased").to(device)
fsner_tokenizer = FSNERTokenizer.from_pretrained("sayef/fsner-bert-base-uncased")

# size of query and supports must be same. If you want to find all the entitites in one particular query, just repeat same query n times where n is the size of supports (or entities).


query = [
    'KWE 4000 can reach with a maximum speed from up to 450 P/min an accuracy from 50 mg',
    'I would like to order a computer from eBay.',
]

# each list in supports are the examples of one entity type

supports = [
        [
           'Horizontal flow wrapper [E] Pack 403 [/E] features the new retrofit-kit „paper-ON-form“',
           '[E] Paloma Pick-and-Place-Roboter [/E] arranges the bakery products for the downstream tray-forming equipment'
,
           'Finally, the new [E] Kliklok ACE [/E] carton former forms cartons and trays without the use of glue',
           'We set up our pilot plant with the right [E] FibreForm® [/E] configuration to make prototypes for your marketi
ng tests and package validation',
           'The [E] CAR-T5 [/E] is a reliable, purely mechanically driven cartoning machine for versatile application fiel
ds'
        ],
        [
            "[E] Walmart [/E] is a leading e-commerce company",
            "I recently ordered a book from [E] Amazon [/E]",
            "I ordered this from [E] ShopClues [/E]",
            "Fridge can be ordered in [E] Amazon [/E]",
            "[E] Flipkart [/E] started it's journey from zero"
        ]
   ]


def tokenize(x):
    return fsner_tokenizer(x, padding='max_length', max_length=384, truncation=True, return_tensors="pt")

W_query = tokenize(query).to(device)
W_supports = tokenize([s for support in supports for s in support]).to(device)

start_prob, end_prob = fsner_model.get_start_end_token_scores(W_query, W_supports)

output = fsner_tokenizer.extract_entity_from_scores(query, W_query, start_prob, end_prob, thresh=0.50)

print(output)

```


</p>
</details>

Would like to have attention of @LysandreJik, @stas00, @sgugger


Comments:
Fix caching issue #12536
Author: ManuelFay
Pull request: yes

# What does this PR do?

This PR is a proposed fix to issue #12536 . It does so by simply logging the unfound file instead of raising an error causing program execution, in the special case of non-existent optional vocab files which are handled in the case `local_files_only=True` (`FileNotFoundError`) and the case where  `local_files_only=False` and user is online, but not the case where `local_files_only=False` and user is offlline.

This needs to be reviewed to ensure this is the direction to go to fix this issue, and that this will not be a problem in other cases.

Fixes #12536


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@LysandreJik 

Comments:
Note: Tests will have to be changed if you want to go this way, I would imagine it's a bit too general of a fix to be honest.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
ImportError: cannot import name 'LineByLineTextDataset' from 'transformers' (unknown location)
Author: ccyccxcl
Pull request: no

 when I tried to run the following codes as in **https://huggingface.co/blog/how-to-train** , it raised a bug:

from transformers import LineByLineTextDataset
dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path="./oscar.eo.txt",
    block_size=128,
)


ImportError: cannot import name 'LineByLineTextDataset' from 'transformers' (unknown location)

@sgugger 


Comments:
Yes this notebook is deprecated, you should look at the new version [here](https://github.com/huggingface/notebooks/blob/master/examples/language_modeling_from_scratch.ipynb) (or on [colab](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/language_modeling_from_scratch.ipynb)).

More generally the up-to-date list of notebooks is in the [documentation](https://huggingface.co/transformers/master/notebooks.html).

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[Work In Progress] SentenceTransformer implementation based on CLIP
Author: minsik-ai
Pull request: yes

# What does this PR do?
Sentence Transformer Flax implementation for Flax/JAX week.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Reviewers will be added when the PR has progressed.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
PEGASUS using ONNX
Author: karimfayed
Pull request: no

Hello  @patrickvonplaten. , I just uploaded my fine-tuned model to the hub and I wanted to use ONNX to convert the pytorch model and be able to use it in a JavaScript back-end.
**I used the following command:**
`!python3 -m transformers.convert_graph_to_onnx --model Karimfayed/pegasus-SAMSum --framework pt pegasus-SAMSum.onnx`
**I receive the following error message:**

> Error while converting the model: Unrecognized configuration class <class 'transformers.configuration_pegasus.PegasusConfig'> for this kind of AutoModel: AutoModel.
Model type should be one of RetriBertConfig, T5Config, DistilBertConfig, AlbertConfig, CamembertConfig, XLMRobertaConfig, BartConfig, LongformerConfig, RobertaConfig, LayoutLMConfig, SqueezeBertConfig, BertConfig, OpenAIGPTConfig, GPT2Config, MobileBertConfig, TransfoXLConfig, XLNetConfig, FlaubertConfig, FSMTConfig, XLMConfig, CTRLConfig, ElectraConfig, ReformerConfig, FunnelConfig, LxmertConfig, BertGenerationConfig, DebertaConfig, DPRConfig, XLMProphetNetConfig, ProphetNetConfig.

**Is PEGASUS going to be added to the list soon or is there any way around it?**


Comments:
Hello @karimfayed! We're in the process of switching our approach relative to using the ONNX converter. See the following PR https://github.com/huggingface/transformers/pull/11786.

It has support for BART, so enabling support for Pegasus should be fairly simple. Please let us know if you run into any issues.

You can see the docs here: https://235542-155220641-gh.circle-artifacts.com/0/docs/_build/html/serialization.html

Please make sure to git checkout the PR first!

> Hello @karimfayed! We're in the process of switching our approach relative to using the ONNX converter. See the following PR #11786.
> 
> It has support for BART, so enabling support for Pegasus should be fairly simple. Please let us know if you run into any issues.
> 
> You can see the docs here: https://235542-155220641-gh.circle-artifacts.com/0/docs/_build/html/serialization.html
> 
> Please make sure to git checkout the PR first!

Hello @LysandreJik , thank you for your help. I read both the [docs](https://235542-155220641-gh.circle-artifacts.com/0/docs/_build/html/serialization.html) and the issue and I used the command : 
`!python3 -m transformers.onnx -f pytorch --model=Karimfayed/pegasus-SAMSum --features=default --optimize --optimization-level=all onnx/Karimfayed/pegasus-SAMSum/`

**but I keep getting this error**

> /usr/bin/python3: No module named transformers.onnx

**Even when I replace ` transformers.onnx` with `transformers.onnx.export ` I get this error:**

> /usr/bin/python3: Error while finding module specification for 'transformers.onnx.export' (ModuleNotFoundError: No module named 'transformers.onnx')

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
push_to_hub related issues (from Google Colab)
Author: ewalia
Pull request: no

I am trying to write a transformer model to a repo at huggingface.co
!git push doesn't work after successful !git add . and !git commit 
fatal: could not read username for https://huggingface.co no such device or address

Comments:
Are you inside a Colab?

You might want to try `huggingface_hub.Repository` and pass in an authentication token


To write to the repo, you'll also need to login with `!huggingface-cli login`

Once you've done that, if you want to use only git commands without passing by the `Repository` class, you can do it as such:

```
!git clone https://user:$(cat /root/.huggingface/token)@huggingface.co/<NAMESPACE>/<MODEL_ID>
```

or, if you'd rather use environment variables:

```py
# Put the token in an environment variable
from huggingface_hub import HfFolder
import os
os.environ['HF_AUTH'] = HfFolder().get_token()
```
```
# Clone the repo with authentication
!git clone https://user:$HF_AUTH@huggingface.co/<NAMESPACE>/<MODEL_ID>
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.