Question:
Has anyone managed to use the code in examples/seq2seq on the wikihow dataset used in the pegasus paper?
If you have a working data dir (one line per example for source and target), but no preprocessing code, I would find a google drive/s3 link to the data useful! Thanks!
If you have preprocessing code, even more useful!

Context:
Here you go Sam @sshleifer
raw article files
gdown -O train_articles.zip --id 1-1CR6jh6StaI69AsbBXD8lQskFbGc2Ez # train
gdown -O valid_articles_.zip --id 1-EGoT5ZKRNHQb_ewNpD9GZCvQ3uHzDSi # val
gdown -O test_articles_.zip --id 1-CxzdzEIuBYzCs06zrglYrLBlLI6kjSZ # test

unzip these
pre-proc code (not super readable, wrote it a while ago)
import os
import glob
import json
import re

def get_art_abs_wikihow(path):
  articles = glob.glob('%s/*' % path)
  for a in articles:
    try:
      with open(a, 'r') as f:
        text = f.read()
      splits  = text.split('@article')
      abstract = splits[0].replace('\n', '').replace('@summary', '').strip()
      article = splits[1].replace('\n', '').replace('@article', '').strip()
      yield article, abstract
    except Exception as e:
      yield None

def write_to_bin(lines, out_prefix):
  print("Making bin file for %s..." % out_prefix)

  with open(out_prefix + '.source', 'at') as source_file, open(out_prefix + '.target', 'at') as target_file:
    for idx,line in enumerate(lines):
      if idx % 1000 == 0:
        print("Writing story %i" % idx)

      # Get the strings to write to .bin file
      if line is None: continue
      article, abstract = line

      #  a threshold is used to remove short articles with long summaries as well as articles with no summary
      if len(abstract) < (0.75*len(article)):
        # remove extra commas in abstracts
        abstract = abstract.replace(".,",".")
        # remove extra commas in articles
        article = article.replace(";,", "")
        article = article.replace(".,",".")
        article = re.sub(r'[.]+[\n]+[,]',".\n", article)

        abstract = abstract.strip().replace("\n", "")
        article  = article.strip().replace("\n", "")

        # Write article and abstract to files
        source_file.write(article + '\n')
        target_file.write(abstract + '\n')

  print("Finished writing files")

def create_stories(save_path='wikihow'):
  # Create some new directories
  if not os.path.exists(save_path): os.makedirs(save_path)

  # write wikihow
  print("Making bin file for wikihow valid set")
  lines = get_art_abs_wikihow('./valid_articles')
  write_to_bin(lines, os.path.join(save_path, "val"))

  print("Making bin file for wikihow train set")
  lines = get_art_abs_wikihow('./train_articles')
  write_to_bin(lines, os.path.join(save_path, "train"))

  print("Making bin file for wikihow test set")
  lines = get_art_abs_wikihow('./test_articles')
  write_to_bin(lines, os.path.join(save_path, "test"))

and processed data using above script. (one line per example)
gdown -O wikihow.zip --id 1_QE1PLJhhugMf2e1edUGJRMiktKsm8YU


Question: What should be the format of the data for pre-training? could it be any raw data (e.g., news articles) in my case and then after I fine-tune, then I need to define it for a specific task e.g., classification?

Context:
It comes in many forms and usually needs processing/adaptation for model input. Look into transformers examples to start.
If you are asking about the datasets project specifically - it makes it even simpler to do the above. There are many prepared datasets ready to go. Read through documentation.


Question:
Hello everyone,
Is it possible to put get_nearest_examples() computation on GPU after having added a FAISS index 4?

Context:
You can specify the GPU ID when you instantiated the index. Will use the CPU by default (None).
add_faiss_index(column, device=0)


.get_nearest_examples() throws ArrowInvalid: offset overflow while concatenating arrays

Question:
Hello everyone,
I am adding a FAISS index on the MSMARCO passages dataset that has ~8.8M passages. I have already created the embeddings with DPRContextTokenizer and DPRContextEncoder.
Dataset info:
ms marco features888×140 18 KB
After adding the FAISS index to it, I tried to retrieve some documents with a query.
q_encoder = DPRQuestionEncoder.from_pretrained("facebook/dpr-question_encoder-single-nq-base")
q_tokenizer = DPRQuestionEncoderTokenizer.from_pretrained("facebook/dpr-question_encoder-single-nq-base")

question = 'what is the difference between a c-corp and a s-corp?'
question_embedding = q_encoder(**q_tokenizer(question, return_tensors="pt"))[0][0].detach().numpy()

scores, retrieved_documents = dataset_embedded_passages['train'].get_nearest_examples('embeddings', question_embedding, k=10)

And then it threw,
ArrowInvalid                              Traceback (most recent call last)

 in
~/.conda/envs/andregodinho/lib/python3.6/site-packages/datasets/search.py in get_nearest_examples(self, index_name, query, k)
    564         self._check_index_is_initialized(index_name)
    565         scores, indices = self.search(index_name, query, k)
--> 566         return NearestExamplesResults(scores, self[[i for i in indices if i >= 0]])
    567
    568     def get_nearest_examples_batch(

~/.conda/envs/andregodinho/lib/python3.6/site-packages/datasets/arrow_dataset.py in __getitem__(self, key)
   1069             format_columns=self._format_columns,
   1070             output_all_columns=self._output_all_columns,
-> 1071             format_kwargs=self._format_kwargs,
   1072         )
   1073

~/.conda/envs/andregodinho/lib/python3.6/site-packages/datasets/arrow_dataset.py in _getitem(self, key, format_type, format_columns, output_all_columns, format_kwargs)
   1037                 )
   1038             else:
-> 1039                 data_subset = self._data.take(indices_array)
   1040
   1041             if format_type is not None:

~/.conda/envs/andregodinho/lib/python3.6/site-packages/pyarrow/table.pxi in pyarrow.lib.Table.take()

~/.conda/envs/andregodinho/lib/python3.6/site-packages/pyarrow/compute.py in take(data, indices, boundscheck)
    266     """
    267     options = TakeOptions(boundscheck)
--> 268     return call_function('take', [data, indices], options)
    269
    270

~/.conda/envs/andregodinho/lib/python3.6/site-packages/pyarrow/_compute.pyx in pyarrow._compute.call_function()

~/.conda/envs/andregodinho/lib/python3.6/site-packages/pyarrow/_compute.pyx in pyarrow._compute.Function.call()

~/.conda/envs/andregodinho/lib/python3.6/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

~/.conda/envs/andregodinho/lib/python3.6/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

It threw ArrowInvalid: offset overflow while concatenating arrays
The dataset is very large. Any workaround?

Context:
Thanks for reporting !
It will be fixed in the datasets release of this week


Question:
Hello everyone, I hope you are all having fun using the Hugging face library.
I am tokenizing the 8.8M passages from MSMARCO dataset. Moreover, I have indexed the dataset with Hugging face dataset because I want to add a FAISS index 1 over it afterwards.
To do all of these, I created the dataset correctly by following these steps 12. Afterwards, I ran this code:
torch.set_grad_enabled(False)
ctx_encoder = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
ctx_tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
dataset_embedded_passages = dataset_passages.map(lambda example: {'embeddings': ctx_encoder(**ctx_tokenizer(example["passage"], return_tensors="pt"))[0][0].numpy()})

This will take a lot of time in the CPU since the dataset has ~8.8M passages. Is it possible to do the tokenization in the GPU? I checked the .map() method and did not find a way to put to the device.

Context:
Hi !
You can indeed put the tokenized text on GPU and give it to the model.
Also you can make it significantly faster by using a batched map:
def embed(examples):
    tokenized_examples = ctx_tokenizer(
        examples["passage"],
        return_tensors="pt",
        padding="longest",
        truncation=True,
        max_length=512
    ).to(device=ctx_encoder.device)
    embeddings = ctx_encoder(**tokenized_examples)[0]
    return {"embeddings": embeddings}

dataset_embedded_passages = dataset_passages.map(embed, batched=True, batch_size=16)

Let me know if it helps


Pipeline with custom dataset tokenizer: when to save/load manually

Question:
I am trying my hand at the datasets library and I am not sure that I understand the flow.
Let’s assume that I have a single file that is a pickled dict.  In that dict, I have two keys that each contain a list of datapoints. One of them is text and the other one is a sentence embedding (yeah, working on a strange project…).
I know that I can create a dataset from this file as follows:
dataset = Dataset.from_dict(torch.load("data.pt"))
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
keys_to_retain = {"input_ids", "sembedding"}
dataset = dataset.map(lambda example: tokenizer(example["text"], padding='max_length'), batched=True)
dataset.remove_columns_(set(dataset.column_names) - keys_to_retain)
dataset.set_format(type="torch", columns=["input_ids", "sembedding"])

My question is, what’s next? Especially considering how caching works. The first thing that should happen is splitting the dataset into a train, dev, test set 1. As a result I would eventually have a dictionary with train, dev, test keys in them. I can then use them in dataloaders and I am ready to go.
The question is, what about subsequent runs (.e. new Python sessions). Will all that code need to be run again? Should I do dataset.save_to_disk, and in a next session not run the whole dataset creation again? In other words, do I have to manually check for the saved files? Something like this (untested).
def create_datasets(dataset_path):
    if Path(dataset_path).exists():
        datasets = {partition: load_from_disk(Path(dataset_path) / partition) for partition in ["train", "dev", "test"]}
    else:
        # the snippet that I posted above
        # assuming we have train, dev, test in datasets
        for key, dataset in datasets.items():
            dataset.save_to_disk(Path(dataset_path).joinpath(key))

    return dataset

Or is the dataset cached somewhere and every time the first snippet is encountered, none of those steps is repeated and the cached dataset is loaded?
In short, it is not clear to me when I can rely on cache that is hidden (probably somewhere in the user directory), and when I should manually use save_to_disk and load a dataset manually.
Thanks!

Context:
The caching should work across sessions, normally you don’t have to use save_to_disk. The cache is indexed by a hash of the operations performed on the dataset, if a new, independent, session performs the same operations, they will use the cache instead of being recomputed. If you change something to the operation performed on the dataset, they will be recomputed instead of using the cache.
I will add a detail on the hashing mechanism to the doc when I have some time (no ETA) but basically it use as hash to store the dataset a complete pickle dump of all the arguments you provide the processing function at each step (including the function provided to map) so if anything changes it will be detected and the operation is recomputed instead of using the cache. If all the arguments and inputs are identical, the hash is the same (whether it’s the same session or not) and the cache file is used if it is found.
save_to_disk is provided as a special utility mostly for people who preprocess a dataset on one machine which has access to the internet and would like to use the dataset on a cluster without any access to the internet (and which thus cannot download the dataset files).


Question:
Hugdatafast: huggingface/nlp  fastai
An integration to make use of hundreds of datasets with fastai, and some handy transforms to make concatenated dataset like language model dataset.
 pip install hugdatafast
 Documentation: https://hugdatafast.readthedocs.io/en/latest/ 38
Doing NLP ?
See if you can turn your data pipeline into just 3 lines.
hugdatafast_fastai1216×871 234 KB
The updates will also be tweeted on my Twitter Richard Wang 3.

Context:
Update: add a example for preparing any hugginface/nlp dataset for (traditional) langugage model, or implement custom context window.
Update the update: I cancel the updates.
— Reason — (just for notes, skipping is ok)
Originally I want to introduce LMTransform 3 and CombineTransform 3, which can do context window over examples. But I suddenly thought there is few cases we need context window across examples. Examples in a dataset are often not consecutive, we don’t need to concatenate texts not related. So these classes might be only useful for my personal use case.


[SOLVED] Dataset.map() is frozen on ELI5

Question:
I have tried to prepare ELI5 to train with T5, based on this wonderful notebook of Suraj Patil 4
However, when I run dataset.map() on ELI5 to prepare input_text, target_text, dataset.map is frozen in the first hundreds examples. On the contrary, this works totally fine on SQUAD (80,000 examples). Both nlp version 0.3.0 and 0.4.0 cause frozen process . Also try various pyarrow versions from 0.16.0 / 0.17.0 / 1.0.0 also have the same frozen process.
Reproducible code can be found on this colab notebook  4, where I also show that the same mapping function works fine on SQUAD, so the problem is likely due to ELI5 somehow.

More Info : instead of map, if I run for loop and apply function by myself, there’s no error and can finish within 10 seconds. However, nlp dataset is immutable , so I could not create a new key-value within the dataset directly ) .
I also notice that SQUAD texts are quite clean while ELI5 texts contain many special characters, not sure if this is the cause ?

Context:
Fixed by amazing Quentin here:

      github.com/huggingface/nlp

      Bugs : dataset.map() is frozen on ELI5 53

        opened

        Aug 7, 2020

          closed

        Aug 12, 2020

          ratthachat

  Hi Huggingface Team!
Thank you guys once again for this amazing repo.
I have tried to prepare ELI5 to train with T5, based...

Thanks very much again!


Question:
I was playing around with nlp Datasets 5 library and was seriously impressed by the speed!!
I figured it would be interesting to test it out to see if it would make more sense to do as much text processing (e.g. cleaning, tokenization, numericalisation) with it, instead of using Fastai’s defaults. I used fastai’s TextDataloader with all of its defaults and tried to replicate all its functionality with nlp Datasets
Full blog post here 13
Curious if anyone has any feedback or how this test might have been done better, especially how any pointers on how to parallelise tokenisation with nlp Datasets
Just tell me the results
Results were…mixed…
Fastai’s initialisation (e.g. load, preprocess, tokenize etc) was faster with the 1.6M row Sentiment140 dataset I used, however I have a few caveats:
Parallelisation
Fastai parallelises the tokenization, which I couldn’t figure out how to do with nlp Datasets (probably my own lack of knowledge and not a limitation of the library though). My guess is that doing so would likely make nlp Datasets much faster than Fastai
Sorting by sample length
To try and replicate SortedDL's behaviour, I sorted the entire dataset in the nlp Dataset trial, which added a significant amount of time, possibly theres a way to better replicated SortedDL's behaviour
Caching
nlp Datasets also uses caching so that the second time around you’d like to do the same pre-processing etc, it is much much faster
10% Data

0.16M ROWS:
Init (s)
1 epoch (s)
1 mini-batch [bs=64] (ms)

Fastai
124
14.3
7.4

Fastai w/sorted
48.1
14.3
7.4

nlp
71.2
11.3
5.6

100% Data

1.6M ROWS:
Init (s)
1 epoch (s)

Fastai w/sorted
484
142

nlp
1024
323

Any and all feedback welcome|!
(the forums auto-correct “nlp” in my post title to “Nlp” haha)

Context:
Hi there
Thanks for doing this speed comparison ! This is important for us to make sure we achieve the fastest read/write/process actions we can offer using the power of apache arrow, and with the minimum memory.
We plan to add multiprocessing in the very short term that will speed up processing significantly


Question: I want to fine tune pre-trained Roberta or Electra for multiclass classification (sentiment classify) imbalance data set . How handling problem ??

Context:
For class imbalance, one aspect to consider is that each batch has enough signal to provide some coverage of all the classes, even the unbalanced ones. Otherwise, it may degenerate during training.
When evaluating test performance though, you will need to keep the real proportions as you would observe in the real world.


Question:
Features:
New methods to transform a dataset:

dataset.shuffle: create a shuffled dataset

dataset.train_test_split: create a train and a test split (similar to sklearn)

dataset.sort: create a dataset sorted according to a certain column

dataset.select: create a dataset with rows selected following the given list of indices

Other features:

Better instructions for datasets that require manual download

Important: if you load datasets that require manual downloads with an older version of nlp, instructions won’t be shown and an error will be raised

Better access to dataset information (for instance dataset.feature['label'] or dataset.dataset_size)

Datasets:

New: cos_e v1.0
New: rotten_tomatoes
New: german and italian wikipedia

New docs:

documentation about splitting a dataset

Bug fixes:

fix metric.compute that couldn’t write on file
fix squad_v2 imports

Context:
Nice, enjoying using nlp already!
Quick question, what is the vision for the nlp library? Will it’s main focus be in curating existing datasets or might it evolve into a more general purpose PyArrow wrapper for any (text?) dataset? I’m just blown away by its speed and it would be amazing to be able to do the same with my own text datasets.
I know I could already just start using PyArrow (as below) but I have a feeling that the nlp library might have more text-specific functionality coming down the line that would be amazing to be able to use with my own data…
table = pa.Table.from_pandas(df)


How to ensure that tokenizers never truncate partial words?

Question:
Is there a way to ensure tokenizers never partially truncate a word, as illustrated below:
tokenizer= SomeTokenizer.from_pretrained('some/path')
tokenizer.decode(tokenizer('I am Nasheed and I like xylophones.', truncation=True, max_length=12)['input_ids'])

The output is the above sentence truncated like: “’[CLS] I am Nasheed and I like xylo [CLS]’”
I want it to be truncated as:  “’[CLS] I am Nasheed and I like [CLS]’”
Is there a way to enforce this?

Context:
Hi Nasheed, I’m quite curious about your use case and why you’re interested in never partially truncating, if you don’t mind sharing!
In any case, here is how I would do it: Increase max_length by 1. Tokenize the text. Decode the tokenized text. Check if the second to last token (the one before the final [CLS] token) starts with ## (the prefix that signifies that a longer token was split). If yes, remove both tokens, the one that starts with ## and the one before that. If not, just remove the one before the [CLS] token.
In your example it would be
[CLS] I am Nasheed and I like xylo ##phones [CLS]

Because the second to last token starts with ## you would remove that token and the token before it.
Hope that helps.
Cheers
Heiko


Adding new tokens to a BERT tokenizer - Getting ValueError

Question:
I have a Python list named unique_list that contains new words that will be added to my tokenizer using tokenizer.add_tokens. However, when I run my code I’m getting the following error:
File "/home/kaan/anaconda3/envs/env_backup/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 937, in add_tokens
    if not new_tokens:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

When I tested with a test array that contains 10 random words, it worked fine but the larger unique_list is causing a problem.
What am I doing wrong here?

Context:
Not sure if you already solved this issue, but I stumbled upon it today.
Looking closely at the error message, looks like the add_tokens() method expects the new_tokens passed to it as a python list rather than an numpy array. Converting new_tokens from numpy array to a list and then passing it resolved the issue.
added_tokens = tokenizer.add_tokens(new_tokens.tolist())


Import distilbert-base-uncased tokenizer to an android app along with the tflite model

Question:
I have converted the model (.h5) to tflite using:
converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS,
                                       tf.lite.OpsSet.SELECT_TF_OPS]
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()
open("/models/tflite_models/5th_Jan/distilbert_sms_60_5_jan.tflite", "wb").write(tflite_model)

but I also need tokenizer to run the model locally on the android app (independent of internet availability).
According to the articles on internet and question answered on stackoverflow How to tokenize input text in android studio to process in NLP model? 4 we need json file of tokenizers to tokenize words in new inputs.
When I run the following code:
import json

with open( 'android/word_dict.json' , 'w' ) as file:
    json.dump( tokenizer.word_index , file )

The following error comes:
AttributeError: 'DistilBertTokenizer' object has no attribute 'word_index

I am unable to find solution to use tokenizer of distilbert-base-uncased in android app. Any help will be appreciated. Thanks.

Context: excuse me did you solve it ?


ERROR?why encoding [MASK] before ‘.’ would gain a idx 13?

Question:
I found that if I use tokenizer for BERT,and decode some sentence with [MASK] before ‘.’,it would gain an additional idx 13
like this:
tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2') inputs = tokenizer("The capital of France is [MASK].", return_tensors="pt")
the ‘input_ids’ would be:
tensor([[   2,   14, 1057,   16,  714,   25,    4,   13,    9,    3]])
4 refers to [MASK], 9 refers to ‘.’, but 13 refers to None
this would be happen when [MASK] is before ‘.’
is there something wrong?

Context: You don’t generally mask works by replacing the text of the word with “[MASK]”. You usually encode the text firest “The capital of France is Paris.” and then replace the token for Paris with the Mask token. Perhaps that’s the reason for the addition of toke 13?


How to know if a subtoken is a word or part of a word?

Question:
For example, using BERT in a token classification task, I get something like this …
[('Darüber', 17), ('hinaus', 17), ('fanden', 17), ('die', 17), ('Er', 17), ('##mitt', -100), ('##ler', -100), ('eine', 17), ('Ver', 17), ('##legung', -100), ('##sli', -100), ('##ste', -100), (',', 17), ('die', 17), ('bestätigt', 17), (',', 17), ('dass', 17), ('Dem', 8), ('##jan', -100), ('##juk', -100), ('am', 17), ('27', 17), ('.', -100), ('März', 17), ('1943', 17), ('an', 17), ('die', 17), ('Dienst', 17), ('##stelle', -100), ('So', 0), ('##bi', -100), ('##bor', -100), ('ab', 17), ('##kom', -100), ('##mand', -100), ('##iert', -100), ('wurde', 17), ('.', -100)]

… in the format of (sub-token, label id).
Is there a way I can automatically know that “##mitt” and “##ler” are part of “Er” (thus making up the word “Ermittler”) that would work across all tokenizers (not just BERT)?

Context:
what do you mean by “automatically know”?
I guess you already know that ##xx tokens are continuation tokens.
I don’t think it is possible to detect from (‘Er’, 17)  that it has a continuation.
If you feed the data into an untrained  Bert model, [I think] the embedding layer will create an embedding vector for (‘Er’,17) that does not depend on the continuation tokens.
If you feed the data into a trained Bert model, the embedding layer might create different embedding vectors for different instances of (‘Er’, 17), depending on their context, which includes depending on any continuation tokens.
There is a nice tutorial by Chris McCormick here https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/ 25 that discusses embeddings in more detail.
If you were asking about something else, please clarify.
[I am not an expert, and I could be wrong]


Question: In python, BertTokenizerFast has batch_encode_plus, is there a similar method in rust?

Context: I will assume due to the lack of reply that there’s no way to do this.


Adding new tokens while preserving tokenization of adjacent tokens

Question:
I’m trying to add some new tokens to BERT and RoBERTa tokenizers so that I can fine-tune the models on a new word. The idea is to fine-tune the models on a limited set of sentences with the new word, and then see what it predicts about the word in other, different contexts, to examine the state of the model’s knowledge of certain properties of language.
In order to do this, I’d like to add the new tokens and essentially treat them like new ordinary words (that the model just hasn’t happened to encounter yet). They should behave exactly like normal words once added, with the exception that their embedding matrices will be randomly initialized and then be learned during fine-tuning.
However, I’m running into some issues doing this. In particular, the tokens surrounding the newly added tokens do not behave as expected when initializing the tokenizer with do_basic_tokenize=False. The problem can be observed in the following example; in the case of BERT, the period following the newly added token is not tokenized as a subword (i.e., it is tokenized as . instead of as the expected ##.), and in the case of RoBERTa, the word following the newly added subword is treated as though it does not have a preceding space (i.e., it is tokenized as a instead of as Ġa.
from transformers import BertTokenizer, RobertaTokenizer

new_word = 'mynewword'
bert = BertTokenizer.from_pretrained('bert-base-uncased', do_basic_tokenize = False)
bert.tokenize('mynewword') # does not exist yet
# ['my', '##ne', '##w', '##word']
bert.tokenize('testing.')
# ['testing', '##.']

bert.add_tokens(new_word)
bert.tokenize('mynewword') # now it does
# ['mynewword']
bert.tokenize('mynewword.')
# ['mynewword', '.']

roberta = RobertaTokenizer.from_pretrained('roberta-base', do_basic_tokenize = False)
roberta.tokenize('mynewword') # does not exist yet
# ['my', 'new', 'word']
roberta.tokenize('A testing a')
# ['A', 'Ġtesting', 'Ġa']

roberta.add_tokens(new_word)
roberta.tokenize('mynewword') # now it does
# ['mynewword']
roberta.tokenize('A mynewword a')
# ['A', 'mynewword', 'a']

Is there a way for me to add the new tokens while getting the behavior of the surrounding tokens to match what it would be if there were not an added token there? I feel like it’s important because the model could end up learning that (for instance), the new token can occur before ., while most others can only occur before ##. That seems like it would affect how it generalizes. In addition, I could turn on basic tokenization to solve the BERT problem here, but that wouldn’t really reflect the full state of the model’s knowledge, since it collapses the distinction between different tokens. And that doesn’t help with the RoBERTa problem, which is still there regardless.
In addition, I’d ideally be able to add the RoBERTa token as Ġmynewword, but I’m assuming that as long as it never occurs as the first word in a sentence, that shouldn’t matter.

Context:
Hey @mawilson if you want to add new tokens to the vocabulary, then in general you’ll need to resize the embedding layers with
model.resize_token_embeddings(len(tokenizer))

You can see a full example in the docs 3 - does that help solve your problem?


Implementing custom tokenizer components (normalizers, processors)

Question:
I’m wondering if there is an easy way to tweak the individual components of a tokenizer. Specifically, I’d like to implement a custom normalizer and post-processor.
Just to provide some context, I’m trying to train a Danish tokenizer. Danish has a lot of compound nouns (e.g., the Danish translation of “house owner” is “husejer”, with “hus” being “house” and “ejer” being “owner”), so a tokenizer should split these accordingly. A standard BPE or WordPiece can deal with this just fine.
The issue is that for some compound nouns, we impose an “s” in between the two words. For instance, “birthday greeting” is “fødselsdagshilsen”, with “fødselsdag” being “birthday” and “hilsen” being “greeting”. This messes up the tokenizer completely, tokenizing it as [‘fødselsdag’, ‘shi’, ‘l’, ‘sen’] rather than the ideal [‘fødselsdag’, ‘s’, ‘hilsen’].
I think I can solve it by imposing a new special token, <conn>, and at the normaliser stage I check if the word is of the form <word1>s<word2> where <word1> and <word2> are known words, and if so, replaces the “s” by <conn>. At the post-processing stage, I then replace the <conn> instances with “s” again.
Long story short, is there a way to simply subclass the normaliser/processor classes to implement such behaviours?

Context:
For anyone else looking, this can be done, and it’s answered in this question:

    How to add additional custom pre-tokenization processing? 🤗Tokenizers

    I would like to add a few custom functions for pre-tokenization. For example, I would like to split numerical text from any non-numerical test.
Eg
‘1000mg’ would become [‘1000’, ‘mg’].
I am trying to figure out the proper way to do this for the python binding; I think it may be a bit tricky since its a binding for the original rust version.
I am looking at the pretokenizer function
/huggingface/tokenizers/blob/2ccd16bf5c3dd97759d7bdf5229e2feeba314b4a/bindings/python/py_src/tokenizers/pre_to…


Question:
Does T5Tokenizer support the Greek language?
When I run the 3 lines of code below, then the input_ids are just 2 and 3 which correspond to the unknown token and the underscore respectively. This is the same for any input text of Greek letters.
from transformers import T5Tokenizer
tokenizer = T5Tokenizer.from_pretrained(“t5-small”)
input_ids = tokenizer(‘Γειά σου Κόσμε’, return_tensors=‘pt’).input_ids

Context:
Hi,
T5 itself was trained on English data only. However, there’s a multilingual variant called mT5 2 which supports Greek.


How padding in huggingface tokenizer works?

Question:
I tried following tokenization example:
tokenizer = BertTokenizer.from_pretrained(MODEL_TYPE, do_lower_case=True)
sent = "I hate this. Not that.",
_tokenized = tokenizer(sent, padding=True, max_length=20, truncation=True)
print(_tknzr.decode(_tokenized['input_ids'][0]))
print(len(_tokenized['input_ids'][0]))

The output was:
[CLS] i hate this. not that. [SEP]
9

Notice the parameter to tokenizer: max_length=20. How can I make Bert tokenizer to append 11 [PAD] tokens to this sentence to make it total 20?

Context: You need to change padding to "max_length". The default behavior (with padding=True) is to pad to the length of the longest sentence in the batch, meanwhile sentences longer than specified length are getting truncated to the specified max_length. In your example you have only one sentence, thus there’s no padding (the only sentence is the longest one). Your sentence is shorter than max length, so there’s no truncation either.


How to configure TokenizerFast for AutoTokenizer

Question:
Hi there,
I made a custom model and tokenizer for Retribert architecture. For some reason, when using AutoTokenizer.from_pretrained method, the tokenizer does not initialize model_max_len tokenizer attribute to 512 but to a default of a very large integer. If I invoke AutoTokenizer.from_pretrained with an additional max_len=512 kwarg then the model_max_len gets set to 512 as expected. However, as you might expect I don’t want users to pass this additional kwarg but would prefer to somehow set this value by default.
I figured out that TokenizerFast gets initialized from tokenizer.json and I attempted to add model_max_len attribute to tokenizer.json. However, as soon as I do that AutoTokenizer complains that it can not load the JSON file any longer. Perhaps this property can’t be set via tokenizer.json or perhaps I am not adding it at the right JSON node.
Any ideas on how to set model_max_len tokenizer property so that AutoTokenizer picks it up without additional kwargs?
Best,
Vladimir

Context:
Hi, I figured this one out; leaving a small note if you stumble on this issue yourself. All you need to do is add tokenizer_config.json file with additional configs for the tokenizer. I added a simple tokenizer_config.json with the following contents:
{“model_max_length”: 512}
That’s all.
Cheers,
Vladimir


Question:
What would be the best strategy to mask only specific words during the LM training?
My aim is to mask only words of interest which I have previously collected in a list.
The issue arises since the tokenizer, not only splits a single word in multiple tokens, but it also adds special characters if the word does not occur at the begging of a sentence.
E.g.:

The word “Valkyria”:

at the beginning of a sentences gets split as [‘V’, ‘alky’, ‘ria’] with corresponding IDs: [846, 44068, 6374].
while in the middle of a sentence as [‘ĠV’, ‘alky’, ‘ria’] with corresponding IDs: [468, 44068, 6374],

This is just one of the issues forcing me to have multiple entries in my list of to-be-filtered IDs.
I have already had a look at the mask_tokens() function into the DataCollatorForLanguageModeling class, which is the function actually masking the tokens during each batch, but I cannot find any efficient and smart way to mask only specific words and their corresponding IDs.

Context:
When using the “fast” variant of the tokenizers available in huggingface/transformers 1, whenever you encode some text, you get back a BatchEncoding 3.
This BatchEncoding provides some helpful mappings that we can use in this kind of situation. So, you should be able to:

Find the word associated with any token using token_to_word 8. This method returns the index of the word in the input sequence.
Once you know the word’s index, you can actually retrieve its span with word_to_chars 2. This will let you extract the word from the input sequence.


Question:
I’m trying to evaluate a QA model on a custom dataset. This is how I prepared the velidation features:
def prepare_validation_features(examples):
    # Tokenize our examples with truncation and maybe padding, but keep the overflows using a stride. This results
    # in one example possible giving several features when a context is long, each of those features having a
    # context that overlaps a bit the context of the previous feature.
    tokenized_examples = tokenizer(
        examples["question" if pad_on_right else "context"],
        examples["context" if pad_on_right else "question"],
        truncation="only_second" if pad_on_right else "only_first",
        max_length=max_length,
        stride=doc_stride,
        return_overflowing_tokens=True,
        return_offsets_mapping=True,
        padding="max_length",
    )

    # Since one example might give us several features if it has a long context, we need a map from a feature to
    # its corresponding example. This key gives us just that.
    sample_mapping = tokenized_examples.pop("overflow_to_sample_mapping")

    # We keep the example_id that gave us this feature and we will store the offset mappings.
    tokenized_examples["example_id"] = []

    for i in range(len(tokenized_examples["input_ids"])):
        # Grab the sequence corresponding to that example (to know what is the context and what is the question).
        sequence_ids = tokenized_examples.sequence_ids(i)
        context_index = 1 if pad_on_right else 0

        # One example can give several spans, this is the index of the example containing this span of text.
        sample_index = sample_mapping[i]
        tokenized_examples["example_id"].append(examples["id"][sample_index])

        # Set to None the offset_mapping that are not part of the context so it's easy to determine if a token
        # position is part of the context or not.
        tokenized_examples["offset_mapping"][i] = [
            (o if sequence_ids[k] == context_index else None)
            for k, o in enumerate(tokenized_examples["offset_mapping"][i])
        ]

    return tokenized_examples

But as I try to apply the function to my dataset:
test_features = test_dataset.map(
    prepare_validation_features,
    batched=True,
)

at a certain moment (more or less 23% of the process) it return me this error:
23%
5/22 [00:19<00:51, 3.04s/ba]
---------------------------------------------------------------------------
ArrowInvalid                              Traceback (most recent call last)
<ipython-input-156-6658d0dc57be> in <module>()
      1 test_features = test_dataset.map(
      2     prepare_validation_features,
----> 3     batched=True,
      4 )

8 frames
/usr/local/lib/python3.7/dist-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowInvalid: Column 3 named attention_mask expected length 1000 but got length 1076

How can I fix it?

Context: Hello! I have the same issue. Did you fix it?


Is that possible to embed the tokenizer into the model to have it running on GCP using TensorFlow Serving?

Question:
Hello!
Thanks in advance for your help!
At the beginning I’ve created an issue on github with this question:
Question: Is that possible to embed a tokenizer into the model for tensorflow serving? · Issue #13843 · huggingface/transformers · GitHub 6 and I’ve got a suggestion to tag @Rocketknight1 who is an expert in TensorFlow for questions like this.
I already using TF-BERT model (uncased version) with tensorflow serving. I found that I need to modify some inputs to get something like that:
    callable = tf.function(self.model.call)
    concrete_function = callable.get_concrete_function([
        tf.TensorSpec([None, self.max_input_length], tf.int32, name="input_ids"),
        tf.TensorSpec([None, self.max_input_length], tf.int32, name="attention_mask")
    ])
    self.model.save(save_directory, signatures=concrete_function)

Also I found the following example (blog/tf-serving.md at master · huggingface/blog · GitHub 6), that allows me to change input signature of a model especially for serving:
from transformers import TFBertForSequenceClassification
import tensorflow as tf

# Creation of a subclass in order to define a new serving signature
class MyOwnModel(TFBertForSequenceClassification):
    # Decorate the serving method with the new input_signature
    # an input_signature represents the name, the data type and the shape of an expected input
    @tf.function(input_signature=[{
        "input_ids": tf.TensorSpec((None, None), tf.int32, name="input_ids"),
        "attention_mask": tf.TensorSpec((None, None), tf.int32, name="attention_mask"),
        "token_type_ids": tf.TensorSpec((None, None), tf.int32, name="token_type_ids"),
    }])
    def serving(self, inputs):
        # call the model to process the inputs
        output = self.call(inputs)
        test_out = self.serving_output(output)
        # return the formated output
        return test_out

# Instantiate the model with the new serving method
model = MyOwnModel.from_pretrained("bert-base-cased")
# save it with saved_model=True in order to have a SavedModel version along with the h5 weights.
model.save_pretrained("/tmp/my_model6", saved_model=True)

In my current workflow I have a need of python because I have to prepare input for a model by using tokenizer. It does mean that for now I need to have a REST service that gets text request and then sends it to the serving instance. After I switched to GCP AI Platform I think that it is reasonable and worth to try to embed the tokenizer inside the model and let GCP AI Platform serving it.
I did some tries and looks like that it more difficult than it looks like.
The goal is to have the model with tokenizer on GCP AI platform and get rid of python REST API service because all other infrastructure is written using Erlang/Rust. I need to supply the text to the model serving instance (not the object with input_ids, attention_mask, etc.) and get logits. Or softmaxed logits.
So could someone please answer is that possible and if it is possible to provide some guidance how to achieve this?
Thanks a lot for your help!
/Dmitriy

Context: I have been looking for something like this but couldn’t find it either. Based on the blog you found blog/tf-serving.md at master · huggingface/blog · GitHub 26, the author mentioned that it’s a possible next step improvement so not sure if this is already possible.


Question:
Hi All,
My goal is to add a set of starting tokens to a pre-trained AlbertTokenizerFast.
In the Albert Pre-Trained Vocab (SentencePiece Model), all start tokens are preceded with the meta-symbol: ▁ (e.g. ▁hamburger).
I tried adding tokens, prefixed with the meta symbol:
new_tokens = [AddedToken("▁hamburger",), AddedToken("▁pizza")]
num_added_tokens = tokenizer.add_tokens(new_tokens)

However, as this forum post shows, input text to AddedToken is treated literally; so manually adding the meta-symbol prefixes doesn’t achieve the desired effect.
Instead, I tried using the single_word parameter:
new_tokens = [AddedToken("hamburger", single_word=True, lstrip=True), AddedToken("pizza", single_word=True, lstrip=True)]
num_added_tokens = tokenizer.add_tokens(new_tokens)

This solution successfully encodes the new tokens where hamburger is being encoded by token 30001:
tokenizer('This hamburger tastes great')
>> [2, 15, 30001, 53, 8, 345,3]

However, when I try to decode these ids, no space appears between “this” and “hamburger”:
tokenizer.decode([2, 15, 30001, 53, 8, 345,3])
>> ('Thishamburger tastes great')

I was wondering if anybody had any thoughts about how to fix this.

Context: Does the same occur when setting lstrip=False when defining the new tokens?


Tokenizer.encode not returning encodings

Question:
How do you get token encodings? The method isn’t working in this case.
I need the token offsets in order to translate my labels to a normal list of token tags (my labels are in this format [{start_index: int, end_index: int, tag: str} … ]
Thank you!
Maybe it’s because this is a pre-trained fast tokenizer?

Context:
You need to use the tokenizer directly on your text, not the encode method:
tokenizer("hello world")


Question:
I can only find 0.10.3 through pip install. Am I doing something wrong or I can only install 0.11.0 through source?
Thank you!

Context: There is no version 0.11.0 of Tokenizers that has been released.


Using a fixed vocab.txt with AutoTokenizer?

Question:
Hello,
I have a special case where I want to use a hand-written vocab with a notebook that’s using AutoTokenizer but I can’t find a way to do this (it’s for a non-language sequence problem, where I’m pretraining very small models with a vocab designed to optimize sequence length, vocab size, and legibility).
If it’s not possible, what’s the best way to use my fixed vocab? In the past I used BertWordPieceTokenizer, loaded directly with the vocab.txt path, but I don’t know how to use this approach with newer Trainer-based approach in the notebook.
UPDATE: More specifically, if I try my old method of using BertWordPieceTokenizer(vocab='vocab.txt') it fails later with:
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_3379/2783002494.py in <module>
      3 # Setup train dataset if `do_train` is set.
      4 print('Creating train dataset...')
----> 5 train_dataset = get_dataset(model_data_args, tokenizer=tokenizer, evaluate=False) if training_args.do_train else None
      6
      7 # Setup evaluation dataset if `do_eval` is set.

/tmp/ipykernel_3379/2486475202.py in get_dataset(args, tokenizer, evaluate)
     32   if args.line_by_line:
     33     # Each example in data file is on each line.
---> 34     return LineByLineTextDataset(tokenizer=tokenizer, file_path=file_path,
     35                                  block_size=args.block_size)
     36

~/anaconda3/envs/torch_17/lib/python3.8/site-packages/transformers/data/datasets/language_modeling.py in __init__(self, tokenizer, file_path, block_size)
    133             lines = [line for line in f.read().splitlines() if (len(line) > 0 and not line.isspace())]
    134
--> 135         batch_encoding = tokenizer(lines, add_special_tokens=True, truncation=True, max_length=block_size)
    136         self.examples = batch_encoding["input_ids"]
    137         self.examples = [{"input_ids": torch.tensor(e, dtype=torch.long)} for e in self.examples]

TypeError: 'BertWordPieceTokenizer' object is not callable

The notebook I’m trying to use is from: github.com/gmihaila/ml_things.git 1

Context:
Okay, so obviously I’m not a Python guy… I see there’s some insanity in the language that allows class instances to be callable… (why, Python… WHY???)    …so I’m a bit stumped, but presumably it has to do with the fact that BertWordPieceTokenizer is not a subclass of PreTrainedTokenizer (which has the crazy attribute of being callable).
I’m really stuck. I’d just like to plug in my custom tokenizer, but it seems that when I hit “LineByLineTextDataset”, I’m going to hit the same callable error. I tried running with the default tokenization and although my vocab went down from 1073 to 399 tokens, my sequence length went from 128 to 833 tokens. Hence the desire to load my tokenizer from the hand-written vocab.
Aack!
UPDATE: Okay, I hadn’t realized I could do it with BertTokenizerFast. I haven’t totally verified that this is working, but so far it looks correct.


Question:
Hi,
I am pre training a Bert model from scratch. For that I first need to train a wordpiece tokenizer, I am using BertWordPieceTokenizer for this.
My question:
Should I train the tokenizer on the whole corpus which is huge, or training it on a sample is enough?
Is there a way to tell the tokenizer to take train only on a sample?
Thanks.

Context:
Kamel:

Should I train the tokenizer on the whole corpus which is huge

Yes. With HuggingFace Tokenizers 5, it takes seconds. From the README: “Takes less than 20 seconds to tokenize a GB of text on a server’s CPU”.


How to save a fast tokenizer using the transformer library and then load it using Tokenizers?

Question:
I want to avoid importing the transformer library during inference with my model, for that reason I want to export the fast tokenizer and later import it using the Tokenizers library.
On Transformers side, this is as easy as tokenizer.save_pretrained(“tok”), however when loading it from Tokenizers, I am not sure what to do.
from tokenizers import Tokenizer
Tokenizer.from_file(“tok/tokenizer.json”)
Seems to work, but it is ignoring the two other files in the directory: tokenizer_config.json and special_tokens_map.json, for that reason I believe it won’t give me the same tokens.
Is there a way to import a tokenizer using the whole directory files ? Or better, can we import a pretrained fast tokenizer from the hub ?
Thanks

Context:
you can load tokenizer from directory with from_pretrained method:
tokenizer = Tokenizer.from_pretrained("your_tok_directory")


Cannot create an identical PretrainedTokenizerFast object from a Tokenizer created by tokenizers library

Question:
I can’t seem to create a “PreTrainedTokenizerFast” object from my original tokenizers tokenizer object that has the same proporties. This is the code for a byte pair tokenizer I have experimented on. The resulting fast tokenizer does not have a [PAD] token, and does not have any special tokens at all.
    tokenizer = ByteLevelBPETokenizer()
    tokenizer.preprocessor = pre_tokenizers.BertPreTokenizer()
    tokenizer.normalizer = normalizers.BertNormalizer()
    tokenizer.train_from_iterator(docs, vocab_size=16_000, min_frequency=15, special_tokens = ["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"])
    tokenizer._tokenizer.post_processor = processors.BertProcessing(
        ("[SEP]", tokenizer.token_to_id("[SEP]")),
        ("[CLS]", tokenizer.token_to_id("[CLS]")),
    )
    tokenizer.enable_truncation(max_length=256)
    tokenizer.enable_padding(pad_id=3, pad_token="[PAD]")
    fast_tokenizer = PreTrainedTokenizerFast(tokenizer_object=tokenizer)

The result of printing the fast_tokenizer is:
PreTrainedTokenizerFast(name_or_path='', vocab_size=16000, model_max_len=1000000000000000019884624838656, is_fast=True, padding_side='right', special_tokens={})

Which model_max_len and special_tokens are wrong in it. Also, there is no pad_token and pad_token_id in the fast_tokenizer object. (the warning for pad_token for example: Using pad_token, but it is not set yet. ) Have I done anything wrong, or is this not supposed to happen?
The versions of libraries I’m using:
'tokenizers                    0.10.3',
'transformers                  4.10.0.dev0'

I have also tested with these versions:
'tokenizers                    0.10.3',
'transformers                  4.9.2'

Context: When using PreTrainedTokenizerFast directly and not one of the subclasses, you have to manually set all the attributes specific to Transformers: the model_max_length as well as all the special tokens. The reason is that the Tokenizer has no concept of associated model (so it doesn’t know the model max length) and even if it has a concept of special tokens, it doesn’t know the differences between them, so you have to indicate which one is the pad token, which one the mask token etc.


Question:
Hi, I am trying to train a basic Word Level tokenizer based on a file data.txt containing
5174 5155 4749 4814 4832 4761 4523 4999 4860 4699 5024 4788 [UNK]

When I run my code
from tokenizers import Tokenizer
from tokenizers.models import WordLevel

tokenizer = Tokenizer(WordLevel(unk_token='[UNK]'))
tokenizer.train(files=['data.txt'])
tokenizer.encode('5155')

I get the error

Exception: WordLevel error: Missing [UNK] token from the vocabulary

Why is it still missing despite having [UNK] in data.txt and also setting unk_token='[UNK]'?
Any help is very appreciated!

Context: Hi Athena, I’m having the same issue… did you find the root of the problem?


Extracting embedding values of NLP pertained models from tokenized strings

Question:
I am using huggingface’s pipeline to extract embeddings of words in a sentence. As far as I know, first a sentence will be turned into a tokenized strings. I think the length of the tokenized string might not be equal to the number of words in the original sentence. I need to retrieve word embedding of a particular sentence.
For example, here is my code:
#https://discuss.huggingface.co/t/extracting-token-embeddings-from-pretrained-language-models/6834/6

from transformers import pipeline, AutoTokenizer, AutoModel
import numpy as np
import re

model_name = "xlnet-base-cased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)

tokenizer.add_special_tokens({'pad_token': '[PAD]'})
model.resize_token_embeddings(len(tokenizer))

model_pipeline = pipeline('feature-extraction', model=model_name, tokenizer=tokenizer)

def find_wordNo_sentence(word, sentence):

    print(sentence)
    splitted_sen = sentence.split(" ")
    print(splitted_sen)
    index = splitted_sen.index(word)

    for i,w in enumerate(splitted_sen):
        if(word == w):
            return i

    print("not found") #0 base

def return_xlnet_embedding(word, sentence):

    word = re.sub(r'[^\w]', " ", word)
    word = " ".join(word.split())

    sentence = re.sub(r'[^\w]', ' ', sentence)
    sentence = " ".join(sentence.split())

    id_word = find_wordNo_sentence(word, sentence)

    try:
        data = model_pipeline(sentence)

        n_words = len(sentence.split(" "))
        #print(sentence_emb.shape)
        n_embs  = len(data[0])
        print(n_embs, n_words)
        print(len(data[0]))

        if (n_words != n_embs):
            "There is extra tokenized word"

        results = data[0][id_word]
        return np.array(results)

    except:
        return "word not found"

return_xlnet_embedding('your', "what is your name?")

Then the output is:

what is your name [‘what’, ‘is’, ‘your’, ‘name’] 6 4 6

So the length of tokenized string that is fed to the pipeline is two more than number of my words. How can I find which one (among these 6 values) are the embedding of my word?

Context: More especifically, I want to know when I call the model_pipeline(sentence), how should I understand how the sentence was tokenized? Because I think some words in the sentence might be tokenized into several parts, so I need to understand them.


Question:
As I’ve building tiny-modes for hf-internal-testing (HF Internal Testing) 4 I need to shrink/truncate the original tokenizers and the vocab in order to get truly tiny models and often it was taking quite a long time to figure out. So I reached out for help and got several great recipes which I thought I’d share here in case others might need something similar.

Anthony Moi’s version
@anthony’s tokenizer shrinker:
import json
from transformers import AutoTokenizer
from tokenizers import Tokenizer

vocab_keep_items = 5000
mname = "microsoft/deberta-base"

tokenizer = AutoTokenizer.from_pretrained(mname, use_fast=True)
assert tokenizer.is_fast, "This only works for fast tokenizers."
tokenizer_json = json.loads(tokenizer._tokenizer.to_str())
vocab = tokenizer_json["model"]["vocab"]
if tokenizer_json["model"]["type"] == "BPE":
    new_vocab = { token: i for token, i in vocab.items() if i < vocab_keep_items }
    merges = tokenizer_json["model"]["merges"]
    new_merges = []
    for i in range(len(merges)):
        a, b = merges[i].split()
        new_token = "".join((a, b))
        if a in new_vocab and b in new_vocab and new_token in new_vocab:
            new_merges.append(merges[i])
    tokenizer_json["model"]["merges"] = new_merges
elif tokenizer_json["model"]["type"] == "Unigram":
    new_vocab = vocab[:vocab_keep_items]
elif tokenizer_json["model"]["type"] == "WordPiece" or tokenizer_json["model"]["type"] == "WordLevel":
    new_vocab = { token: i for token, i in vocab.items() if i < vocab_keep_items }
else:
    raise ValueError(f"don't know how to handle {tokenizer_json['model']['type']}")
tokenizer_json["model"]["vocab"] = new_vocab
tokenizer._tokenizer = Tokenizer.from_str(json.dumps(tokenizer_json))
tokenizer.save_pretrained(".")

LysandreJik’s version
Using the recently added train_new_from_iterator suggested by @lysandre
from transformers import AutoTokenizer

mname = "microsoft/deberta-base" # or any checkpoint that has a fast tokenizer.
vocab_keep_items = 5000

tokenizer = AutoTokenizer.from_pretrained(mname)
assert tokenizer.is_fast, "This only works for fast tokenizers."
tokenizer.save_pretrained("big-tokenizer")
# Should be a generator of list of texts.
training_corpus = [
    ["This is the first sentence.", "This is the second one."],
    ["This sentence (contains #) over symbols and numbers 12 3.", "But not this one."],
]
new_tokenizer = tokenizer.train_new_from_iterator(training_corpus, vocab_size=vocab_keep_items)
new_tokenizer.save_pretrained("small-tokenizer")

but this one requires a training corpus, so I had an idea to cheat and train the new tokenizer on its own original vocab:
from transformers import AutoTokenizer

mname = "microsoft/deberta-base"
vocab_keep_items = 5000

tokenizer = AutoTokenizer.from_pretrained(mname)
assert tokenizer.is_fast, "This only works for fast tokenizers."
vocab = tokenizer.get_vocab()
training_corpus = [ vocab.keys() ] # Should be a generator of list of texts.
new_tokenizer = tokenizer.train_new_from_iterator(training_corpus, vocab_size=vocab_keep_items)
new_tokenizer.save_pretrained("small-tokenizer")

which is almost perfect, except it now doesn’t have any information about the frequency for each word/char (that’s how most tokenizers compute their vocab, which if you need this info you can fix by
having each key appearing len(vocab) - ID times, i.e.:
training_corpus = [ (k for i in range(vocab_len-v)) for k,v in vocab.items() ]

which will make the script much much longer to complete.
But for the needs of a tiny model (testing) the frequency doesn’t matter at all.

hack the tokenizer file version
Some tokenizers can be be just manually truncated at the file level, e.g. Electra:
# Shrink the orig vocab to keep things small (just enough to tokenize any word, so letters+symbols)
# ElectraTokenizerFast is fully defined by a tokenizer.json, which contains the vocab and the ids, so we just need to truncate it wisely
import subprocess
from transformers import ElectraTokenizerFast

mname = "google/electra-small-generator"
vocab_keep_items = 3000

tokenizer_fast = ElectraTokenizerFast.from_pretrained(mname)
tmp_dir = f"/tmp/{mname}"
tokenizer_fast.save_pretrained(tmp_dir)
# resize tokenizer.json (vocab.txt will be automatically resized on save_pretrained)
# perl -pi -e 's|(2999).*|$1}}}|' tokenizer.json # 0-indexed, so vocab_keep_items-1!
closing_pat = "}}}"
cmd = (f"perl -pi -e s|({vocab_keep_items-1}).*|$1{closing_pat}| {tmp_dir}/tokenizer.json").split()
result = subprocess.run(cmd, capture_output=True, text=True)
# reload with modified tokenizer
tokenizer_fast_tiny = ElectraTokenizerFast.from_pretrained(tmp_dir)
tokenizer_fast_tiny.save_pretrained(".")

spm vocab shrinking
First clone sentencepiece into a parent dir:
git clone https://github.com/google/sentencepiece

now to the shrinking
# workaround for fast tokenizer protobuf issue, and it's much faster too!
os.environ["PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION"] = "python"

from transformers import XLMRobertaTokenizerFast

mname = "xlm-roberta-base"

# Shrink the orig vocab to keep things small
vocab_keep_items = 5000
tmp_dir = f"/tmp/{mname}"
vocab_orig_path = f"{tmp_dir}/sentencepiece.bpe.model" # this name can be different
vocab_short_path = f"{tmp_dir}/spiece-short.model"
# HACK: need the sentencepiece source to get sentencepiece_model_pb2, as it doesn't get installed
sys.path.append("../sentencepiece/python/src/sentencepiece")
import sentencepiece_model_pb2 as model
tokenizer_orig = XLMRobertaTokenizerFast.from_pretrained(mname)
tokenizer_orig.save_pretrained(tmp_dir)
with open(vocab_orig_path, 'rb') as f: data = f.read()
# adapted from https://blog.ceshine.net/post/trim-down-sentencepiece-vocabulary/
m = model.ModelProto()
m.ParseFromString(data)
print(f"Shrinking vocab from original {len(m.pieces)} dict items")
for i in range(len(m.pieces) - vocab_keep_items): _ = m.pieces.pop()
print(f"new dict {len(m.pieces)}")
with open(vocab_short_path, 'wb') as f: f.write(m.SerializeToString())
m = None

tokenizer_fast_tiny = XLMRobertaTokenizerFast(vocab_file=vocab_short_path)
tokenizer_fast_tiny.save_pretrained(".")

If you have other related recipes please don’t hesitate to add those in the comments below.
p.s. if you create custom models that are derivations from original ones if possible please upload the script that created the derivative with the model files, so that in the future it’s easy to update or replicate or adapt to other models. e.g. make-tiny-deberta.py · hf-internal-testing/tiny-deberta at main created hf-internal-testing/tiny-deberta · Hugging Face.

Context:
gpt2 seems to have a special token "<|endoftext|>" stashed at the very end of the vocab, so it gets dropped and code breaks. So I hacked it back in with:
    if "gpt2" in mname:
        new_vocab = { token: i for token, i in vocab.items() if i < vocab_keep_items-1 }
        new_vocab["<|endoftext|>"] = vocab_keep_items-1
    else:
        new_vocab = { token: i for token, i in vocab.items() if i < vocab_keep_items }


Question:
Hello everyone, I am trying to understand how to use the tokenizers in a NER context.
Basically, I have a text corpus with entities annotations, usually in IOB format [1], which can be seen as a mapping f: word → tag (annotators are working on a non-tokenized text and we ask them to annotate entire words).
When I am using any modern tokenizer, basically, I will get several tokens for a single word (for instance “huggingface” might produce something like [“hugging#”, “face”]). I need to transfer the original annotations to each token in order to have a new labelling fonction g: token → tag
E.g. what I have in input
text  = "Huggingface is amazing"
labels = [B_org, O, O]"

what I need to produce if the tokenizer output is ["Hugging#", "face", "is", "amazin"] is
labels_per_tokens = [B_org, I_org, O, O]
```

To do so I need to backtrack for every token produced by the tokenizer what is the original word / annotation that I got in input but it seems not so easy to do (especially with [UNK] tokens). Am I missing something obvious ? Are there some good practice or solution to my problem ?

Thanks a lot for your help !

[1] https://en.wikipedia.org/wiki/Inside%E2%80%93outside%E2%80%93beginning_(tagging)

Context: hey @Thrix you can see how to align the NER tags and tokens in the tokenize_and_align_labels function in this tutorial: https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/token_classification.ipynb#scrollTo=n9qywopnIrJH 182


Question:
Hi! I would like to train a sentencePiece tokenizer from scratch but I’m a bit lost from the documentation and don’t know where to start. There are already examples on how to train a BPE tokenizer on the huggingface website but I don’t know if I can simply transfer it 1 to 1. Also I don’t even know where to find the trainable class for sentencePiece.
Have you already trained a sentencePiece tokenizer?

Context:
@Johncwok check this page: Using tokenizers from 🤗 Tokenizers — transformers 4.7.0 documentation 69
You can train a SentencePiece tokenizer
from tokenizers import SentencePieceBPETokenizer

tokenizer = SentencePieceBPETokenizer()
tokenizer.train_from_iterator(
    text,
    vocab_size=30_000,
    min_frequency=5,
    show_progress=True,
    limit_alphabet=500,
)

and then just wrap it with a PreTrainedTokenizerFast
from transformers import PreTrainedTokenizerFast

transformer_tokenizer = PreTrainedTokenizerFast(
    tokenizer_object=tokenizer
)

Documentation is not quite clear about this


Best way to mask a multi-token word when using `.*ForMaskedLM` models

Question:
For example, in a context where the model is likely to predict the word seaplane (which gets decomposed into two tokens), should I include a single mask or two masks in the contextual sentence?
Here is a complete example: Google Colaboratory 3
Below is the predicted top 6 words for a single mask (where the word seaplane should go). Here it seems reasonable to concatenate the top two predicted vocab words, but this doesn’t seem to extend into the less probable words in the list below.
top_vocab_idxes = torch.topk(torch.softmax(single_mask_token_logits[masked_idx], dim=0), 6)
for token_id in top_vocab_idxes[1]:
    print (tokenizer.decode([token_id]))
sea
plane
hangar
helicopter
lake
river

Below is result for using two masks in the contextual sentence, printing out the top 6 most likely combos for first and second masked tokens in each line.
top_vocab_idxes = torch.topk(probs, 6)
for token_id in torch.transpose(top_vocab_idxes[1], 1, 0):
    print (tokenizer.decode(token_id))
sea plane
water area
mountain hangar
land dive
landing aircraft
flying field

In this particular case the top 3 most probable combos above seem like reasonable predictions for the two masked tokens given context:
double_mask_sentence = f"""When taking off in a seaplane, flying in a seaplane,
and then landing in a {tokenizer.mask_token} {tokenizer.mask_token},
remember to fashion your seat belt."""

It seems likely that I should use the second method above for my inference and possible later fine-tuning, however, I doubt this is what is done during pretraining.
Thank you for any feedback on what might be best practice here.

Context:
This is something of interest for me too!
This might be of help: [2009.07118] It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners 5


TypeError when loading tokenizer with from_pretrained method for bart-large-mnli model

Question:
Hello everyone.
Here is my problem, (I wish someone can help me, I try so hard in vain to resolve it T.T) :
I use transformers 4.2.1 lib, and I am in a context where I only can use it in offline mode (no internet).
I want to use the bart-large-mnli model so I upload it on a specific server and I download the model with the following link :

      huggingface.co

facebook/bart-large-mnli at main 2

Then I try to use from_pretrained method like this :
tokenizer = BartTokenizerFast.from_pretrained(’/appli/pretrainedModel/bart-large-mnli’)
or like this  :
tokenizer = AutoTokenizer.from_pretrained(’/appli/pretrainedModel/bart-large-mnli’)
But every time I do this I get the following error (more detailed log at the end of my post, I truncate the last line with […] to replace all the content of merges.txt) :
“TypeError: Can’t convert [(‘Ä’, ‘t’), (‘Ä’, ‘a’), (‘h’, ‘e’), […] ] (list) to Union[Merges, Filename]”
I definitively don’t know what’s going wrong with merges.txt but it seems like there is a problem…
The content of /appli/pretrainedModel/bart-large-mnli is :
config.json
merges.txt
pytorch_model.bin
rust_model.ot
tokenizer_config.json
vocab.json
Does someone have any idea where is the problem ?
Thanks in advance.
More detailed error log :
---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)
 in
----> 1 tokenizer = AutoTokenizer.from_pretrained(’/appli/pretrainedModel/bart-large-mnli’)
2 pipeline(‘zero-shot-classification’, model=model, tokenizer=tokenizer)
/appli/.conda/envs/bf_verbatim/lib/python3.7/site-packages/transformers/models/auto/tokenization_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
383             tokenizer_class_py, tokenizer_class_fast = TOKENIZER_MAPPING[type(config)]
384             if tokenizer_class_fast and (use_fast or tokenizer_class_py is None):
 → 385                 return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
386             else:
387                 if tokenizer_class_py is not None:
/appli/.conda/envs/bf_verbatim/lib/python3.7/site-packages/transformers/tokenization_utils_base.py in from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs)
1767
1768         return cls._from_pretrained(
 → 1769             resolved_vocab_files, pretrained_model_name_or_path, init_configuration, *init_inputs, **kwargs
1770         )
1771
/appli/.conda/envs/bf_verbatim/lib/python3.7/site-packages/transformers/tokenization_utils_base.py in _from_pretrained(cls, resolved_vocab_files, pretrained_model_name_or_path, init_configuration, *init_inputs, **kwargs)
1839         # Instantiate tokenizer.
1840         try:
 → 1841             tokenizer = cls(*init_inputs, **init_kwargs)
1842         except OSError:
1843             raise OSError(
/appli/.conda/envs/bf_verbatim/lib/python3.7/site-packages/transformers/models/roberta/tokenization_roberta_fast.py in init(self, vocab_file, merges_file, tokenizer_file, errors, bos_token, eos_token, sep_token, cls_token, unk_token, pad_token, mask_token, add_prefix_space, **kwargs)
171             mask_token=mask_token,
172             add_prefix_space=add_prefix_space,
 → 173             **kwargs,
174         )
175
/appli/.conda/envs/bf_verbatim/lib/python3.7/site-packages/transformers/models/gpt2/tokenization_gpt2_fast.py in init(self, vocab_file, merges_file, tokenizer_file, unk_token, bos_token, eos_token, add_prefix_space, **kwargs)
139             eos_token=eos_token,
140             add_prefix_space=add_prefix_space,
 → 141             **kwargs,
142         )
143
/appli/.conda/envs/bf_verbatim/lib/python3.7/site-packages/transformers/tokenization_utils_fast.py in init(self, *args, **kwargs)
87         elif slow_tokenizer is not None:
88             # We need to convert a slow tokenizer to build the backend
—> 89             fast_tokenizer = convert_slow_tokenizer(slow_tokenizer)
90         elif self.slow_tokenizer_class is not None:
91             # We need to create and convert a slow tokenizer to build the backend
/appli/.conda/envs/bf_verbatim/lib/python3.7/site-packages/transformers/convert_slow_tokenizer.py in convert_slow_tokenizer(transformer_tokenizer)
657     converter_class = SLOW_TO_FAST_CONVERTERS[tokenizer_class_name]
658
 → 659     return converter_class(transformer_tokenizer).converted()
/appli/.conda/envs/bf_verbatim/lib/python3.7/site-packages/transformers/convert_slow_tokenizer.py in converted(self)
281                 continuing_subword_prefix="",
282                 end_of_word_suffix="",
 → 283                 fuse_unk=False,
284             )
285         )
TypeError: Can’t convert [(‘Ä’, ‘t’), (‘Ä’, ‘a’), (‘h’, ‘e’), […] ] (list) to Union[Merges, Filename]

Context:
Hi,
Were you able to resolve the issue?
Thanks in advance


Using truncated fragments as input samples in training

Question:
Hi!
I am using the tokenizers library, roughly following the run_mlm.py script to train a Masked Language Model (MobileBert) from scratch.
Since I am training an unsupervised model using truncated sentences, I was wondering if the truncated (left-out) fragments are included by default in the dataset for training since they would be valid examples for my use case (MLM setting). If they are not used (which I believe to be the case) I wanted to ask if there is any easy way in which I might include them in my training dataset (maybe by using return_overflowing_tokens and stride in a smart way?).
As an additional related question, I would like to know if there is any native way of sorting by length before batching to reduce the dataset size to the minimum. Something along these lines: pommedeterresautee gist and McCormickML blogpost.
EDIT: The best way I have found to do the smart batching is to create an ‘sample_length’ column and use the .sort method to sort by that column before tokenizing.
Thanks in advance!

Context:
By default, those are not included (unless you use the --line_by_line option which will concatenate all the samples then create block of the size you picked). Using return_overflowing_tokens is definitely an option to get those truncated part! stride is only if you want some overlap between the two parts of a long sentence, which is useful for question answering, but not necessarily for masked language modeling pretraining.
For the sorting by length before batching, we have the --group_by_length option in the Trainer, though it’s for the dataset so it happens after tokenization, which may not be what you are looking for.


Using whitespace tokenizer for training models

Question:
I have a dataset for which I wanted to use a tokenizer based on whitespace rather than any subword segmentation approach.
This snippet I got off github has a way to construct and use the custom tokenizer that operates on whitespaces:-
from tokenizers import Tokenizer, trainers
from tokenizers.models import BPE
from tokenizers.normalizers import Lowercase
from tokenizers.pre_tokenizers import CharDelimiterSplit

# We build our custom tokenizer:
tokenizer = Tokenizer(BPE())
tokenizer.normalizer = Lowercase()
tokenizer.pre_tokenizer = CharDelimiterSplit(' ')

# We can train this tokenizer by giving it a list of path to text files:
trainer = trainers.BpeTrainer(special_tokens=["[UNK]"], show_progress=True)
tokenizer.train(files=['/content/dataset.txt'], trainer=trainer)

I wanted to use it for pre-training the BigBird attention model, but facing two issues:

I can’t seem to be able to use this snippet with the custom tokenizer above to convert tokenized sentences in model-friendly sequences

from tokenizers.processors import BertProcessing

tokenizer._tokenizer.post_processor = tokenizers.processors.BertProcessing(
    ("</s>", tokenizer.token_to_id("</s>")),
    ("<s>", tokenizer.token_to_id("<s>")),
)
tokenizer.enable_truncation(max_length=16000)

This returns me an error, and without any preprocessing the output does not contain the sequence start and end tokens (<s>; </s>) as expected.

Next problem arises, when I save the tokenizer state in the specified folder, I am unable to use it via:

tokenizer = BigBirdTokenizerFast.from_pretrained("./tok", max_len=16000)

since it yields the error that my directory does not ‘reference’ the tokenizer files, which shouldn’t be an issue since using RobertaTokenizerFast does work - I assume it has something to do in the tokenization post-processing phase.
If anyone wants, I can create a reproducible colab notebook to speed up the issue being solved.
Thanks in advance,
N

Context: I have created a fully reproducible colab notebook, with commented problems and synthetic data. Please find it here 12. Thanx


Question:
I am following the Trainer example 31 to fine-tune a Bert model on my data for text classification, using the pre-trained tokenizer (bert-base-uncased).
In all examples I have found, the input texts are either single sentences or lists of sentences. However, my data is one string per document, comprising multiple sentences. When I inspect the tokenizer output, there are no [SEP] tokens put in between the sentences, e.g.:
This is how I tokenize my dataset:
def encode(examples):
  return tokenizer(examples['text'], truncation=True, padding='max_length')

train_dataset = train_dataset.map(encode, batched=True)

And this is an example result of the tokenization:
tokenizer.decode(train_dataset[0]["input_ids"])

[CLS] this is the first sentence . this is the second sentence. [SEP]

Given the special tokens in the beginning and the end, and that the output is lower-cased, I see that the input has been tokenized as expected. However, I was expecting to see a [SEP] between each sentence, as is the case when the input comprises a list of sentences.
What is the recommended approach? Should I split the input documents into sentences, and run the tokenizer on each of them? Or does the Transformer model handle the continuous stream of sentences?
I have seen posts like this:

Split document into sentences for sentence embedding

Just use a parser like stanza or spacy to tokenize/sentence segment your data. This is typically the first step in many NLP tasks.

And:

Summarization on long documents

The disadvantage is that there is no sentence boundary detection. You can theoretically solve that with the NLTK (or SpaCy) approach and splitting sentences.

However, it is not clear to me if this applied for a standard pipeline.

Context:
hey @carschno, how long are your documents (on average) and what kind of performance do you get with the current approach (i.e. tokenizing + truncating the whole document)?
if performance is a problem then since you’re doing text classification, you could try chunking each document into smaller passages with a sliding window (see this tutorial 109 for details on how that’s done), and then aggregate the [CLS] representations for each window in a manner similar to this 23 paper.
it’s not the most memory efficient strategy, but if your documents are not super long it might be a viable alternative to simple truncation


Question: Based on the CTRL approach on GPT2, i’m trying to add tokens in order to control my text generation style. Is there a difference between adding a token as a regular one and adding it as a special token?

Context: hey @Felipehonorato, as far as i know, special tokens won’t be split by the tokenizer which might be handy in your case where you’re trying to incorporate control tokens. you can find more information in the docs here 5.


How do you use SentencePiece for BPE of sequences with no whitespace

Question:
I am trying to use byte pair encoding on amino acid sequences which have no spaces:
ADNRRPIWNLGHMVNALKQIPTFLXDGANA
the tokenizers summary section of the docs states suggests SentencePiece could be useful, as it treats the input as a raw stream, includes the space in the set of characters to use, then uses BPE or unigram to construct the appropriate vocabulary.
How would I train a tokenizer from scratch using SentencePiece? The tokenizer library seems to only support
WordPiece.

Context:
In original sentencepiece model, white space is considered as a regular character. Please read the description here.

      github.com

google/sentencepiece - Whitespace is treated as a basic symbol 17

  Unsupervised text tokenizer for Neural Network-based text generation.

I am not totally familier with the huggingface implementation of the sentencepiece. But you can use the original sentencepiece library for that and then try loading that sentencepiece model by huggingface wrapper if needed.


Question:
Default mBERT tokenizes a sentence as ['[CLS]', 'This', 'is', 'a', 'sample', 'sentence', '[SEP]']. I want to change this behaviour and add a language specific token after the CLS token like this: ['[CLS]', '__en__', 'This', 'is', 'a', 'sample', 'sentence', '[SEP]']
I know TemplateProcessing can be used to achieve this if the language token doesn’t change
from tokenizers.processors import TemplateProcessing
tokenizer._tokenizer.post_processor = TemplateProcessing(
    single=f"{_lang_token} $A [SEP]",
    pair=f"{_lang_token} $A [SEP] $B:1 [SEP]:1",
    special_tokens=[("[SEP]", tokenizer.convert_tokens_to_ids("[SEP]")),
                    (_lang_token, tokenizer.convert_tokens_to_ids(_lang_token))],
)

But in my case, the language token changes with every batch. What is the best way to add these tokens? Creating TemplateProcessing objects every time seems inefficient.

Context: @sgugger any suggestions?


Question:
Hi guys, I am trying to work with a FairSeq model converted to  but I have some issues with tokenizer. I am trying to fine-tune it for POS tagging so I have the text split to words already and I want to use the offset_mapping to detect first token for each word. I do it like this:
tokenizer = RobertaTokenizerFast.from_pretrained('path', add_prefix_space=True)
ids = tokenizer([['drieme', 'drieme'], ['drieme']],
    is_split_into_words=True,
    padding=True,
    return_offsets_mapping=True)

The tokenization looks like this:
['<s>', 'Ġd', 'rieme', 'Ġd', 'rieme', '</s>']

But the output from the command looks like this:
{
  'input_ids': [[0, 543, 24209, 543, 24209, 2], [0, 543, 24209, 2, 1, 1]],
  'attention_mask': [[1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 0, 0]],
  'offset_mapping': [
    [(0, 0), (0, 1), (1, 6), (1, 1), (1, 6), (0, 0)],
    [(0, 0), (0, 1), (1, 6), (0, 0), (0, 0), (0, 0)]
  ]
}

Notice the offset mapping for the word drieme in the first case. First word has mappings (0, 1) and (1, 6). This looks reasonable, however the second drieme is (1, 1) and (1, 6). Suddenly, there is 1 at the first position. This 1 is there for all but first word for any sentence I try to parse. I feel like it might have something to do with handling the start of the sentence vs all the other words, but I am not sure how to solve this issue so that I have proper offset mappings. What am I doing wrong?

Context: Thanks for reporting, it’s definitely a bug. Could you open an issue on tokenizers 18 with your snippet?


BertTokenizerFast for stsb-xlm-r-multilingual model

Question:
Hi community,
Would there be a fast tokenizer for the stsb-xlm-r-multilingual model?
Thanks !

Context:
Hi community and @lewtun,
Could anyone have an idea on how to get a fast tokenizer for stsb-xlm-r-multilingual model?
I am blocked with low latency response due to tokenizer computation. Is there a fast tokenizer model as BertTokenizerFast or is there a way to run tokenizer on GPU ?


How to add additional custom pre-tokenization processing?

Question:
I would like to add a few custom functions for pre-tokenization. For example, I would like to split numerical text from any non-numerical test.
Eg
‘1000mg’ would become [‘1000’, ‘mg’].
I am trying to figure out the proper way to do this for the python binding; I think it may be a bit tricky since its a binding for the original rust version.
I am looking at the pretokenizer function
/huggingface/tokenizers/blob/2ccd16bf5c3dd97759d7bdf5229e2feeba314b4a/bindings/python/py_src/tokenizers/pre_tokenizers/init.pyi#L6
Which I am guessing may be where I could potentially as some pretokenization functions, but it doesn’t seem to return anything. I noticed that it’s expecting an instance of the PreTokenizedString defined here
/huggingface/tokenizers/blob/2ccd16bf5c3dd97759d7bdf5229e2feeba314b4a/bindings/python/py_src/tokenizers/init.pyi#L55
Which does seem to have some text processing functions. But they don’t seem to return anything. I am guessing that any additional rules need to be implemented in the original rust version itself?
I am looking at the rust pretokenizers code, it seems that I have to add any additional preprocessing code here

      github.com

    huggingface/tokenizers/blob/master/tokenizers/src/pre_tokenizers/unicode_scripts/pre_tokenizer.rs 3
use crate::pre_tokenizers::unicode_scripts::scripts::{get_script, Script};
use crate::tokenizer::{normalizer::Range, PreTokenizedString, PreTokenizer, Result};

#[derive(Clone, Debug)]
pub struct UnicodeScripts;
impl_serde_unit_struct!(UnicodeScriptsVisitor, UnicodeScripts);

impl UnicodeScripts {
    pub fn new() -> Self {
        Self {}
    }
}

impl Default for UnicodeScripts {
    fn default() -> Self {
        Self::new()
    }
}

// This code exists in the Unigram default IsValidSentencePiece.

  This file has been truncated. show original

Does this seem like the right track for adding additional preprocessing code?
It it makes a difference, what I am trying to do is train a brand new tokenizer.

Context:
Hi @reSearch2vec
There are multiple ways to customize the pre-tokenization process:

Using existing components
The tokenizers library provides many different PreTokenizer that you can use, and even combine as you wish to. There is a list of components in the official documentation 54

Using custom components written in Python
It is possible to customize some of the components (Normalizer, PreTokenizer, and Decoder) using Python code. This hasn’t been documented yet, but you can find an example here 67. It lets you directly manipulate the NormalizedString 15 or PreTokenizedString 13 to normalize and pre-tokenize as you wish.

Now for the example you mentioned (ie ‘1000mg’ would become [‘1000’, ‘mg’]), you can probably use the Digits PreTokenizer that does exactly this.
If you didn’t get a chance to familiarize yourself with the Getting started part of our documentation 35, I think you will love it as it explains a bit more how to customize your tokenizer, and gives concrete examples.


Using a BertWordPieceTokenizer trained from scratch from transformers

Question:
Hey everyone,
I’d like to load a BertWordPieceTokenizer I trained from scratch using the interface built in transformers, either with BertTokenizer or BertTokenizerFast. It looks like those two tokenizers in transformers expect different ways of loading in the saved data from BertWordPieceTokenizer, and I am wondering what is the best way to go about things.
Example
I am training on a couple test files, saving the tokenizer, and the reloading it in tokenizers.BertTokenizer (there is a bit of ceremony here creating the test data, but this is everything you need to reproduce the behavior I am seeing):
from pathlib import Path

from tokenizers import BertWordPieceTokenizer
from transformers import BertTokenizer

def test_text():
    text = [
        "This is a test, just a test",
        "nothing more, nothing less"
    ]

    return text

def create_test_files():
    test_path = Path("tmp")
    test_path.mkdir()

    test_data = test_text()

    for idx, text in enumerate(test_data):
        file = test_path.joinpath(f"file{idx}.txt")
        with open(file, "w") as f:
            f.write(text)

    return test_path

def cleanup_test(path):
    path = Path(path)

    for child in path.iterdir():
        if child.is_file():
            child.unlink()
        else:
            rm_tree(child)

    path.rmdir()

def create_tokenizer_savepath():
    savepath = Path("./bert")
    savepath.mkdir()
    return str(savepath)

def main():
    # Saving two text files to train the tokenizer
    test_path = create_test_files()

    files = test_path.glob("**/*.txt")
    files = [str(f) for f in files]

    tokenizer = BertWordPieceTokenizer(
        clean_text=True,
        strip_accents=True,
        lowercase=True,
    )

    tokenizer.train(
        files,
        vocab_size=15,
        min_frequency=1,
        show_progress=True,
        special_tokens=["[PAD]", "[UNK]", "[CLS]", "[SEP]", "[MASK]"],
        limit_alphabet=1000,
        wordpieces_prefix="##",
    )

    savepath = create_tokenizer_savepath()
    tokenizer.save_model(savepath, "pubmed_bert")

    tokenizer = BertTokenizer.from_pretrained(
        f"{savepath}/pubmed_bert-vocab.txt",
        max_len=512
    )

    print(tokenizer)

    cleanup_test(test_path)
    cleanup_test(savepath)

if __name__ == "__main__":
    main()

Loading the Trained Tokenizer
Specifying the path to the pubmed_bert-vocab.txt is deprecated:
Calling BertTokenizer.from_pretrained() with the path to a single file or url is deprecated
PreTrainedTokenizer(name_or_path='bert/pubmed_bert-vocab.txt', vocab_size=30, model_max_len=512, is_fast=False, padding_side='right', special_tokens={'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'})

But, if I just specify the path to the directory containing pubmed_bert-vocab.txt:
Traceback (most recent call last):
  File "minimal_tokenizer.py", line 86, in <module>
    main()
  File "minimal_tokenizer.py", line 76, in main
    max_len=512
  File "/home/ygx/opt/local/anaconda3/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 1777, in from_pretrained
    raise EnvironmentError(msg)
OSError: Can't load tokenizer for 'bert'. Make sure that:

- 'bert' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'bert' is the correct path to a directory containing relevant tokenizer files

The directory I am saving to only contains pubmed_bert-vocab.txt. If specifying the full path to that vocab is deprecated, what is the best way to load that tokenizer?
Using BertTokenizerFast
If I swap out BertTokenizer for BertTokenizerFast, and pass in the path to the directory where I have saved my tokenizer trained from scratch, I get the same error:
Traceback (most recent call last):
  File "minimal_tokenizer.py", line 86, in <module>
    main()
  File "minimal_tokenizer.py", line 76, in main
    max_len=512
  File "/home/ygx/opt/local/anaconda3/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 1777, in from_pretrained
    raise EnvironmentError(msg)
OSError: Can't load tokenizer for 'bert'. Make sure that:

- 'bert' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'bert' is the correct path to a directory containing relevant tokenizer files

And if I specify the path to file saved by my tokenizer (pubmed_bert-vocab.txt), I get a ValueError (vs the deprecation warning I was getting using BertTokenizer):
Traceback (most recent call last):
  File "minimal_tokenizer.py", line 86, in <module>
    main()
  File "minimal_tokenizer.py", line 76, in main
    max_len=512
  File "/home/ygx/opt/local/anaconda3/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 1696, in from_pretrained
    "Use a model identifier or the path to a directory instead.".format(cls.__name__)
ValueError: Calling BertTokenizerFast.from_pretrained() with the path to a single file or url is not supported.Use a model identifier or the path to a directory instead.

Current Approach
I am currently using BertTokenizer, specifying the full path the the pubmed_bert-vocab.txt and am ignoring the deprecation warning, but ideally I would like use BertTokenizerFast but don’t know how to load in my saved tokenizer. What is the best way to go forward on this?

Context: pinging @anthony


Token Classification: How to tokenize and align labels with overflow and stride?

Question:
Hello Huggingface,
I try to solve a token classification task where the documents are longer than the model’s max length.
I modified the tokenize_and_align_labels function from example token classification notebook 41. I set the tokenizer option return_overflowing_tokens=True and rewrote the function to map labels for the overflowing tokens:
tokenizer_settings = {'is_split_into_words':True,'return_offsets_mapping':True,
                        'padding':True, 'truncation':True, 'stride':0,
                        'max_length':tokenizer.model_max_length, 'return_overflowing_tokens':True}

def tokenize_and_align_labels(examples):
    tokenized_inputs = tokenizer(examples["tokens"], **tokenizer_settings)

    labels = []
    for i,document in enumerate(tokenized_inputs.encodings):
       doc_encoded_labels = []
       last_word_id = None
       for word_id  in document.word_ids:
           if word_id == None: #or last_word_id == word_id:
               doc_encoded_labels.append(-100)
           else:
               document_id = tokenized_inputs.overflow_to_sample_mapping[i]
               label = examples[task][document_id][word_id]
               doc_encoded_labels.append(int(label))
           last_word_id = word_id
       labels.append(doc_encoded_labels)

    tokenized_inputs["labels"] = labels
    return tokenized_inputs

Executing this code will result in an error:
exception has occurred: ArrowInvalid
Column 5 named task1 expected length 820 but got length 30

It looks like the input 30 examples can’t be mapped to the 820 examples after the slicing. How can I solve this issue?
Environment info
Google Colab running this notbook 41
To reproduce
Steps to reproduce the behaviour:

Replace the tokenize_and_align_labels function with the function given above.
Add examples longer than max_length
run tokenized_datasets = datasets.map(tokenize_and_align_labels, batched=True) cell.

Context: cc @sgugger


Space token ’ ’ cannot be add when is_split_into_words = True

Question:
for example,
>>> tokenizer = BertTokenizer.from_pretrained('bert-base-chinese')
>>> tokenizer.add_tokens(' ')
1

>>> tokenizer.encode('你好 世界', add_special_tokens=False)
[872, 1962, 21128, 686, 4518]
>>> tokenizer.encode(['你','好',' ', '世', '界'], is_split_into_words=True, add_special_tokens=False)
 [872, 1962, 686, 4518]

Obviously, the blank token is ignored. But if you change it to another token like ‘[balabala]’, it works.
So what is the proper way to do this?

Context: I found that one way is to use convert_tokens_to_ids, yet by which I cannot use the convenient features in encode and __call__ such as padding and automatically generating attention_mask


Does AutoTokenizer.from_pretrained add [cls] tokens?

Question:
Hello,
I am currently working on a classification problem using ProtBERT and I am following the Fine-Tuning Tutorial 4. I have called the tokenised using
tokenizer = AutoTokenizer.from_pretrained

and then tokenised like the tutorial  says
train_encodings = tokenizer(seq_train, truncation=True, padding=True,
max_length=1024, return_tensors="pt")

Unfortunately, the model doesn’t seem to be learning (I froze the BERT layers).  From reading around, I saw that I need to add the [CLS] token and found such an option using
tokenised.encode(add_special_tokens=True)

Yet the tutorial I am following doesn’t seem to require and I was wondering wyy is there a discrepancy and perhaps maybe this is why my model isn’t learning.
Thank you

Context:
Hi @theudster, I’m pretty sure that ProtBERT has a CLS token since you can see it in the tokenizer’s special tokens map:
from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("Rostlab/prot_bert")
# returns {'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'}
tokenizer.special_tokens_map

You can also see it by encoding a text and then decoding it:
text = "I love Adelaide!"
# add_special_tokens=True is set by default
text_enc = tokenizer.encode(text)

for tok in text_enc:
    print(tok, tokenizer.decode(tok))

You say you froze the BERT layers, so I’m wondering how you’re doing fine-tuning? I’ve sometimes found that the tutorials in the docs aren’t always complete, so for fine-tuning with text classification I would recommend following Sylvain’s tutorial here: https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/token_classification.ipynb 9


Combine multiple sentences together during tokenization

Question:
I want my output to be like [CLS] [SEP] text1 [SEP] text2 [SEP] text3 [SEP] eos token. As per the default behaviour, tokenizer expects either a string or a pair of string.
tokenizer(sentence1, sentence2) # returns a single vector value for input_ids. I want this but for three sentences
I want the pair of string behavior for three sentences. I can pass a list of sentences, but that creates 3 lists of input_ids.
tokenizer([sentence1, sentence2, sentence3]) # returns three tensors for input_ids
I want a single tensor representing the output I wrote above.
Is there any good way of doing it ?

Context:
I don’t think tokenizer handles this case directly.
You could directly join the sentences using [SEP] and then encode it as one single text.
tok = BertTokenizer.from_pretrained("bert-base-cased")
text = "sent1 [SEP] sent2 [SEP] sent3"
ids = tok(text, add_special_tokens=True).input_ids
tok.decode(ids)
=> '[CLS] sent1 [SEP] sent2 [SEP] sent3 [SEP]'


“OSError: Model name ‘./XX’ was not found in tokenizers model name list” - cannot load custom tokenizer in Transformers

Question:
I’m trying to create tokenizer with my own dataset/vocabulary using Sentencepiece and then use it with AlbertTokenizer transformers.
I followed really closely the tutorial on how to train a model from scratch: https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb#scrollTo=hO5M3vrAhcuj 7
    # import relevant libraries
    from pathlib import Path
    from tokenizers import SentencePieceBPETokenizer
    from tokenizers.implementations import SentencePieceBPETokenizer
    from tokenizers.processors import BertProcessing
    from transformers import AlbertTokenizer

    paths = [str(x) for x in Path("./data").glob("**/*.txt")]

    # Initialize a tokenizer
    tokenizer = SentencePieceBPETokenizer(add_prefix_space=True)

    # Customize training
    tokenizer.train(files=paths,
                    vocab_size=32000,
                    min_frequency=2,
                    show_progress=True,
                    special_tokens=['<unk>'],)

    # Saving model
    tokenizer.save_model("Sent-AlBERT")

    tokenizer = SentencePieceBPETokenizer(
        "./Sent-AlBERT/vocab.json",
        "./Sent-AlBERT/merges.txt",)

    tokenizer.enable_truncation(max_length=512)

Everything is fine up until this point when I tried to re-create the tokenizer in transformers
    # Re-create our tokenizer in transformers
        tokenizer = AlbertTokenizer.from_pretrained("./Sent-AlBERT", do_lower_case=True)

This is the error message I kept receiving:
OSError: Model name './Sent-AlBERT' was not found in tokenizers model name list (albert-base-v1, albert-large-v1, albert-xlarge-v1, albert-xxlarge-v1, albert-base-v2, albert-large-v2, albert-xlarge-v2, albert-xxlarge-v2). We assumed './Sent-AlBERT' was a path, a model identifier, or url to a directory containing vocabulary files named ['spiece.model'] but couldn't find such vocabulary files at this path or url.

For some reason, it works with RobertaTokenizerFast but not with AlbertTokenzier.
If anyone could give me a suggestion or any sort of direction on how to use Sentencepiece with AlberTokenizer I would really appreciate it.
P.S: I also tried to use ByteLevelBPETokenizer with DistilBertTokenizer, but it couldn’t recognize the tokenizer in the transformer either. I’m not sure what I am missing here.

Context:
You can’t directly use this tokenizer with a “slow” tokenizer (not backed by rust) there is a conversion step to do (not super versed in it but maybe @thomwolf can chime in?).
It should work with AlbertTokenizerFast (which has more functionality and is faster so it should be a win-win over all!)