how to reduce original model's tokenizer vocabulary
Author: masterwang22327
Pull request: no
Labels: name: Feature request

`###` Feature request

I am working on model distillation. I am currently using the nllb-distilled-600M model, but the parameters of this model are still too large, and the vocabulary supports more than 100 languages. My use case is single language translation, such as English to Hebrew. Therefore, I need to reduce the redundant vocabulary of the original model and only keep the English and Hebrew vocabulary. I noticed that transformers do not use the sentencepiece.bpe.model file, and I don't want to retrain a tokenizer, because the trained tokenizer will be inconsistent with the original tokenizer result, which will lead to the subsequent model weight migration and model distillation process cannot be carried out. Therefore, my idea is to quickly replace the tokenizer.json and tokenizer_config.json files in the original model, and then migrate the model weights at the model level to get a pruned model. What I am doing now is to load the original model tokenizer, tokenize the corpus I prepared, count the registered tokens, regain a reduced vocabulary, and change the corresponding json file. Is there any better strategy to quickly replace the tokenizer vocabulary?

![Image](https://github.com/user-attachments/assets/0433f4df-766d-4804-a752-e02a104d3cfa)

### Motivation

quick modify model vocabulary for beater application

### Your contribution

> `def modify_tokenizer():

    for sentences in tqdm.tqdm(range(100,len(en_corpus),100)):
        enc = teacher_tokenizer(en_corpus[sentences-100:sentences],
                        add_special_tokens=False,
                        return_attention_mask=False,
                        return_token_type_ids=False)
        for ids in enc['input_ids']:
            selected_ids.update(ids)
    print('all english tokens nums is ',len(selected_ids))
    for sentences in tqdm.tqdm(range(100,len(he_corpus),100)):
        enc = teacher_tokenizer(he_corpus[sentences-100:sentences],
                        add_special_tokens=False,
                        return_attention_mask=False,
                        return_token_type_ids=False)
        for ids in enc['input_ids']:
            selected_ids.update(ids)
    print('all english+Hebrew tokens nums is ',len(selected_ids))
    for tok in teacher_tokenizer.all_special_tokens:
        # print('special_token ',tok)
        selected_ids.add(teacher_tokenizer.convert_tokens_to_ids(tok))
    print('all english+Hebrew_special tokens nums is ',len(selected_ids))
    #  从原 vocab 中反查出对应 token
    orig_vocab = teacher_tokenizer.get_vocab()
    new_tokens = []
    for tok, idx in sorted(orig_vocab.items(), key=lambda kv: kv[1]):
        if idx in selected_ids:
            new_tokens.append(tok)
    # 写出新的 vocab.json（Hugging Face 格式）
    new_vocab = {tok: i for i, tok in enumerate(new_tokens)}
    #修改原有tokenizer和tokenizer_config
    teacher_tokenizer_path='/workspace/nllb-200-distilled-600M/tokenizer.json'
    teacher_tokenizer_config_path='/workspace/nllb-200-distilled-600M/tokenizer_config.json'
    student_tokenizer_path='/workspace/distilled_model_test/tokenizer.json'
    student_tokenizer_config_path='/workspace/distilled_model_test/tokenizer_config.json'
    def _read_json(path):
        with open(path, "r", encoding="utf-8") as f:
            data = json.load(f)
        return data
    def _write_json(path,data):
        with open(path, "w", encoding="utf-8") as f:
            json.dump(data, f, ensure_ascii=False, indent=2)
    #change tokenizer 
    student_tokenizer_data=_read_json(teacher_tokenizer_path)
    student_tokenizer_data['model']['vocab']=new_vocab
    for single_added_token in student_tokenizer_data['added_tokens']:
        single_added_token['id']=new_vocab[single_added_token['content']]
    new_merges=[]
    #change merges
    for merge_pair in student_tokenizer_data['model']['merges']:
        _temp_merge=merge_pair[0]+merge_pair[1]
        if _temp_merge in new_vocab.keys():
            new_merges.append(merge_pair)
    student_tokenizer_data['model']['merges']=new_merges
    _write_json(student_tokenizer_path,student_tokenizer_data)
    #change tokenizer_config`

Comments: