RagTokenizer Missing patch_token_id, patch_token, and encode Functionality
Author: hanshengzhu0001
Pull request: no
Labels: name: Good Second Issue

name: Feature request

### Feature request

I propose adding the following functionalities to the RagTokenizer in the Hugging Face Transformers library:

Support for patch_token_id and patch_token attributes: These attributes are essential for specifying special tokens that can be used during tokenization, particularly for Retrieval-Augmented Generation (RAG) models.
Implementation of the encode function: This function is critical for converting input text into token IDs, which are a standard input for Transformer-based models.
These additions would bring RagTokenizer in line with other tokenizers in the library, making it easier to use in preprocessing pipelines for training and inference.

Paper reference: [RAG: Retrieval-Augmented Generation](https://arxiv.org/abs/2005.11401)
Current RagTokenizer documentation: Hugging Face Transformers

### Motivation

The absence of the patch_token_id, patch_token, and encode functionalities in RagTokenizer introduces several limitations:

It is challenging to preprocess data for RAG models without a way to specify and use special tokens like patch_token.
The lack of an encode function makes it cumbersome to tokenize text into input IDs, which is a critical step for training and inference. This is a deviation from the expected behavior of tokenizers in the Transformers library.
This can cause confusion and inefficiency for users accustomed to the functionality available in other tokenizers like BertTokenizer or GPT2Tokenizer.
Addressing these issues will make RagTokenizer more consistent with the rest of the library and improve usability in RAG-related workflows.

### Your contribution

I am willing to contribute by:

Submitting a Pull Request (PR) to implement these functionalities, given guidance on the expected behavior and the existing code structure.
Writing unit tests to verify the behavior of the patch_token_id, patch_token, and encode functionalities.
Updating the documentation to reflect these changes.
Let me know if this aligns with your vision for the RagTokenizer, and I’d be happy to assist further!

cc @ArthurZucker @itazap 

Comments:
cc @arthurzucker @itazap for tokenizers

@hanshengzhu0001 have you already started or ready with the codebase to merge? 

We follow what the community want so in general if this gets a lot of traction, happy to merge a PR that adds this!

Hello, I would love to help contribute to this, opening up a PR with a naive implementation, happy to iterate upon based on feedback!

After doing some more digging around, I found that #37967 already attempts this issue, so I don't want to duplicate existing work.

To my understanding, the paper's encoding step refers to the DPR question encoder, the query side of the bi-encoder, producing a dense query representation. This is already implemented through `model.question_encoder()`. Separately, `RagTokenizer` lacks the usual tokenizer convenience method `.encode()`.

I was also ambiguous on the patch_token implementation in the RAG paper. Could you clarify its intended role? Is there another use case for patch_token? 

Additionally, since #37967 has been inactive since May 2025, I plan on taking over with a narrowly scoped change that adds tokenizer-method delegation and tests, assuming that the behavior is still desired. Please let me know if I can help. Thanks.

Hi, I did love to work on this! 
My plan: add patch_token / patch_token_id as properties on RagTokenizer (following the pattern used in BertTokenizer/GPT2Tokenizer for special token attributes), and implement encode() by delegating to the underlying question_encoder/generator tokenizers the way __call__ already does. Let me know if this approach sounds right before I open a PR.