Repository: huggingface/transformers.js (language: javascript)

Issue:
[Model request] Add support for donut models
**Describe the bug**
*A clear and concise description of what the bug is.*

```
InitializersAndNodeArgs] Removing initializer '/decoder/decoder/layers.1/self_attn_layer_norm/Constant_output_0'. It is not used by any node and should be removed from the model.
- error Error [TypeError]: Cannot convert undefined to a BigInt
    at BigInt (<anonymous>)
    at file:///Users/datduyn/Documents/mygithub.com/embedcloud-server-interference/vercel_serverless/node_modules/.pnpm/@xenova+transformers@2.6.0/node_modules/@xenova/transformers/src/models.js:252:47
    at Array.map (<anonymous>)
    at toI64Tensor (file:///Users/datduyn/Documents/mygithub.com/embedcloud-server-interference/vercel_serverless/node_modules/.pnpm/@xenova+transformers@2.6.0/node_modules/@xenova/transformers/src/models.js:252:38)
    at Function.seq2seqRunBeam [as _runBeam] (file:///Users/datduyn/Documents/mygithub.com/embedcloud-server-interference/vercel_serverless/node_modules/.pnpm/@xenova+transformers@2.6.0/node_modules/@xenova/transformers/src/models.js:404:28)
    at Function.runBeam (file:///Users/datduyn/Documents/mygithub.com/embedcloud-server-interference/vercel_serverless/node_modules/.pnpm/@xenova+transformers@2.6.0/node_modules/@xenova/transformers/src/models.js:1260:27)
    at Function.generate (file:///Users/datduyn/Documents/mygithub.com/embedcloud-server-interference/vercel_serverless/node_modules/.pnpm/@xenova+transformers@2.6.0/node_modules/@xenova/transformers/src/models.js:980:41)
    at Function._call (file:///Users/datduyn/Documents/mygithub.com/embedcloud-server-interference/vercel_serverless/node_modules/.pnpm/@xenova+transformers@2.6.0/node_modules/@xenova/transformers/src/pipelines.js:1361:43)
    at async handler (webpack-internal:///(api)/./src/lib/base-model-api-handler.ts:32:28) {
  digest: undefined
}
```


**How to reproduce**

```
let url = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/cats.jpg';
let captioner = await pipeline('image-to-text', 'Xenova/donut-base-finetuned-cord-v2');
let output = await captioner(url);
// [{ generated_text: 'a cat laying on a couch with another cat' }]
```

Running on Nextjs


**Expected behavior**
*A clear and concise description of what you expected to happen.*

<!-- ANSWER GOES HERE -->

**Logs/screenshots**
*If applicable, add logs/screenshots to help explain your problem.*

**Environment**
- Transformers.js version: 2.6.0
- MacOS M2



Seems to be related to #10 


Discussion / hints:
Debugging context:

![image](https://github.com/xenova/transformers.js/assets/5236548/937fe5fa-8c72-4a28-ba71-15f673e91d3a)

Because `getStartBeams` doesn't fill `output_token_ids` here:

![image](https://github.com/xenova/transformers.js/assets/5236548/b2ce6273-8e9a-49cd-8fb6-b0be075994bd)

Hi there 👋 Thanks for the report. It looks like this is due to `decoder_start_token_id` being set to null in [config.json](https://huggingface.co/naver-clova-ix/donut-base-finetuned-cord-v2/blob/main/config.json#L23). Also, it should be noted that donut-swin models are technically [not yet supported by transformers.js](https://github.com/xenova/transformers.js#models), which is why there are issues (I haven't tested them before for correctness). I've only converted the models so far (because I intended to add support for them soon).



Fix patch:
diff --git a/README.md b/README.md
index 502784a69..3a2fc81c3 100644
--- a/README.md
+++ b/README.md
@@ -237,7 +237,7 @@ You can refine your search by selecting the task you're interested in (e.g., [te
 
 | Task                     | ID | Description | Supported? |
 |--------------------------|----|-------------|------------|
-| [Document Question Answering](https://huggingface.co/tasks/document-question-answering)         | `document-question-answering`  | Answering questions on document images. | ❌ |
+| [Document Question Answering](https://huggingface.co/tasks/document-question-answering)         | `document-question-answering`  | Answering questions on document images. | ✅ [(docs)](https://huggingface.co/docs/transformers.js/api/pipelines#module_pipelines.DocumentQuestionAnsweringPipeline)<br>[(models)](https://huggingface.co/models?pipeline_tag=document-question-answering&library=transformers.js) |
 | [Feature Extraction](https://huggingface.co/tasks/feature-extraction)         |  `feature-extraction`  | Transforming raw data into numerical features that can be processed while preserving the information in the original dataset. | ✅ [(docs)](https://huggingface.co/docs/transformers.js/api/pipelines#module_pipelines.FeatureExtractionPipeline)<br>[(models)](https://huggingface.co/models?pipeline_tag=feature-extraction&library=transformers.js) |
 | [Image-to-Text](https://huggingface.co/tasks/image-to-text)         |  `image-to-text`  | Output text from a given image. | ✅ [(docs)](https://huggingface.co/docs/transformers.js/api/pipelines#module_pipelines.ImageToTextPipeline)<br>[(models)](https://huggingface.co/models?pipeline_tag=image-to-text&library=transformers.js) |
 | [Text-to-Image](https://huggingface.co/tasks/text-to-image)         |  `text-to-image`  | Generates images from input text.  | ❌ |
@@ -271,6 +271,7 @@ You can refine your search by selecting the task you're interested in (e.g., [te
 1. **[DeiT](https://huggingface.co/docs/transformers/model_doc/deit)** (from Facebook) released with the paper [Training data-efficient image transformers & distillation through attention](https://arxiv.org/abs/2012.12877) by Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, Hervé Jégou.
 1. **[DETR](https://huggingface.co/docs/transformers/model_doc/detr)** (from Facebook) released with the paper [End-to-End Object Detection with Transformers](https://arxiv.org/abs/2005.12872) by Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko.
 1. **[DistilBERT](https://huggingface.co/docs/transformers/model_doc/distilbert)** (from HuggingFace), released together with the paper [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://arxiv.org/abs/1910.01108) by Victor Sanh, Lysandre Debut and Thomas Wolf. The same method has been applied to compress GPT2 into [DistilGPT2](https://github.com/huggingface/transformers/tree/main/examples/research_projects/distillation), RoBERTa into [DistilRoBERTa](https://github.com/huggingface/transformers/tree/main/examples/research_projects/distillation), Multilingual BERT into [DistilmBERT](https://github.com/huggingface/transformers/tree/main/examples/research_projects/distillation) and a German version of DistilBERT.
+1. **[Donut](https://huggingface.co/docs/transformers/model_doc/donut)** (from NAVER), released together with the paper [OCR-free Document Understanding Transformer](https://arxiv.org/abs/2111.15664) by Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeongyeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, Seunghyun Park.
 1. **[FLAN-T5](https://huggingface.co/docs/transformers/model_doc/flan-t5)** (from Google AI) released in the repository [google-research/t5x](https://github.com/google-research/t5x/blob/main/docs/models.md#flan-t5-checkpoints) by Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei
 1. **[GPT Neo](https://huggingface.co/docs/transformers/model_doc/gpt_neo)** (from EleutherAI) released in the repository [EleutherAI/gpt-neo](https://github.com/EleutherAI/gpt-neo) by Sid Black, Stella Biderman, Leo Gao, Phil Wang and Connor Leahy.
 1. **[GPT NeoX](https://huggingface.co/docs/transformers/model_doc/gpt_neox)** (from EleutherAI) released with the paper [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://arxiv.org/abs/2204.06745) by Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, Samuel Weinbach
diff --git a/docs/snippets/5_supported-tasks.snippet b/docs/snippets/5_supported-tasks.snippet
index 7a91cbf87..e2da1636a 100644
--- a/docs/snippets/5_supported-tasks.snippet
+++ b/docs/snippets/5_supported-tasks.snippet
@@ -53,7 +53,7 @@
 
 | Task                     | ID | Description | Supported? |
 |--------------------------|----|-------------|------------|
-| [Document Question Answering](https://huggingface.co/tasks/document-question-answering)         | `document-question-answering`  | Answering questions on document images. | ❌ |
+| [Document Question Answering](https://huggingface.co/tasks/document-question-answering)         | `document-question-answering`  | Answering questions on document images. | ✅ [(docs)](https://huggingface.co/docs/transformers.js/api/pipelines#module_pipelines.DocumentQuestionAnsweringPipeline)<br>[(models)](https://huggingface.co/models?pipeline_tag=document-question-answering&library=transformers.js) |
 | [Feature Extraction](https://huggingface.co/tasks/feature-extraction)         |  `feature-extraction`  | Transforming raw data into numerical features that can be processed while preserving the information in the original dataset. | ✅ [(docs)](https://huggingface.co/docs/transformers.js/api/pipelines#module_pipelines.FeatureExtractionPipeline)<br>[(models)](https://huggingface.co/models?pipeline_tag=feature-extraction&library=transformers.js) |
 | [Image-to-Text](https://huggingface.co/tasks/image-to-text)         |  `image-to-text`  | Output text from a given image. | ✅ [(docs)](https://huggingface.co/docs/transformers.js/api/pipelines#module_pipelines.ImageToTextPipeline)<br>[(models)](https://huggingface.co/models?pipeline_tag=image-to-text&library=transformers.js) |
 | [Text-to-Image](https://huggingface.co/tasks/text-to-image)         |  `text-to-image`  | Generates images from input text.  | ❌ |
diff --git a/docs/snippets/6_supported-models.snippet b/docs/snippets/6_supported-models.snippet
index d3c448692..c203530cd 100644
--- a/docs/snippets/6_supported-models.snippet
+++ b/docs/snippets/6_supported-models.snippet
@@ -17,6 +17,7 @@
 1. **[DeiT](https://huggingface.co/docs/transformers/model_doc/deit)** (from Facebook) released with the paper [Training data-efficient image transformers & distillation through attention](https://arxiv.org/abs/2012.12877) by Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, Hervé Jégou.
 1. **[DETR](https://huggingface.co/docs/transformers/model_doc/detr)** (from Facebook) released with the paper [End-to-End Object Detection with Transformers](https://arxiv.org/abs/2005.12872) by Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko.
 1. **[DistilBERT](https://huggingface.co/docs/transformers/model_doc/distilbert)** (from HuggingFace), released together with the paper [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://arxiv.org/abs/1910.01108) by Victor Sanh, Lysandre Debut and Thomas Wolf. The same method has been applied to compress GPT2 into [DistilGPT2](https://github.com/huggingface/transformers/tree/main/examples/research_projects/distillation), RoBERTa into [DistilRoBERTa](https://github.com/huggingface/transformers/tree/main/examples/research_projects/distillation), Multilingual BERT into [DistilmBERT](https://github.com/huggingface/transformers/tree/main/examples/research_projects/distillation) and a German version of DistilBERT.
+1. **[Donut](https://huggingface.co/docs/transformers/model_doc/donut)** (from NAVER), released together with the paper [OCR-free Document Understanding Transformer](https://arxiv.org/abs/2111.15664) by Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeongyeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, Seunghyun Park.
 1. **[FLAN-T5](https://huggingface.co/docs/transformers/model_doc/flan-t5)** (from Google AI) released in the repository [google-research/t5x](https://github.com/google-research/t5x/blob/main/docs/models.md#flan-t5-checkpoints) by Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei
 1. **[GPT Neo](https://huggingface.co/docs/transformers/model_doc/gpt_neo)** (from EleutherAI) released in the repository [EleutherAI/gpt-neo](https://github.com/EleutherAI/gpt-neo) by Sid Black, Stella Biderman, Leo Gao, Phil Wang and Connor Leahy.
 1. **[GPT NeoX](https://huggingface.co/docs/transformers/model_doc/gpt_neox)** (from EleutherAI) released with the paper [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://arxiv.org/abs/2204.06745) by Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, Samuel Weinbach
diff --git a/scripts/supported_models.py b/scripts/supported_models.py
index c998ef954..f481dda41 100644
--- a/scripts/supported_models.py
+++ b/scripts/supported_models.py
@@ -203,6 +203,7 @@
     'donut': [
         # Image-to-text
         'naver-clova-ix/donut-base-finetuned-cord-v2',
+        'naver-clova-ix/donut-base-finetuned-zhtrainticket',
 
         # Document Question Answering
         'naver-clova-ix/donut-base-finetuned-docvqa',
diff --git a/src/models.js b/src/models.js
index c71730990..d473f4413 100644
--- a/src/models.js
+++ b/src/models.js
@@ -80,7 +80,7 @@ import {
 } from './utils/tensor.js';
 
 import { executionProviders, ONNX } from './backends/onnx.js';
-import { medianFilter, round } from './transformers.js';
+import { medianFilter } from './transformers.js';
 const { InferenceSession, Tensor: ONNXTensor } = ONNX;
 
 /**
@@ -341,12 +341,13 @@ async function seq2seqForward(self, model_inputs) {
 /**
  * Start the beam search process for the seq2seq model.
  * @param {PreTrainedModel} self The seq2seq model object.
- * @param {Object[]} inputTokenIds Array of input token ids for each input sequence.
+ * @param {Tensor} inputTokenIds Array of input token ids for each input sequence.
+ * @param {Object} generation_config The generation config.
  * @param {number} numOutputTokens The maximum number of output tokens for the model.
  * @returns {Object[]} Array of beam search objects.
  * @private
  */
-function seq2seqStartBeams(self, inputTokenIds, numOutputTokens) {
+function seq2seqStartBeams(self, inputTokenIds, generation_config, numOutputTokens) {
     let beams = [];
     let beamId = 0;
 
@@ -354,8 +355,17 @@ function seq2seqStartBeams(self, inputTokenIds, numOutputTokens) {
     const requires_attention_mask = self.requires_attention_mask ?? true;
 
     // decoder_input_ids == output_token_ids
-    let decoder_input_ids = self.config.decoder_start_token_id;
-    if (!Array.isArray(decoder_input_ids)) {
+    let decoder_input_ids =
+        generation_config.decoder_input_ids
+        ?? generation_config.decoder_start_token_id
+        ?? generation_config.bos_token_id
+        ?? generation_config.eos_token_id;
+
+    // Support input as tensor or list
+    // TODO support batched decoder_input_ids
+    if (decoder_input_ids instanceof Tensor) {
+        decoder_input_ids = decoder_input_ids.tolist().flat();
+    } else if (!Array.isArray(decoder_input_ids)) {
         decoder_input_ids = [decoder_input_ids];
     }
 
@@ -399,10 +409,17 @@ function seq2seqStartBeams(self, inputTokenIds, numOutputTokens) {
 async function seq2seqRunBeam(self, beam) {
     const input_name = self.main_input_name;
 
+    let decoder_input_ids = beam.output_token_ids;
+    if (beam.prev_model_outputs) {
+        // After the first step, `prev_model_outputs` won't be null.
+        // So, we cut decoder_input_ids if past is used
+        decoder_input_ids = decoder_input_ids.slice(-1);
+    }
+
     // 1. Prepare
     let model_inputs = {
         [input_name]: beam.inputs,
-        decoder_input_ids: toI64Tensor(beam.output_token_ids.slice(-1)),
+        decoder_input_ids: toI64Tensor(decoder_input_ids),
         encoder_outputs: beam.encoder_outputs,
         past_key_values: beam.prev_model_outputs?.past_key_values,
     }
@@ -474,13 +491,14 @@ async function decoderForward(self, model_inputs) {
 /**
  * Starts the generation of text by initializing the beams for the given input token IDs.
  * @param {Object} self The text generation model object.
- * @param {any} inputTokenIds An array of input token IDs to generate text from.
+ * @param {Tensor} inputTokenIds An tensor of input token IDs to generate text from.
+ * @param {Object} generation_config The generation config.
  * @param {number} numOutputTokens The maximum number of tokens to generate for each beam.
  * @param {Tensor} [inputs_attention_mask] The attention mask tensor for the input token IDs.
  * @returns {Object[]} An array of beams initialized with the given inputs and parameters.
  * @private
  */
-function decoderStartBeams(self, inputTokenIds, numOutputTokens, inputs_attention_mask) {
+function decoderStartBeams(self, inputTokenIds, generation_config, numOutputTokens, inputs_attention_mask) {
     let beams = [];
 
     let beamId = 0;
@@ -975,7 +993,7 @@ export class PreTrainedModel extends Callable {
         let sampler = Sampler.getSampler(generation_config);
 
         // @ts-ignore
-        let beams = this.getStartBeams(inputs, numOutputTokens, inputs_attention_mask);
+        let beams = this.getStartBeams(inputs, generation_config, numOutputTokens, inputs_attention_mask);
 
         while (beams.some(x => !x.done) && numOutputTokens < maxOutputTokens) {
             let newest_beams = [];
@@ -1251,13 +1269,14 @@ export class PreTrainedModel extends Callable {
     /**
      * Initializes and returns the beam for text generation task
      * @param {Tensor} inputTokenIds The input token ids.
+     * @param {Object} generation_config The generation config.
      * @param {number} numOutputTokens The number of tokens to be generated.
      * @param {Tensor} inputs_attention_mask Optional input attention mask.
      * @returns {any} A Beam object representing the initialized beam.
      * @private
      */
-    getStartBeams(inputTokenIds, numOutputTokens, inputs_attention_mask) {
-        return this._getStartBeams(this, inputTokenIds, numOutputTokens, inputs_attention_mask)
+    getStartBeams(inputTokenIds, generation_config, numOutputTokens, inputs_attention_mask) {
+        return this._getStartBeams(this, inputTokenIds, generation_config, numOutputTokens, inputs_attention_mask)
     }
 
     /**
@@ -2024,6 +2043,27 @@ export class MBartForSequenceClassification extends MBartPreTrainedModel {
     }
 }
 
+
+export class MBartForCausalLM extends MBartPreTrainedModel {
+    /**
+     * Creates a new instance of the `MBartForCausalLM` class.
+     * @param {Object} config Configuration object for the model.
+     * @param {Object} decoder_merged_session ONNX Session object for the decoder.
+     * @param {Object} generation_config Configuration object for the generation process.
+     */
+    constructor(config, decoder_merged_session, generation_config) {
+        super(config, decoder_merged_session);
+        this.generation_config = generation_config;
+
+        this.num_decoder_layers = this.config.decoder_layers;
+        this.num_decoder_heads = this.config.decoder_attention_heads;
+        this.decoder_dim_kv = this.config.d_model / this.num_decoder_heads;
+
+        this.num_encoder_layers = this.config.encoder_layers;
+        this.num_encoder_heads = this.config.encoder_attention_heads;
+        this.encoder_dim_kv = this.config.d_model / this.num_encoder_heads;
+    }
+}
 //////////////////////////////////////////////////
 
 
@@ -2528,7 +2568,6 @@ export class WhisperForConditionalGeneration extends WhisperPreTrainedModel {
  */
 export class VisionEncoderDecoderModel extends PreTrainedModel {
     main_input_name = 'pixel_values';
-    add_encoder_pkv = false;
 
     /**
      * Creates a new instance of the `VisionEncoderDecoderModel` class.
@@ -2542,9 +2581,47 @@ export class VisionEncoderDecoderModel extends PreTrainedModel {
         this.decoder_merged_session = decoder_merged_session;
         this.generation_config = generation_config;
 
-        this.num_layers = this.config.decoder.n_layer;
-        this.num_heads = this.config.decoder.n_head;
-        this.dim_kv = this.config.decoder.n_embd / this.num_heads;
+        // Extract configs
+        const encoderConfig = this.config.encoder;
+        const decoderConfig = this.config.decoder;
+
+        // Validate encoder
+        const encoderModelType = encoderConfig.model_type;
+        const encoderModel =
+            MODEL_MAPPING_NAMES_ENCODER_ONLY.get(encoderModelType)
+            ?? MODEL_MAPPING_NAMES_ENCODER_DECODER.get(encoderModelType);
+        if (!encoderModel) {
+            console.warn(`Model type for encoder '${encoderModelType}' not found, assuming encoder-only architecture. Please report this at https://github.com/xenova/transformers.js/issues/new/choose.`);
+        }
+
+        // Validate decoder
+        const decoderModel = MODEL_WITH_LM_HEAD_MAPPING_NAMES.get(decoderConfig.model_type);
+        if (!decoderModel) {
+            throw new Error(`Unable to construct \`VisionEncoderDecoder\` due to unsupported decoder: "${this.config.decoder.model_type}"`);
+        }
+
+        // @ts-ignore
+        const decoderModelClass = decoderModel[1];
+        // @ts-ignore
+        const decoder = new decoderModelClass(decoderConfig, decoder_merged_session, generation_config);
+
+        this.add_encoder_pkv = 'num_decoder_layers' in decoder;
+        if (this.add_encoder_pkv) {
+            // Decoder is part of an encoder-decoder model
+            this.num_decoder_layers = decoder.num_decoder_layers;
+            this.num_decoder_heads = decoder.num_decoder_heads;
+            this.decoder_dim_kv = decoder.decoder_dim_kv;
+
+            this.num_encoder_layers = decoder.num_encoder_layers;
+            this.num_encoder_heads = decoder.num_encoder_heads;
+            this.encoder_dim_kv = decoder.encoder_dim_kv;
+
+        } else {
+            // Decoder is a decoder-only model
+            this.num_layers = decoder.num_layers;
+            this.num_heads = decoder.num_heads;
+            this.dim_kv = decoder.dim_kv;
+        }
     }
 }
 //////////////////////////////////////////////////
@@ -3130,6 +3207,86 @@ export class SwinForImageClassification extends SwinPreTrainedModel {
 }
 //////////////////////////////////////////////////
 
+//////////////////////////////////////////////////
+export class DonutSwinPreTrainedModel extends PreTrainedModel { }
+
+/**
+ * The bare Donut Swin Model transformer outputting raw hidden-states without any specific head on top.
+ * 
+ * **Example:** Step-by-step Document Parsing.
+ * 
+ * ```javascript
+ * import { AutoProcessor, AutoTokenizer, AutoModelForVision2Seq, RawImage } from '@xenova/transformers';
+ * 
+ * // Choose model to use
+ * const model_id = 'Xenova/donut-base-finetuned-cord-v2';
+ * 
+ * // Prepare image inputs
+ * const processor = await AutoProcessor.from_pretrained(model_id);
+ * const url = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/receipt.png';
+ * const image = await RawImage.read(url);
+ * const image_inputs = await processor(image);
+ * 
+ * // Prepare decoder inputs
+ * const tokenizer = await AutoTokenizer.from_pretrained(model_id);
+ * const task_prompt = '<s_cord-v2>';
+ * const decoder_input_ids = tokenizer(task_prompt, {
+ *   add_special_tokens: false,
+ * }).input_ids;
+ * 
+ * // Create the model
+ * const model = await AutoModelForVision2Seq.from_pretrained(model_id);
+ * 
+ * // Run inference
+ * const output = await model.generate(image_inputs.pixel_values, {
+ *   decoder_input_ids,
+ *   max_length: model.config.decoder.max_position_embeddings,
+ * });
+ * 
+ * // Decode output
+ * const decoded = tokenizer.batch_decode(output)[0];
+ * // <s_cord-v2><s_menu><s_nm> CINNAMON SUGAR</s_nm><s_unitprice> 17,000</s_unitprice><s_cnt> 1 x</s_cnt><s_price> 17,000</s_price></s_menu><s_sub_total><s_subtotal_price> 17,000</s_subtotal_price></s_sub_total><s_total><s_total_price> 17,000</s_total_price><s_cashprice> 20,000</s_cashprice><s_changeprice> 3,000</s_changeprice></s_total>

+ * ```
+ * 
+ * **Example:** Step-by-step Document Visual Question Answering (DocVQA)
+ * 
+ * ```javascript
+ * import { AutoProcessor, AutoTokenizer, AutoModelForVision2Seq, RawImage } from '@xenova/transformers';
+ * 
+ * // Choose model to use
+ * const model_id = 'Xenova/donut-base-finetuned-docvqa';
+ * 
+ * // Prepare image inputs
+ * const processor = await AutoProcessor.from_pretrained(model_id);
+ * const url = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/invoice.png';
+ * const image = await RawImage.read(url);
+ * const image_inputs = await processor(image);
+ * 
+ * // Prepare decoder inputs
+ * const tokenizer = await AutoTokenizer.from_pretrained(model_id);
+ * const question = 'What is the invoice number?';
+ * const task_prompt = `<s_docvqa><s_question>${question}</s_question><s_answer>`;
+ * const decoder_input_ids = tokenizer(task_prompt, {
+ *   add_special_tokens: false,
+ * }).input_ids;
+ * 
+ * // Create the model
+ * const model = await AutoModelForVision2Seq.from_pretrained(model_id);
+ * 
+ * // Run inference
+ * const output = await model.generate(image_inputs.pixel_values, {
+ *   decoder_input_ids,
+ *   max_length: model.config.decoder.max_position_embeddings,
+ * });
+ * 
+ * // Decode output
+ * const decoded = tokenizer.batch_decode(output)[0];
+ * // <s_docvqa><s_question> What is the invoice number?</s_question><s_answer> us-001</s_answer>

+ * ```
+ */
+export class DonutSwinModel extends DonutSwinPreTrainedModel { }
+//////////////////////////////////////////////////
+
 //////////////////////////////////////////////////
 export class YolosPreTrainedModel extends PreTrainedModel { }
 export class YolosModel extends YolosPreTrainedModel { }
@@ -3219,15 +3376,6 @@ export class MarianMTModel extends MarianPreTrainedModel {
         this.num_encoder_heads = this.config.encoder_attention_heads;
         this.encoder_dim_kv = this.config.d_model / this.num_encoder_heads;
     }
-
-
-    /**
-     * @param {any} model_inputs
-     * @returns {Promise<Seq2SeqLMOutput>}
-     */
-    async forward(model_inputs) {
-        return await seq2seqForward(this, model_inputs);
-    }
 }
 //////////////////////////////////////////////////
 
@@ -3260,13 +3408,6 @@ export class M2M100ForConditionalGeneration extends M2M100PreTrainedModel {
         this.encoder_dim_kv = this.config.d_model / this.num_encoder_heads;
     }
 
-    /**
-     * @param {any} model_inputs
-     * @returns {Promise<Seq2SeqLMOutput>}
-     */
-    async forward(model_inputs) {
-        return await seq2seqForward(this, model_inputs);
-    }
 }
 //////////////////////////////////////////////////
 
@@ -3479,6 +3620,7 @@ const MODEL_MAPPING_NAMES_ENCODER_ONLY = new Map([
     ['deit', ['DeiTModel', DeiTModel]],
     ['resnet', ['ResNetModel', ResNetModel]],
     ['swin', ['SwinModel', SwinModel]],
+    ['donut-swin', ['DonutSwinModel', DonutSwinModel]],
     ['yolos', ['YolosModel', YolosModel]],
 
     ['sam', ['SamModel', SamModel]], // TODO change to encoder-decoder when model is split correctly
@@ -3562,6 +3704,7 @@ const MODEL_WITH_LM_HEAD_MAPPING_NAMES = new Map([
     ['llama', ['LlamaForCausalLM', LlamaForCausalLM]],
     ['mpt', ['MptForCausalLM', MptForCausalLM]],
     ['opt', ['OPTForCausalLM', OPTForCausalLM]],
+    ['mbart', ['MBartForCausalLM', MBartForCausalLM]],
 ]);
 
 const MODEL_FOR_MASKED_LM_MAPPING_NAMES = new Map([
@@ -3598,6 +3741,10 @@ const MODEL_FOR_VISION_2_SEQ_MAPPING_NAMES = new Map([
     ['vision-encoder-decoder', ['VisionEncoderDecoderModel', VisionEncoderDecoderModel]],
 ]);
 
+const MODEL_FOR_DOCUMENT_QUESTION_ANSWERING_MAPPING_NAMES = new Map([
+    ['vision-encoder-decoder', ['VisionEncoderDecoderModel', VisionEncoderDecoderModel]],
+]);
+
 const MODEL_FOR_IMAGE_CLASSIFICATION_MAPPING_NAMES = new Map([
     ['vit', ['ViTForImageClassification', ViTForImageClassification]],
     ['mobilevit', ['MobileViTForImageClassification', MobileViTForImageClassification]],
@@ -3811,6 +3958,9 @@ export class AutoModelForAudioClassification extends PretrainedMixin {
     static MODEL_CLASS_MAPPINGS = [MODEL_FOR_AUDIO_CLASSIFICATION_MAPPING_NAMES];
 }
 
+export class AutoModelForDocumentQuestionAnswering extends PretrainedMixin {
+    static MODEL_CLASS_MAPPINGS = [MODEL_FOR_DOCUMENT_QUESTION_ANSWERING_MAPPING_NAMES];
+}
 
 //////////////////////////////////////////////////
 
diff --git a/src/pipelines.js b/src/pipelines.js
index e2be02fcf..2c076e856 100644
--- a/src/pipelines.js
+++ b/src/pipelines.js
@@ -31,6 +31,7 @@ import {
     AutoModelForImageClassification,
     AutoModelForImageSegmentation,
     AutoModelForObjectDetection,
+    AutoModelForDocumentQuestionAnswering,
     PreTrainedModel,
 } from './models.js';
 import {
@@ -1761,6 +1762,79 @@ export class ObjectDetectionPipeline extends Pipeline {
     }
 }
 
+/**
+ * Document Question Answering pipeline using any `AutoModelForDocumentQuestionAnswering`.
+ * The inputs/outputs are similar to the (extractive) question answering pipeline; however,
+ * the pipeline takes an image (and optional OCR'd words/boxes) as input instead of text context.
+ * 
+ * **Example:** Answer questions about a document with `Xenova/donut-base-finetuned-docvqa`.
+ * ```javascript
+ * let image = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/invoice.png';
+ * let question = 'What is the invoice number?';
+ * 
+ * let qa_pipeline = await pipeline('document-question-answering', 'Xenova/donut-base-finetuned-docvqa');
+ * let output = await qa_pipeline(image, question);
+ * // [{ answer: 'us-001' }]
+ * ```
+ */
+export class DocumentQuestionAnsweringPipeline extends Pipeline {
+    /**
+     * Create a new DocumentQuestionAnsweringPipeline.
+     * @param {Object} options An object containing the following properties:
+     * @param {string} [options.task] The task of the pipeline. Useful for specifying subtasks.
+     * @param {PreTrainedModel} [options.model] The model to use.
+     * @param {PreTrainedTokenizer} [options.tokenizer] The tokenizer to use.
+     * @param {Processor} [options.processor] The processor to use.
+     */
+    constructor(options) {
+        super(options);
+    }
+
+    /**
+     * Answer the question given as input by using the document.
+     * @param {any} image The image of the document to use.
+     * @param {string} question A question to ask of the document.
+     * @param {Object} [generate_kwargs={}] Optional generation arguments.
+     * @returns {Promise<Object|Object[]>} A Promise that resolves to an object (or array of objects) containing the generated text(s).
+     */
+    async _call(image, question, generate_kwargs = {}) {
+        // NOTE: For now, we only support a batch size of 1
+
+        // Preprocess image
+        image = (await prepareImages(image))[0];
+        const { pixel_values } = await this.processor(image);
+
+        // Run tokenization
+        const task_prompt = `<s_docvqa><s_question>${question}</s_question><s_answer>`;
+        const decoder_input_ids = this.tokenizer(task_prompt, {
+            add_special_tokens: false,
+            padding: true,
+            truncation: true
+        }).input_ids;
+
+        // Run model
+        const output = await this.model.generate(
+            pixel_values,
+            {
+                ...generate_kwargs,
+                decoder_input_ids,
+                max_length: this.model.config.decoder.max_position_embeddings,
+            }
+        );
+
+        // Decode output
+        const decoded = this.tokenizer.batch_decode(output)[0];
+
+        // Parse answer
+        const match = decoded.match(/<s_answer>(.*?)<\/s_answer>/);
+        let answer = null;
+        if (match && match.length >= 2) {
+            answer = match[1].trim();
+        }
+        return [{ answer }];
+    }
+}
+
 const SUPPORTED_TASKS = {
     "text-classification": {
         "tokenizer": AutoTokenizer,
@@ -1951,6 +2025,18 @@ const SUPPORTED_TASKS = {
         },
         "type": "multimodal",
     },
+    "document-question-answering": {
+        "tokenizer": AutoTokenizer,
+        "pipeline": DocumentQuestionAnsweringPipeline,
+        "model": AutoModelForDocumentQuestionAnswering,
+        "processor": AutoProcessor,
+        "default": {
+            // TODO: replace with original
+            // "model": "naver-clova-ix/donut-base-finetuned-docvqa",
+            "model": "Xenova/donut-base-finetuned-docvqa",
+        },
+        "type": "multimodal",
+    },
 
     // This task serves as a useful interface for dealing with sentence-transformers (https://huggingface.co/sentence-transformers).
     "feature-extraction": {
@@ -1987,6 +2073,7 @@ const TASK_ALIASES = {
  * @param {string} task The task defining which pipeline will be returned. Currently accepted tasks are:
  *  - `"audio-classification"`: will return a `AudioClassificationPipeline`.
  *  - `"automatic-speech-recognition"`: will return a `AutomaticSpeechRecognitionPipeline`.
+ *  - `"document-question-answering"`: will return a `DocumentQuestionAnsweringPipeline`.
  *  - `"feature-extraction"`: will return a `FeatureExtractionPipeline`.
  *  - `"fill-mask"`: will return a `FillMaskPipeline`.
  *  - `"image-classification"`: will return a `ImageClassificationPipeline`.
diff --git a/src/processors.js b/src/processors.js
index 593792713..4a7e11226 100644
--- a/src/processors.js
+++ b/src/processors.js
@@ -31,7 +31,7 @@ import {
 import {
     max,
     softmax,
-    FFT
+    FFT,
 } from './utils/maths.js';
 
 
@@ -169,6 +169,7 @@ export class ImageFeatureExtractor extends FeatureExtractor {
         this.do_normalize = this.config.do_normalize;
 
         this.do_resize = this.config.do_resize;
+        this.do_thumbnail = this.config.do_thumbnail;
         this.size = this.config.size;
 
         this.do_center_crop = this.config.do_center_crop;
@@ -176,7 +177,43 @@ export class ImageFeatureExtractor extends FeatureExtractor {
         this.do_convert_rgb = this.config.do_convert_rgb ?? true;
 
         this.pad_size = this.config.pad_size;
-        this.do_pad = (this.config.do_pad ?? false) && this.pad_size;
+        this.do_pad = this.config.do_pad;
+
+        if (this.do_pad && !this.pad_size && this.size.width !== undefined && this.size.height !== undefined) {
+            // Should pad, but no pad size specified
+            // We infer the pad size from the resize size
+            this.pad_size = this.size
+        }
+    }
+
+    /**
+     * Resize the image to make a thumbnail. The image is resized so that no dimension is larger than any
+     * corresponding dimension of the specified size.
+     * @param {RawImage} image The image to be resized.
+     * @param {{height:number, width:number}} size The size `{"height": h, "width": w}` to resize the image to.
+     * @param {string | 0 | 1 | 2 | 3 | 4 | 5} [resample=2] The resampling filter to use.
+     * @returns {Promise<RawImage>} The resized image.
+     */
+    async thumbnail(image, size, resample = 2) {
+        const input_height = image.height;
+        const input_width = image.width;
+
+        const output_height = size.height;
+        const output_width = size.width;
+
+        // We always resize to the smallest of either the input or output size.
+        let height = Math.min(input_height, output_height)
+        let width = Math.min(input_width, output_width)
+
+        if (height === input_height && width === input_width) {
+            return image;
+        }
+        if (input_height > input_width) {
+            width = Math.floor(input_width * height / input_height);
+        } else if (input_width > input_height) {
+            height = Math.floor(input_height * width / input_width);
+        }
+        return await image.resize(width, height, { resample });
     }
 
     /**
@@ -206,8 +243,13 @@ export class ImageFeatureExtractor extends FeatureExtractor {
             let shortest_edge;
             let longest_edge;
 
+            if (this.do_thumbnail) {
+                // NOTE: custom logic for `Donut` models
+                const { height, width } = this.size;
+                shortest_edge = Math.min(height, width)
+            }
             // Support both formats for backwards compatibility
-            if (Number.isInteger(this.size)) {
+            else if (Number.isInteger(this.size)) {
                 shortest_edge = this.size;
                 longest_edge = this.config.max_size ?? shortest_edge;
 
@@ -235,9 +277,9 @@ export class ImageFeatureExtractor extends FeatureExtractor {
                     ? 1 // If `longest_edge` is not set, don't downscale
                     : Math.min(longest_edge / newWidth, longest_edge / newHeight);
 
-                // To avoid certain floating point precision issues, we round to 3 decimal places
-                const finalWidth = Math.floor(Number((newWidth * longResizeFactor).toPrecision(3)));
-                const finalHeight = Math.floor(Number((newHeight * longResizeFactor).toPrecision(3)));
+                // To avoid certain floating point precision issues, we round to 2 decimal places
+                const finalWidth = Math.floor(Number((newWidth * longResizeFactor).toFixed(2)));
+                const finalHeight = Math.floor(Number((newHeight * longResizeFactor).toFixed(2)));
 
                 // Perform resize
                 image = await image.resize(finalWidth, finalHeight, {
@@ -254,6 +296,11 @@ export class ImageFeatureExtractor extends FeatureExtractor {
             }
         }
 
+        // Resize the image using thumbnail method.
+        if (this.do_thumbnail) {
+            image = await this.thumbnail(image, this.size, this.resample);
+        }
+
         if (this.do_center_crop) {
 
             let crop_width;
@@ -272,7 +319,7 @@ export class ImageFeatureExtractor extends FeatureExtractor {
         let reshaped_input_size = [image.height, image.width];
 
         // TODO is it okay to pad before rescaling/normalizing?
-        if (this.do_pad) {
+        if (this.do_pad && this.pad_size) {
             let left = 0;
             let right = this.pad_size.width - image.width;
             let top = 0;
@@ -362,6 +409,7 @@ export class ViTFeatureExtractor extends ImageFeatureExtractor { }
 export class MobileViTFeatureExtractor extends ImageFeatureExtractor { }
 export class DeiTFeatureExtractor extends ImageFeatureExtractor { }
 export class BeitFeatureExtractor extends ImageFeatureExtractor { }
+export class DonutFeatureExtractor extends ImageFeatureExtractor { }
 
 /**
  * Detr Feature Extractor.
@@ -1329,6 +1377,7 @@ export class AutoProcessor {
         DeiTFeatureExtractor,
         DetrFeatureExtractor,
         YolosFeatureExtractor,
+        DonutFeatureExtractor,
 
         SamImageProcessor,
         Wav2Vec2FeatureExtractor,
@@ -1379,10 +1428,10 @@ export class AutoProcessor {
         if (!feature_extractor_class) {
             if (preprocessorConfig.size !== undefined) {
                 // Assume ImageFeatureExtractor
-                console.warn('Feature extractor type not specified, assuming ImageFeatureExtractor due to size parameter in config.');
+                console.warn(`Feature extractor type "${key}" not found, assuming ImageFeatureExtractor due to size parameter in config.`);
                 feature_extractor_class = ImageFeatureExtractor;
             } else {
-                throw new Error(`Unknown Feature Extractor type: ${preprocessorConfig.feature_extractor_type}`);
+                throw new Error(`Unknown Feature Extractor type: ${key}`);
             }
         }
 
diff --git a/src/tokenizers.js b/src/tokenizers.js
index 9a0d295eb..879914e39 100644
--- a/src/tokenizers.js
+++ b/src/tokenizers.js
@@ -2269,6 +2269,7 @@ export class PreTrainedTokenizer extends Callable {
      * @param {Object} options An optional object containing the following properties:
      * @param {string|string[]} [options.text_pair=null] Optional second sequence to be encoded. If set, must be the same type as text.
      * @param {boolean} [options.padding=false] Whether to pad the input sequences.
+     * @param {boolean} [options.add_special_tokens=true] Whether or not to add the special tokens associated with the corresponding model.
      * @param {boolean} [options.truncation=null] Whether to truncate the input sequences.
      * @param {number} [options.max_length=null] Maximum length of the returned list and optionally padding length.
      * @param {boolean} [options.return_tensor=true] Whether to return the results as Tensors or arrays.
@@ -2281,7 +2282,7 @@ export class PreTrainedTokenizer extends Callable {
         // Optional keyword arguments
         {
             text_pair = null,
-            // add_special_tokens = true, // TODO
+            add_special_tokens = true,
             padding = false,
             truncation = null,
             max_length = null,
@@ -2306,11 +2307,11 @@ export class PreTrainedTokenizer extends Callable {
                 }
 
                 tokens = text.map(
-                    (t, i) => this.encode(t, text_pair[i])
+                    (t, i) => this.encode(t, text_pair[i], { add_special_tokens })
                 )
 
             } else {
-                tokens = text.map(x => this.encode(x));
+                tokens = text.map(x => this.encode(x, null, { add_special_tokens }));
             }
 
         } else {
@@ -2323,7 +2324,7 @@ export class PreTrainedTokenizer extends Callable {
             }
 
             // For single input, we just wrap in an array, and then unwrap later.
-            tokens = [this.encode(text, text_pair)];
+            tokens = [this.encode(text, text_pair, { add_special_tokens })];
         }
         // At this point, tokens is batched: [batch_size, tokens]
         // However, array may be jagged. So, we pad to max_length
@@ -2474,14 +2475,19 @@ export class PreTrainedTokenizer extends Callable {
      *
      * @param {string} text The text to encode.
      * @param {string|null} text_pair The optional second text to encode.
+     * @param {Object} options An optional object containing the following properties:
+     * @param {boolean} [options.add_special_tokens=true] Whether or not to add the special tokens associated with the corresponding model.
      * @returns {number[]} An array of token IDs representing the encoded text(s).
      */
-    encode(text, text_pair = null) {
+    encode(text, text_pair = null, {
+        add_special_tokens = true,
+    } = {}) {
         // Function called by users to encode possibly multiple texts
         let tokens = this._encode_text(text);
         let tokens2 = this._encode_text(text_pair);
 
-        let combinedTokens = (this.post_processor !== null)
+        // TODO improve `add_special_tokens` and ensure correctness
+        let combinedTokens = (this.post_processor !== null && add_special_tokens)
             ? this.post_processor(tokens, tokens2)
             : mergeArrays(tokens ?? [], tokens2 ?? []);
 


Test patch:
diff --git a/tests/pipelines.test.js b/tests/pipelines.test.js
index efab2d98d..26a905779 100644
--- a/tests/pipelines.test.js
+++ b/tests/pipelines.test.js
@@ -1325,4 +1325,29 @@ describe('Pipelines', () => {
             await detector.dispose();
         }, MAX_TEST_EXECUTION_TIME);
     });
+
+    describe('Document question answering', () => {
+
+        // List all models which will be tested
+        const models = [
+            'naver-clova-ix/donut-base-finetuned-docvqa',
+        ];
+        const image = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/invoice.png';
+        const question = 'What is the invoice number?';
+
+        it(models[0], async () => {
+            let qa_pipeline = await pipeline('document-question-answering', m(models[0]));
+
+            // basic
+            {
+                let output = await qa_pipeline(image, question);
+                let expected = [{ answer: 'us-001' }];
+                compare(output, expected);
+            }
+
+            await qa_pipeline.dispose();
+
+        }, MAX_TEST_EXECUTION_TIME);
+    });
+
 });