Mapaor/pix2text-mfr-1.5-web

Pix2Text MFR-1.5 but with additional features

This model is meant to be used with pix2text-mfr-web.

The end goal is to support client-side math formula recognition. The need to fork the model arised from KV cache not being set up for the original breezedeus/pix2text-mfr.

Release target

The first Hugging Face release will be:

• Repository: Mapaor/pix2text-mfr-1.5-web

• Initial variant: FP32 with real decoder KV caching

• Layout: one repository with versioned variant directories:

  fp32/
  fp16/  # future release
  int8/  # future release
  

• Baseline: the original FP32 model is preserved for output comparison

• WebGPU/WASM: an API/runtime execution-provider choice, not a model variant

The cache-enabled model must be exported from a PyTorch checkpoint. (Setting use_cache: true in configuration alone is not sufficient: the exported decoder graph must expose past-key-value inputs and present-key-value outputs).

Implementation plan

1. Obtain and record the source checkpoint

• Obtain the original/upstream PyTorch MFR-1.5 checkpoint.
• Record its source revision, configuration, tokenizer files and license.
• Preserve the existing FP32 ONNX model as the baseline artifact.

Gate: the source checkpoint and export environment can be reproduced without changing the baseline model.

2. Export the cache-enabled FP32 ONNX model

• Export separate encoder and decoder graphs for browser inference.
• Add decoder past_key_values inputs and present_key_values outputs for every decoder layer.
• Keep the initial decoder pass compatible with an empty cache and subsequent passes compatible with one-token decoding.
• Store the exported model and configuration under fp32/.

Gate: graph inspection confirms the expected cache inputs and outputs, dynamic sequence dimensions, FP32 tensors, and matching tokenizer/configuration metadata.

3. Implement cache-aware decoding

Update pix2text-mfr-web to:

• Run the initial decoder pass with the encoder states and no prior cache.
• Feed only the newly generated token on each subsequent pass.
• Carry each layer's present_key_values into the next pass as past_key_values.
• Retain the current full-sequence decoder path as the baseline comparison path until parity is established.
• Support model URLs rooted at the variant directory without assuming that the model is stored at repository root.

4. Validate output parity

Compare the cache-aware and baseline decoders on a fixed corpus of representative formula images.

Required checks:

• identical generated token IDs for greedy decoding;
• identical decoded LaTeX text;
• correct EOS and maxNewTokens handling;
• correct behavior for empty, short and long outputs;
• clear errors for missing or incompatible cache tensors.

Any intentional numerical difference must be documented with an explicit tolerance and reviewed before publication.

5. Benchmark browser execution

Benchmark both wasm and webgpu execution providers using the same browser, model revision and input corpus. Record:

• model download and initialization time;
• first-token latency;
• total recognition latency;
• peak memory where the browser exposes it;
• fallback behavior when WebGPU is unavailable.

The benchmark must demonstrate that cache-aware decoding improves or does not regress the supported browser experience before release.

6. Publish and document the model

• Publish the validated artifacts to HuggingFace Mapaor/pix2text-mfr-1.5-web. (Read how to publish a repo to HF)
• Keep the original FP32 baseline identifiable and separate from fp32/.
• Document model specifications, expected inputs/outputs, tokenizer details, licensing, revision pinning and browser requirements.
• Add FP16 and INT8 only after the FP32 cache-enabled release is validated; they must follow the same directory and metadata conventions.

7. Update the npm package and demo

• Point the package and demo at the published fp32/ directory.
• Verify model loading, browser cache behavior, WebGPU selection and WASM fallback against the published revision.
• Update package and API documentation with the new model URL and layout.

Definition of done

The initial release is complete when:

1. The Hugging Face fp32/ decoder exposes working past/present KV-cache tensors.
2. Cache-aware decoding matches the preserved baseline token-for-token on the validation corpus.
3. WASM and WebGPU benchmarks and fallback results are recorded.
4. The package and demo load the published revision from the new layout.
5. Model specifications, reproducibility details and usage instructions are documented.