InkSight: Offline-to-Online Handwriting Conversion by Teaching Vision-Language Models to Read and Write
Authors: Blagoj Mitrevski
Arina Rak
Julian Schnitzler
Chengkun Li
Andrii Maksai
Jesse Berent
Claudiu Cristian Musat
Venue: Accepted by TMLR ()

Abstract: Digital note-taking is gaining popularity, offering a durable, editable, and easily indexable way of storing notes in a vectorized form, known as digital ink. However, a substantial gap remains between this way of note-taking and traditional pen-and-paper note-taking, a practice that is still favored by a vast majority. Our work InkSight, aims to bridge the gap by empowering physical note-takers to effortlessly convert their work (offline handwriting) to digital ink (online handwriting), a process we refer to as derendering. Prior research on the topic has focused on the geometric properties of images, resulting in limited generalization beyond their training domains. Our approach combines reading and writing priors, allowing training a model in the absence of large amounts of paired samples, which are difficult to obtain. To our knowledge, this is the first work that effectively derenders handwritten text in arbitrary photos with diverse visual characteristics and backgrounds. Furthermore, it generalizes beyond its training domain into simple sketches. Our human evaluation reveals that 87% of the samples produced by our model on the challenging HierText dataset are considered as a valid tracing of the input image and 67% look like a pen trajectory traced by a human.

Published in Transactions on Machine Learning Research (06/2025)
InkSight: Offline-to-Online Handwriting Conversion by Teaching Vision-Language Models to Read and Write
Blagoj Mitrevski 9 Arina Rak < Julian Schnitzler
O Chengkun Li + Andrii Maksai
Jesse Berent, Claudiu Musat
Google DeepMind, EEFL, work done as student researcher.
@ Finst authors: random order decided AEA tool, Projeec lead: amakeai@ooogee.coo
Reviewed OpenReview: https:// https://openrrrieew httt:/ooruur::ddssSSEEEEaa
Saon. the na babbit noticed Alice, as she stood looking cariously but her, and at once Rid in quick ingry tone, why, Marry Ann what are. you doing out here ! Go home this momeat, and look
Door the na rablit bbit noturll Alee as sha stall looking curiously about ham, and at once Soul in guick Regry Tone, why,
Blary Ann what aoy you deing wat
bern ! Go home this moment aadh look
A photo of handwritten text (offline handwriting), Right: Output digital ink (online handwriting). In every word, character colors transition from red to purple, following the rainbow sequence, ROYGBIV, which reflects the stroke order; within each stroke, the shade progresses from darker to lighter. More visualizations in Appendix B.
Figure 1: Results of InkSight [Animated visualization). Left:
Abstract
Digital note-taking is gaining popularity, offering a durable, editable, and easily indexable way of storing notes a vectorized form, known as digital ink. However, substantial gap remains between this way of note-taking and traditional pen-and-paper note-taking, a practice that is still favored by vast majority. Our work InkSight?, aims to bridge the gap by empowering physical note-takers to effortlessly convert their work (offline handwriting) to digital ink (online handwriting), process we refer to as derendering. Prior rescarch on the topic has focused on the geometric properties of images, resulting in limited generalization beyond their training domains. Our approach leverages vision-language models to combine reading and wriling priors, allowing training a model the absence large amounts of paired samples, which are difficult to obtain. To our knowledge, this is the first work that effectively derenders handwritten text in arbitrary photos with diverse visual characteristics and backgrounds. Furthermore, it generalizes beyond its training domain into simple sketches. Our human evaluation reveals that 87% of the samples produced by our model on the challenging HierText dataset are considered as a valid tracing of the input image and 67% look like pen trajectory traced by a human.
1 Introduction
Handwritten notes have been cornerstone of information storage for centuries. Today, with the rise of stylus and digital pen technologies, digital inking is a compelling alternative. This modern approach to
GitHub: http:///gthhb..ommggggeeeeee
Hugging Face: Tink Project page: [link]

---

Published in Transactions on Machine Learning Research (06/2025)
storing information offers several advantages: Digital ink offers enhanced durability, structured organization, and fine-grained editability. Unlike pen-and-paper notes, it supports post hoc modifications users can move, resize, or restyle individual strokes or sections. This enables dynamic reuse, seamless integration with digital workflows, and compatibility with assistive systems. Despite these apparent bencfits, many people still prefer traditional handwrilten notes over the digital format.
Traditional OCR extracts textual content from handwriting but discards the rich structural and temporal information embedded in the handwritten form. In contrast, our approach captures handwriting as digital ink, preserving stroke-level trajectories. This enables users to retain the familiarity of pen-and-paper note-taking while gaining the flexibility of digital representations without requiring stylus.
The topic of converting offline handwriting to online handwriling has gained significant interest in both academia (Nguyen et al., 2021; Chen et al., 2022b; Mohamed Moussa et al., 2023) and industry (Not, 2023), with the introduction of software solutions that digitize handwriting and hardware solutions that require smart pens and/or or special gadgets (Nuw, 2025; Liv, 2024; Roc, 2024; Neo, 2024). Our approach requires only image of the handwritten note, without any specialized equipment. It differs from the methods that rely on geometric priors, where gradients, contours, and shapes in image are utilized to extract writing strokes. Instead, we harness the power of learned reading and writing priors, where:
The learned reading prior enhances the model's capability to precisely locate and extract textual elements from images. This is achieved either through the model's textual understanding ability or aided with external textual input, e.g. from an OCR engine.
The integration of the writing prior ensures that the resulting vector representation, the digital ink, closely aligus with the typical human approach to writing in terms of physical dynamics and the order of strokes.
To the best of our knowledge, our work is the first to incorporate these priors, resulting in robust model that is capable of derendering handwriting across diverse scenarios and appearances, including challenging lighting conditions, notale occlusions, etc. Our model adopts a simple architecture combining the widely popular and readily available VIT (Dosovitskiy et al., 2021) encoder and an m'15 (Xue et al., 2021) encoder-decoder, fostering reproducibility, reusability, and case of adoption. To summarize, the major contributions of our work are:
We propose the first system to perform derendering: transforming arbitrary photos of handwritten text into digital ink. It relies only on the standard architecture components and the training and inference setup that works without expensive data collections and scales to arbitrarily large input images.
2. We show that the inks produced by our system are both semantically and geometrically similar to the input images. We demonstrate that they are similar to real digital ink data, as measured by both automatic and human evaluations.
3. We show that, with the learned reading and writing priors, our approach is robust and works on various types of handwriting, simple sketches, and full pages of notes.
4. We release the public model Small-p, trained entirely on publicly available datasets, along with a subset of model-generated inks in universal inkML format and expert-traced online handwriting data to support future research on the topic.
2 Related Work
Pen trajectory recovery has been a task of interest to many researchers due to its utility for tasks that can benefit from stroke order / temporal information such online handwriting recognition (Lallican et al., 2004; Viard-Gaudin et al., 2005; Zhang et al., 2015; Chan, 2020). Another point of interest is the ability to efficiently store, index, and edit. handwrillen notes within existing online note-taking systems. Classical approaches commonly consist of domain-specific preprocessing (e.g. noise removal, image binarization, skeletonization), local (sub-)stroke level processing (c.g. identification of junction points) and global aggregation (commonly as a graph- based optimization problem) (Jager, 1996; Kato & Yasuhara, 2000; Qiao al., 2006; Chan, 2020; Doermann et al., 2002). These approaches often rely on the quality of the preprocessing and hand-
2

---

Published in Transactions on Machine Learning Research (06/2025)
designed heuristics and do not generalize well to other scripts and domains. More recent methods use convolutional (Nguyen et al., 2021; Archibald et al., 2021) and/or recurrent (Bhunia et al., 2018; Chen et al., 2022b) neural networks to translate an image into sequence of coordinates. Set, sort! (Mohamed Moussa et al., 2023) use two Transformer models to first encode the substrokes of the input and then reorder them, given the encodings. Those methods produce promising results, but focus on the simplified setups of rendered online handwriting and/or single characters.
Line drawing vectorization shares many features with geometric approaches to pen trajectory recovery, but does not include stroke order reconstruction. Recent techniques for this problem involve solving an optimization task of fitting set of geometric primitives (e.g. Bezier curves) to match the geometric and/or semantic content of the input image (Vinker et al., 2022), sometimes relying on differentiable rendering to maintain the imputs and outputs in the image space (Mo et al., 2021). These approaches focus on converting raster images (perfectly aligned, clean, clear backgrounds) to vector ones. Thus they avoid dealing with the realistic photo artifacts relatod to lighting, noisy backgrounds or photo-taking angles. These artifacts are however unavoidable in weal-world setting.
Dataset availability is limiting the research in trajectory recovery there are few datasets of images and their corresponding digital inks. These include IRONOFF (Viard-Gaudin et al., 1999), CASIA (Liu et al., 2011), IBM-UB (Shivram et al., 2013) for handwritten text (images there exhibit limited variability being always black-on-white line writing, usually on the same input device) and Sketchy (Sangkloy et al., 2016) for crowd-sourced sketches of photos from 125 object categories.
Combining language models with new modalities is related to this work since convert the pen trajectory to sequence of coordinates on fixed size canvas and represent the coordinates with ink tokens (more details in Section 3.3). Hence, we consider our approach exlending Visual Language Model (VLM) with a new modality. Recent research in this area has been pointed towards training adapters for merging the pretrained unimodal models (e.g. Alayrac et al. (2022)) or, fully or partially fine-tuning a system with multi-modal tasks (e.g. Chen et al. (2022x)). Works most similar to ours are AudioPaLM (Rubenstein et al., 2023), extending LLM with audio tokens, Painter (Pourreza et al., 2023) using coordinate-based representation of strokes to perform generation and understanding of sketches, and PixeILLM (Xu al., 2024) using the same representation to localize words in the image caption.
3 Method
While the fundamental concept of derendering appears straightforward - training a model that generates digital ink representations from input images the practical implementation for arbitrary imput images presents two significant challenges:
Limited Supervised Data: Acquiring paired data with corresponding images and ground truth digital ink for supervised training can be expensive and time-consuming and no datasets with sufficient. diversity exist for this task yet.
2. Scalability to Large Images: The model must effectively handle potentially arbitrary large input images with varying resolutions and amounts of content.
To address the first problem without expensive data collection, we propose multi-task training setup that combines recognition and derendering tasks. We show that this setup enables the model to generalize on derendering tasks with various styles of images as input, and injects the model with both semantic understanding and writing priors of handwritten text. We discuss our data and multi-task training sctup in Section 3.2, and ablate the design choices in Section 4.6. The exact model structure, which consists of standard publicly available off- the-shelf components, is discussed in Section 3.1. In Section 3.3, cover the representation of digital ink, the normalization of the data, and the tokenization scheme we employ.
The second problem can be addressed by training a model with very high-resolution input images and very long output sequences, but this is computationally prohibitive. Instead, we break down the derendering of a page of notes into three steps: running OCR to extract word-level bounding boxes, derendering each of the words separately and pasting the derendered words back, as shown in Figure 2. An example of full page inference is shown in Figure 1, more full page results are presented in Appendix N.
3

---

Published in Transactions on Machine Learning Research (06/2025)
InkSight
started
ViT
MAR JPM
Started
mT5 Encoder
mT5
inkSight
Deronder the ink.
Derender the inio started Recognize end derender.
Output Outgut Tect
Fullaage Inguut
Word evel processing
Fullaagn Outgutt
Inforencc Taska
Figure 2: Left [Animated visualization): Illustration model inference. Inputs include an image and a prompt specifying the task. Outputs may consist of ink alone or combination of ink and text, as indicated by matching colors in both the imput prompt and the output. Ink tokens detokenized into digital ink using proposed ink tokenizer explained in Section 3.3. Right [Animated visualization): Diagram of the Full Page System.
3.1 Vision-Language Model for Digital Ink
Our model architecture, inspired by that of PaLl (Chen et al., 2022a; 2023b;a), integrates a Vision Transformer (ViT) encoder (Dosovitskiy et al., 2021) with an mT5 encoder-decoder Transformer model (Xue et al., 2021). We provide the lask-specific instructions as lext, as described in Section 3.2. We use the same set. of tokens for both imput and output, which contains the individual character tokens from the original mT5 vocabulary and specific tokens reserved for the ink representation (details in Sec. 3.3). To initialize the model, use the pre-trained weights of the ViT encoder. The mT55bbaedd encoder-decoder weights are initialized randomly, which is motivated by our customized token dictionary differing from the one used in mT5's original training. In our training, we freeze the weights of the ViT encoder we justify this choice empirically in Section 4.6.
Demunder with Text
Veniilla Denender
Recognize and Derender
Resounition LSymthetic]
Recegnition DRaal
Input
the
Input
Derender
Pluls and
the donencer
Input Text
Imput
writton?
PIZZA
Input What
Ourpput
Output
Output dnvassn
Dutput PIZZA.
Outppt
vil, Jaben con col-unn Uliie
Labe der Jahn Crta
wa cas vas sleo
coian
Training
Figure Illustration of multi-task training mixture. The training mixture comprises five different task types: two derendering tasks (ink output), two recognition tasks (text output), and one mixed task (text-and-ink output). Each type of task utilizes task-specific input text, enabling the model to distinguish between tasks during both training and inference.
3.2 Training Task Mixture
To circumvent the challenge of not having diverse paired image and ink samples as training data, we propose a multi-task training setup comprising two derendering tasks, two recognition tasks, and one hybrid task (shown in Figure 3 and described in Table 1).
We show that this setup helps the model to: (1) generalize to the derendering of real photos; (2) learn useful priors that help deal with occlusions and generate realistic ink; (3) allow using different inference setups that enable, c.g. high-quality text derendering with OCR result (Derender with Text) or derendering without textual input for non-textual elements (Vanilla Derender). Training tasks are shuffled and assigned equal appearance probability (20%). However, future work could explore adjusting this ratio dynamically during training and towards the end of training. For best performance during inference, we use Derender with Text (this choice ablated Section 4.6 where show that our model performs well without reliance on OCR well).
4

---

Published in Transactions on Machine Learning Research (06/2025)
Table Summary of tasks in our proposed multi-task training.
Task
Description
Vanilla Derender
The model receives synthetic ink image with conditioning task prompt and outputs ink tokens corresponding to the image content.
Derender with Text
The model receives synthetic ink image and task conditioning prompt including the larget textual content, and outputs ink tokens corresponding the provided text within the image.
Recoguition (Syn/Real)
The model receives a handwriting image (either syuthetic or real) and task conditioning prompt and outputs recognized textual content. This task uses both synthetic ink images and real images from OCR dataseta.
Recognize and Derender
This task combines recognition and derendering. The model receives a synthetic ink image and a prompt and outputs both the recognized text and the corresponding ink tokens.
Data augmentation. To narrow the domain gap between the synthetic images of rendered inks and the real photos, we augment the data in tasks that take rendered ink as input. This done by randomizing the ink angle, color, stroke width, and by adding Gaussian noise and cluttered backgrounds. Examples are provided in Figure 3, more details are given in Appendix F, and Section 4.6 highlights the importance of having the data augmentation.
3.3 Data representation
Digital Ink tokenization. Digital ink is usually represented a sequence of strokes 7= {81, 82, 8n), where each stroke 8; consists of sequence of m: (length of the i-th stroke) coordinate-time triplets, represented as 8; = {(x,, Mi, (m) To enable sequence models like mT5 to generate ink strokes, we first normalize the ink and then tokenize into a set of discrete tokens from a pre-defined dictionary. Normalizing the ink includes:
Resampling: Resample the ink at fixed frequency (20 ms) to account for potential variations
between input devices.
Sequence Reduction: Apply Ramer- Dowg-oooooeeeeeeeee resampling (Visvalingam & Whyatt, 1990) to reduce the sequence length while preserving the overall shape of the strokes.
3. Normalization: Shift and scale the ink strokes to position them at the center of fixed-size canvas. Each point in stroke is scaled to fit within the range [0, N], where we use N = 222 in our implementation.
Ink Tokens
Beginning of Stroke Taken
k-th Sampled Point
Text Label
Fixed Size Canvas
Figure 4: Ink tokenization for single-stroke digital ink [Animated visualization]. The dark red ink depicts the normalized ink stroke, with numbered circles marking sampled points after time resampling. The color gradient of the sampled points indicates the point order. Each point is represented with two tokens encoding its and y coordinates. The token sequence for this ink bogins with b, signifying the beginning of the stroke, followed by the Lokens for coordinates of sampled points.
As shown in Figure& 4, tokenization of the normalizod ink involves prepending a beginning of stroke token followed by the and y coordinates of each point inside the stroke, rounded to the nearest integer value.
5

---

Published in Transactions on Machine Learning Research (06/2025)
This is done for every stroke inside the digital ink. The total dictionary size is 2N 3 with separate set of tokens for z and y, including N + 1 possible x and values and a token indicating start of the stroke. N controls the tradeoff between rounding error and vocabulary size, and we use N 224 in practice.
Image representation. For the derendering tasks of the training mixture, we render digital inks in the center of an M M image with stroke width, color, and background color selected via random augmentation (see Appendix F). For the recognition task training and inference with real images, apply similar transformation: the input image is scaled, centered, and padded in black to produce an M x M image. Although we use M = 224, by design there are no constraints to have N = M.
Expanding text vocabulary with Ink. Our set of tokens contains all individual symbols from the multilingual mT5 tokenizer (Xue et al., 2021) (approx. 20k), with additional 2N + 3 tokens reserved for the representation of the digital ink, described above. Removing all multi-symbol tokens allows to reduce the size of the input text embedding and final softmax of the decoder model by ~ 80%, while maintaining the ability to input and output arbitrary text and not affecting the quality of derendering or recognition.
4
Experiments
In this section, we discuss the training datasets and implementation details, and present the qualitative and quantitative results, followed by an ablation study on training tasks and design choices.
4.1 Datasets
We train Iwo different models using two lypes of datasets: one trained only on publicly available data and one trained on our in-house proprietary data. Below, we detail the public datasets and provide aggregated statistics for the in-house counterparts. Additional details are provided in Appendix C.
OCR training data. As public OCR training data, use RIMES (Augustin et al., 2006; Grosicki et al., 2009), HierText (Long et al., 2022; 2023), IMGUR5K (Krishnan et al., 2023). ICDAR'15 historical documents (Murdock et al., 2015), and IAM (Marti & Bunke, 1999). We crop out word-level images where possible, acquiring 295 000 samples with text in Latin script. The in-house dataset consists of images of handwritten and printed text (67% and 33% respectively) written on a diverse set of backgrounds, with 95% of the labels in English.
For data sources that contain word-level segmentation, extract images of individual words based on their bounding boxes. For others, we found it beneficial to heuristically filter the training data to exclude samples that are too short, too long, or of too low resolution to be rendered on 224 x 224 image. The filtering rules cnsure that the aspect ratio satisfics (0.5 < width / height < 4.0), and that the image size is at least 25 pixels per side.
Digital ink training data. As public digital ink training data, use VNOnDB (Nguyen et al., 2018), SCUT-Couch (Ti et al., 2008), and Deep Writing (Aksan et al., 2018). For DeepWriting we use the available segmentation information to produce 3 datasets with character-level, word-level, and line-level croppings, and use all 3 for training. For VNOnDB we extract individual words to be used as training data. The public dataset size totals ~2.7M samples. The in-house multilingual dataset primarily comprises short snippets of text in Mandarin (37%) and Japanese (23%), while the remaining languages cach contribute less than 5%.
Evaluation Data. Since there are no available diverse datascts of paired inks and images (as discussed in Section 2), we perform the evaluation on OCR data and additionally do a small data collection to obtain paired samples. To automatically assess our models, we evaluate the quality of derendering on the test splits of OCR datasots: IAM (testset_f, ~17.66 samples), IMGUR5K (~~3.77k 7k samples), and HierText. For HierText which was not originally designed around handwritten samples, we apply the same filtering as the OCR training data, and additionally only consider words marked as handwritten (~1.3k samples).

---

Published in Transactions on Machine Learning Research (06/2025)
Additionally, we have performed small annotation campaign, asking people to trace ~00 samples from the Text test set. We use these as the "golden" set to find the reference point in the automated evaluation and as a control group when doing the human evaluation.
4.2 Models
We train variants of our model: Small-p (~340M parameters, -p stands for public setup) trained on publically available datasets with ViT B/16 encoder pretrained ImageNet-21lk (Deng et al., 2009) paired with b'se-base encoder-decoder; Small-i (-i stands for in-house setup), built with the same architecture Small-p, but using JFT-300M (Sun et al., 2017) pretrained ViT B/16 checkpoint; Large-i (~1B paramcters) with a ViT L/16 encoder pretrained on JFT-300M, paired with an m mT55-argg encoder-decoder. All models employ a context length of 1024 for the output, and 128 for the input. Implementation details are provided in Appendix G, and a model card for Small-p Appendix M.
4.3 Baseline Method
Despite pen trajectory recovery being a popular research direction in the last decades, available comparisons are limited due to the scarcity of open-sourced code, training data, and pre-trained model weights. Appendix D offers a detailed analysis of recent work and their reproducibility. We choose the General Virtual Sketching (GVS) Framework (Mo et al., 2021) as our baseline. Originally trained for sketch and photo vectorization, GVS offers valuable reference point despite the domain discrepancy between sketches and handwritten notes. We found that providing the GVS line drawing vectorization model with binarized inputs yields best performance, we compare all 3 available models original and binarized inputs in Appendix H.
4.4 Qualitative Evaluation
We compare the performance of our models and GVS on 3 public evaluation datasets (mentioned in Section 4.1) in Figures and 6. Our models mostly produce results that accurately reflect the text content, disregarding semantically irrclovant background. They can also handle occlusions, highlighting the benefit of the learned reading prior, in contrast to GVS, which produces multiple duplicate strokes, and does not distinguish between background and foreground (which is reasonable, given it was trained for sketch vectorization). Large-i is able to retain more details and accommodate more diverse image styles.
Input Image
GVS
Small-p
Small-i
Large-i
C Couldnn ECc Could dnn E Couldn & Caaaall lan'' & Souudd t6
nigh ha fought feuight fought
rig h- Frig 6.
:3g 67 hi
Lought fou
THINK
TWINK
TRTNA
TWENA
SUBSTANCE
SUSTANLE
SUBSTAIICE
SUBSTANCE
SUBSTANCE
AiERIN
GTARIN
GiTRRAN
IBRAM
TTBRAN
Figure 5: Comparison between performance of GVS, Small-i, Small-p, and Large-i IAM (top) and IMGUR5K (boliom), more visualizations in Appendix N.1
7

---

Published in Transactions on Machine Learning Research (06/2025)
Input Image
GVS
Small-p
Small-i
Large-i
HISTO MISTRY! HIMPY HISTORY /
HHITORY !
day
T day
Maydayday
THE The THE The THE
Figure 6: Comparison between performance of GVS, Small-i, Small-p, and Large-i on HierTText, which includes data with varying styles and occlusions.
Generalization to Sketches. We additionally study the performance of models on out-of-domain samples, such as simple sketches. For this, we use the Vanilla Derender inference mode to obtain the inks. In Figure 7, observe that our models partly generalize lo sketches, but the performance varies siguificantly depending on the sample and has noticeable artifacts such as missing details or over-focusing on a segment and over-tracing it. They, however, demonstrate robustness to complicated backgrounds (c.g. the lighting of the coffee sample).
Input Image
GVS
Small-p
Small-I
Large-1
Figure 7: Sketch derendering for Small-p, Small-i, Large-i and GVS. Our models are mostly able to derender simple sketches, however with significant. artifacts: missing strokes (e.g the cat), creating cycles (e.g the eye of the penguin for Small-i, the paws of the cat for Large-i) or unnatural strokes (e.g. for Small-p).
8

---

Published in Transactions on Machine Learning Research (06/2025)
4.5 Quantitative Comparison
To support the claims about the performance of our model, we conduct both human evaluation and automated evaluation that compare the similarity of our model output to the original image and to real digital inks. To the best of our knowlodge, there are no established metrics and benchmarks for this problem yet.
4.5.1 Human evaluation
We performed a human evaluation of the quality of the derendered inks produced by the three variants of our model. We used the "golden" human traced data on the Hier'Text dataset as the control group and the output of our model on these samples as the experiment group. Evaluators were shown the original image alongside rendered digital ink, which was either model-generated or human-traced (unknown to the evaluators). They were asked answer two questions:
the digital ink output a reasonable tracing of the imput image? (Answers: Yes, it's a good tracing; It's an okay tracing, but has some small errors; It's bad tracing, has some major artifacts); Could this digital ink output. have been produced by human? (Answers: Yes; No).
We provide more detailed description of the task and the instructions in Appendix L. The evaluation included 16 individuals familiar with digital ink, but not involved in this research. Each sample was evaluated by 3 raters and aggregated with majority voting (with inter-rater reliability : (1) 0.46, (2) 0.44 for the respective questions).
We demonstrate the results of our evaluation campaign Figure We observe better performance (good or okay rating, more realistic samples) when both the data and model size are scaled up. The most common improcisions that lead to choosing "okay tracing" over "good tracing" include: small number of extra strokes (e. g. derendering irrelevant elements present in the image), missing details (e.g. punctuation, dot over j), and unnecessary double strokes (see Appendix L for examples).
Is ReasomaNie?
Is Human?
100
95.5
80
822
87..5
$2.5
Yos, it's good tracing
asa bad tracing, has somte major aniface
No
Small-p
Small-i
Large-
Small-p
Small-
Larg-l
Figure Human evaluation results Small-p, Small-i and Large-i on 200 "golden" samples of HeerText dataset.
4.5.2 Automated Evaluation
Automated metrics (described below) are presented in Table Most importantly, the model ranking using the automaled metrics matches the results of the human evaluation.
Similarity to the input image. To capture how similar the output of our model is to the input image semantically and geometrically, compute the Character Level F1 score by following the protocol of the Robust Reading Challenge (Rob, 2023). This measure captures both geometry and semantics by counting the character bounding boxes found by an OCR model (in the original input image and in the output of the model rendered as an image) that both overlap significantly and match content-wise. To obtain those character bounding boxes, we use the publicly available API (Clo, 2024). We see that our models perform similarly GVS on simpler black-on-white IAM datasct, but outperforms on HierText which has more diverse background. We also note that the OCR engine used is not perfect, and due errors the OCR model and its sensitivity to line width, the value of this metric 0.64 for the "golden" human traced data

---

Published in Transactions on Machine Learning Research (06/2025)
Table Automated metrics comparison between three variants of our model, GVS, and the "golden" human traced data. We compare both Character Level Fl score of the rendered characters and Accuracy of recognizing the generated digital ink with state-of-the-ar recognizers.
IAM
IMGURSK
HierText
Model
F1
Acc.
F1
Acc.
P1
Acc.
Small-p
0.65
0.60
0.47
0.33
0.53
0.37
Small-i
0.65
0.59
0.51
0.33
0.61
0.44
Large-1
0.66
0.58
0.51
0.32
0.61
0.46
GVS
0.69
0.02
0.51
0.01
0.55
0.01
Human"
0.64
0.74
Computed on the human-traced subset HierText data.
Hext Therefore, can only be used to meaningfully distinguish models as long as their F1 score is fairly low.
Similarity to the real digital ink data. To assess the semantic consistency and geometric properties of our generated digital inks, we compute recognition Exact Match Accuracy with the state-of-the-art online handwriting recognizer trained on real digital ink data (Clo, 2024). As shown in Table 2, our models demonstrate superior accuracy compared to GVS (on which the online handwriting recognizer is failing completely due to unnatural stroke number, length and order). We also observed that while the three model variants perform similarly the IAM and IMGUR5K datasets, the Large variant outperforms Small variants on the Hier'Text dataset which has greater diversity, which aligns with our human evaluation.
To further validate the similarity between real and derendered digital inks, we train an online handwriting recognizer on inks derendered from the TAM training set. Testing on the testset f of TAMOnDB (Liwicki & Bunke, 2005) and compare with the performance of model trained on LAMOnDB train set, in terms of Character Error Rate metric (CER). As shown in Figure 9, the model trained only on dcrendered inks achieves slightly worse CER performance compared to the model trained on real data. Furthermore, the combination of derendered TAM inks and real IAMOnDB inks allows to acquire a more diverse training set, resulting in significantly lower CER. This results holds for Small-p and Large-i as well (see Appendix I). While our CER trails state-of-the-ar CER due to limited training data, our work demonstrates that offline handwriting datasets can effectively augment online recognizer training. The details of the online handwriting recognizer we train are given in Appendix I.
IAMOnDB
6.1
IAM derendered
7.8
IAM derendered + IAMOnDB
4.6
2
Character Error Rate (%)
Figure 9: Online handwriting recognition results IAMOnDB test set for Small-i trained real digital inks (TAMOnDB train set), derendered digital inks (Trom TAM train set) and their combination.
4.6 Ablation Studies
This section examines the performance variation across inference tasks and ablates the impact of augmentation, data mixture, and training strategies on the Small-i model's performance. Conclusions and numerical results of the ablation study presented in Table and summarized below. For more details, Appendix E.
10

---

Published in Transactions on Machine Learning Research (06/2025)
Table 3: Ablation studies on Small-i. The first row shows the best performance setup (with inference mode Derender with Text). Subsequent rows display results for the Vanilla Derender and Recognize and Derender (R D) tasks (Figure 3), followed by the impact of various design choices on performance.
IAM
IMGURTK
HierTText
Setup
Acc.
F1
Acc.
Acc.
Small-i
0.66
0.59toe1
0.51
0.331m08
0.61
0.45a0o
Vanilla
0.61 L
0.53 L
0.44
0.28 /
0.53
0.35
R+D
0.62 |
0.57
0.46.
0.32
0.57
0.42
Remove
data aug
0,42 +
0,33 |
0.21
0.09 1
0.23
0.13
rec
0.58 |
0.50 |
0.50
0.25 |
0.56
0.38
real rec
0.64 |
0.50 |
0.55
0.19
0.59
0.36
all rec
0.65
0.51 4
01.55
0.22 |
0.61
0.38
frosen VIT! 1.65
0.53
0.43
a.31
0.59
0.41 =00s]
Digital ink rendered fixed-width white strokes black background. The results aggregated from three random Initializations, all other results are acquired with the best. performing initislizations with the same three seeds.
DESIGNÉRS
Cilantro
breakfast-nook
hobbits
Inpor Imagee
DBiGNO'S
allntro
1 biakkesl--oo
hobbits
Vanilla
DESiGNES
Allantd
I. wotk/at nox
hotross
Recouniast DESKGNER'S
Recognized: Cllantro
Recognized: Lesthhilbadd
Recognized hossttes
O/OCR Input w/ Recogaltton
Cllatro
b eat/sst
hoTSiZTS
Designers
OCR Cilantra
OCR Input: break
Unper MiRRirs
Gvess
allntro
beak oo0k hOBBITS
Figure 10: Varied digital ink outputs from Small-i inference tasks on samples with ambiguous transcription. Column titles correspond to the ground truth labels. Image titles correspond to either the recognition result from our model or OCR system.
Inference task matters in difficult cascs (Rows 2-3). We compare the performance between three types of inference tasks, find Derender with Texl is slightly better than Recogwize and Derender (R+D), which itself outperforms Vanilla Derender all three datasets, it generates more semantically consistent. Figure 10 shows a collection of derendered inks from Small-i where each input image has ambiguous or difficult-to-read characters. We lake the first column, DESiGNÉRS, as an example: Inference with Vanilla Derender captures the overall geometry of the input image but often lacks textual understanding (e.g., the letter and ÉR are drawn as rough strokes rather than following the precise structure of the characters). Derender with Text produces results consistent with the text input the model provided by OCR (upper recognition, resulting in lowercase and er). More examples in Appendix E.2.
attribute this difference to the fact that
and ER), while Recognize and Derender aligns with the model's
The necessity of data augmentation (Row 5). Removing the data augmentation leads to significantly worse performance across all metrics on all datasets. While data augmentation (as shown in Figure 3) does not visually align rendered inks with real ink photos, diversifies the rendered ink distribution and we find that it is essential for the model to perform valid derendering real images with training setup.
11

---

Published in Transactions on Machine Learning Research (06/2025)
Recognition tasks improve derendering quality (Rows 6-8). Removing recognition tasks from the multi-task training mixture notably reduces overall performance across all evaluation datasets, particularly impacting accuracy which reflects the model's semantic understanding ability. We hypothesize that the increase in F1 and drop in accuracy (such as the onc observed on IMGUR5K) may be attributed to the model locusing less on the textual content and interpreting more background noise as strokes, which leads to digital inks that closely mimic the style of the input imagos, resulting higher likelihood of overlapping bounding boxes in Fl calculations.
Impact of frozen VIT in multi-task training. (Last row). Unfroezing the Vision Transformer (ViT) in multi-task training typically incurs training instability, as evidenced by increase in variance in model evaluations and an increased tendency to misinterpret background noise as textual content. Explaining the observed increase in F1 scores in Table 3 can be attributed to the model misinterpreting background noise as strokes, generating digital ink that visually closely resemble the input images, including the noise. This behavior results in a higher likelihood of overlapping bounding boxes, inflating F1 calculations. More details in Appendix E.1.
4.7 Limitations
We acknowledge the following two main limitations our approach:
(1) Due to the limits in context length of transformer-based models, it can derender limited amount of characters words in one model inference call; While our system incorporates efficient design choices such as resampling and sequence reduction to support full-page derendering, this remains a constraint.
(2) Consequently, it requires some form of segmentation of the handwrilten page to run derendering on rather short individual segments (words in our implementation) and in the current form cannot work on complex sketches or other objects with lots of strokes, extending to handle complex sketches or dense stroke- based objects presents an exciting direction for future work.
One thing that is not limitation of our model is the use of OCR to provide the word labels as shown in the first 2 rows of Table 3, the model performance degrades only slightly if the label is not provided or is inferred by the model. We also discuss some specific failure cases in Appendix J.
5 Conclusion
In this work we present the first approach to converting photos of handwriting into digital ink. We propose a training setup that works without paired training data, which can be difficult to acquire. We show that our method is robust to a variety of inputs, can work on full handwritten notes, and generalizes to out-of-domain sketches to some extent. Furthermore, our approach does not require complex modeling and can be constructed from standard building blocks. We relcase the weights and inference code of the Small-p model, with a subset of model-generated and human expert-traced digital ink data. Future work may address the main limitations of our model stated above.
Acknowledgments
The authors thank Leandro Kicliger for the help on synthotic ink rendering, foodback on initial versions of the paper, Philippe Schlattner on human evaluation protocol, Anastasiaa Fadeeva, Mircea Traichioiu for the multiple discussions, Efi Kokiopoulou, Diego Antognini, Henry Rowley, Reeve Ingle, Manuel Drazyk for feedback on initial versions of the paper, Sebastian Goodman, Jialin Wu for the help on implementing deterministic training and PaLI related questions, Xiao Wang on ViT related questions, Tom Duerig and Tomáš Tzo for leadership support.
12

---

Published in Transactions on Machine Learning Research (06/2025)
References
Notability Ink to Image conversion.
https://support..gaee http://sppoot.gggggrlalaaa http:/ggaaaaa...ooooooeeoaaeeaee htt.aaa...aocaaoeaaauuaaaataaaal
2023. [Online; accessed 10-Jan-2024J.
Hierarchical Text: Challenge Unified OCR and Layout Analysis. com=tasks, 2023. [Online; accessed 10-Jan-2024).
Google Cloud Vision Handwriting Text Detection API. handwriting, 2024. [Online; accessed 10-Jan-2024].
LiveScribe. https::/ns..1iveerrbb.oomm 2024. [Online; accessed 10-Jan-2024|, 2024].
NeoSmartPen. https:/heeomarteen.com 2024. [Online; accessed 10-Jan 2024].
Rocketbook. https://getrrocketbookccc com/, 2024. [Ouline; accessed [i-Jan-2024]
Nuwa. https://nuwapen.coom 2025. [Online; accessed 12-Feb-2025).
Emre Aksan, Fabrizio Poee, and Otmar Hilliges. Deepwriting: Making digital ink editable via deep generative modeling. In Proceedings of the 2018 CHI conference on human factors in compuling systems, pp. 1 14, 2018.
lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 0123716-237380 2022.
Fakhraddin Alwajih, Eman Badr, and Sherif Abdou. Transformer-based models for arabic online handwriting recognition. International Journal of Advanced Computer Science and Applications, 13(5), 2022.
Taylor Archibald, Mason Poggemann, Aaron Chan, and Tony Martinez. Trace: a differentiable approach to linc-level stroke recovery for offline handwritten text. In Document Analysis and Recognition-ICDA. 2021: 16th International Conference, Lausanne, Switzerland, September 5 5-10, 2021, Proceedings, Port III 16, pp. 414-429. Springer, 2021.
Emmanuel Augustin, Matthicu Carré, Emmanuèle Grosicki, J-M Brodin, Edouard Gcoffrois, and Françoisc Prêteux. Rimes evaluation campaign for handwritten mail processing. In International Workshop Frontiers in Handwriting Recognition (IWFHR 06),, pp. 231-235, 2006.
Ayan Kumar Bhunia, Abir Bhowmick, Ankan Kumar Bhunia, Aishik Konwer, Prithaj Banerjee, Partha Pratim Roy, and Umapada Pal. Handwriting trajectory recovery using end-to-ond deep encoder-docoder network. 2018 24th International Conference on Pallern Recognition (ICPR), pp. 3639 3644. IEEE, 2018.
Ayan Kumar Bhunia, Pinaki Nath Chowdhury, Yongxin Yang, Timothy M Hospedales, Tao Xiang, and Yi-Zhe Song. Vectorization and rasterization: Self-supervised learning for sketch and handwriting, In Proceedings of the IEEE/CC Conference Computer Vision and Pattern Recognation, pp. 5672-5681, 2021.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, et al. Jax: composable transformations of python+ numpy programs. 2018.
Chungkwong Chan. Stroke extraction for offline handwritten mathematical expression recognition. IEFF Access, 8:61565 1-61565-61575, 2020.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Scbastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: jointly-scaled multilingual language-image model. aix preprint 0rx::220000001 06794, 2022a.
13

---

Published in Transactions on Machine Learning Research (06/2025)
Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al. Pali-x: On scaling up multilingual vision and language model. erXiv preprint wr.Xi:2235.11860 2023a.
Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Scbastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, et al. Pali-3 vision language models: Smaller, faster, stronger. arXiv preprint orXiv:2310.0919 091199 2023b.
Zhounan Chen, Daihui Yang, Jinglin Liang, Xinwu Liu, Yuyi Wang, Zhenghua Peng, and Shuangping Huang. Complex handwriting trajectory recovery: Evaluation metrics and algorithm. In Proceedings of the asian conference on computer vision, pp. 1060-1076, 2022b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEFE conference on computer vision and pottern recognàtion, pp. 248 188-255. leee, 2009.
David Doermann, Nathan Intrator, E Rivin, and Tal Steinherz. Hidden loop recovery for handwriting recognition. In Proceedings Eighth International Workshop on Frontiers in Handuriting Recognition, pp. 375 380. IEEE, 2002.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Usækoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
David Etter, Stephen Rawls, Cameron Carpenter, and Gregory Sell. A synthetic recipe for ocr. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 864 869, 2019. doi:
10.1109/CCDAR.2019.000
Emmanuèle Grosicki, Matthicu Carré, Jean-Maric Brodin, and Edouard Geoffrois. Results of the rimes evaluation campaign for handwritten mail processing. In 2009 10th International Conference on Document Analysis and Recognition, pp. 941-945. IEEE, 2009.
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. Flax: A neural network library and ecosystem for JAX, 2023. URL http://github.com com/ google/flax.
Stefan Jager. Recovering writing traces in off-line handwriting recognition: Using a global optimization technique. In Proceedings of 13th international conference on pallern recognition, volume 3, pp. 150 154.
TEFE, 1996.
Yoshiharu Kato and Makoto Yasuhara. Recovery of drawing order from single-stroke handwriting images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(9):938 949, 2000.
Praveen Krishnan, Rama Kovvuri, Guan Pang, Boris Vassilev, and Tal Hassner. Textstylebrush: Transfer of text aesthetics from a single example. IEFF Trunsactions on Pattern Analysis and Machine Intelligence, 2023.
P Lallican, Christian Viard-Gaudin and Stefan Knerr. From off-line to on-line handwriting recognition. In EPRINTS-BOOK-TITE 2004.
Yunyang Li, Lianwen Jin, Xinghua Zhu, and Teng Long. Scut-couch200s: A comprehensive online uncon- strained chinese handwriting dataset. ICFHR, 2008:165 170, 2008.
Cheng-Lin Liu, Fei Yin, Da-Han Wang, and Qiu-Feng Wang. Casia online and offline chinese handwriting databases. In 2011 international conference on document analysis and recognition, pp. 37 41. IEEE, 2011.
M. Liwicki and H. Bunke. Tam-ondb- an on-line english sentence database acquired from handwritten text on a whiteboard. In Eighth International Conference on Document Analysis and Recognition ([CDAR'05) pp. 956-90600 961 Vol. 2, 2005. doi: 10. 10.1100/TCDARR22055.3222
14

---

Published in Transactions on Machine Learning Research (06/2025)
Shangbang Long, Siyang Qin, Dmitry Panteleev, Alessandro Bissacco, Yasuhisa Fujii, and Michalis Raptis. Towards end-to-end unified scene text detection and layout analysis. In Proceedings of the TEEE/CVE /VFF Conference on Computer Vision and Pattern Recognition, pp. 1049 1059, 2022.
Shangbang Long, Siyang Qin, Dmitry Panteleev, Alessandro Bissacco, Yasuhisa Fujii, and Michalis Raptis. Icdar 2023 competition on hicrarchical text detection and recognition. arXiv preprint XXix:3305.09750 2023.
U-V Marti and Horst Bunke. A full english sentence database for off-line handwriting recognition. In Proceedings of the Fifth International Conference on Document Analysis and Recognition. ICDAR '99 (Cal. No. PR00318), pp. 105-708. IEEE, 1999.
Margaret Mitchell, Simone WU, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the conference on faimess, accoundability, and transparency, pp. 220 229, 2019.
Haoran Mo, Edgar Simo-Serra, Chengying Gao, Changqing Zou, and Ruomei Wang. General virtual sketching framework for vector linc art. ACM Transactions on Graphics (TOG), 40(4):1-14, 2021.
Elmokhtar Mohamed Moussa, Thibault Lelore, and Harold Mouchère. Set, sort! novel sub-stroke level transformers for offline handwriting to online conversion. In International Conference on Document Analysis and Recognition, pp. 81 $1-97. Springer, 2023.
Michael Murdock, Shawn Reid, Blaine Hamilton, and Jackson Reese. Icdar 2015 competition on text line detection in historical documents. In 2015 13th international conference on document analysis and recognition (ICDAR), pp. 1171-1175. TEEE, 2015.
Hung Tuan Nguyen, Cuong Tuan Nguyen, and Masaki Nakagawa. Tefhr 2018 competition on vietnamese online handwritten text recognition using hands (vohtr2018). In 2018 16th International Conference on Frontiers in Handuriting Recognition (ICFHR), pp. 494-499. IEEE, 2018.
Hung Tuan Nguyen, Tsubasa. Nakamura, Cuong Tuan Nguyen, and Masaki Nakawaga. Online trajectory recovery from offline handwritten japanese kanji characters of multiple strokes. In 2020 25th International Conference on Pallern Recognition (ICPR), pp. 8320- 8327. IEFF, 2021.
Reza Pourreza, Apratim Bhattacharyya, Sunny Panchal, Mingu Lee, Pulkit. Madan, and Roland Memisevie. Painter: Teaching reto-regressive language models to draw sketches. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV) Workshops, pp. 305-314, October 2023.
Yu Qiao, Mikihiko Nishiara, and Makoto Yasuhara. A framework toward restoration of writing order from single-stroked handwriting image. IEEE transactions on pattern analysis and machine intelligence, 28(11): 1724-1737, 2006.
Adam Roberts, Hyung Won Chung, Gaurav Mishra, Anselm Levskaya, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, et al. Scaling up models and data with t5x and seqio. Journal of Machine Learning Research, 24(377):1-8, 2023.
Paul K Rubenstein, Chulayuth Asswaroengchaii, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306. 12925, 2023.
Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. The sketchy database: learning to retrieve badly drawn bunnies. ACM Transactions Graphics (TOG), 35(4): 12, 2016.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp. 4596 4604. PMLR, 2018.
Arti Shivram, Chetan Ramaiah, Srirangaraj Sctlur, and Venu Govindaraju. Tbm ub A dual mode unconstrained english handwriting dataset. In 2013 12th International Conference on Document Analysis and Recognition, pp. 13 17. IEFF, 2013.
15

---

Published in Transactions on Machine Learning Research (06/2025)
Taichi Sumi, Brian Kenji Iwana, Hideaki Hayashi, and Seiichi Uchida. Modality conversion of handwritten patterns by cross variational autoencoders. In 2019 international conference on document analysis and recognition (ICDAR), pp. 407 412. IEEE, 2019.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning cra. In Proceedings of the IEEE international conference computer vision, pp. 843 852, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in nearol information processing systems, 30, 2017.
Christian Viard-Gaudin Pierre Michel Lallican, Stefan Knerr, and Philippe Binter. The ireste on/oft (ironoff) dual handwriting databasc. In Proceedings of the Fifth International Conference on Document Analysis and Recognition. ICDAR '99 (Cal. No. PR00318), pp. 455 458. IEEE, 1999.
Christian Viurd-Gaudin, Pierre-Michel Lallican, and Stefan Knerr. Revognition-direetee recovering of temporal information from handwriting images. Pattern Recognition Letters, 26(16):2537-2555 -248,, 2005.
Yael Vinker, Ehsan Pajouheshgar, Jessica Y Bo, Roman Christian Bachmann, Amit Haim Bermano, Daniel Cohen- Or, Amir Zamir, and Ariel Shamir. Clipasso: Semautically-aware object sketching. ACM Transactions on Graphics (TOG), 41(4):1 11, 2022.
Mahes Visvalingam and J Duncan Whyatt. The douglas-peucker algorithm for line simplification: re-evaluation through visualization. In Computer graphics forum, volume 9, pp. 213 225. Wiley Online Library, 1990.
Jiarui Xu, Xingyi Zhou, Shen Yan, Xiuye Gu, Anurag Arnab, Chen Sun, Xiaolong Wang, and Cordelia Schmid. Pixcl-aligned language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognition, pp. 10030-13033 2024.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. m A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 483 498, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v http2221.naaccee 41. URL https: hatps////cccaaaoooo//// org/2021
41.
Xiaohua Zhai, Xiao Wang, Basil Mustala, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with lockod-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 18123 18133, 2022.
Xiong Zhang, Min Wang, Lijuan Wang, Qiang Huo, and Haifeng Li. Building handwriting recognizers by leveraging skeletons both offline and online samples. 2015 13th International Conference on Document Analysis and Recognition
(IC (ICDAR), pp. 406-410 410. TEFE, 2015.
16

---

A Appendix
CONTENTS
B Full-page Results
18
Data Statistics and Preprocessing
29
D Survey of Derendering Models
29
E Further Details of Ablation Studies
31
Rendering and Data Augmentation
34
G Additional Model Details
36
General Virtual Sketching Framework Performance
37
I Online Handwriting Recognizer Details
J Expansion on Failure Cases
39
K Expansion on Out-of-Domaal Generalization
40
L Human Evaluation
40
M Model Card
42
N Additional Visualizations
43
Impact Statements
48
This oppendia prowides comprehensive supplementary materials supporting the main findings and methodology presented in this work.

---

Published in Transactions on Machine Learning Research (06/2025)
B Full- Results
We show the full-page results on samples that resemble real-life inputs (mostly from Unsplash with keyword search for "handwriting and others collected by the authors with consent from the writers) produced by three variants of our models.
Vaver theee Nlaer Heur lbeer
neer Near Nest Nov Hate
ou Shane Nour Hove Sheet
Neve
Sover Jaee Been New
Jast Viur Dy sysii
Never New Moten Teur Hemr hew
Stayscory
Nosts Nove Vov Swaitten
Styssors
EVER EVER EVER E EVER EVER EVER
EVERIEVE PRRVEVER VERR VERR EVER EVER
L. YORR EVVRR EVER
EVEREVER GIVE UP Hang foona"
EVEREVER EVER
E EVEREVER GAVE UP
GIVE DA
Hay Kar faves
Figure 11: Large-i full page results of handwritten notes in a real-life scenario, credit: Unsplash.
ber Near Nest Never Host
Vaver theee Nearr Heur lovr
Never New Note Hemm hem
Stayscor
ae Shnre Nour Hove Sheet
New
Stayspor
EVER EVER EVER E EVER EVER EVER
EVERIEVE REVERVEEN WEER E VER EVER EVER E EVER VVER
Ie YARR AVDRR EVEER
EVEREVER GIVE UP Hang foena
EVER VEERS GOVE UP
GIVE DA
SO
Hay Kan faves
Figure 12: Small-i full page results of handwritten notes in a real-life scenario, credit: Unsplash.
Vever Sheee Neur Hour bler
oeer Nese Nlase Never Hest
Mo Shene Nour Nave Sheat
Nevr
Nome Shar Herr How
les Mhat Nait
Never New Neter tevr Hemr bee
Stayseor
Nest Nene Near Seateer
Staysors
Neee Seam bet MAm
EVER EVER EVER
EVER VERVEVERS EIEEEE EVER
ELVES, YOUR LIVEA
E EVER EVER EVER
E VVER EVER EVE
E WEE ETRER ENER
EVEREVER GIVE UP Haug foons"
EVER EVER
E EVER GAVE UP
GIVE DA
SO
How and forrs
Figure 13: Small-p full page results of handwritten notes a real-life scenario, credit: Unsplash.
18

---

Published in Transactions on Machine Learning Research (06/2025)
JE VEUX DE L'AMOUR
JE VEUX DE LAMOUR
JE YEUX DE NAMURR
Figure 14: Large-i full page results of French sample. credit: Unsplash
JE VEUX DE L'AMOUR
JE VEUX DE L'MOURt
JE VEOR DE AAMOR-
Figure 15: Small-i full page results of French sample. credit: Unsplash
JE VEUX DE L'AMUUR
JE VEUX DE LAMOUR
Voor DE NAMUR
Figure 16: Small-p full page results of French sample. credit: Unsplash
19

---

Published in Transactions on Machine Learning Research (06/2025)
kuncle Heag
is Beaelifd and searide Hane hopper, cnuestorr
Bradiity and cerrie way capper. wo
Lted praiiad
Figure 17: Large-i full page results of handwritten notes in a real-life scenario with low resolution, credit: Unsplash.
Here Beautitdt and tearile Haie and te kfonid. cRuecterr
12 and 11
Beat te and Foreich Hard to sfaec Cmersonn
Beantiz pstiby
Figure 18: Small-i full page results of handwritten notes in a real-life scenario with low resolution, credit: Unsplash.
Hare Readtifd and secvile Hane with happer le afonid. cpaeetorr
Bead
Horo Go Bead ond coirie wid "appen Dell be sooi claecce
Beudlife
pialee 11
Figure 19: Small-p full page results of handwritten notes in real-life scenario with low resolution, credit: Unsplash.
20

---

Published in Transactions on Machine Learning Research (06/2025)
OFFESTIEE
rouse stufft
romiC shuft
commin loudly recycle shff
Cocally
VeCupka
racyle ttuff
Figure 20: Large-i full page results of Post-It note.
(FERTUEE
rousse stufft
re shuB2
gumae boudly
PICA Valya
co-tuAL Koally
resulke shuff
recycla st.ff
Figure 21: Small-i full page results of Post-It note.
(FFETTIEE
rouse stuff
mon shupp. COMDEECO locn Mig raycle stuff
oumees Coully meclle sufff
Figure 22: Small-p full page results of a Post-It note.
21

---

Published in Transactions on Machine Learning Research (06/2025)
eaery aprisions
Aorboleed
mangs Speaiine
Adssioaly oetnin moll cappy
Yaluively
Schetar
Rubtimiy polono mate Longo Velio oring pontile fane, wher, the no gomties only misepluk and thi witt of ravsnennl
Petioss awaay Jomtcle in fire. vhher the too querllies diBer wey by anittiioess and the watemmnee
Vinte
Red
Figure 23: Large-i full page results of handwritten notes of mass mass-energy energy equivalence.
Mas carry apiniou
cndey dip
felatinnly oehoton mass capp
Sobetor
jmsii
Adbriinyy setesin Hole Brugh
Veloes oweyy dumill fire, ahhere the guanthes Jifter rey by anttioooee uorter and the wosn wasemeen
peanyy
Vellie oreny of semmle vabara Ahe gormebis Adfion only by un coultpoonn conteny and the unit monemmteet
Hend
Figure 24: Small-i full page results of handwritten notes of mass-energy equivalence.
Mos cantyy spriiin
Man conny sprindone
felotienly botuton mass Carpy
Iobetor
maks erayy)
Escme
Es
Veliess oneny of Sumill rest fore, Jifter myy by autiidiieee contert and the unit
Serey
Aarivilk
Pelnn onayy of pomnis net riens
goanhhies
Fand
quaatii do J.Rie only by wwwlipaatie condint and eia onth
Sondirrd
Figure 25: Small-p full page results of handwritten notes of mass-energy equivalence.
22

---

Published in Transactions on Machine Learning Research (06/2025)
Mit # 1...
R
AA Ro &
Pe A 46 A9
Hoppy Birblay + you!
Happy Biehady
you!
Hopy Hullyyy + you
Best withos
Best Luishes
Best withrs
Tayeus saniersaire
Tagers mnnittraire
Jogars ancniigare
Figure 26: Large-i full page results of multilingual handwritten notes.
AR.4T. # A
7. 4T. #
# A ta A
th A
Happy Birbyn + you!
Happy Bethyy
you!
Marpy Bidly
you
Best wishes
Best
buithes
heer with
Joyour snaivesaire.!
Joypus annidersave
Jangus ond inbrgeiie
Figure 27: Small-i full page results of multilingual handwritten notes.
NR.4 2 Ad7A1
6
1
M
A s6 1
8 46 48
AAMAHAA
Happ Birblyy + you!
Happy
Bithbby
you
Happ Betabay
yo
Best withes
Best wuithos
Beer with
Jayeur snaiversaire!
Iagerss anniveccave.
Joyans awsiirame
Figure 28: Small-p full page results of multilingual handwritten notes.
23

---

Published in Transactions on Machine Learning Research (06/2025)
Digink
Digink
Digink
Figure 29: Large-i full page results of OOD sample.
Digine
Digink
Diginl
Figure 30: Small-i full page results of OOD sample.
Digink
Digink
Digink
Figure 31: Small-p full page results of OOD sample.
24

---

Published in Transactions on Machine Learning Research (06/2025)
PANAEE.
DANKE
C DANKE
Figure 32: Large-i full page results of OOD sample, credit: Unsplash.
DARANK..
CARAEEE
DANKE.
Figure 33: Small-i full page results of OOD sample, credit: Unsplash.
PRDAKK....
DANEE
M2 DANKE..
Figure 34: Small-p full page results of OOD sample, credit: Unsplash.
25

---

Published in Transactions on Machine Learning Research (06/2025)
We have vody made the decision to rempurarly dase dhinng hhss difiout times, hope thet for short base daing that our emoning byoa Cuatavers, sray sate heavty Stay tiuen on Instagram @e coloa. sisters
Tpul
we have vodly mads the Jeasson to Homipromill clsee chinny TreEo anriott time dhet siove too woss wking uhat we Suvrunded our on eociing lyy cuemmrrs Stoy coffe veans on ingrigrinn @ card catvi
Hod
close
Theie
Hinse S
Rount
is Ferr
Hhead
Soling
Moatt
Rlmarting
Stony
Lata
Shony
Boush
Inghanrane
Kia
Gattell Sieters
vp Jaite famiy
xovonov
Kewoxay
Figure 35: Large-i full page results of handwritten notes in a real-life scenario, credit: Unsplash.
We have sooty mode the decision to rempurarly casee duiing mese défieul times v. hope thot for shert
W
We have Jocty mads the deanon to "erppamnll clese deing "veso diriual times inops dhent for shart imm
Tempurately
Cllgee
There
Chratrell
Hinar S
IS Por
bose doing whet suroonned by our emasing layai custumers.
ploing
Sount
hook doing whar surroin car dossann loyal custmatis, Stay Life
stay safe heavthy
Stery
Leie
Stay louan on Instagram
Bourc
Sto newar on besigar (B a riiire
@ solon, sisters
v Siate Jicce
V
disted
famiy xovokoic
fomelyy foverave
Figure 36: Small-i full page results of handwritten notes in a real-life scenario, credit: Unsplash.
26

---

Published in Transactions on Machine Learning Research (06/2025)
We have vody made the decision to tempurarly dase dhiing thss difiout times. hope thet for short base daing that we by our emoning byoa Cuatavers, sryy sate heavty Stay tiuen on Instagram @e coloa. sisters
leav
Tpul
We have vodt vode the deoien de snavly clese clinng treed oxt ooui time Viee dhet fer Sioee too
Hod
close
Theiel
Hinse S
Rount
is Ferr
Head
Soiing
Moatt
Ico bn doing shat was dell our dioittnng eapa austaates Siawy lot's
Rlmakting
Stony
Lata
Shony
Boush
Inghanrane
Atey on inn inssin
Gatcell S1eters
aa addis iite
vp Juete famiy
b Toalo poc owmly
xovosov
Figure 37: Small-p full page results of handwritten notes in a real-life scenario, credit: Unsplash.
200t ure
de
14 doenn
and
apern ortaree
1991
020
ight
and -etorm geog wves nist
or was.
1208
2ut -gg- vie the <nnd
194
118
210 15a ze 36 dr
<re / 4o0 gut 6oM
1932
1ng 6-mm aet man
020 keeen ds yeu
a.
1944 gas guitt
199 Lirrp -
1997
7644 450
de seep
1959
19
god
-0 6.6
20 thd
tse
.010
//c
Figure 38: Large-i full page results handwritten notes in a real-life scenario in Korean, credit: Unsplash.
27

---

Published in Transactions on Machine Learning Research (06/2025)
an To -a
*.g Aur "Mo ve may tye
que -4 aing
-rga. of *ves agey
1258
Hi
1922
.nd 8.mm aut 1-14 mxw
<te / 4uth gui, Epm 84
any
188
0
764
zis. the
dien "gnee "a)
ze de dy
24 de
r0L4 pew toiat anw
my
3a9 - +
2oone bee
T04e +30
T449
1630
setgens thd pwg)
os geny
<17 4.6
Figure 39: Small-i full page results of handwritten notes in a real-life scenario in Korean, credit: Unsplash.
de
1866
1991
and -epeem grng wnes ugit
4-t -igl gome 2ut -gg- vie <nnd
ot uas.
(208)
194
500 15a
<re / 4ow qut tom ds
atl andt
the
198 0
1n4 s4m0 aet ze 36 dg
Tee
T.
yia una igens wast
eu
4. Jin
de
2084. gas goits yes
sa9 Aaar -
4f 1444
2.
seep de en paol
see
T644 450 si
gem the
200
deggeas the
-10 6.6
-016
//c
Figure 40: Small-p full page results of handwritten notes in a real-life scenario in Korean, credit: Unsplash.
28

---

Published in Transactions on Machine Learning Research (06/2025)
C Data statistics and Preprocessing
We present the language distribution of the digital ink datasets used to train our in-house (-i suffix) and publicly available (-p sullix) models in Figures 41 and 42, datasets and corresponding number of samples used training our public model in Table The in-house OCR dataset comprises of approximately 500k samples, and in-house Digital Ink dataset comprises approximately 16M samples.
Latin Mixed
2.8%
Oujaroti
Japanese
1.7%
Hind
3.1%
Amenian
1.7%
Jopanese 22.7%
Tomil
Georgian 1.5%
13%
Malayalam
English
Figure 41: In-house Datasets Language Distributions. Left: Digital inks, Right: OCR for Recognition. These datasets are used to train both in-house models Small-i and Large-i.
Pinyln
English
2.0%
IAM
3%
2.4%
Chinsss
77.8%
IMOURSK
Figure 42: Public Datasets. Language distribution of public Digital ink datasets used to train our public models (on the left) and public OCR datasets used to train public available model Small-p (on the right).
D Survey of Derendering Models
In this section we provide a small survey of recent published pen trajectory recovery approaches and show that none of them release sufficient materials reproduce their work with reasonable amount of effort. We illustrate this with Table 5.
29

---

Published in Transactions on Machine Learning Research (06/2025)
Table 4: Public training datasets used for training small-p.
Task Type
Dataset
Samples
License
DeepWriting (words)
89,565
Custom
Deep Beeiiig (lines)
33,933
Custom
(characters)
359,643
Custom'
Derendering
VNOnDB
66,991
MIT
SCUT-COUCH (chars)
1,998,784
Custom
SCUT-COUCH (pinyin)
15,653
Custom
1AM word-level (train)
53,839
Custom
IMGUR5k (train)
181,792
CC BY-NC"
Recognition
RIMES word-level (train)
51,738
CC BY-NC"
Hier (train)
5,978
CC BY-SA
ICDAR-2015 (train)
1,535
Custom
Custom license (CC BY-NC-SA 4.0 + non-distribation). See: URL.
MIT License. Sec: URL.
Custom license (research-only, attribution). See: URL
Custom license (reseacch-nnly, non-commercial, attribution). See: URL.
CC BY-NC 4.0. See: URL.
CC BY-SA
URL.
Custom license (research-only, attribution). See: URL.
Table 5: Existing pen trajectory recovery approaches and their reproducibility.
Study
Code Availability
Data Availabil-
Model Weights Availabil-
Year
ity
ity
Mohamed Moussa et al. (2023)
X
2023
X
Chen et al. (2022b)
Repo missing core elements:
X
2022
data preprocessing,
patible configs
Archibald al. (2021)
X
2021
X
Bhunia et al. (2021)
Repo missing core elements: training scripts, data prepro cessing
X
2021
Nguyen et al. (2021)
X
X
2021
Sumi et al. (2019)
X
X
2019
Blunia et al. (2018)
X
2018
30

---

Published in Transactions on Machine Learning Research (06/2025)
E Further Details of Ablation Studies
E.1 Frozen ViT helps training stability
We investigated the impact of freezing the vision encoder during training model stability and derendering performance. To assess training stability, analyzed the ratio of empty ink predictions across training steps on the "golden' human traced dataset (detailed in Section 4.5.1). Empty inks typically occur when the model confuses tasks, mistakenly outputting text instead of ink or both (different tasks depicted Figure 3). Our analysis revealed significantly greater variance between runs with an unfrozen selup compared to the consistent learning observed with frozen vision encoder (Figure 43).
Unfrozen VIT
Frozen ViT
60%
Random seeds
Empty ink ratio [%]
46
40%
64
2024
20%
0%
0
100k
200k
300k
100k
200k
300k
Total training steps
Figure 43: Comparison between frozen and unfrozen VIT Small-i trainings. Comparison is executed with 3 random seeds. Left: ratio empty ink outputs for unfrozen VIT multi-task training setup. Right: ratio of empty ink outputs for the frozen ViT multi-task training setup, both use inference task Derender with Tezt.
Furthermore, in terms of final model derendering performance, we identified that certain model initializations (controlled by seeds) within the unfrozen setup could also converge to functional models with comparable abilities to those trained with frozen setup. These models demonstrate superior preservation of visual ink details but exhibit increased sensitivity to background noise. We present visual comparison of their output characteristies in Figure 44.
OCR Imput heut
OCR Ieput: the
OCR Input Mi
OCR Impat:
OCR Inpot Mamiy
OCR Ieput: Lecture
OCR Impot: light
Inpurt Image
Mamp
the
Ww
I
L Lecrve
fight
Small-i-anfrooem
Mamp
heart
the
ww
Leer
fighs
Many
Small-i
heart
the
ww
I
L Leerme
4 fight
Difference between Small-i-unfrozen which was trained with an unfrozen VIT (seed 46 Figure 43) and Small-i which trained with frozen ViT. This shows that Small-i-unfrozen is sensitive to background noise compared to Small-i.
Figure 44:
31

---

Published in Transactions on Machine Learning Research (06/2025)
E.2 Inference type matters
As we first. illustrated in Section 4.6, different. inference tasks can produce different. output digital inks especially when the imput image of text is semantically or geometrically ambiguous due to either image quality or how the texts were written. For example the first column of Figure 45 where the word "wich" is removed by the writer with a strike through line but still recognized by both Recognize and Derender (model intrinsic) and Derender with Texl (extrinsic OCR system) inference, and the difference understanding and locating the textual information in input image results in different output digital inks. Additionally, the output from Vanilla Derender where semantics are not intentionally involved during training is more robust to ambiguous inputs but shows poorer semantic consistency with the input image.
which
Parsons
trouble
jumps
ILLUSION
however,
history
Input Image Pastasd
juyps
hifory
sieL whicl
Do traetble
ILLUsION
however,
Vanilla
snce wiiie
fasinn
Jumps
ilcusion
& aavean
s Mory
Recognized: sick which
Recognized: Passons
Recognized: tralle
Recognized: jumps
Recogtized: iLLUSIOn
Recogtized: nowelez.
Recognized: hitery.
w/ Reeognition
Paston
fwyps llusion raveln
ory
OCR Input: wich w which
OCR Input: Parsons
OCR Input: trouble
OCR Input: Jumps
OCR Input: ILLUSION
OCR Input: bowever.
OCR Input: history
WOOC Input
snch whiceh Pastons
fumps iLusion
haseier
v itry
Figure 45: Difference between inference tasks of Small-i on samples from public evaluation datasets, where the texts on top are the ground truth labels for the images in these datasets. Vanilla stands for inference with Vanilla Derendering, w w/Recognition stands for inference with Recognize and Derender, and w/ OCR Input stands for inference with Derender with Text where we resort. to an OCR system.
For Derender with Tezt inference, we have shown that how external textual input could benefit the model's semantic understanding. We explore the behavior of the model cases where the external text different. than the ground-truth textual information in Figure 46. The results reveal that the model's output is not solely determined by input text. It demonstrates the ability to produce valid outputs when external textual information is incomplete erroneous.
32

---

Published in Transactions on Machine Learning Research (06/2025)
RMA
XyZABCDE
of
year
S8
Imput Image
"
"as
of
year
SS
xxx &
Lo
KMA
year
Ss
Vanilla
Recognized:
LyzARGDE.
Recognized:
Recognized: year
Recognizod:
w/ Recognition
Cs
year
as
GG3
OCR Input:
xyz CDD
OCR Input:
OCR Inpot
OCR
w/ OCR Input
RM
Ss
year
AZZAB
Figure Cases when there is a mismatch between ground-truth textual information and external textual information provided to the model. We evaluated this using Small-i on samples from 3 public evaluation datascts, where the texts top the ground truth labels for the images in these datasets. Vanilla stands for inference with Vanillo Derender, w/ w/ #/Recognition stands for inference with Recognize and Derender, and w/ OCR Input stands for inference with Derender with Text where we resort. to an OCR system.
33

---

Published in Transactions on Machine Learning Research (06/2025)
F Rendering and Data Augmentation
To create synthetic rendered inks of our online samples, we use the Cairo graphics library to render online ink samples onto a 224 x 224 canvas. We then add several augmenlations to rendered samples.
Before rendering, we first randomly rotate the samples. Then, we pick the color of the strokes and the background uniformly random from the RGB color space and render the ink with a random width. Furthermore, we add lines, grids, or Gaussian noise into the background with a fixed probability. Finally, potentially add box blur the resulting image. This approach from synthetic OCR generation, e.g, described in (Etter et al., 2019). The detailed parameters of data augmentation and some samples are shown in Table 6. An example of how samples augmented with all parameters chosen at random could appear in the training set can be seen in Figure 47. Other approaches, such as perspective skew, shifting, scaling, or padding, did not show any improvements in the model performance and were discarded.
mainly inspired by approaches
mi Milie
milie milies c
milie
milie s
Figure 47: Demonstration of possible ways combine all used augmentations. (Sample from (Aksan al., 2018))
34

---

Published in Transactions on Machine Learning Research (06/2025)
Table 6: Data Augmentations Overview (sample from Aksan et al. 2018).
Augmentation
Possible Values
Examples
Rotation
angle (rad): [- 4:
milie milie milie milie
Stroke color
RGB: 1]s
milie milie milie c milie c milie
Background color
RGB: 1]
milie milie milie c milie milie
Stroke Width
width: [1px, 12px]
milie c milie milie c milii millie c milie
Lines
line width: [1px, 6px] line dist: [10px, 100px] line
color: RGB: 1]a
millee v mi Lie s mi (ie
Grids
line width: [1px, 6px] line
dist: [10px, 100px]
color: RGB: [0, 1]
m Li Comilee m tie
Gaussian Noise
standard dev.: [50, 500]
neu milie mi tie iive mittee
Box blur
radius: [Opx, 5px)
milie c milie milie c milie c milie
35

---

Published in Transactions on Machine Learning Research (06/2025)
G
Additional Model Details
The overview and individual components of three versions of our model Small-i, Small-p, and Large-i is shown in Table 7.
Table Model overview and components.
Model
Components
Parameters
Training Data
ViT-L Large
303M
In-house proprictary datasets shown in Appendix C.
Large-i
783M
ViT-B
85M
In-house proprietary datasets as shown in Appendix C.
Small-i
m mT5-Base
247M
ViT-B
85M
Publicly available datasets as shown Appendix C.
Small-p
T5-Base
247M
Implementation Details. Similar to PaLl models (Chen et al., 2022a; 2023a;b), models together with the training mixtures are implemented in JAX/Flax (Bradbury et al., 2018) using the open-source T5X, SeqIO (Roberts et al., 2023) and Flaxformer (Heek et al., 2023) frameworks. For training, we use the Adafactor optimizer (Shazeer & Stern, 2018) with 31 = O, second order moment of 0.8, and a language- model style teacher forcing with softmax cross-entropy loss. For the learning rate schedule, we use the linear decay scheduler with a peak learning rate of 0.01, 5k steps of warmup and a linear decay with decay factor of 3e - 6, and dropout of 0.1 both ViT encoder and the mT5 encoder-decoder. We train our models for 340k steps with batch size 512. With frozen ViT encoders, the training of 340M parameter model (such Small-i or Small-p) takes ~33h 64 TPU v5e chips and the training of 1B parameter Large-i model takes 105h on 64 TPU v5e chips, both run on an internal TPU v5e cluster and should be reproducible with public GCP TPU vōc. Inference with a 340M parameter model takes around hour for all 3 datasets, and 3 hours with Large-i.
Compute. For the results reported in Table 3, we train one 340M parameter model per ablation. For entries with estimated variances, we train each model with different seeds, respectively. This totals in ~330h 64 TPU v5e chips of training. We additionally train Small-p and Large-i to obtain the results in Table 2, accounting for 138h on 64 TPU v5e chips. In total, all experiments in this paper require a total of 5500h of compute for training the models and running inference.
Decoding. In our experiments, we observe marginal performance improvements in derendering tasks through adjustments in temperature sampling. However, for the sake of reproducibility and framework simplicity, we employ a greedy decoding strategy during inference for all tasks, with model checkpoints selected from the final step.
Training Mixture. As described in Section 3.2, the training mixture consists of 5 tasks constructed using Seq1O (Roberts et al., 2023). These tasks are pre-shuffled before training, ensuring each task appears with equal probability throughout the entire training process. And to foster reproducibility, the input pipeline as well as the model initialization are designed to be deterministic, reducing randonness-nnduee variations that could affect our design choices.
Inference Speed. We benchmarked inference speed on three different GPU devices by measuring tokens processed per second over five runs. The Titan RTX achieved an average of 114.43 tokens/s, the T4 averaged 47.14 tokens/s, and the V100 averaged 139.14 tokens/s. All measurements were conducted using greedy decoding under consistent conditions with batch size 32 across devices.
36

---

Published in Transactions on Machine Learning Research (06/2025)
H General Virtual Sketching Framework Performance
We demonstrate the performance of three GVS models from the official GitHub repository: Line Drawing Vectorization, Rough Sketch Simplification, and Photo to Line Drawing Conversion in two setups (original image as input and binarized image as input) in Figure 48. We inspect the samples visually on the more challenging HierText dataset. The photo to line drawing model was trained on people portrait photos and fails to generalize the images of text. Binarization is beneficial for both remaining setups and reduces the noise when derendering texts. Line Drawing Vectorization and Sketch Simplification perform similarly on the inspected samples, we choose Line Drawing Vectorization binarized images baseline since this model was trained on black & white raster images and therefore treats binarized samples in-domain. For derendering of the sketches (Figure 7) we use the Sketch Simplification model as intended by the authors.
SUPER
Inpot image
Mamy
106
IRAQ
your
CABBAGE
Line drawing
06
Line drawing bitsaiized
Mam
AR
CABBAGE
Mamy
06
ARAY
Sketch
CABCAGE
06
Sketch Iinurized
ARAO
ABBLAGE
Mamy
O
TRAQ
auur
Photo
CABLEGE
Sup
Photo binurtzed
MEM
IRAQ
your
CABBAGE
Figure 48: Comparison between General Virtual Sketching Framework inference setups.
37

---

Published in Transactions on Machine Learning Research (06/2025)
Online Handwriting Recognizer Details
In this section we provide the result of online handwriting recognizers used to evaluate the low-level feature similarity of real inks for IAMOnDB dataset and inks derendered by Small-p, Small-i and Large-i (see Table 8). We observe that the inks acquired with all of our models perform similarly when used as training data both on their own and in combination with real digital inks.
Table 8: Online handwriting recognition results on IAMOnDB test set for models trained on real digital inks (IAMOnDB train set), derendered digital inks (from IAM train set) and the combination of the two.
Small-i IAMOnDB
Small-p
Large-i LAMOnDB TAM derendered
Small-i
Small-p TAM derendered
Large-i TAM derendered
IAMOnDB
IAMOnDB
TAM derendered
IAM derendered
TAM
derendered
CER
4.5
Additionally, describe the architecture and our recognition training procedure. We choose common approach of training a (relatively small, 9.2M, due lo the amount of the training data) Translormer encoder combined with a CTC loss, similar to Alwajih et al. 2022. We fix the same preprocessing steps for both real and derendered inks. Those include shifting and scaling the inks to start at the origin and have fixed side, time resampling at 20 ms for real inks and time hallucination (assuming the predicted ink tokens are produced at a fixed sampling rate of 20ms, matching the training data) and adding pen up strokes. We apply random rotation within + with probability 0.5 data augmentation account for the small size of the dataset. The points from the ink are encoded into Bezier curves. The input features then processed by 7 transformer attention layers with 8 attention heads.
We train the models on a training set of LAMOnDB only (baseline), derendered IAM train set only and the combination of the two for 17.5k steps with batch size 64. We perform a hyper-parameler search for dropout and learning rate on the same grid for all 3 setups. The best parameters for each model can be found in Table
Table 9: Recognizer parameters found through hyper-parameter search.
Small-i derendered
Small-p
Large-i IAMOnDB +
Small-i IAM
Small-p IAM
Large-i IAM
IAMOnDB +
IAMOnDB +
Parameter
IAMOnDB
LAM
IAM
IAM derendered
derendered
derendered
derendered
derendered
Dropout
0.25 0.005
0.25
0.3
0.3 0.005
0.25
0.25 0.001
0.25
Learning rate
0.001
0.001
0.005
0.001
38

---

Published in Transactions on Machine Learning Research (06/2025)
J Expansion on Failure Cases
While our system exhibits satisfactory performance across various tested inputs, we have identified specific failure patterns that warrant further atlention.
Extreme Stroke Width /Syle Variations As observed in Figures 44 and 46, performance can degrade when processing samples with stroke widths or styles significantly outside the training distribution. While training with our data augmentation improves robustness, it does not fully cover extreme cases. Enhancing robustness to diverse and extreme stylistic variations such through the inclusion of harder examples in the training mixture the use of expert models, remains an area for future work.
Disproportionate Component Scaling While scaling the vision (VIT) component enhances detail preservation (Figure 5, Section 4.5.1), we observed that semantic understanding for tasks requiring recognition can be negatively impacted if the text-ink (mT5) component. is not scaled proportionally. Finding the optimal balance between visual feature extraction and sequence modeling capabilities during scaling an important consideration.
Vision Head Training Trade-offs As detailed in Section 4.6, unfreezing the ViT heads allows capturing finer image details but introduces training instability and increased confusion between text and background noise. Future work could explore more sophisticated fine-tuning strategies, potentially adapting techniques like (Zhai et al., 2022) to balance detail capture with stability and semantic accuracy.
Untrained Languages When presented with handwriting in a language completely absent from its training data (as illustrated with the Korean example for Small-p in Figure 40), the model cannot leverage semantic understanding. these scenarios, defaults to derendering based primarily on visual features and learned writing dynamies, which can lead less accurate stroke reconstruction and an inability to utilize text-conditioned generation effectively. This highlights limitation in applying text-conditioned modes to truly unseen scripts without further adaptation or fine-tuning.
Task Confusion during Decoding Our multi-task training setup, while crucial for injecting priors, can occasionally lead to task confusion during decoding. We observed instances, particularly with the Recognize and Derender Lask, where the model is prompted lo perform semantic understanding might mistakenly prioritize the recognition sub-task and fail to generate the corresponding ink sequence (we use greedy docoding), resulting an empty ink output. This suggests potential ambiguity in the model's intornal representation or decoding process when asked to perform both tasks simultaneously. To mitigate this during inference, implemented fallback mechanism: if the Recognize and Derender task produces empty ink output, automatically re-run inference using the Vanilla Derender mode to ensure ink sequence is generated based solely on the visual input. While effective as practical workaround, addressing the root cause of this task confusion within the model architecture or decoding strategy represents an area for future investigation.
39

---

Published in Transactions on Machine Learning Research (06/2025)
K Expansion on Out-of-Domail Generalization
In addition to the Generalization to Sketches as we described for Figure 7, we analyze more evidence of Out-of-Domain Generalization that we see in selected samples.
Gencralization to untrained scripts and languages As noted in Appendix J, even though the model was trained only on Chinese, English, and Vielnamese, it can still partially derender Korean text. similar extension to German is shown in Figure 34, though both cases the derendering quality is noticeably worse especially when comparing Small-p with Small-//Lage-i Because Small-i/Lagge- was trained on both languages while Small-p not, these examples highlight the model's limited generalization ability to untrained scripts.
Generalization to unseen textures and backgrounds We present additional evidence for generalization on a variety of textures and backgrounds, such as a neon sign on a window ledge in Figure 31 or handwriting written in sand Figure 34.
L Human Evaluation
We provide the text of our evaluation instructions in Table 10 and the screenshot of the interface in Figure 49. Furthermore, we show the samples and corresponding rating (after majority voling) Figure 50.
Table 10: Human evaluation campaign instructions.
Description
Instruction
Task setup
You will be looking at input images of photos of handwriting in variety of styles and the corresponding digital ink tracing performed either by human by one of our derendering models.
Each stroke is rendered in different color and with color gradient (darker + whiter) to reflect the direction.
You will see the derendered ink as well as the derendered ink overlaid on top of the image and can use either to make the judgment. each sample answer the two questions on the right and click submit proceed to the next sample.
Is Reasonable
Is the digital ink a reasonable tracing of the input image?
Yes, it's a good tracing
It's an okay tracing, but has some small errors
It's bad tracing, has some major artifacts One interpretation of the quality of the ink could be like this: If you were to use in a note taking app, would you
keep it as good tracing
need to make some edits: okay tracing
not use it at all: bad tracing
If you need some examples to get a better feeling, you can use the ones below, however there is some ambiguity within the task since we don't want to impose any strict rules but allow for a natural variety of what people find good derendering (it designed be subjective).
Is Real
Could this digital ink have been produced by a human?
Yes
No
In this question go with your intuition, do you think the ink could have been traced by [reasonable] human?
40

---

Published in Transactions on Machine Learning Research (06/2025)
the digital Ink reasonable tracing of the input image?
good tracing
okay tracing.
small
tracing,
artifacts
Input image
Derendered Ink
Overlaid Ink
Could this digital ink have been produced by human?
Yes O No
Eman aw (mniiingg)
Inman EV
(mailing
Annanbing
Imaaee
Immanee
Figure 49: A screenshot of the human evaluation interface.
a good tracing
okay tracing,
small Overlald
bad tracing, major artifacts
Overlaid
Overlaid
Overlaid
Overlald
Overlald
Small-p
2PT
twesorne
Grits
OUT SISSNEE
Overlaid Ink
Overlaid Ink
Overiald Ink
Overiald Ink
Overlaid Ink
Overiald Ink
Small-i
Janly
Bigger
Outsider
MARCH-TO
Mizaiin
Overlaid Ink
Overlaid
Overlald lnk
Overlald Inne
Overleid
Overield
Large-i
WHO
MARCHé
Durm
Apales
Srup
FLORIDA
M
Figure 50: Examples of samples produced by our models and ratings given them by our evaluators.
41

---

Published in Transactions on Machine Learning Research (06/2025)
M
Model Card
We present the model card (Mitchell et al., 2019) of our public release model Small-p in Table 11.
Table 11: Model card of publicly available Small-p.
Model Summary
Model Architecture
multimodal scquence-to-sequene TTann- former (Vaswani et al., 2017) model with mT5 (Xue et al., 2021) encoder-decoder architecture. takes text tokens and ViT (Dosovitskly et al., 2021) dense image embeddings inputs to encoder and autoregressively predicts diacrete text and ink tokens with decoder.
Input(s) Output(s)
pair image and text. Generated digital ink.
Usage
Application
The model for research protolype, and the public version Small-p is planned to be released and available for the public. None.
Known Caveats
System Type
System Description
This is standalone model. None.
Upstream Dependencies
Downstream Dependencies
None.
Implementation Frameworks
Hardware & Software
Hardware: TPU võe.
Software: T5X (Roberts et al., 2023). JAX/Flax (Bradbury et al., 2018), Flaxformer (Heek ct al., 2023)
For implementation details please refer to Ap- pendix G.
Compute Requirements
Please refer Appendix G.
Data Overview
Training Datasets
The VIT encoder (Dusuvitskiy al., 2021) is pre- trained ImageNet-21k (Deng al., 2009), mT5 encoder and decoder initialized from scratch. The entire model is trained on the mixture of publicly available datasets described in Appendix C.
Evaluation Results
Evaluation Methods
Human evaluation (reported in Section 4.5.1) and automated evaluations (reported in Section 4.5.2).
Model Usage Limitations
Sensitive Use
The model is capable of converting images to digital inks. This model should not be used for any of the privacy intruding use cases, e.g. forging handwritings. Reported in Section 4.7. Reported Appendix
Known Limitations
Ethical Considcrations Potential Socictal Consequences
42

---

Published in Transactions on Machine Learning Research (06/2025)
N
Additional Visualizations
Iagott Image
Pl Pantra
THER
RAFEE
Quckily
nhovarive
PROFiTS? PLANT
Mantro
CHER
Creasuty
RAFVE
Buchil
Wing Jerile
PROFITS?
PLAN
Hoogtene
Kenggrert
Hitegtot
Betoriet
Jasplee
W Recogattinn
antra
Treesury
RATHE
Buckil
Who Who Upriie
PROFITS?
PLANT
Clicss
OCt
Teen Terioe
OCRlEF. PROFTTS
faza PLANT
OOR Engor
lattr
reasury
RAFIE
Res
luckill
ihot JariVe
PROFITS?
PLANN
Syudas
gate
52
IS
Shooling
C
toditon
never
IS
Slaasting
22 Nac
Emeggton
2
IS
3
IS Shank
Figure 51: Results of Small-p randomly selected samples from public evaluation datasets.
43

---

Published in Transactions on Machine Learning Research (06/2025)
to fear
CHECK
ARE
E Expla Spllar
Goord
burzt
Lao
O
RAC
to
feaer Explur
Korbic Cepss
to
feeos
to
feaen
AACO
E Explaca
ber
Sorth
hardrr
Mina
hex
get
more
Hapily
SOuth
The
here
hardey
Adiire
hez
9et
HIMR
South
Lhee
LEe
Sdiia
Kea
MOR
Haplyy
South
Konder
0CS
qott
Saplly
South
Nere
Figure 52: Results of Small-i on randomly selected samples from 3 public evaluation datasets.
44

---

Published in Transactions on Machine Learning Research (06/2025)
is
eln
feeling
God
H
the
enjoy
is
chn
feeling
God
the
Vire
SS
cli
Reeling
the
Vine
is
eln
feeling
Loo
the
priel
le ad
weddding
EEEETO
came
wedding
UN
Ahava Snibe
li
GRI
TO
medding
UN
ahyya
ERETO
Came
wedding
aric
redding
abova
Figure 53: Results of Large-i on randomly selected samples from 3 public evaluation datasets.
45

---

Published in Transactions on Machine Learning Research (06/2025)
N.1
More Visualizations Public Evaluation Datasets
/ h poud abooud ty g Htanedospectie 2
Ingus Imtae
GVS
Crepous pours Po alesnd
granging Hard
aofeetive
href Pous try K dl Mojechoe
2
Smatt-p
Poummban the sangin Stared
2
Smalks
he parisaaan teenessallee alooud tru manginn Raare
?
Largei
t
Figure 54: Comparison between Small-i, Small-p, and Large-i on LAM Dataset.
Ingut Image
Surgery
buckles
stream
Med...
started QUoTE
x Beatibul
ONGE
Jujj +
GVS
Surgery
Buckles
stiram
Mec
Started
Bertefell
Pj
Smallpp
Surger
Dockles
stream
Matt...
started aoore
Beatefule
Pj sitei.
Small-4
Surgery
GNDE ENDE
DUcKles
Mot...
starled
QOOTE
Beatiful
Pi stream.
Larget
Sureery
UNDF
Duckles
Mut....
stated
QUOTE
Beatifule
Figure 55: Comparison between Small-i, Small-p, and Large-i on IMGUR5K Dataset.
46

---

Published in Transactions on Machine Learning Research (06/2025)
Toport Imaage
VO
The
CHRISTIAN MIAM
Walter
72-20 SAMPA
DREAM
TRAQ
M
CIRSS MAKTMMIANN MIMM
GVS
Smullyp
The
ARISTAN MIAM
Salter
72-20
SAMPADKAARRA
IRAQ
Smaal-44
The
LIRISTANN MAMMMMMMMM
Talter
12-20SAMM SAMPAUKEAM
ARAQ
Largei
The
CIRISTANN MIAMI Welter 72-20 SAMPA DREAM IRAQ
Figure 56: Comparison between Small-i, Small-p, and Large-i on HierText Dataset.
47

---

Published in Transactions on Machine Learning Research (06/2025)
O Impact Statements
We acknowledge the importance of considering the ethical implications of InkSight. This section outlines potential concerns and benefits to promole responsible use.
0.1 Ethical Considerations
Forgery A key concern is forgery. However, InkSight is designed for ofline-to-online handwriting conversion, not open-ended generation. We verify that proposed models are unable to perform open-ended ink generation. We provide random texts in task prompts of task Derender with Text (in Figure 3) with input images cither empty or containing handwritings that do not match the label. We observe that the generated inks do not match the labels for all 100 cases inspected, signifying that the model is unable to generate the prompted text that not present in the input image. More details in Appendix O.3.
Bias Handwriting varies significantly across demographics, geographic regions, langunges/scripts, age groups, and individuals with disabilities. Models trained primarily on data from dominant groups may exhibit bias, performing poorly for underrepresented styles or scripts. Ongoing efforts should focus on curating more diverse and representative training datasets encompassing a wider range of scripts, writing styles, and demographic groups. Evaluating model performance across different subgroups crucial to identify and mitigate biases. Designing models with fairness metrics in mind an important direction for future research.
O.2 Positive Societal Impacts
Despite the risks, InkSight offers significant potential benefits:
Document Digitization Facilitates the preservation and indexing of personal notes, historical documents, and cultural heritage artifacts beyond their textual form.
Support for Low-Resource Languages By enabling the conversion of existing handwritten docunents in low-resource languages into digital ink, InkSight can help generate valuable training data for online handwriting recognition systems, improving digital tool availability for these communities.
Educational Tools Digital ink conversion can potentially enable new interactive learning and teaching experiences for students and teachers.
Unsupervised Analysis via Geometric Pseudo-Labels Beyond supervised training data generation, InkSight's output offers a unique advantage for unsupervised analysis, particularly in low-resource or challenging scenarios. The generated digital ink sequence itself, even when semantic recognition is imperfect (e.g,, for unseen languages or styles, as illustrated by the Korean example in Figure 40)captures valuable geometric and dynamic properties of the handwriting (shape, curvature, stroke connections). This ink sequence can serve as powerful geometric pseudo-label. Offline handwriting images can then be clustered or compared based on the similarity of these ink pseudo-labels (e.g., using sequence comparison methods like DTW), leveraging purely structural information. This provides language-agnostic alternative to OCR-based analysis, which relies on accurate text recognition and language models, allowing for grouping or analysis based on visual writing patterns even when the text content is unknown or poorly recognized.
0.3 Open-ended Generation
As illustrated in Figure 57, our experiments with Small-p, Small-i, and Large-i models demonstrate that when prompted with text not present in the original image, the models fail to generate accurate corresponding handwritten strokes. We regard this behavior as natural safeguard against potential misuse, such as forgery unauthorized reproduction of handwriting styles. Since the models can only reconstruct handwriting palterns from the input image rather than generating arbitrary new content, they inherently restrict attempts to synthesize fraudulent handwritten content.
48

---

Published in Transactions on Machine Learning Research (06/2025)
Small-p Input text: result
Small-i Input text: result
Large-i Input text: result
16 &
rasie the
Fews as
Input text: carrion
Input text: carrion
Inpul text: carrion
Input December
Input December
Input text: December
Peccesember
JELEEE mm
Input text: used
Input text: used
Input text: used
Weer thedi
Nas Sسدة Se ب
Input text: Carnivora
Input text: Carnivora
Input text: Carnivora
Figure 57: Inability to perform open-ended generation. Prompting Small-p, Small-i and Large-i with text that not presented the image does not produce digital inks that match the textual input.
49


Reviews:
[{"review_id": "JUGsAEpKlv", "reviewer": "Reviewer_2tDr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength:**\n+ **Novel Task and Effective Solution:** The proposed task is novel and holds significant potential for preserving handwritten notes and historical documents while enhancing the machine readability of handwritten text. Although the approach involves a straightforward architectural combination of mT5 and a pretrained ViT (similar to PaLI), it proves to be highly effective, demonstrating exceptional performance in derendering.\n\n+ **Comprehensive Experimental Analysis:** The experiments are comprehensive and strongly support the claims made in the paper. Ablation studies and visual results further validate the design choices, emphasizing the importance of the Derendering with Text task during inference for achieving optimal results. Qualitative analysis highlights that the small variant (small-i) achieves notable performance gains over the baseline GVS, while the best-performing variant (Large-i) exhibits greater robustness to diverse visual conditions and image styles. \n\n+ **Generalization to out-of-domain data:** Additionally, the proposed approach generalises well to out-of-domain simple sketches. Both human and automatic evaluations (character-level F1 score, exact match accuracy) substantiate the model's performance. Furthermore, the paper experimentally demonstrates that combining derendered data with real online handwritten text enhances online handwritten recognition.\n\nOverall, the paper is well-structured, well-presented, and a pleasure to read, and it has a lot of value for the handwritten image analysis community.\n\n\n**Weakness:**\n+ The paper does not provide empirical evidence on whether the proposed derendering approach improves offline handwriting recognition. Since online handwriting recognition is generally considered easier than offline recognition, experiments in this direction would strengthen the work’s impact.\n\n+ The generalization of the proposed model to unknown writers, writing styles, and new languages or scripts is not thoroughly analyzed. A detailed quantitative and qualitative evaluation in this regard would enhance the completeness of the study.\n\n+ While the limitations of the model are acknowledged, the paper offers limited insight into failure case analysis, making it difficult to understand where and why the approach struggles.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces InkSight, a novel and robust method for converting images of handwritten notes into digital ink by leveraging vision-language models, specifically mT5 and a pretrained ViT. To the best of my knowledge, this is the first work addressing offline-to-online derendering. Extensive experiments validate the effectiveness of the proposed system, demonstrating that it successfully reconstructs online handwritten text that is both semantically and geometrically aligned with the input images. This derendering process proves valuable for generating additional training data to enhance handwriting recognition. Moreover, the approach generalizes well to unseen data, handling diverse visual conditions beyond the training distribution.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "+ Consider adding experiments to evaluate whether the proposed derendering approach improves offline handwriting recognition. Since online handwriting recognition is generally easier, demonstrating its impact on offline recognition would further highlight the practical utility of the method.\n\n+ Historical degraded documents such as H-DIBCO can also be considered for evaluation. \n\n+ Include a detailed analysis of the model’s generalisation to unknown writers, diverse writing styles, and new languages or scripts. Providing both quantitative metrics and qualitative examples would strengthen the paper’s claims of robustness.\n\n+ Expand the discussion on failure cases by offering deeper insights into the model’s limitations. Analysing where and why the approach struggles could help understand its boundaries and potential areas for improvement.\n\n+ Add a section on ethical implications and broader impact. (Refer Broader Impact Concerns)", "broader_impact_concerns": "A section on ethical implications and a Broader Impact Statement is essential, given the work’s focus on handwritten text processing. Privacy concerns arise as handwritten documents may contain personally identifiable information, requiring proper data anonymization and compliance with regulations. Bias in handwriting recognition must also be discussed, e.g., across diverse scripts, demographics, and writing styles, including for individuals with disabilities. The system’s potential misuse in forgery, document tampering, or identity theft necessitates safeguards. At the same time, the technology can benefit document digitization and accessibility, particularly for low-resource languages and historical documents. Responsible deployment should include clear licensing. Ensuring open-source accessibility while preventing misuse is critical. Adding these discussions will strengthen the paper’s ethical considerations and promote the responsible adoption of the proposed approach.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "K0jZ577QTA", "reviewer": "Reviewer_iQTZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "As firstly, I'd like to kindly point out some weakness in my mind:\n- I did not see particularly technical contributions to address the mentioned two challenges: Limited Supervised Data and Scalability to Large Images. \n    - The paper indeed proposed some data augmentation strategies in Section 3.2. However, even the data augmentation includes changing of stroke width, but as mentioned in Appendix J (page 36), \"model’s performance deteriorates when processing samples with wider stroke widths or significant variations in stroke width\". This shows the proposed strategy is not that successful even the numbers in ablation studies table 3 are good.\n    - As mentioned in Section 4.7 \"Limitations\", \"the proposed method can derender only several characters or words in one model inference call\", while this is exactly the second challenge mentioned in the beginning. \n\n- The whole paper gives me the feeling that some domain-specific data is being collected and the VLM is being fine-tuned. I am unsure if any of the studied components can benefit other domains.\n\nBesides the above fundamental weaknesses, this paper studied the particular question very deep. Figure 4 is intriguing, and overall the paper is easy to follow. Also, they included failure cases analysis which are my particularly interested. \n\nIn general, I believe this paper can serve as a great reference for future research in this particular direction.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a system capable of transforming arbitrary photos of handwritten notes into digital ink (derendering), enabling conversion from offline handwriting to online handwriting without specialized equipment. Unlike prior methods limited by geometric properties, this approach utilizes learned reading and writing priors, enhancing precision in stroke order and dynamics. Evaluations on the HierText dataset show the model’s effectiveness, with 87% of outputs rated as accurate tracings and 67% indistinguishable from human pen trajectories. Additionally, the method demonstrates robustness beyond its training domain, generalizing to various handwriting styles, sketches, and complex scenarios. The authors publicly release model weights and inference code.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "My main requested changes would be: can you address the above mentioned limitations \n- thoroughly discussing the inference speed given different image condition, using FLOPS or Token/s. Also, how to improve the overall system performance if handle a page full of notes. Even we need multiple times of model inference, we may can still optimize the whole system performance.\n- how to improve the system performance in handling very wider stroke width.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cbKCopNcGC", "reviewer": "Reviewer_NKQU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n1. Extending pen trajectory recovery from individual characters to text in wild images is a non-trivial challenge.\n\n2. The qualitative results are impressive.\nThe proposed model appears to successfully convert handwritten text into digital ink.\n\n3. The training scheme, which utilizes both synthetic and real data, appears to be plausible.\n\n## Weakness\n1. I am unsure about the practical applications of converting handwritten text into digital ink.\nPrevious studies on pen trajectory recovery have primarily focused on applications such as signature verification and calligraphy synthesis.\nHow important is the extension from individual characters to full text in pen trajectory?\n\n2. The evaluation dataset does not appear to contain ground truth digital ink.\nAs a result, the quantitative evaluation relies on the recognition results of the traced pen trajectory.\nIt is unclear whether the quantitative evaluation truly correlates with the quality of the digital ink.\n\n3. The baseline is weak, as the method is not specifically designed for pen trajectory.\nThe suitability of the proposed language model-based approach for this domain remains unclear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores a method for converting handwritten text in images into digital ink, which represents the pen trajectory for multiple characters. \nTo accomplish this, this paper proposes a model called InkSight, which consists of a ViT encoder and an mT5 encoder-decoder.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Major Concerns\n### weakness 1\nThis paper should discuss specific applications that require digital ink rather than just pen trajectory for individual characters.\nIt would be beneficial to demonstrate a concrete application that necessitates digital ink.\n\n### weakness 2\nI recommend creating a small dataset that includes ground truth digital ink and using it to evaluate the proposed method.\n\n## Minor Concerns\n### weakness 3\nI recommend evaluating the proposed method on the datasets for pen trajectory recovery of individual characters.", "broader_impact_concerns": "I have not ethical implications for this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors are recommended to include all the clarifications mentioned during rebuttal and prepare the final paper taking all the points into account.
Decision: Accept (poster)