ProtoSnap: Prototype Alignment For Cuneiform Signs
Authors: Rachel Mikulinsky
Morris Alper
Shai Gordin
Enrique Jiménez
Yoram Cohen
Hadar Averbuch-Elor
Venue: ICLR 2025 Poster (2025)
Keywords: Machine learning for social sciences
Ancient character recognition
generative models
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: An unsupervised approach for recovering the fine-grained internal configuration of cuneiform signs using diffusion-based generative models.

Abstract: The cuneiform writing system served as the medium for transmitting knowledge
in the ancient Near East for a period of over three thousand years. Cuneiform
signs have a complex internal structure which is the subject of expert paleographic
analysis, as variations in sign shapes bear witness to historical developments and
transmission of writing and culture over time. However, prior automated techniques
mostly treat sign types as categorical and do not explicitly model their highly varied
internal configurations. In this work, we present an unsupervised approach for
recovering the fine-grained internal configuration of cuneiform signs by leveraging
powerful generative models and the appearance and structure of prototype font
images as priors. Our approach, ProtoSnap, enforces structural consistency on
matches found with deep image features to estimate the diverse configurations
of cuneiform characters, snapping a skeleton-based template to photographed
cuneiform signs. We provide a new benchmark of expert annotations and evaluate
our method on this task. Our evaluation shows that our approach succeeds in
aligning prototype skeletons to a wide variety of cuneiform signs. Moreover, we
show that conditioning on structures produced by our method allows for generating
synthetic data with correct structural configurations, significantly boosting the
performance of cuneiform sign recognition beyond existing techniques, in particular
over rare signs. Our code, data, and trained models are available at the project page:
https://tau-vailab.github.io/ProtoSnap/

Published as a conference paper at ICLR 2025
PROTOSNAP:
PROTOTYPE ALIGNMENT FOR CUNEIFORM SIGNS
Rachel Mikulinsky
Morris Alper
Shai 2ordin Hadar Averbuch-Elor"
Enrique Jimenez'
Yoram Cohen'
'Tel Aviv University
Ariel University
LMU
*Cornell University
ABSTRACT
The cuneiform writing system served as the medium for transmitting knowledge in the ancient Near East for period of over three thousand years. Cuneiform signs have a complex internal structure which is the subject of expert paleo- graphic analysis, as variations in sign shapes bear witness to historical devel- opments and transmission of writing and culture over time. However, prior automated techniques mostly treat sign types as categorical and do not explic- itly model their highly varied internal configurations. In this work, we present an unsupervised approach for recovering the fine-grained internal configura- tion of cuneiform signs by leveraging powerful generative models and the ap- pearance and structure of prototype font images as priors. Our approach, Pro- toSaa,, enforces structural consistency on matches found with deep image features estimate the diverse configurations of cuneiform characters, snap- ping a skeleton- based template to photographed cuneiform signs. We provide new benchmark of expert annotations and evaluate our method on this task. Our evaluation shows that our approach succeeds in aligning prototype skele- tons wide variety of cuneiform signs. Moreover, show that condition- ing on structures produced by our method allows for generating synthetic data with correct structural configurations. significantly boosting the performance of cuneiform sign recognition beyond existing techniques, in particular over rare signs. Our code, data, and trained models are available at the project page: https://tau-vailab.a www.gggii....oooooiioo io/ProtoSnap/
INTRODUCTION
The earliest forms of decipherable scripts date back to the late 4th millennium BCE, with the inven- tion of the cuneiform writing system in ancient Mesopotamia, which came to be used for a number of historically significant ancient languages such as Sumerian and Akkadian (Radner and Robson 202T). Cuneiform signs have complex internal structures which varied significantly across the eras, cultures, and geographic regions among which cunciform writing was used. The study of these variations is part of field called paleography, which is crucial for understanding the historical context of attested writing (Biggs) 1973 Homburg 2021). However, while compu- tational methods show promise for aiding experts in analyzing cuneiform texts (Bogacz and Mara 2022), they are challenged by the vast variety of complex sign variants and their visual nature: Represented as wedge-shaped imprints in clay tablets which have often sustained physical dam- age, cuneiform appears as shadows on a non- con-uniform clay surface which may even be difficult for human experts to identify under non-optimal lighting conditions (Taylor 2015).
Prior work has focused on digitization of cunciform tablets at coarse resolution, localizing and classifying signs from photographs of whole tablets (Dencker et al.) 2020 Stötzner et al.) 2023a). However, these methods treat sign types as categorical while neglecting sign- inteenal configura- tions of strokes in each character, which provides crucial information for identifying rare signs and distinguishing between sign variants. In this work, we aim to recover the fine-graines internal configuration of real cuneiform signs, given coarse- marse-grained categorical information as input. In
"Equal contribution

---

Published as a conference paper at ICLR 2025
the
Reol Cunciform Tablet
Conciiorim sign Prottstype (font4skoletor)) Our ourtpot Application: hand copy genorattion
ProtoSnap applied to a full tablet by cropping each sign using existing bounding boxes (such those depicted in unique colors), and matching prototypes of the signs (illustrated in the center). Our technique "snaps" the skeletons of the prototypes to the target images depicting real cunciform signs. These aligned results can be used to produce an automatic digital hand copy (right). We also show that our approach can be used to boost performance of cunciform sign recog- nition.
Figure I:
particular, our method is provided with a prototype image and its associated skeleton indicating the canonical structure of sign, and aligns this structure target image depicting corresponding real cuneiform sign. As illustrated in Figure [ our technique is analogous to the laborious hand copies produced by expert Assyriologists; when applied to tablet with existing character-level annotations, this outputs the outlines of signs in the style of the original document. Furthermore, show that these aligned skeletons may be used to boost optical character recognition (OCR) performance, by training a generative model with structural conditioning as detailed below.
To this aim, we present ProtoSnap, an unsupervised approach leveraging deep diffusion features to snap a skeleton-basee prototype to a target cunciform sign, revealing its structure without requiring any labelled examples of real photographed signs. By using fine-tuned generative model as prior on the appearance of cuneiform images, and enforcing global and local consistency, we are able to localize the constituent strokes real cuneiform images. We make use of the key insight that pairwise similarities between regions of the prototype and target images encode information about both coarse global alignment and fine-grained local deviations of each stroke from its canon- ical pose. Our technique distills this information with a mulli-stage process performing global alignment followed by local refinement of stroke positions.
We provide new benchmark of expert-annotated photographed signs for evaluation, and show that our system succeeds at identifying their internal structures, significantly outperforming generic correspondence matching techniques. We also show the downstream utility of ProtoSnap for au- tomatic digitization of cuneiform texts, by using aligned prototype skeletons as a condition for generative model produce structurally-corres synthetic data to train cuneiform OCR. Our results show that this achieves state-of of satee-of-theaatt ttt-hhhrttt art results on cunciform sign recognition, particularly enhancing performance on rare signs where naive synthetic data generation struggles to produce instances of the correct sign.
Stated explicitly, our key contribution are:
ProtoSnap, an unsupervised prototype alignment method capturing the structure of pho- tographed cuneiform signs.
A novel benchmark with expert annotations, and results showing that our method outper- forms generic correspondence methods this task.
State-of- of-te-of-the-ar- OCR results for cunciform when using synthetic data produced using our method's alignments for conditional generation.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORKS
Machine learning (ML) for cunciform. Ancient texts provide a window into our history, but their decipherment and interpretation require painstaking work and expert knowledge of esoteric languages, complex writing systems, and historical context which serve as barrier to their trans- lation and analysis on scale. Due to the high societal value of these tasks and the scarcity of expert knowledge and time, machine learning promises to provide an invaluable aid for understanding the ancient world. In the context of a number of works on MI applied to diverse ancient inscrip- tions (Hassner et al.] [2013a] Assael et al., 2019] Yin et al., 2019 Huang et al. | [uo et al.) 2021 Hayon al. 2024), the cunciform script poses particular challenges. These include the na- ture of the physical writing media (indentations in textured and often damaged clay under various lighting conditions), and the diverse nature of cuneiform signs which changed over thousands of
years of use in vast geographical regions.
Various approaches have been applied to modeling cuneiform signs for the purpose of down-
stream tasks such as optical character recognition. Some works apply recognition directly to image data (Bogacz et al. 2017 Dencker et al. [2200] Stötzner et al.) 2023a), while others have treated cuneiform as structured graphs to recognize signs based on their internal configurations (Kriege al., 2018 Chen al. 2024). A handful of works have explicitly modeled cuneiform signs as
compositions of wedges, though these mainly focus on segmentation from 3D meshes or localiz- ing strokes on the bounding box level (Bogacz and Mara 2022, Stötzner al. [023b] Hamplová et al. 2024). By contrast, our approach provides pixel-alignes skeleton indicating the relative positions, sizes, and orientations of the strokes, and unlike prior works we operate exclusively on 2D photographs of cuneiform signs and without any strong supervision.
Skeleton-based template alignment. Our approach to inferring the configuration of cuneiform images as compositions of strokes by aligning a skeleton- based template bears partial similarity to various existing methods that typically operate over generic natural images.
Our method resembles template matching in that we use a template image (our font prototype) and search for relevant regions in the target image. While earlier approaches used naive comparisons of image intensities low-level features when sliding the template across the target image (Ben-
Arie and Rao 1993, Cole et al. 2004) Kim et af., (201T. this is not robust to complex relations between the template and target. To handle these challenges, more recent works have adopted
deep image features along with modeling complex non-rigid deformations (Oron et al.) 2017]
Talmi et al., 2017 Cheng al. 2019 Gao and Spratling 2022). Our method similarly searches for matches to our template using deep features and allowing for deformations, although we differ from conventional template matching in explicitly using the skeletonized graph structure of the template and matching each of its constituent strokes separately.
We also note similarity to pose estimation methods, as we infer the structure of sign by localiz- ing keypoints. Pose estimation methods align graph of keypoints to an image, most commonly applied to a single category such as humans (Fang et al.) [022] Zheng et al., (023), animals (Li and Leel | Yang et al. 2022), or vehicles (Reddy et al.) [0188] López et al. 2019), where the
same fixed graph applies to all instances. However, in our setting the number and connectivity of keypoints depends on the sign type under consideration. In this respect our method resembles
category-agnostic pose estimation methods (Xu et al., 2022 Hirschorn and Avidan 2023), though our input also includes template image rather than only using an abstract graph.
In the context of images depicting text, a number of works address the problem of text spotting,
which searches for matches to given visual text representation in images (He et al. 2018 Huang et 20222 Ye et al. 2023). This may include alignment or dewarping of detected text, but does not typically explicitly leverage the internal shape of symbols as in our method. We also note several works performing transcript alignment using dense correspondence methods, which align visual text but do not explicitly handle suuracter-interaal structure (Hassner al. 2013b 2016).
3 THE CUNEIFORM WRITING SYSTEM
Cuneiform, one of the earliest known writing systems, was a logo-syllabic script indicating both units of sound and meaning with signs. Unlike the Latin alphabet which uses less than thirty basic
3

---

Published as a conference paper at ICLR 2025
Prototype Sign (Font Skeleton)
Target Image
Bestt-buddy Correspondences for Gloval Aligywmnn
Local Refinement
Figure 2: Method Overview. Given prototype image with annotated skeleton and target image of a real cuneiform sign, ProtoSnap first extracts best-buddy correspondences from deep diffusion features (extracted with fine-tuned SID-ff model), globally aligning the target image to the skeleton the prototype. Our method then "snaps" the individual strokes into place with a local refinement stage by optimizing a per-stroke transform.
letters indicate sounds, cuneiform signs numbered upwards of one thousand unique types, which varied dramatically in their realizations across eras, languages, and geographical regions (Walker) 1987). See, for instance, the right-most examples in Figure[5] both variants of the same sign AN from different eras. Scholars have attempted to collate lists of known signs and their vari- ants (Labat and Malbran-Labal 1988 Borger 2003), and canonical representations of the most common variants for different time periods have been encoded in digital fonts.
Cuneiform was written on clay tablets by scribes using a stylus to create wedge-shaped impres- sions, also known as strokes, which combine to form signs. Scribes used styli with triangular edges to create impressions on the moist clay in three possible directions: horizontal, vertical, or oblique (Cammarosano) 2014 Cammarosano et al. 2014). Various encoding schemes proposed
to encode these strokes digitally (Bogacz and Mara 2022): : we adopt the four-keypoint scheme,
treating all strokes being composed of triangular head indicated with three keypoints and fourth keypoint indicating the stroke's tail; see Figure 2 (second to left) for an example. We refer to the graph of these keypoints and the edges connecting them as a sign's skeleton. Our method's prototype input consists of both rasterized font image along with its skeleton encoding the con- figuration of its strokes; collect these skeletons via manual annotation described in the ap- pendix.
METHOD
Given a prototype consisting of a font image annotated with a skeleton composed of strokes and a target image of real cuneiform sign, system aligns the prototype skeleton match the sign structure in the target image. An overview of this process is shown in Figure 2
Our system consists of three steps: First, calculate semantically- which encodes the pairwise feature similarities of each pair of regions in the two images. This similarity volume cuneiform images. We then calculate a global alignment between the prototype font image and the target image, using best-buddies sparse correspondences, extracted from the similarity volume, as robust signal to fit this alignment. Finally, we perform local refinement to optimize the relative positions of each stroke and their internal configurations, when deviation from the canonical con- figuration is necessary. We describe each step in turn below, with further implementation details provided in the appendix.
4D similarity volume
diffusion features to encapsulate the complex geometry and semanties of
4.1 SEMANTICALLY-ADAPII 4D SIMILARITY VOLUME
Our system is based on the guiding assumption that local similarities between regions in the two images can be used to compute a structurally-consise ture and the target cuneiform sign. As backbone for computing meaningful similarity scores, we use diffusion features (DIFT) Tang et al. (202))), )), which leverage the strong geometric and semantic understanding of generative text-to- toxt-to-image model to represent image features for discriminative tasks. These features are calculated as intermediate activations the model's denoising compo- nent, applied the input image with added random noise. However, standard generative models
matching between the prototype sign struc-
4

---

Published as a conference paper at ICLR 2025
SD- TF
Target Image
(.) H
Diffusion Features
H
SD-IT
4D Similarity Volumo
2D slice i
Figure 3: DIFT-Based Best-Buddies Correspondences. Noised images passed through fine-tuned denoising diffusion model SD-Tf to extract deep Diffusion Features (DIFT), used to calculate the 4D similarity volume S. For each region (i.j) in the target image. we examine the 2D slice Sii, j, J. and determine the indices (k: f) which maximize its value. Symmetrically, for each region (ks, e) the prototype we find the corresponding region in the target by maximizing the 2D slice S k, f]. If these two regions correspond to each other, they are identified as best buldies.
are typically pretrained on natural images from the Internet, with cuneiform scans being out-of- distribution. We thus fine-tune the generative model Stable Diffusion (Rombach al., 2022) on cuneiform image scans, which we indicate as SD-ff, using the cuneiform sign name as its accompa- nying text prompt. use this as our vision backbone for calculating DIFT features.
We apply DIFT with SD-If to our prototype and target images to obtain feature vector maps F(e) (r(P)) Jay and F(t) (C()) )i. respectively of unit-normalized feature vectors. Each feature map is C H x W tensor, where C' is the number of feature dimensions and H and W are the spa- tial dimension of the feature map (lower-reoolution than the original image, as each feature vector has larger receptive field in the image). Using these features, we calculate the four-dimensional similarity volume (r(2) E.e.e.... This N W И W tensor, visualized in Figure contains the pairwise cosine similarities between features encoding patches of the prototype and target images.
4.2 GLOBAL ALIGNMENT FROM BEST-BUDDIES CORRESPONDENCES
While S provides a strong similarity measure, does not encode geometric constraints on the over- all matching between the images. For instance, multiple regions in image may all have high similarity scores with a single region in the other image, but mapping them all to the same target region will result in degenerate solution. To robustly identify sparse set of best-matching pairs of regions, we follow prior work (Oron et al., 2017 Drory et al. (2020) by identifying best buddies, defined as pairs of patches in the two images which are mutual nearest-neighhors ac- cording to their similarities scores in S. Formally, this is defined as pairs of coordinates (i,j) and (ks, E) such that (i,j) arg max,,; S.,,k. and (k: E) arg max St.jkke See Figure for an illustration.
Using these sparse correspondences, we fit an affine transformation with least squares estimation, defining global alignment G of the prototype the target image. This transformation allows for basic deformations while preserving the overall structure of the prototype. We learn the parameters
911
912
913
G
=
J21
922
J23
0
permitting scaling, rotation, and shear (f1,, 912, (or.g21.900 922) as well as translation (gia. 923). This is applied in projective space p2, i.e. mapping point v = [r. 1)" E p2 to the point v' Gv [a',, v/, 1
5

---

Published as a conference paper at ICLR 2025
L sim
(k,, 8)
Lsal
S
Salieneyy Map
Target Immage
Prototype
Figure 4: Local Refinement via Skeleton-Based Optimization, To adjust the positioning of indi- vidual strokes in sign, our global alignment is followed by local refinement stage which learns transformations for each stroke. The loss function encourages positioning on salient regions (Laal) while semantically matching the corresponding regions in the prototype image, as measured by feature similarity (Csim). For each stroke (exemplified by the stroke red above), these objectives are calculated along points sampled from the skeleton (red dots above). The loss also includes a regularization term (Creg) preventing excessive deviation from the global transformation.
To handle outliers in a robust manner, we perform the fitting with RANSAC. For further robust- ness, incorporate prior inlier points being spread over the majority of the area of prototype and target images, by performing this procedure multiple times (including the stochastic calcula- tion of DIFT features and correspondences from Section (41) and selecting the result with the best spread of inlier points across the relevant regions in the images, following Hassner et al. (2014) and as further described in the appendix.
4.3 LOCAL REFINEMENT VIA SKELETON-BASED OPTIMIZATION
Our global alignment procedure can roughly align the prototype to the target image. However, as the target image was written by hand, each stroke's location may deviate from the canonical rela- tive position given by the prototype font image. Therefore, we introduce a local refinement stage, illustrated Figure which allows each stroke to move from the canonical prototype structure and "snap" into place, while avoiding excessive deviations from the global structure representing the sign's 's identity. We model each stroke's 's deviation from the global alignment as projective transformation, allowing for a higher degree of deformation than the affine global transformation. The local transformation of stroke is parameterized by the matrix
(
Pi1
P12
P13
p(o) = I+
(2)
(1)
P21
Pa2
P23
(1)
P31
P32
0
where / is the 3 x 3 identity matrix. These are applied on top of the global transformation, resulting in per-stroke transformation of the form p. As projective transformation, this maps a point
[z, y, 1)" I2 to Gv v/'.v/22] zero (i.e. each local transformation is initialized as the identity).
(c) [x' /2/, 3//2', 11" The Pi are all initialized to
We optimize the parameters pjkk via gradient descent with the loss function
L Asim Ceim + Asat Caal + Areg
where Agim, Asal, Areg are constant weights. We proceed to define each loss term.
Featural Similarity Loss Csim. To encourage semantically-comee corrrct positioning of strokes, we define a loss to maximize feature similarities between matching points on the prototype and target images under the local transformation. Using the similarity volume from 4.1 we sample points along the lines connecting skeleton keypoints in the prototype image, calculate their correspond- ing points under the current global and local transformations, and evaluate their similarity via with temperature- weighted softmax applied to each slice of S over the prototype image. This uses differentiable grid sampling to interpolate values of S, as has a lower spatial resolution in comparison the images.
Saliency Map Loss Ceat- To encourage the strokes to cover salient regions (i.e. areas which ap- pear contain writing), calculate saliency map over the target image and use it to define a

---

Published as a conference paper at ICLR 2025
Prototype
If
Input
*Refinement Global
Figure 5: Qualitative alignment results, aligning the prototypes (first row) to target cuneiform images (second row). We demonstrate the results after performing global alignment (third row), and the final result after local refinement (fourth row). As illustrated above, the global alignment stage provides a coarse placement of the prototype template, while the refinement stage allows each stroke to slightly diverge from the original prototype, resulting in more accurate alignments.
loss. The saliency map is calculated using the 4D similarity tensor S from Section 4.1 for each region in the target image, we calculate the difference between mean similarities to foreground (black) and background (white) regions in the prototype font image, and post-process by applying adaptive histogram equalization, scaling and setting low values to zero. This yields an approximate segmentation map of the cuneiform sign visible in the scanned image. The loss L is calculated as the mean value of the map sampled at points along the transformed skeleton, selecting points and using temperature scaled differentiable grid sampling as in Section 4.1
Regularization Loss Creg. To avoid invalid solutions that over-optimize the previous objectives, add regularization term that penalizes excessive deformations and solutions that stray from the boundaries of the image. This is defined as Freg = LL1 + Loos LI regularization loss is given by
1 N
LL
pp(
Ill21
N
jk
where N is the number of strokes and / is the 3 x 3 identity matrix. This penalizes local transforma- tions which greatly deviate from the identity.
The out-of-bounds loss Loob is defined as zero if all transformed keypoints are within the image boundaries, otherwise as the maximum absolute difference between each transformed coordinate and the image bounds. This models the soft constraint that all keypoints must lie within the image, handling edge cases where the global transformation pushes part of stroke outside the image bounds.
EXPERIMENTS
We conduct various experiments, evaluating our method both directly on our benchmark expert- annotated real photographed signs (Section 5.1) and by testing it downstream OCR bench- mark (Section 5.2p. We also present qualitative results (Figure 5 and appendix), which further highlight the complexity and unique challenges of the task and setting addressed in our work. Fi- nally, we discuss limitations of our approach (Section 5.3). 3. Further implementation details and results provided in the appendix.

---

Published as a conference paper at ICLR 2025
Method
FI @0
FI @
FI @40
SIFT Lowe, 1999
RANSAC
2.56%
3.78%
5.01%
DINOV2 (Quab et al.
2024)
12.33%
21.88%
31.04%
DINO RANSAC
16.12%
28.19%
37.93%
DIFT (Tang et al. 2023)
16.14%
25.80%
33.79%
DIFT RANSAC
13.13%
21.96%
30.15%
Ours (w/o refinement)
21.31 %
37.08%
50.13%
Ours (full)
27.14%
42.09%
52.43%
Table I: Alignment evaluation, measuring keypoint localization at various distance thresholds. We compare against several generic correspondence matching baselines, including geometry- based method (SIFT) and deep feature-based methods (DINOv2, DIFT), As illustrated above, our method significantly outperforms these baselines. Furthermore, our local refinement stage provides a performance boost beyond learning simply global transform.
5.1 ALIGNMENT EVALUATION
To evaluate the quality of alignment, we curate test set of ground-truth (GT) alignments from manual annotations by expert archeologists. Provided with photographs of cuneiform signs from the eBL dataset Cobanoglu al. (2024), the experts annotated the position of strokes by indicating groups of four keypoints. Images were marked for exclusion from the test set if they were poor quality or show sign variants differing from the corresponding prototype font images. total, test set contains 272 images with expert annotations, covering 25 different sign types, which were not seen in training of SD-IF. A breakdown of this test set and more details on our annotation procedure are provided in the appendix.
To quantify performance on this benchmark, we compare the predicted and GT positions of stroke keypoints; given fixed distance threshold, consider predicted keypoint true positive prediction it is within the given threshold a GT keypoint. We report F1 scores for several distance thresholds (along with precision and recall in the appendix).
In Table 1 we compare our solution to generic correspondence matching techniques. We com- pare against traditional geometry-basse baseline that computes SIFT (Lowe 1999) features and aligns the images using RANSAC. We also compare against two deep feature- based techniques (DINOv2 (Qquab al. 2024), DIFT Tang al. (023)), applied by assigning each keypoint the corresponding point of maximum similarity in the target image.
We see that ProtoSnap significantly outperforms the baseline methods at localization, our so- lution explicitly considers the complex structure of the sign. We also report performance of our approach without local refinement (i.e performing the first step of global alignment alone); we see that the final step of local optimization indeed achieves better alignment as reflected in our metrics. This may also be seen visually in Figure [ illustrating the overall global alignment and more precise results yielded by local refinement. In the appendix, we show an ablation study of the different components of our system (such as using fine-tuned SD-If and each of our loss terms). These results provide further motivation for the design of our full system. We also provide addi- tional qualitative results.
To further evaluate the practical usability and effectiveness of ProtoSnap, conducted a survey of 12 assyriologists, finding that users are approximately twice as likely to prefer scans with our aligned skeleton overlaid an overlay without our alignment applied. Further details regarding this user study can be found in the appendix.
5.2 LEARNING OCR FROM ALIGNED DATA
We show the downstream benefit of our approach by using ProtoSnap alignments to produce structurally-divee synthetic training data for an OCR system. As cuneiform signs are highly diverse, with number and arrangement of strokes varying significantly between eras, regions, and individuals, a model trained to produce synthetic data conditioned on sign type alone may strug- gle depict the correct configurations of signs, particularly rare signs with few attestations
8

---

Published as a conference paper at ICLR 2025
Accuracy
Balanced Accuracy
Training Data
All
Rare
All
Rare
CSDD* (Dencker al. 2020)
58.43%
25.84%
34.57%
16.45%
+SD-I data
61.56%
38.56%
39.02%
31.13%
+CN-TF data
64.14%
53.17%
43.57%
39.98% %
Table 2:
Sign classification performance when training on previously collected real data (CSDD), and with added data generated using our fine-tuned diffusion model (SD-TF) ControlNet trained using aligmments from ProtoSnap (CN-IT). Our solution demonstrates improved OCR results, with structural augmentations showing increased performance relative to direct unconditional image generation, especially on rare signs (signs that have less than 50 occurrences in the real train set). denotes our reproduced model, as further described in the appendix.
in the existing data. Furthermore, conditioning on sign type does not allow to specify the exact variant of the sign, resulting in generation of the most prevalent variant in the data. As we show, conditioning
skeleton-based structures provides control over the generated fine-grained struc- tures yielding synthetic data which better captures the real configurations. To demonstrate the benefit of our approach for the downstream task of OCR, we align prototype skeletons to set of cuneiform images from the eBL dataset (Cobanoglu et al. 2024). and fine-tune ControlNet (Zhang condition, instead of
al. (023) to generate new cuneiform images using such skeletons text prompt. This model, denoted as CN-TT, allows us to produce new cuneiform sign images with any input structure, not limited to a list of predefined categories or specific structural configura- tions. We then use this model to generate synthetic training data, as shown in Figure 0 Further details about the model training are provided in the appendix.
We examine the benefit of this generated data for learning cuneiform sign classification. Dencker al.] (2020) report sign classification performance on the CSDD dataset when training 18 (He et al., 2016) model with supervision from the CSDD training set alone. We compare this to augmenting the training set with the I-g-ee-geeertte synthetic data described above. As an additional baseline, we also compare with using SD-TF (see Section 4.1) generations for augment- ing sign types which uses categorical sign names as a textual condition rather than structural con- ditions. We report classification accuracy following Dencker et al. (2020); this dataset is highly imbalanced, we also report balanced accuracy and performance on rare sign types (signs with less than 50 occurrences in the real training set). Results are summarized in Table 2 As seen there, aug- menting the CSDD dataset with synthetic data significantly improves classification performance, with cructurally-conddtionee data from CN-If providing significant boost over synthetic data from SD-IF. These results reflect that structurally-controll eration of the correct sign variant, while SD-If struggles to produce correct configurations from the sign category alone, as seen in Figure 6
generation with CN-If guarantees gen-
5.3 LIMITATIONS
We note various limitations of our work (visualized in Figure 7 Our alignment procedure requires a canonical sign image and will still fail if the scan displays a structurally different variant of the sign in question. Additionally, our method may fail under extreme deformations, or on low- quality tablets or scans which cannot be feasibly interpreted. Future work might investigate how to calculate confidence measure detect such failure cases.
6 CONCLUSION
We have proposed the ProtoSnap method for estimating the internal structure of cuneiform signs without any direct supervision, by harnessing pairwise comparisons of deep image features in re- gions of photographed cuneiform images and skeleton-based prototypes. We have curated expert annotations to provide a new benchmark for this task, and have shown that our method signifi- techniques. Beyond its direct application for
cantly outperforms generic correspondence-basee palcographic analysis, we have also shown our method's 's utility for the downstream task of OCR, by using aligned skeletons for conditional synthetic data generation.

---

Published as a conference paper at ICLR 2025
Name
LU
IM
BA
E2
SD-ff
Prototype
H
CN-I
(1) Unseen signs
(2) Signs with multiple variants
synthetic data (denoted as CN-If above), in comparison to text-conditional generation (denoted as SD-IF), using two differ- ent scenarios: (1) Text-conditional generation cannot succeed in generating signs unseen during training, while our method can correctly adhere to the structural conditioning provided as input, even for rare or unseen signs. (2) text-conditional model often generates the sign variation most prevalent during training (e.g., the variants the right sides above), while our method can gener- ate different variants, yielding a more diverse synthetic set for training downstream models.
Figure 6: We demonstrate the benefit of producing cructurally-control/
Prototype
Input
Protonnap
Figure 7: Limitations of our method, illustrating examples with significant deformations from the prototype skeleton (left) and structurally different sign variants (middle) and corrupt sign image (right). We visualize correctly alignnd strokes in green, and misaligned strokes in red.
We foresee range of applications and possible extensions of our work. Our method represents a step towards paleographic analysis on scale, tracking allographs (sign variants) for each sign produced by scribes between time periods, cities, archives, and personal handwriting styles. It could also be used to automate the production of hand copies, currently produced manually by experts to illustrate tablet contents their original context. While we have focused on the case of single sign scans, we foresee future work extending these results to lines of text, as our method could be incorporated into pipeline including text detection and localization of individual signs applied either to images or directly to 3D scans of inscriptions. Finally, our per-stroke optimiza- tion process is designed for cunciform signs composed wedge shapes parametrized by four keypoints, future work might extend this to other ancient scripts with different geometric config- urations for example, by using more flexible primitives such as Bezier splines to model curved lines found in ancient Chinese oracle bone inscriptions. We hope that our work will spur future research on using the internal structure of glyphs in complex scripts such as cunciform to advance downstream tasks.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Karen Radner and Eleanor Robson, editors. The oxford handbook of cuneiform culture. Oxford
University Press, Oxford, 2011. doi: 10.10003woffftthhh/7770999999999 000100001.....1 URL
Michael P. Streck, editor. Sprachen des Alten Orients. wbg Academic, Darmstadt, 4., überarbeitete und aktualisierte auflage edition, 2021.
Robert D Biggs. On regional cuneiform handwritings in third millennium mesopotamia. Orientalia, 42:39-46, 1973.
Timo Homburg. Palcocodage-cchhnccinn machine-readable cuneiform descriptions using muchine-readable paleographic encoding. Digital Scholarship in the Humanities, 36 (Supplement (Supplement_2;iil27--ii 2):i127--i154, 2021.
Bartosz Bogacz and Hubert Mara. Digital assyriology- advances in visual cuneiform analysis. Journal on Computing and Cultural Heritage (JOCCH), 15(2):1-22, 2022.
Jon Taylor. Wedge Order in Cuneiform: Preliminary Survey, page 1-30. PeWe-Verlag, Gladbeck, 2015.
Tobias Dencker, Pablo Klinkisch, Stefan M Maul, and Björn Ommer. Deep learning of cuneiform sign detection with weak supervision using transliteration alignment. Plos one, 15(12):e0243039 2020.
Ersst Stötzner, Timo Homburg, and Hubert Mara. Can based cuneiform sign detection learned from annotated 3d renderings and mapped photographs with illumination augmentation. Proceedings of the IEEF/CVF International Conference Computer Vision, pages 1680-1688, 2023a.
Hassner, Malte Rehbein, Peter A. Stokes, and Lior Wolf. Computation and Palaeography: Potentials and Limits (Dagstuhl Perspectives Workshop 12382). Dagstuhl Reports, 2(9):1841199 2013a. ISSN 2192-5283, doi: 10.4230/DagRRp.229.844. URL hatps:: /dropsrpppppee/// dagstuhl de/entities /wwp:///cccccccooocco 10. 4234 4230/DagRee.
Yannis Assacl, Thea Sommerschield, and Jonathan Prag. Restoring ancient text using deep learning: study Greek epigraphy. In Empirical Methods in Natural Language Processing, pages 6369-6376, 2019.
Xusen Yin, Nada Aldarrab, Beáta Megyesi, and Kevin Knight. Decipherment of historical manuseript images. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 78-85, IEEE, 2019.
Shuangping Huang, Haobin Wang, Yongge Liu, Xiaosong Shi, and Lianwen Jin. Obc306: A large- scale oracle bone character recognition dataset. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 681-688 IEEE, 2019.
Jiaming Luo, Frederik Hartmann, Enrico Santus, Regina Barzilay, and Yuan Cao. Deciphering under- segmented ancient scripts using phonetic prior. Transactions of the Association for Computational Linguistics, 9:69 11, 2021.
Offry Hayon, Stefan Münger, Ilan Shimshoni, and Ayellet Tal. Arcaid: Analysis of archaeological artifacts using drawings. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 7264-7274, 2024.
Bartosz Bogacz, Maximilian Klingmann, and Hubert Mara. Automating transliteration of cuneiform from parallel lines with sparse data. In 2017 14th IAPR International Conference Document Analysis and Recognition (ICDAR), volume 1, pages 1-0020 IEEE, 2017.
Nils M Kriege, Matthias Fey, Denis Fisseler, Petra Mutzel, and Frank Weichert. Recognizing cunciform signs using graph based methods. International Workshop on Cost-Sensitive Learning, pages 31-44. PMLR, 2018
11

---

Published as a conference paper at ICLR 2025
Yizhou Chen, Anxiang Zeng, Qingtao Yu, Kerui Zhang, Cao Yuanpeng, Kangle Wu, Guangda Huzhang, Han Yu, and Zhiming Zhou. Recurrent temporal revision graph networks. Advances in Neural Information Processing Systems, 36, 2024.
Ernst Stötzner, Timo Homburg, Jan Philipp Bullenkamp, and Hubert Mara. R-enn based polygonal wedge detection learned from annotated 3d renderings and mapped photographs of open data cuneiform tablets. 2023b.
Adéla Hamplová, Avital Romach, Josef Pavlíčck, Amoöš Veselý, Martin Čejka, David Franc, and Shai Gordin. Cunciform stroke recogmition and vectorization in 2d images. Digital Humanities Quarterly, 18(1), 2024. URL httptttpp////ttt di gigittat human t ies orrg/hh//ol/////
11/000733000033..
Jezekiel Ben-Arie and K Raghunath Rao. A novel approach for template matching by nonorthogonal image expansion. IEEE Transactions on Circuits and Systems for Video Technology. 3(1):71-84, 1993.
Luke Cole, David Austin, Lance Cole, et al. Visual object recognition using template matching. In Australian conference on robotics and automation, 2004.
Hae Yong Kim et al. Ciratefi: An rst-invariant template matching with extension to color images. Integrated Computer-Aided Engineering, 18(1):75-90, 2011.
Shaul Oron, Tali Dekel, Tianfan Xue, William Freeman, and Shai Avidan. Best-buddies similar- ity-robust template matching using mutual nearest neighbors. IEEE transactions on pattern analysis and machine intelligence, 1098):1799118 2017.
Itamar Talmi, Roey Mechrez, and Lihi Zelnik-Manor. Template matching with deformable diversity similarity. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 175-183, 2017.
Jiaxin Cheng, Yue Wu, Wael AbdAlmagced, and Premkumar Natarajan. Qatm: Quality-awarc template matching for deep learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1553-11566 2019.
Bo Gao and Michael W Spratling. Robust template matching via hierarchical convolutional features from a shape biased cnn. The International Conference on Image, Vision and Intelligent Systems (ICIVIS 202/), pages 333-344, Springer, 2022.
Hao-Shu Fang, Jiefeng Li, Hongyang Tang, Chao Xu, Haoyi Zhu, Yuliang Xiu, Yong-Lu Li, and Cewu Lu. Alphapose: Whole body regional multi-person person pose estimation and tracking in real- time. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Ce Zheng, Wenhan Wu, Chen Chen, Taojiannan Yang, Sijie Zhu, Ju Shen, Nasser Kehtarnavaz, and Mubarak Shah. Deep learning- human pose estimation: A survey. ACM Computing Surveys, 56(1):16331 2023.
Chen Li and Gim Hee Lee. From synthetic to real: Unsupervised domain adaptation for animal pose estimation. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 1482-1491, 2021.
Yuxiang Yang, Junjie Yang. Yufei Xu, Jing Zhang, Long Lan, and Dacheng Tao. Apt-36k: A large- scale benchmark for animal pose estimation and tracking. Advances in Neural Information Processing Systems, 35: 35:17301-17313, 2022.
N Dinesh Reddy, Minh Vo, and Srinivasa Narasimhan. Carfusion: Combining point tracking and part detection for dynamic 3d reconstruction of vehicles. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1906-1915, 2018.
Javier García López, Antonio Agudo, and Francese Moreno-Noguer. Vehicle pose estimation via regression of semantic points of interest. 2019 11th International Symposium on Image and Signal Processing and Analysis (ISPA), pages 209-214. IEEF, 2019.
12

---

Published as a conference paper at ICLR 2025
Lumin Xu, Sheng Jin, Wang Zeng. Wentao Liu, Chen Qian, Wanli Ouyang, Ping Luo, and Xiao- gang Wang. Pose for everything: Towards category category-agnostic agnostic pose estimation. In European conference on computer vision, pages 398-416. Springer, 2022.
Or Hirschorn and Shai Avidan. Pose anything: A graph-based approach for category-agnosti pose estimation. arXiv preprint arXiv:2311.1788 2023.
Tong He, Zhi Tian, Weilin Huang, Chunhua Shen, Yu Qiao, and Changming Sun. An end-to-end textspotter with explicit alignment and attention. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5020-500000 10000-509,,000 2018.
Mingxin Huang, Yuliang Liu, Zhenghao Peng. Chongyu Liu, Dahua Lin, Shenggao Zhu, Nicholas Yuan, Kai Ding, and Lianwen Jin. Swintextspotterr Scene text spotting via better synergy between text detection and text recognition. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4593-4603, 2022.
Maoyuan Ye, Jing Zhang, Shanshan Zhao, Juhua Liu, Tongliang Liu, Bo Du, and Dacheng Tao. Deepsolo: Let transformer decoder with explicit points solo for text spotting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19348-19357, 2023.
Tal Hassner, Lior Wolf, and Nachum Dershowitz. Ocr-free transcript alignment. In 2013 12th International Conference on Document Analysis and Recognition, pages 1310-1314, IEEE, 2013b.
Tal Hassner, Lior Wolf, Nachum Dershowitz, Gil Sadeh, and Daniel Stökl Ben-Ezra. Dense cor- respondences and ancient texts. Dense Image Correspondences for Computer Vision, pages 279-295, 2016.
Christopher Bromhead Fleming Walker. Cuneiform, volume 3. Univ of California Press, 1987.
R. Labat and F. Malbran-Labat. Manuel dl E'iiraphii Akkadienne. Paris, sixth edition edition, 1988.
R. Borger. Mesopotamisches Zeichenlexicon. Alter Orient Und Altes Testament: Veröffentlchun- gen Zur Kultur Und Geschichte Des Alten Orients Und Des Alten Testaments. Ugarit Verlag, Münster, 2003.
Michele Cammarosano. The cuneiform stylus. Mesopotamia: Rivista di Archeologia, Epigrafia Storia Orientale Antica, 69:53-90, 2014.
Michele Cammarosano, Gerfrid G. W. Müller, Denis Fisseler, and Frank Weichert. Schriftmetrologie des keils: Dreidimensionale analyse von keileindrücken und handschriften. Die Welt des Orients, 44(1):2-36, 2014.
Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. Advances in Neural Information Processing Systems, 36: 1363-1389, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695 2022.
Amnon Drory, Tal Shomer, Shai Avidan, and Raja Giryes. Best buddies registration for point clouds. In Proceedings of the Asian Conference on Computer Vision, 2020.
Tal Hassner, Liav Assif, and Lior Wolf. When standard ransac is not enough: cross-media visual matching with hypothesis relevancy. Machine Vision and Applications, 25:971 119971-988 1933, 2014.
Yunus Cobanoglu, Luis Sáenz, Tlya Khait, and Enrique Jiménez. Sign detection for cuneiform tablets. it Information Technology, 66 66(1))68-3666 2024. doi:
URL
-024 0028
David G Lowe. Object recognition from local scale-invariant features. In Proceedings of the seventh
IEEE international conference computer vision, volume 2, pages 1 1150-1157. leee, 1999.
13

---

Published as a conference paper at ICLR 2025
Maxime Oquab, Timothée Dareet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po- Yao Huang. Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2024.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Eugen Rusakov, Turna Somel, Gernot Fink, and Gerfrid G.W. Müller. Towards query-by- expression retrieval of cuneiform signs. In 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR), pages 43-48, 2020. doi: 10.1 (OW//C/FF222222200000000
APPENDIX
A EXPERIMENTAL DETAILS
Below we provide additional experimental details. Our code also provided (zipped the supple- mentary material). For all experiments described below, single A5000 GPU was used. Running the method on a single image takes about minute.
A.1 IMAGE AND FONT INFORMATION
For all of our tests,
RGB images with resolution 512 512, resizing images needed.
Font images are rendered from the Santakku and SantakkuM fonts, designed by Sylvie Vanséveren and available the Hethitologie Portal Mainz, For uniform appearance, the white margins of the image are cropped, than 10 pixels of white margins are added to each side, and finally the image is resized 512 512 resolution.
A.2 MODEL DETAILS
We use the CompVi s stabl aattalllee le-dileediiii ffus model. We fine-tune this on cBL classification train dataset (Cobanoglu et al., 2024). for 50K iterations with batch size of 4, learning rate of 10-5 and Adam optimizer. For textual prompts, we use a unique code for each sign type indicated the eBL dataset. This fine-tuned Stable Diffusion model, SD-If, is used for DIFT feature calculations as well as in the synthetic data generation for our OCR application. For tests using ControlNet, we fine-tune the base ill illyasvies e /sddcontrra net operрos checkpoint, 932 samples with their paired alignment created by ProtoSnap. We trained it for 20K iteration, with batch size of 4, learning rate of 10-5 and Adam optimizer. This fine-tuned ControlNet, CN-If, uses another fine-tuned Stable Diffusion model, trained with the same parameters as SD-TF, but without prior from a textual prompt, using the same prompt for all signs: "cuneiform single ancient icon".
checkpoint as our base Stable Diffusion
For DIFT feature calculations, we average an ensemble of results on four random noises, sampled timestamp 261, following Tang et al. (2023). We concatenate features from the second and third upscaling U-Net layers. bilinearly interpolated to the same spatial resolution, yielding set of 64 x 64 feature vectors of dimension 1920.
A.3 GLOBAL ALIGNMENT DETAILS
To fit our global alignment, we apply RANSAC with 2000 iterations. At each iteration, 5 corre- spondences are used to fit a least-squares affine transformation, with distance threshold of 50 pixels used to identify outliers. The transformation with the greatest number of inliers is returned.
14

---

Published as a conference paper at ICLR 2025
As high-quality set of correspondences should explain the relevant regions in both of the images, we incorporate prior on inlier points being spread across the images, following Hassner et al. (2014). In particular, we perform the above procedure 8 times, and assign each result a score using the convex hulls of the inlier points in the prototype and scanned cuneiform sign images. For the prototype image, calculate the proportion Pproto of the prototype font foreground contained within the convex hull of inlier points. For the scanned cunciform sign image, we calculate the proportion Pscan of the total area of the image covered by the convex hull of inlier points. Finally, we select the result with maximum score Pproto Pocan. to encourage the global transformation to be based on matches between regions covering most of the sean and prototype font.
A.4 LOCAL REFINEMENT DETAILS
Our total loss uses coefficients Asim = 1.0, Asol = 10-4, and Xveg = 10-4.
To calculate the saliency map used for the saliency loss, we compute differences in mean similari- ties as described in the main paper, apply CLAHH histogram equalization with clip limit 10.0 and tile grid size (2, 2), set values below the mean to zero, and scale values to the range [0, 1]. This yields a scalar field of resolution 64 x 64.
For both semantic similarity and saliency losses, at each iteration values are sampled at both each of the keypoints in the skeleton, and at audomly-samplee points along each line segment in the skeleton connecting keypoints. Loss values are averaged over all of these points. The sampled points are sampled uniformly from the lines in the prototype images, and then transformed using the current global and local transformations to obtain corresponding points in the target scanned cunciform sign image. Loss values are computed with differentiable grid sampling, using bilinear interpolation over scalar fields (the similarity values in respective slices of S, and saliency map values) each of which is passed through a softmax with fixed temperature parameter 100.
To perform optimization, we apply gradient descent for 100 iterations with learning rate 0.01 and Adam optimizer, updating the the parameters of the local transformations of all strokes.
A.5 DATASET DETAILS
Both the training and the test datasets are taken from the eBL classification dataset Cobanoglu et al. (2024) which consists of scanned imaged of cuneiform signs. The dataset was originally split to train and test by source tablet. The train set included 34,868 sign images, representing 362 sign types and was used to train SD- The test set was used to select the 272 images for our test set (as described in 5. focusing on signs for which we have available prototype, and the samples in the set match the prototype structure variant. The final test set consists of 272 samples from 25 different signs, not seen in training, varying from 2 strokes per sign to 8.
The full dataset comprises around 40% from the Neo- reo-Babylonian period (1000-600 BC), around 20% from the Neo-Assyrian (1000-609 BCE), and less than 10% from the following periods: Ur III (2100-2002 BCE), Old Babylonian (2002-1595 BCE), Old Assyrian (1900-850) BCF), Middle Babylonian (1500-1000 BCE), Late Babylonian (600 BC-100 AD), Persian (539-331 BCE), Hellenistic (331-141 BCE), Parthian (141 BCE-100 CE). The dataset represents the Akkadian and Sumerian languages used at those eras.
A.6 USER SURVEY DETAILS
For the users survey, we have asked experiences assyriologists to review 35 cuneiform signs. For each signs, presented them with options- plain sign, a sign with an overlaid prototype without alignment, and a sign with overlaid prototyped aligned using ProtoSnap. For each sign we asked the user to select the image in which they can most easily identify the sign. 12 assyriologists had answered the survey. In all but three signs, the users preferred an overlaid option to the plain one, and out of those, the users preferred the aligned option % of the time (in 21 signs).
15

---

Published as a conference paper at ICLR 2025
Control
Generated
Control
Generated
Figure 8: Examples for data generated using our fine-tuned ControlNet model CN-IF, where the control an image a prototype sign (with two different added transformations).
A.7 OCR EXPERIMENT DETAILS
For the OCR experiment we have generated 50 samples per each sign in the test set (180 signs total), using our fine-tuned SD-If described above. In addition, we used CN-If to generate 50 samples for each sign have available prototype (124 signs in total). To better mimic human handwriting, we augmented the skeletons by applying small random transformation on the entire skeleton and each stroke individually, creating diverse set of controls for each sign. Figure g shows examples of such generated data. Both generations were done using 50 inference steps and classifier-free guidance scale 7.5.
For the experiment we used 6205 samples from the CSDD data (consisting of data from all URLs which were broken in the dataset), 3299 samples in the train set and 2906 in the test. Training ResNet18 on this dataset alone was done for 10 epochs, using batch size of 64, learning rate of 10 3 and Adam optimizer. Training with generated data (both from SD-IF and CN-If) was done for 8 epochs, and then fine-tuned 10 more on real data only. using batch size of 64, learning rate of 5-1 and Adam.
Note that our reproduced baseline on CSDD achieves slightly higher accuracy than reported by Dencker al. (2020), but the test dataset includes URLs which are no longer operational which may contribute to this slight discrepancy in results.
B ADDITIONAL RESULTS
Figure shows additional examples, illustrating ProtoSnap applied various samples of the same sign. Figure 10 shows ProtoSnap results manually annotated test set, compared to the baseline of applying DIFT directly (assigning each keypoint to the region maximal feature similarity).
Figure shows ProtoSnap results on a new, previously unseen dataset, JOCCH (Rusakov et al. 2020). which contains signs from the Hittite language, opposed to the Akkadian and Sumerian from which the training and test set are composed. Those results show that our method is robust and can be generalizable other usages of the cuneiform writing system.
16

---

Published as a conference paper at ICLR 2025
Font
Skeleton
ProtoSnap applied to scanned cuneiform images
Figure 9: Examples of ProtoSnap applied on photographed cuneiform signs of varying structure, illumination conditions and degrees of intactness.
Imput
GT
DIFT
Poschng
Ours
Figure 10: Results of ProtoSnap on our manually annotated test set, with DIFT and Pose
(Hirschorn and Avidan 2023) shown for comparison. We can see that our method produces align- ments which are much closer to expert annotations and is generally less sensitive to outliers.
17

---

Published as a conference paper at ICLR 2025
Protolype yp
Input
ProtoSnap
Figure 1I: ProtoSnap applied on images from a different dataset and language (Hittite), showing that the method is robust and generalizable to various usages of cuneiform writing system. The images on the left show signs from types (names) unseen in the training data, further emphasizing the generalizability of the method.
threshold = 20
threshold = 40
Method
Precision
Recall
F1
Precision
Recall
F1
SIFT (Lowe 1999
RANSAC
2.59%
2.53%
2.56%
5,49%
4.61% %
4.91%
DINOv2 (Oquab et al. 2024
12.42%
12.25%
12.33%
32.09%
30.05%
31.04%
DINOv2 + RANSAC
16.22%
16.01%
16.12%
38.47%
37.42%
37.93%
DIFT Tang al. 2023)
16.18%
16.10%
16.14%
34.32%
33.27%
33.79%
DIFT + RANSAC
13.15%
13.11%
13.13%
30.43%
29.88%
30.15%
Ours (w/o refinement)
21.38% %
21.23%
21.31% %
50.55%
49.73%
50.13% 52.43
Ours (full)
27.17%
27.10%
27.14%
52.76%
52.10%
Table Precision and recall metrics for the alignment evaluation, on top of Fl metric presented in the main paper, at two distance thresholds.
B. ADDITIONAL METRICS
Table 3 shows precision and recall metrics for the alignment evaluation, on top of F1 metric pre- sented in the main paper. Table Show alignment evaluation breakdown per signs, and also pro- vides the number of samples in the test set per sign.
B.2 ABLATION STUDY
We demonstrate the effect of key parts our system by ablating them and evaluating performance on our test set. In particular, we ablate the following:
Use of our fine-tuned SD-TF (rather than base Stable Diffusion)
Use of best-huddies correspondences for computing the global transformation (rather than using all correspondences between prototype image regions and the best-matching regions in the target image according DIFT)
Each of the three loss terms in our full loss function
As seen Table 5) most these ablations have a significant negative effect on quantitative perfor- mance. Removing Lein slightly improves metrics, but find this reflects qualitative trade-off.
18

---

Published as a conference paper at ICLR 2025
Sign Name
Samples
# Strokes
Fl @22
F1 @1
FI @0
MЕ
20
2
23.13%
33.75%
46.88%
A
19
3
26.75%
40.79%
52.41%
IGI
18
3
12.50%
25.23%
37.50%
AN
19
3
30.04%
46.05%
61.39%
UD
18
3
16.20%
29.17%
44.21%
GISH
19
3
28.29%
42.11%
54.17%
MA
16
4
26.95%
44.73%
57.81%
EN
20.00% 5
36.00%
47.49%
IR
17
22.94% 5
36.17%
42.04%
IB
15
5
35.17%
55.17%
63.33%
HA
5 23.33%
35.81%
48.33%
UR
16
5 32.66%
49.06%
57.50%
RI
13
5
29.81%
44.03%
57.11% %
RU
15
5
24.67%
37.50%
45.83%
DIM2
5 25.00%
35.00%
42.50%
DI
5 24.17%
43.71%
51.25%
U2
5
21.67%
41.39%
53.88%
DIB
6 23.61%
34.01%
46.52%
SA
6 36.79%
45.13%
52.00%
GI
7
28.57%
46.43%
53.57%
DA
7
28.57%
42.86%
50.00%
KA
7
26.64%
44.94%
55.79%
ZI
10
7
31.43%
42.50%
50.35%
A2
8 29.69%
44.92%
55.47%
ZE2
8
39.45%
54.49%
60.35%
Table 4: Performance breakdown by sign type. We report alignment performance our model the different annotated signs. We also provide the number of strokes in each sign (#Strokes) and number of samples of the sign in the test set (#Samples).
We foresee an expansion of our test set or development of additional test metrics for this task to better capture this performance.
F1@20
F1@30
FI @40
ProtoSnap (ours)
27.14%
42.09%
52.43%
SDD
19.01%
31.89%
41.25%
best buddies
26.76%
41.76%
52.76%
Caim
27.37%
42.61%
53.19%
Caat
21.13%
37.48%
50.08%
Creg
26.72%
39.66%
47.41%
Table Ablation study results, demonstrating differences performance when removing key parts of our system. Most ablations negatively impact quantitative results, further explained in Section
C ANNOTATION DETAILS
Our expert annotations were performed by Assyriologists who participated in this research.
Below, provide further details annotations collected via crowdsourcing, used to anno- tate keypoints in prototype font images and in scanned cuneiform signs. We then connected the keypoints manually ourselves, creating the prototype skeleton.
19

---

Published as a conference paper at ICLR 2025
C.I IRB APPROVAL, PARTICIPANT SOURCING, AND COMPENSATION
Our annotation tasks, approved by our institution's ' IRB, were conducted on the Amazon Mechan- ical Turk (MTurk) crowdsourcing platform. We published tasks for MTurk workers with least 1000 completed HITs (MTurk tasks) and HIT approval rate of at least 95%. Workers were compensated $0.25 for each font annotation, corresponding to the duration of this task.
C.2 ANNOTATION TASK INSTRUCTIONS
In this lask, you will indicate keypoints ancient character ("cuneiform") to indicate the location of each stroke. Please indicate each stroke with four keypoints as shown here:
there multiple strokes, please indicate each stroke in separate color using four keypoints per stroke, as in these examples:
Make sure the four keypoints are in the locations shown above three indicating the corners of the stroke's 's triangular head, and one indicating the end of its tail.
Use the point tool to place points the requested target(s) of interest: Four each stroke in the character, place four keypoints of the same color, using your mouse to click on each keypoint. Use the four keypoints described in the instructions: for each stroke, three points indicating the corners of the stroke's 's triangular head, and one indicating the end of its tail. Make sure to indicate every stroke seen in the glyph.
20


Reviews:
[{"review_id": "aVyN4RFMxA", "reviewer": "Reviewer_xssh", "summary": "This paper proposes a novel approach to handle the complex internal structure of cuneiform signs, called ProtoSnap, an unsupervised method that utilizes deep generative models and prototype font images to estimate the fine-grained internal structure of cuneiform signs.", "questions": "1. \"This H ×W×H ×W tensor, visualized in Figure 3, contains the pairwise cosine similarities between features encoding patches of the prototype and target images.\" I'm confused about the H ×W ×H ×W.\n\n2. The proposal includes a user study to assess the practical usability and effectiveness of the ProtoSnap methodology from the perspective of an end user such as an archaeologist or historian.\n\n3. Recommendations for fuller experiments.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Originality: ProtoSnap's use of deep diffusion features and skeleton - based prototypes for unsupervised cuneiform sign alignment is novel.\n\nQuality: The overall flow of the methodology section is sound and logical.\n\nClarity: This paper is clearer on the whole, from the introduction part of the cuneiform research background and the limitations of the existing methods, which naturally leads to the research objective, i.e., to propose the ProtoSnap method to solve the problem of analysing the internal structure of the cuneiform symbols.\n\nSignificance: This work was instrumental in the development of the field of cuneiform writing.", "weaknesses": "1. While this paper presents a new benchmark for evaluation, the current dataset may not be fully representative of the variety of cuneiform symbol variants and writing conditions present in the historical record.\n2. The superiority of the method proposed in this paper is not reflected in the related work.\n3. 4D similarity volumes in section 4.1 are not clearly described.\n4. While the method shows promise for cuneiform signs, its adaptation to other ancient writing systems or complex symbol sets may not be straightforward.\n5. There are too few comparative experiments to adequately demonstrate the superiority of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v626NuJotA", "reviewer": "Reviewer_Bpiu", "summary": "This paper introduces ProtoSnap, an unsupervised method for aligning the internal structure of cuneiform signs using generative models and prototype images. The approach improves the recognition of cuneiform signs by refining the alignment of skeletal templates to real sign images. This method leverages deep learning and generative modeling to interpret the complex internal configurations of cuneiform signs, enhancing optical character recognition (OCR) accuracy, especially for rare signs.", "questions": "1. Can the authors detail any specific preprocessing steps required to prepare the cuneiform images before applying ProtoSnap?\n\n2. What are the limitations in terms of computational resources, and how scalable is this approach when applied to large datasets of cuneiform texts?\n\n3. How does ProtoSnap handle extremely degraded or incomplete cuneiform signs where the prototype may not initially align well?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The application of unsupervised learning and prototype alignment to cuneiform signs is novel and shows significant potential.\n\n+ The technical approach is sound, utilizing SoTA techniques in image processing and machine learning.\n\n+ The method has clear applications in digital humanities, aiding the decipherment and study of ancient texts.", "weaknesses": "1. There’s a potential risk that the method could overfit to the prototypes it has been trained on, especially if those prototypes do not capture the full variability of the signs in the dataset.\n\n2. It would be good if the authors could report on the computational resources required for implementing the ProtoSnap method. Considering that it involves deep learning models and generative processes for aligning prototypes with actual images, understanding the computational demands is crucial.\n\n3. I'd like to hear the authors' opinion on the potential of ProtoSnap to adapt to other ancient scripts, which often present unique challenges in terms of symbol complexity and degradation patterns. This discussion could provide valuable insights into the versatility and scalability of the proposed method beyond cuneiform studies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nolh6dlNQ2", "reviewer": "Reviewer_KZk1", "summary": "The paper present a method for fine-grained structure retrieval in cuneiform sing images, given the canonical form of the depicted signs. In more detail, the method first calculates a global alignment using diffusion features, best buddies correspondences and RANSAC. Then,  the aligned template sing is refined via skeleton-based optimization. The authors demonstrate sota results in cuneiform sign alignment and recognition. Last, a new dataset of expert-annotated cuneiform sign images and will be released.", "questions": "1. I kindly ask the authors to provide more details and motivation regarding experimental results in Table 1.\n2. To my understanding, cuneiform sign data are few if not limited. How far away is the field from data-driven methods that predict accurate keypoints on sign images without known templates? Could your synthetic data be helpful towards this direction?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors have done a good job presenting their method to a reader unfamiliar with the subject. The paper is well written and the ideas well presented.\n2. The method is novel for cuneiform sign alignment, as it adopts a common tactic from pose/keypoint detection problems in the scope of the presented subject.\n3. The method achieves sota results in cuneiform sign alignment, although a more detailed comparisons scheme could have been designed (more details in weaknesses 1). \n4. The method achieves sota results in cuneiform sign recognition.\n5. A new benchmark dataset of cuneiform sign images with expert annotations will be released.", "weaknesses": "1. Comparisons in Table 1 are not clear. To my understanding, for DINOv2 and DIFT the authors directly decide keypoints based on feature similarity without solving RANSAC. On the other hand, the authors employ RANSAC for SIFT features and their method (with or without refinement). In my view, the authors should not focus on a single model for feature extraction, but rather experiment with all of them in the same setting (with or without RANSAC) and present their method as a more general method for template alignment for cuneiform signs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kwaiLWYQi2", "reviewer": "Reviewer_UGZ9", "summary": "The paper presents a method for aligning cuneiform character prototypes to in-the-wild real character images. The prototype consists of an image of the canonical character, as well as an aligned skeleton representation of the character. The method uses deep image features from a finetuned diffusion generation model to measure the patch similarities between the prototype image and the real image. Given the similarity map, the method first applies a global affine transformation to the skeletal representation, such that the feature similarities are maximized. Then the method applies per-stroke projective transformation to the skeleton strokes, to further maximize the alignment of image features. To regularize the alignment optimization, mutual optimal matching between patches, RANSAC for global transform, as well as saliency, identity and boundary constraints for local transform are applied. \n\nThe method has been tested on a new benchmarked collected by expert annotators, and showed improved accuracy than baseline matching algorithms. The aligned image-character pairs also allow for finetuning ControlNet, so that new images can be synthesized for training OCR models, which demonstrates the benefits enabled by the method.\n\nOverall, this paper presents a fluent combination of various image processing and registration tools to solve the problem of cuneiform character recognition to a better state.", "questions": "As mentioned above, I hope the authors can discuss the extension and challenges when applying the method to more types of ancient characters. In particular, oracle bone characters have different organizations than cuneiforms, as an oracle bone character not only consists of multiple strokes, but more importantly the strokes are not fixed in numbers and shapes as the cuneiforms do; instead, the strokes are connected into components depicting certain figures, which can vary to a large extent in terms of the number of strokes and nonrigid deformation. To handle such variations, the assumed projective per-stroke transform may not be sufficient. In particular:\n1. Are there any components of the current method that would need significant modification?\n2. Could the authors provide a brief analysis of how the skeleton structure and deformation models need to be adapted?\n\nFig.6 can be more specific about \"the most prevalent sign variation during training\". Which are these variants and how do you know it? Specifically, the authors can provide more insights into the dataset, by the following means:\n1. Provide quantitative data on the frequency of different sign variants in their training set, if available.\n2. Explain the method used to determine which variants were most prevalent.\n3. Include a brief discussion about how this prevalence might impact the model's performance.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is well written and well illustrated. Technical designs are presented concisely in the main text and discussed in detail in the appendix. Experiments are extensive, using a new benchmark with images labeled by experts and crowdsourcing. Ablation studies are exhaustive and confirmative of the various technical designs. The use of aligned prototypes for generative data synthesis is particularly interesting, by bridging the power of pretrained ControlNet and the parameterized cuneiform images.", "weaknesses": "The main weakness is that only cuneiform characters are considered. The authors did not discuss if the same set of techniques used by the method pipeline can be applied to other types of characters, like oracles. It's desirable to at least discuss the possibilities and challenges, e.g. within the different skeleton structures and permitted deformations.\n\nMore analysis of the collected and training datasets can be done, to provide the readers with more understanding of the common signs and variations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)