Classification of Paleographic Artifacts at Scale: Mitigating Confounds and Distribution Shift in Cuneiform Tablet Dating (2024)

Abstract: Cuneiform is the oldest writing system used for more than 3,000 years in ancient Mesopotamia. Cuneiform is written on clay tablets, which are hard to date because they often lack explicit references to time periods and their paleographic traits are not always reliable as a dating criterion. In this paper, we systematically analyse cuneiform dating problems using machine learning. We build baseline models for both visual and textual features and identify two major issues: confounds and distribution shift. We apply adversarial regularization and deep domain adaptation to mitigate these issues. On tablets from the same museum collections represented in the training set, we achieve accuracies as high as 84.42%. However, when test tablets are taken from held-out collections, models generalize more poorly. This is only partially mitigated by robust learning techniques, highlighting important challenges for future work.

heading: 1 Introduction
text: Computational paleography (Vidal-Gorène and Decours-Perez, 2021; Srivatsan et al., 2021) is a growing interdisciplinary field that uses computational algorithms to decipher and analyse ancient writing systems. We investigate using machine learning to automate large-scale dating of cuneiform1, the oldest writing system from around 3,500 BCE. Similar to general chronicle attribution tasks in paleography, cuneiform dating involves classifying cuneiform tablets into specific time periods rather than precise years. For example, Figure 1 shows a tablet comes from Ur III. Different from other historical languages, such as ancient Greek (Assael et al., 2022) or ancient Arabic (Adam et al., 2018), cuneiform tablets are more challenging to convert into a machine readable format because the writing system continually evolved over the 3,000 years it was in use.
1Code is available at https://github.com/taineleau/ CuneiML/tree/main/ml4al_2024_dating.
For many writing systems, historians and paleographers have been able to identify distinguishing features in textual content and writing style that allow for inferences about date of origin for individual artifacts. For some writing systems, these processes have even been automated with machine learning to some extent. For example, Assael et al. (2022) showed encouraging results using neural networks trained on ancient Greek text to restore and date digitized ancient Greek artifacts.
Can we train similar textual models for cuneiform dating using accompanying manual transcriptions or transliterations? We conduct experiments with a series of light-weight recurrent models that show this is indeed possible. However, relying on manual transcriptions for the purpose of dating is somewhat circular: for Cuneiform, transcription and transliteration is as time-intensive as manually dating tablets. Further, transliterations themselves might exhibit bias—for example, an expert’s approach to transliterating a tablet may al-
30
ready be influenced by preconceived notions about its time period—allowing models to overfit to the tendencies of individual transliterators.
Thus, we also study whether visual representations of Cuneiform tablets can be used effectively for automatic dating. Visual representations skirt the issues of manually-intensive transcription and confounds due to transliteration style. Further, visual representations may even allow models to automatically extract information about the visual style of writing, which paleographers have found useful for manual dating. In past work, Bogacz and Mara (2020) has shown that relatively accurate dating of cuneiform tablets using 3D scans is possible. However, currently it is not feasible to produce 3D scans of over 100,000 remaining tablets, which are dispersed among museums and private collections around the world.
Therefore, instead we explore the use 2D photographs from CDLI (CDLI contributors, 2024) to address the dating problem—a task that as far as we are aware has not been previously studied. Our experiments using convolutional neural models trained on 2D images demonstrate a new problem however: the different imaging setups used by different collections presents a confound that leads to poor generalization (shown in Figure 3). We find that the gap between performance on tablets from collections that were attested in training data versus those that were not is extremely large. Thus, we also evaluate to what extent robust learning methods that attempt to address out-of-distribution (OOD) generalization can mitigate this issue. We find that while these methods do help, they do not increase generalization to the point where accurate dating of tablets from unseen collections can be performed reliably. Thus, our empirical study highlights this important challenge as an area for future research. We summarize our primary contributions below:
1. We identify and analyze several challenging issues in cuneiform dating related to confounds, distribution shift, and domain generalization. These challenges are likely also present in the classification of other ancient artifacts with text.
2. We study a range of modeling approaches including simple methods like Naive Bayes, as well as neural methods for both images and text features. We demonstrate strong performance when using data splits that reduce dis-
tribution shift and OOD effects, but poor performance across museum collections.
3. We applied multiple robust learning techniques to mitigate distribution shift and the effect of confounds. While our results demonstrate improvements from these techniques, overall OOD generalization performance is still prohibitive for broader use.
In the following sections, we first formulate the problem and then describe the data collection splits we created to address our core research questions.

heading: 2 Problem Formulation
text: Technically, the dating task can be formulated as either a classification or a regression problem. However, after careful examination, we concluded that treating inferred dates as continuous variables (using regression) does not make sense in this domain because the annotation standard used for manual dating (the source of supervision for learning and evaluation) includes date categories with overlapping time intervals (see Figure 5). Instead, we represent each time period as a categorical class ID and treat dating as a multi-class classification problem.
Next, we layout the core research questions we attempt to answer in this empirical study. To address each, we will carefully design data splits that contain three separate test sets, each measuring a specific aspect of OOD generalization, along with a train and validation set.
RQ1: What models, configurations, and features—either visual or textual—are most effective for automatically dating cuneiform tablets?
RQ2: How much of a problem do OOD effects pose for generalization in this domain? For example, do models overfit to specific features present in individual museum collections? How well do models generalize to tablets from previously unseen museum collections?
RQ3: How well do existing robust learning techniques address the issue of distribution shift and OOD generalization in the context of cuneiform tablet dating?
In later sections, we will specify the datasets we use, which specific input representations we compare, and which modeling approaches
we evaluate. We will also carefully design test splits to answer specific questions about OOD generalization. Next, we describe and define some of the potential OOD effects in this domain and distribution shifts we seek to analyze.
Generally speaking, distribution shift occurs whenever the underlying distribution that generated the training data diverges from the distribution that will generate future test instances. Distribution shift poses a substantial challenge for learning systems: patterns that hold true on the training data may not generalize to the test set, leading to poor generalization performance. In the domain of cuneiform data the are two important types of distribution shift.
First, cuneiform datasets tend to exhibit substantial label shift due to how tablets are distributed across museum collections. We depict the distribution of tablet dates in museum collections in the
CuneiML dataset (Chen et al., 2023) (which we use in experiments) in Table 2. Most museums contain tablets from a small range of time periods. Thus, if train and test setups for validating computational approaches are selected based on i.i.d. sampling from this dataset, the test performance may not accurately reflect expected performance on tablets from new, unseen museum collections. In Figure 4 we visualize actual label distribution shift in a i.i.d. train/test split.
Second, the input representations from individual museum collections may have properties that make the collection itself identifiable. For instance, as shown in Figures 1 and 3, the scanning methodologies used by separate museums leave artifacts like different amounts of color saturation and blurring. Similarly, it is possible that different transliteration styles may also be identifiable. Because individual collections are biased towards specific date ranges, the confounds mentioned above may cause covariate shift—a type of distribution shift where the distribution on the input variables and the relationship between input and output vary between train and test. For example, a model may learn to identify the collection based on properties of the scanning hardware in order to determine date. This may work on training data, but will not generalize
to new collections. Thus, one of our primary goals is to measure the effects of label and covariate shift for cuneiform dating and to evaluate to what extent robust learning methods may address these issues.

heading: 3 Data
text: We obtain 38,937 tablet images with transliterations from CDLI (CDLI contributors, 2024), using prepossessing from CuneiML (Chen et al., 2023). An example is shown in Figure 6. Besides transliteration and 2D images, we use several other attributes from the metadata entries, including provenience, collection, and genre, which we use in later experiments for both simple baselines and as additional supervision to mitigate distribution shift.
Split % Count Note all 100% 38,937 - train 80% 30,626 -

heading: 3.1 Data split
text: Inspired by Koh et al. (2021), we identify two kinds of distribution shift and would like to create splits that disentangle the issues and better answer the research questions. As we can see in Figure 2, most museum collections only own tablets from one or two time periods and most time periods are collected by a specific museum. To better study the distribution shift across collections, we split the data with regard to the collection id, i.e. tablets from the same collection only present in one split. This split we call OOD test split (test 1). We use p(y) shift to denote a split where the label distri-
bution p(y) is significantly different from that of the training set. We describe briefly how we split the data (Table 1) below.
1. Step 1: Getting an OOD and p(y) shift set S1 from the full data. We sampled about 10% from the full dataset using the following rules: (i) We sample by collections, meaning tablets from an entire collection are either included or excluded. (ii) For a given time period, we do not select collections that constitute more than 30% of the data for that time period, ensuring that we do not remove most of the tablets for certain time periods from the training set. We named the remaining 90% of full data S2. Figure 4 shows the shift of p(Y ).
2. Step 2: Getting valid and test 1 set. we evenly split S1 We obtained from step 1 and we now have valid and test 1 set.
3. Step 3: Getting test 2. We sampled 5% of the data from the subset S2 against the label distribution of test 1. Therefore, test 2 has the same sub-population shift from the training set as test 1, but consists of in-domain (ID)
data instead of OOD data. We named the remaining 85% of the full data S3.
4. Step 4: Getting test 3 and train set. We randomly sampled 5% of the data from S3 to constitute test 3, and the remaining 80% is the final training set.
Therefore, we have three testing splits setup as shown in Table 1.

heading: 4 Methods
text: We describe the baseline models we used in experiments and also several training strategies, adversarial regularization and , to mitigate the distribution shift issues.

heading: 4.1 Baseline models
text: 1. Naive Bayes. We use discrete categorical features, including genre, collection, provenance, and size, to predict the time period as a categorical prediction problem. Note that when there is only one feature, the performance indicates a correlation between the feature and the predicted class.
2. Char-LSTMs. We use a character-level two-layer bi-directional LSTMs to process cuneiform transliterations and sign tokens for dating ancient texts. The model has a hidden size of 128 and an embedding size of 256. We train for 200 epochs using the ADAMW optimizer with a learning rate of 5e-4 and a weight decay of 1e-3.
3. ResNet. Our study utilizes the ResNet (He et al., 2016) architecture, specifically ResNet50 and ResNet-101. We apply these models to classify images of cuneiform inscriptions, leveraging their powerful feature extraction capabilities. The models are trained using a cross-entropy loss function, with adjustments made to the final layer to suit our specific class labels. The training regimen includes a batch size of 16, 30 epochs, ADAM with a learning rate of 3e-5, and no weight decay.

heading: 4.2 Baseline Objective
text: For all the neural models, we use cross entropy (CE) loss to train the models.
L = CE(y(t), p(t))

heading: 4.3 Advanced Algorithms
text: To address the aforementioned issues, we explore several different robust training algorithms in this paper.
Adversarial Regularization. We use other attributes such as provenience and genre, to optimize a min-max objective. We attach a new branch of MLP to calculate the p(adv). L = CE(y(t), p(t)) + KLD(y(const), p(adv)) where CE is cross entropy loss and KLD is the KL Divergence loss.
Correlation Alignment for Deep Domain Adaptation (CORAL). CORAL (Sun and Saenko, 2016) measures the divergence of means and covariance between batches of feature representations. The goal of CORAL is to match the feature distributions from different domains.
Invariant risk minimization (IRM). IRM (Arjovsky et al., 2019) penalizes feature distributions that result in different optimal linear classifiers across different domains. where where Φ is the entire invariant predictor, w = 1.0 is a fixed classifier, and the gradient norm penalty is the measure of the classifier at each environment.

heading: 5 Experiments and Results

heading: 5.1 Input Features
text: We have four different input features for training, describing as below.
1. Raw image. The raw images downloaded from CDLI. Each image usually contains photographs of six faces for each tablet.
2. Major-face image. The major-face cutout of the raw images, which are usually the front faces of the tablets.
3. Raw transliteration. We use the postprocessed version from CuneiML, which removes formatting string such as line numbers, broken markers and etc.
4. Cuneiform sign (glyph) token. We tokenize cuneiform glyph at a character-level, with a vocabulary size of 764. See Figure 6 for an example. We keep the space between words and line break.
ones are underlined. Colored background highlight the best overall model for each setting.
The bounding boxes for major-face images and the Cuneiform sign (glyph) tokens are obtained from Chen et al. (2023)2.

heading: 5.2 Metrics
text: As the label distribution p(y) imbalance exists and there is a distribution shift, we primarily use the F1 score and accuracy to evaluate our methods. Specifically, we use Macro F1 and accuracy3 as our major evaluation metrics.
Macro F1 score computes the F1 score independently for each class and then takes the average, thus treating all classes equally regardless of their frequency. This dual approach allows us to address both the overall accuracy and the individual class performance, ensuring a thorough evaluation in the face of skewed class distributions and shifts.

heading: 5.3 Results and Analysis
text: The main results for two split settings are shown in Table 2 and several key observation are summarized as follows.
1. Random and Majority Baseline Models. These models provide basic benchmarks with the majority model performing based on the most frequent class, note that the majority class contains more than 70% of the models, which accounts for the big discrepency between macro F1 and accuracy. The low F1
2https://github.com/taineleau/CuneiML 3For single-label classification, Micro F1 is equal to accu-
racy
scores, indicating poor performance across all classes evenly.
2. Neural models perform the best across all settings. Both visual and textual neural models work fairly good in ID setting (test 3), showing that both textual and visual features provide sufficient information to date tablets.
3. Raw images contain confounded undesired features: collection. When using a ResNet50 model, features extracted from the raw images outperformed those obtained from front face cutouts on ID split (test 3). However, this performance was reversed on an OOD split (test 1). This reversal clearly indicates that raw images include collections as a confounding factor.
4. Textual features are not effective for dating when label shift exists. From test 3 to test 2, only the label distribution changes, while the data remains in-domain. However, textual models experience a dramatic drop in performance by 57.9%, revealing that textual features are not robust to label imbalance issues. In contrast, image models are not affected as significantly.
5. Textual models are not robust to OOD shift; visual models are better but still have room for improvement. Textual models exhibit nearly a 50% relative decrease in macro F1 for the OOD setting (test 1) compared to visual models. With the application of OOD
mitigating algorithms (see section 6.3 for details), visual models improve from 34.82% to 41.06%, achieving the best F1 score on test 1. This aligns with our earlier concerns that textual features do not capture any writing style of the tablet, making it difficult to determine the time period under OOD shift conditions.

heading: 6 Further Analysis

heading: 6.1 Zooming in on Textual Models
text: As shown in Figure 7, we conducted extensive experiments on the number of lines per example fed into the models. As mentioned earlier, we use majority vote by default to ensemble predictions when we divide a full document. The performance of the glyph token features (sign token) increases as the number of lines in an example increases, while the transliteration features typically achieve the best performance with only one or two lines. This observation aligns with our understanding that transliteration already encodes some contextual knowledge, as signs are transliterated into Latin depending on the context. In contrast, for sign token features, the machine learning model requires more lines to discern the underlying information effectively.

heading: 6.2 Mitigating Label Imbalance Issues
text: Table 3 presents the results of label imbalance methods using char-LSTM on transliteration and glyph token features, with loss reweighing (LR) and upsampling (US). While both methods show varied effects on the performance metrics, loss reweighing generally improves F1 scores and accuracy across the test sets, particularly for transliteration features, achieving a F1 86.06% on test 3.

heading: 6.3 Distribution shift and Confounds
text: Adversarial Regularization. Table 4 show results using adversarial regularization. Macro F1 does not change as much as the accuracy. We also found that adversarial training requires very careful hyper-parameter tuning; otherwise, the model may completely underfit due to the noisy gradients provided by the adversarial branch.
OOD mitigation. Table 5 shows results using OOD methods. Among the OOD mitigating algorithms, CORAL consistently improves the performance across all test sets for both raw and cutout features. Notably, CORAL achieves the best F1 scores of 29.42% and 40.46% on test 1 for raw and cutout features, respectively. The other algorithms, IRM and groupDRO, generally show a decline in performance, with groupDRO performing the worst, especially for the cutout features. Overall, the results indicate that while textual models struggle with domain shifts, visual models, particularly those enhanced with cutout features and CORAL, demonstrate a more robust performance, albeit with room for further improvement.

heading: 6.4 Cuneiform Dating at Scale
text: It is not possible to make an apple-to-apple comparison on 2D and 3D scans features because most of the HeiCuBeDa dataset (Bogacz and Mara, 2020) does not accompany with a 2D photo. The paper reported a weighted F1 of 83% (which is roughly comparable to accuracy in our case). We conduct a set of experiments by varying the number of training examples, as shown in Table 6. Both models show a clear trend of improved performance with increased training data.

heading: 7 Related work

heading: 7.1 Automated classification for ancient languages
text: Sommerschield et al. (2023) provides a detailed overview of ancient languages processing using machine learning. Resler et al. (2021) classified artifact images using CNNs and nearest neighbors. Assael et al. (2022) train a BE to restore ancient Greek. There have been work on dating documents in various ancient languages, like Arabic, Korean and Chinese oracles bones among others (Sommer-
schield et al., 2023)

heading: 7.2 Cuneiform studies
text: There have been important efforts in cuneiform sign recognition, language identification (BernierColborne et al., 2019), and machine translation for Akkadian have been explored (Gutherz et al., 2023). Bogacz and Mara (2020) use high resolution 3D scans to classify time periods, and more recently Yugay et al. (2024) have explored the dating of first millennium Assyrian and Babylonian documents, using stylistic criteria and CNN. As mentioned earlier, it is non-trivial to tokenize the transliteration. Gordin et al. (2020) uses HMM and neural models to automatically transliterate Unicode cuneiform signs. On the contrary, in our paper, we reverse this process by converting the transliteration back to Unicode cuneiform signs to reduce transliteration bias.

heading: 7.3 Distribution shift
text: Historical data always suffers from noise and therefore it is hard to have good generalization on held out data. Specially for cuneiform, the systematic distribution shift is the most salient one. The systematic distribution shift is a special cases in domain adaptation, and therefore can be mitigated by general domain adaptation methods (Koh et al., 2021)). Ahmed et al. (2020) analyses group invariant predictions, where dominant simpler correlations with the target variable. Zare and Nguyen (2022) studied similar scenario in medical diagnosis, which has a shift on several attributes such as sex, age and race. They use invariant risk minimization (IRM) (Arjovsky et al., 2019) to learn invariant features. Another branch of methods is adversarial regularization, which uses adversarial training (Gokhale et al., 2021) to improve the generalization ability. Li et al. (2018) uses Maximum Mean Discrepancy (MMD) to align loss in different class.

heading: 8 Conclusion
text: In this paper, we explore end-to-end cuneiform dating at scale using machine learning. We have identified three major challenges—label imbalance, distribution shift, and circular reasoning—that are prevalent in cuneiform dating. These issues and solutions explored in our paper are broadly applicable to the classification of other ancient artifacts as well. We hope our initial analysis will inspire the
community to further adopt machine learning for addressing problems in ancient language processing.

heading: Acknowledgments
text: All the images and annotations are from CDLI (CDLI contributors, 2024), and our work would not have been possible without the numerous annotations and editorial work provided by their team and collaborators.

heading: A Appendix
text: A.1 Tokenization
There are 7, 000 glyphs across different time periods. We use Chen et al. (2023) tokenization of text.
1. word boundary. Empty space is manually inserted between word. We by default keep the space by inserting.
2. Logogram. A tilde sign before a sign indicate it is a logogram. By default we differentiate whether a sign is syllable or logogram.
3. Intrusions. (...) indicates unknown number of signs is missing.
4. Modifier. In ATF, at-sign precedes a sign or group. For example, @c means curved.
5. Compound. |GA2 ∼ a×EN|, means: “the a-allograph of the sign GA2 containing sign EN”.
6. Breakage. Hash tag is used to mark breakage.

heading: B Details
text: B.1 OOD Experiments Details
1. Raw
(a) IRM. We train for 30 epochs with a learning rate of 3e-5, an IRM lambda 1, and seed 2.
(b) CORAL. We train for 30 epochs with a penalty weight 10 and seed 0.
(c) groupDRO We train for 30 epochs with a learning rate of 3e-5 and seed 1.
2. Front
(a) IRM. We train for 30 epochs with a learning rate of 3e-5, an IRM lambda 1, and seed 2.
(b) CORAL. We train for 30 epochs with a penalty weight 10 and seed 2.
(c) groupDRO We train for 30 epochs with a learning rate of 3e-5 and seed 2.
B.2 Hyperparameters and ablation study
We provide further analysis and conduct a comprehensive ablation study in the following section,
exploring the effects of hyperparameters, input feature selection, and the number of training examples on our model’s performance.
As shown in Table ??, larger model or larger resolution of input can boost model performance.
C Visualization of tablets counts
A full resolution with number annotated heatmap for the time periods preserved in each museum collection is shown in Figure 8.

heading: Future Work
text: Generally speaking, distribution shift occurs whenever the underlying distribution that generated the training data diverges from the distribution that will generate future test instances. Distribution shift poses a substantial challenge for learning systems: patterns that hold true on the training data may not generalize to the test set, leading to poor generalization performance. In the domain of cuneiform data the are two important types of distribution shift. First, cuneiform datasets tend to exhibit substantial label shift due to how tablets are distributed across museum collections. We depict the distribution of tablet dates in museum collections in the
CuneiML dataset (Chen et al. , 2023) (which we use in experiments) in Table 2. Most museums contain tablets from a small range of time periods. Thus, if train and test setups for validating computational approaches are selected based on i. i. d. sampling from this dataset, the test performance may not accurately reflect expected performance on tablets from new, unseen museum collections. In Figure 4 we visualize actual label distribution shift in a i. i. d. train/test split. Second, the input representations from individual museum collections may have properties that make the collection itself identifiable. For instance, as shown in Figures 1 and 3, the scanning methodologies used by separate museums leave artifacts like different amounts of color saturation and blurring. Similarly, it is possible that different transliteration styles may also be identifiable. Because individual collections are biased towards specific date ranges, the confounds mentioned above may cause covariate shift—a type of distribution shift where the distribution on the input variables and the relationship between input and output vary between train and test. For example, a model may learn to identify the collection based on properties of the scanning hardware in order to determine date. This may work on training data, but will not generalize
to new collections. Thus, one of our primary goals is to measure the effects of label and covariate shift for cuneiform dating and to evaluate to what extent robust learning methods may address these issues