OBI-Bench: Can LMMs Aid in Study of Ancient Script on Oracle Bones?
Authors: Zijian Chen
tingzhu chen
Wenjun Zhang
Guangtao Zhai
Venue: ICLR 2025 Poster (2025)
Keywords: oracle bone inscriptions
ancient character deciphering
large multi-modal models
benchmark
Primary area: datasets and benchmarks
TL;DR: Using large multi-modal models to solve problems in the oracle bone inscription area

Abstract: We introduce OBI-Bench, a holistic benchmark crafted to systematically evaluate large multi-modal models (LMMs) on whole-process oracle bone inscriptions (OBI) processing tasks demanding expert-level domain knowledge and deliberate cognition. OBI-Bench includes 5,523 meticulously collected diverse-sourced images, covering five key domain problems: recognition, rejoining, classification, retrieval, and deciphering. These images span centuries of archaeological findings and years of research by front-line scholars, comprising multi-stage font appearances from excavation to synthesis, such as original oracle bone, inked rubbings, oracle bone fragments, cropped single characters, and handprinted characters. Unlike existing benchmarks, OBI-Bench focuses on advanced visual perception and reasoning with OBI-specific knowledge, challenging LMMs to perform tasks akin to those faced by experts. The evaluation of 6 proprietary LMMs as well as 17 open-source LMMs highlights the substantial challenges and demands posed by OBI-Bench. Even the latest versions of GPT-4o, Gemini 1.5 Pro, and Qwen-VL-Max are still far from public-level humans in some fine-grained perception tasks. However, they perform at a level comparable to untrained humans in deciphering tasks, indicating remarkable capabilities in offering new interpretative perspectives and generating creative guesses. We hope OBI-Bench can facilitate the community to develop domain-specific multi-modal foundation models towards ancient language research and delve deeper to discover and enhance these untapped potentials of LMMs.

Published as a conference paper at ICLR 2025
OBI-BENCH: CAN LMMS AID IN STUDY OF ANCIENT SCRIPT ON ORACLE BONES?
Zijian Chen', Tingzhu Chen Wenjun Zhang', Guangtao Zhai Institute of Image Communication and Information Processing, Shanghai Jiao Tong University School of Humanities, Shanghai Jiao Tong University
[zi jian. chen, tingzhuchen, zhangwenjun, zhaiguangtao @s edu cn
"Corresponding authors https: / /tt/tgtttttbbbb com/ zi www.:iiannnnnnn/--ne--
ABSTRACT
We introduce OBI-Bench, holistic benchmark crafted systematically evaluate large multi-moda models (LMMs) on whole-process oracle bone inscriptions (OBI) processing tasks demanding expert-level domain knowledge and deliberate cogni- tion. OBI-Bench includes 5,523 meticulously collected diverse-ssourel images, covering five key domain problems: recognition, rejoining, classífication, retrieval, and deciphering. These images span centuries of archaeological findings and years research by front-line scholars, comprising multi-stage font appearances from excavation to synthesis, such original oracle bone, inked rubbings, oracle bone fragments, cropped single characters, and handprinted characters. Unlike existing benchmarks, OBI-Bench focuses advanced visual perception and reasoning with OBI-specific knowledge, challenging LMMs to perform tasks akin to those faced by experts. The evaluation of 6 proprietary LMMs well as 17 open-source LMMs highlights the substantial challenges and demands posed by OBI-Bench. Even the latest versions of GPT-4o, Gemini 1.5 Pro, and Qwen-VLL VL-Max are still far from public-level humans in some fine- fineggrined perception tasks. However, they perform at a level comparable to untrained humans deciphering tasks, indicating remarkable capabilities in offering new interpretative perspectives and generating creative guesses. We hope OBI-Bench can facilitate the community to develop domain- sspacffc multi-modal foundation models towards ancient language research and delve deeper to discover and enhance these untapped potentials of LMMs.
Large Mulli- modal Models (LMM
OBI-Bench
Evel
Ixplosaece deneliatiiooos
Recwsling steued
Netioation
Expert review
Diedict tonks
Difivont
OBI taodos
Excavation
Synthesis
H
Original oracle bone
Inked rubbing
Oradle bone fragments
Cropped sample
Handprinted
Recognition Denoognttoo/DDecttion DDteccion
Rejoining
Classification
Retrieval
Deciphering
HOBI
Cheracler
English
Visual
guide
Cafte
Sun
"enee doedem
Figure 1: Overview the OBI-Bench. OBI-Bench presents five in-process tasks: 1) recognition: locating dense oracle bone characters from original oracle bone rubbings; 2) rejoining: reconstruct- ing fragmented text fragments into coherent texts; 3) classification: categorizing individual characters into their respective meanings; 4) retrieval: returning relevant results according to the given query OBI images; 5) deciphering: interpreting the OBI for historical and cultural investigation.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Oracle bone inscriptions (OBI) have been recognized as valuable records of making divination and praying to gods by the late Shang people from 1400 B.C. to 1100 B.C., in addition to being the carliest evidence of a Chinese writing system (UNESCO, 2017). Their discovery and interpretation provide unique window for evidence of the thought, language, society, and history of past civilizations. All along, there have been several challenges from the excavation to the interpretation of OBI. First, after thousands of years buried in the ground, the original OBIs are inherently eroded, fragmented, and susceptible to damage. Recognizing oracle bone characters from scattered pieces not only helps in initially determining the existence of the characters but also aids in their digital archiving with minimal physical handling. Second, the rejoining work of OBI is an indispensable prerequisite for follow-up research, which restores the original appearance of oracle Bones and offers complete and accurate information for OBI scholars (Zhang et al., 2022). Current rejoining procedures are non-trivial and involve time-consuming and specialized workflows (Men, 2008), thus posing demands to develop automatic or Al-assisted rejoining techniques. Third, OBIs exhibit wide range of stylistic variations in font and normally suffer from various types of noise in real rubbings (e.g., stroke broken, edge-cracked, and spindles), making it arduous to distinguish from distortions and accurately classify them. Fourth, OBI retrieval, as a core component for building large-scale databases, its difficulty lies in differentiating the inter-class and intra-class discrepancies of oracle bone characters. Fifth, among the thousands of OBIs that have been excavated, over half of them remain undeciphered (Wang& & Deng, 2024). The traditional methods for oracle character deciphering typically involve a combination of historical linguistics and archaeological context. More recently, lots of efforts (Jiang et al., 2023; Gao et al., 2024; Qiao al., 2024; Guan et al., 2024b) have been made leverage the representation capability of neural networks so as to establish correspondences between OBIs and modern Chinese. However, many characters have evolved in meaning through cultural and historical changes, and not all oracle bone characters can be translated into modern Chinese with synonymous expressions. Another problem lies in the absence of a comprehensive corpus and the lack of OBI-encoding systems, which limits the applicability of language models. These five tasks are crucial steps towards placing an OBI both in history and within the world of the people who read and study it.
Nowadays, the emergent large multi-modal models (LMMs) have brought opportunities for solving multidisciplinary tasks with powerful visual perception, understanding. and reasoning abilities (Tian et al., 2024; Yue et al., 2024; Caffagni et al., 2024; OpenAL, 2024b). Meanwhile, the ability of LMMs remains unclear on fine-grained perception and cognition (i.e., high-level perception), which play significant roles in interpreting ancient texts and their associated tasks on image material pre- processing. The perception at different granularities is strongly associated with a wide range of OBI tasks, such as distinguishing noise from real oracle bone character recognition (Wang et al., 2022), marginal sealability checking in rejoining (Zhang et al., 2021c) and component-leve structural decomposition in retrieval (Hu et al., 2024). Henceforth, it worth evaluating the current abilities of LMMs in fine-gr tne-grained perception and cognition based on actual needs to relieve extensive human resources and seek feasible solutions for future OBI research.
In this work, we propose the first comprehensive OBI-focused benchmark, OBI-Bench, to evaluate the recent LMMs in whole-process OBI tasks including recognition, rejoining, classification, retrieval, and deciphering under different settings. Our benchmark is constructed around key question:
Can LMMs aid in study of ancient script on oracle bones?
Specifically, we define two general evaluation principles for LMMs in targeting OBI problems:
Task-oriented Perception. As shown in the bottom of Fig. 1, an LMM is expected to answer accurately to achieve the objectives of five in in-process tasks like an OBI specialist, such as bounding the position of each oracle bone character for the recognition task or providing a reasonable interpretation of OBIs for the deciphering task.
Spanning from excavation to synthesis. As shown in the middle of Fig. 1, an LMM should be able to handle five different forms of oracle bone inscriptions and show good adaptiveness to the appearance or structural variants of OBI within the same task.
To systematically evaluate the whole-process process perception ability on various visual granularity under diverse OBI tasks, we collect 5,523 OBI images from 11 distinct sources. Due to the lack of publicly available OBI recognition datasets real oracle bones and OBI rejoining datasets, we propose the
2

---

Published as a conference paper at ICLR 2025
Table 1: Overview of the different image source datasets the OBI-Bench, and the respective
dataset attributes among recognition, rejoining, classification, retrieval, and deciphering tasks.
Task
Image Source Dataset
Type
Sampled Size
Avg. Res.
Remarkss
SiuQiWenYuanaanar (AYNU, 2020)
Fragments (inked)
2,000
435x538
Coordinate
Recognition
02BR (Ours)
Pragments (original)
800
1529x1192
Coordinate
Rejoining
OBI-rejoin [Ours]
Frugments (mixed)
483
265x313
Adjucency
HWOBC 2022)
Handprinted
300
400
Class label
Classification
Oracle-5OK (Han et 2020)
Handprinted
500
50x.0
Class label
OB1125 ala 2022)
Cropped words
500
77x135
Class label
OBC306 (Huing et aT. 2019)
Cropped words
500
66x111
Class label
Retrieval
OBI-UDH (Fujikawa & Meng, 2020)
Cropped words
100
60x91
Class label
EVOBC (Guan et al,, 2024a)
Handprinted
50
207x212
Interpretation
Deciphering
OBI Component 20 (Hu al, 2024)
Handprinted
50
66x64
Interpretation
HUST-OBS (Wang al., 2024a)
Handprinted
40
112x172
Interpretation
original oracle bone recognition (02BR) dataset and OBI-rejoin dataset as complements. Aligned with existing practices (Yue al., 2024; Wu al., 2024), each image in OBI-Bench equipped with question alongside correct answer. Moreover, we craft spectrum of questions: Whul question, Yes-or-No question, How question, and Where question, evaluating from coarse-grained perception to finer-grained perception. A total of 23 popular LMMs are selected for evaluation including 6 proprictary LMMs (e.g., GPT-4o, Qwen- VL-Max, and Gemini 1.5 Pro) and 17 open-source LMMs (e.g., LLaVA -Next, InternVL2, and mPLUG-OW(3). We showcase some empirical findings here: I) LMMs still have much room for improvement in fine-grained OBI recognition tasks such as quantity detection locating; 2) LMMs are not sensitive enough to local information such the borders of the fragmented margin to directly meet the requirements of rejoining. Meanwhile, part of LMMs can help OBI scholars narrow down the range of qualified fragments (GPT-40 and Gemini 1.5 Pro achieve an average performance of 77.57% on the Acc@ 10 metric); 3) LMMs can be applied in classification and retrieval tasks, especially for cleaner handprinted datasets; partial LMMs rival the performance of public-level humans in deciphering tasks and exhibit remarkable visual reasoning ability interpreting those genuine undeciphered oracle bone characters. The full list of our findings from different OBI tasks is in Sec. 3.
By evaluating up-to-date LMMs from different perspectives (OBI research demands), we gain insights into their strengths, limitations, and potential directions for improvement. Ultimately, our objective is to facilitate the field of paleography by fostering the development of more reliable, unbiased, and task-oriented LMMs that meet the needs of OBI experts while upholding trustworthy standards and contributing to historical debates.
2
CONSTRUCTING THE OBI-BENCH
2.1 GENERAL PRINCIPLES
Focusing OBI Tosk-oriented Abilities of LMMs. Different from existing LMM benchmarks (Liu et al., 2023b; Li ct al., 2024b; Liu et al., 2024b; Yue et al., 2024) that either aim at all-round abilities or generic scenarios, the evaluation principles in OBI-Bench are OBI task-oriented. Specifically, we focus on five major issues in the field of oracle bone inscription research: 1) Recognition, involves the positioning and identification of OBI characters on different carriers. 2) Rejoining, refers the process of stitching broken OBI fragments together to rebuild complete inscriptions. 3) Classification, means categorizing the recognized OBI characters into groups based on their shapes or meanings. 4) Retrieval, is to find a cognate character from large collections of OBIs. 5) Deciphering, includes interpreting the meaning of the oracle bone characters even the contextual semantic information. We adhere to the principles in designing the visual and cognitive tasks, making the proposed OBI-Bench a focused reflection on the OBI processing abilities of L.Ms.
Covering Multi-stage Font Appearances. To cover the diverse appearances of the oracle bones since their excavation, we collect multi-sourced images for each task, as depicted in Tab. 1. In particular, considering that there are currently no available original oracle bone recognition datasets and rejoining datasets, we proposed the O2BR and OBI-rejoin datasets respectively (See more details in App. B). Due to the specific requirement of domain knowledge in OBI research, three domain experts and one senior OBI scholar were involved in the annotation process of O2BR and OBI-rejoin well as the sampling process of all source datasets. We include all the manifestations of
3

---

Published as a conference paper at ICLR 2025
Recognition
Rejoining
Coarse-g graind Fine- spaalii
Parception Perceptii Pecepiion
Classification
Retrieval
Deciphering
Where: Faetun bounding now for
Correc (tei4, 110, 1143 Arower gDA 0933
How: How much probatillyys is
How: How much protusiiity i
How:
How: how predit
sech debecaed enecla laaw cheradar in jemn, yman, arme, ymaud formal
there the theoe tan crecie boee
there thot these two cracle tono
meening of the cradi boin
474,
Frogmnnts rojened togethor?
charadirrs ong the seme dess?
charader??
How: How many orade bono thanaders are thes image?
Coneet foeear mareel
cas roaee loee werk see
rodia cane
Commant cfriuen
beking tottin
Coreuit a reforacc
Yes-ar-No: is this onide
boow ineorption image??
Conrect drower (M): DNV
Weong Answer:
Yes-ar-No: Whother
rejoined together?
lbolongg the some class? K
Yes- ou- No: Whathor
crede charndless
Yes-or-No:
Sanwe
What: What the mocn ng of thtir cracle boee chamider??
Camsuc without discution
What' What Image?
imuge urail rectipaa
roud some
Wh denwee PPo)
Canrelt Resueer der (Pod
Durresl Assumc (i) (rid)
Wounn Aeewee indd
cerioit apooo te ina
Image (I)
the diee loca la reggl
muceen twen
sure Noie war]
OBI
Modern 15
Chinees
Excavation
Synthesized
Figure 2: Sampled OBI-Bench examples from each task. 5,523 (I, Q A) tuples span two quadrants of OBI concerns and encompass four types of questions, providing an all-around evaluation of the ability of LMMs on OBI tasks. Note that OBI classification and retrieval tasks share the same queries.
the entire process of OBI processing from excavation (i.e., original oracle bone) to artificial synthesis (i.e., handprinted or computer-generated The diverse and multiple sources of images morph the OBI-Bench into a holistic and balanced benchmark to fairly evaluate the competency of LMMs in OBI-related tasks.
2.2 BENCHMARK FROM VISUAL ABILITY TO COGNITIVE ABILITY
In the five tasks of OBI-Bench. we evaluate the visual perception and high-level cognitive abilities of LMMs to examine whether they can handle OBI problems by using simple natural queries. For this purpose, we collect 5,523 images (T) from multiple sources (Tab. 1) with diverse task concerns. Then, we design different question prompts (Q) based on the type of tasks and obtain the correct answer (A) from the corresponding dataset for each image. The 5,523 (I, Q, A) tuples compose into the first visual question answering (VQA) testbench (Fig. 2) in the OBI field. Specifically,, the questions in OBI-Bench cover two quadrants of concerns for evaluation (Sec. 2.2.1) and four question types (Sec. 2.2.2). The details are elaborated as follows.
2.2.1 QUADRANTS FOR EVALUATION CONCERNS
Axis From Coarse-grained Perception to Fine-grained Perception. The primary axis differenti- ates the visual perception granularity of OBI tasks. First, coarse-grained perception focuses on the basic forms, themes, or overall structures of the oracle bone inscriptions, which distinguishes OBIs from other symbols or ancient characters (Qiao et al., 2024). Second, fine-grained perception (Xing et al., 2019; Fujikawa et al., 2023) not only involves identifying subtle characteristics, quantity, and spatial positions of OBIs but also reflects the high-level cognitive ability of LMMs (e.g., OBI domain expertise and interdisciplinary reasoning). Several studies (Liu et al., 2023a; He et al., 2024; Tong al., 2024) followed this paradigm and built on it to refine pixel-level perception while extending interactions beyond text-specific inputs.
Axis 2: From Excavation to Synthesis. In the era predating the prevalence of artificial intelligence (AI), unearthed oracle bone fragments were normally processed through the manual rubbing and image cropping processes for preservation. After thousands of years of natural weathering, corrosion, and man-made made destruction, there are various types of noise in oracle bone rubbings. To improve the readability of the original OBIs and expand the data size that could be processed by deep learning algorithms, researcher obtained pseudo-oracle bone characters through handwriting or generative models (Han et al., 2020; Li et al., 2020; Wang et al., 2024b). Acknowledging these forms of OBIs, curate 5.523 OBI images from different stages of processing, that require LMMs to grasp the content or other relevant domain knowledge to answer correctly.
2.2.2 QUESTION TYPES
In OBI-Bench, we design four question types, What, Yes-or-No, How, and Where, to simulate multiple human queries in various OBI tasks, each tailored to different levels of granularity follows.
What Question. The What questions are a common type of query in some LMM benchmarks (Wu et al., 2024; Zhang et al., 2024b). In OBI-Bench, they serve as global graiinnd perception in
4

---

Published as a conference paper at ICLR 2025
Table 2: Performance comparison on the OBI recognition task. The best result is marked in bold and the second-bes result is underlined for both proprictary and open-source LMMs respectively.
Datasets
02BR
YinQiWen YuanaWeeaanneetetee
LSM (Variant)
Wuiif 0.9364
Nos
Move
Wherel
Whaat
Yos
Row?
Wheree
HUMAN (pwulicc)
0.0033
0.9272
0,9189
100%
0.0060
0.8776
Proprietary LMMs:
GEMINI 1.5 PRO
0,5726
98.75%
0.4932
0.1126
0.3557
99.85%
0.3811
0.0586
GEMINI 1.5 FLASH
0.4123
96.50%
1.7857
0.0962
0.3425
97.35%
0.4522
0.0276
GPT-4V
0.5408
99.30%
0.4223
0.0022
0.3701
99.65%
0.4583
0.0165
GPT-40 (ver. 0896)
0.6114
99.95%
0.4016
0.0038
0.3734
99.90%
0,3458
0.0182
QWEN-VI. (ver. (8009)
0.6071
99.63%
0.4799
0.0086
0.3375
99.55%
0.4845
0.0131
GLM-4V
0.5319
39.17%
0.3681
0.0041
0.3635
52.45%
0.3632
0.0124
Open-source LMMs:
XGen-MM Trattrac-innereeeree
0.5236
81.75%
1.2437
0.0233
0.3669
100%
".4121
0.0515
mPLUG-OwI3 (Qwen2-78)
0.3342
99.88% %
0.4474
0,0811
0,2505
99.95%
0.6593
0.0527
MiniCPM-V 2.6 (Qwen2-78)
0,5576
88.75%
0.4829
0.0384
0,3781
99.10%
1.1793
0
Moondream2 (ver. 0728)
0.4818
98.25%
0.6795
0,0400
0.3049
91.30%
0.4436
0.0653
InterVVT.2-4 www.V..--.mmm-37771 (70m
0,5833
99.65%
0.4328
0.0976
0.3892
99.75%
0.5344
0,0623
Intern 40B (Noux Hermes2 Yi 34-4
0.5664
98, 98.35%
0.4561
0.0766
0.3637
99 99.05%
0.6733
0.0487
InternVT.2.80 (fnterddM2.5.. M275577)
0.5232
95 00%
0.4618
0.0020
0.3429
97.95%
1.1146
0.0152
GLM-4V-9B (GLM-4499)) 98)
0,5388
29.50%
0.4825
<10c-4
0,2839
15.70%
0.3934
<10c-4
CogVLM2--Limm2-198 (Llama3-88)
0,5321
61 61.00%
0.6928
<10c-4
0.3966
91.75%
0.5568
0.0002
VA-NN-NeX (Qwen1_55722F)
0.4846
99, 75%
1.1011
0.0445
0.3297
99. 75%
1.0561
0.0591
LLaVA-NeXT (Llama3-8B)
0.4764
93.13%
0.5512
0.0001
0,3120
93.90%
0.4268
0,0189
IDFFICS-2-88 (Mistral-77)) 7
0.3175
85.
0.4916
<10c-4
0.2658
95%
0.5119
<10e-4
DeepSeek (DeepSeek- LLMS5ee--LLM-BB
0.5111
92.75%
0.5657
0.0449
0.3386
98.00%
0.6263
DDeennLMXCCoppsr22-Y VL (IniemLLM2-78)
0,5106
99.88 %
0.6641
0.0049
0,2661
99.95%
1.5231
0.0281
LLaVA-v1.5 Vicuua-vl..-11300
0.4416
99.00%
2.7553
0.0751
0.2875
98.05%
1.7662
0.0493
LLaVA-v1.5 (Vicuma-v1.5-78)
0.4239
91.88%
0.8861
0.0656
0,2729
81.90%
1,9621
0.0465
Owen-VI.cc (Qwenn)))
0.4489
74.13%
1.7416
0.0003
0,3137
86.81%
3.5694
0.0069
recognition task (e.g., What is in this image?), unbound the answers in deciphering task (e.g. What the meaning of this oracle hone character?). contrast to other question types, the What questions examine more comprehensive visual understanding and cognitive abilities of LMMs, by requiring correct attribute perception and knowledge integration.
Yes-or-No Question. As a fundamental type of judgment, Yes-or-No represents a binary output. This design minimizes the ambiguity of the model answers and is closer to the task objectives than the open- open-ended responses of What questions.
How Question. Except from two common types, we also include the How questions to further refine the responses as extension to Yes-or-No questions. As shown in Fig. 2, we can query How many oracle bone characters are in this image? for the recognition task (Fu et al., 2023) or query How much probability is there that these two oracle bone characters belong to the same class? for the classification task to derive quantitative results predicted by LMMs.
Where Question. We specifically employ the Where question for the recognition task. For instance, we can query Return a bounding box for each detected oracle bone character in [xmin, ymin, xmaxx ymax] format to achieve pixel-level OBI anchoring, which requires the LMM to have fine- -ne-grained perception, including spatial location, shape, and relationships between detected objects.
3 EXPERIMENTS
In OBI-Bench, we evaluate 17 up-to-date popular and competitive open-source LMMs, together with 6 proprictary LMMs, under zero-shot settings. More results and analyses are appended in App. C.
3.1 EVALUATION ON RECOGNITION
In this section, we examine the recognition ability of LMMs, comparing them from coarse-grained global content perception to fine-grained OBI locating.
Setup. To evaluate how well LMMs perform under various OBI recognition scenarios, we design four evaluation schemes across two image sources (i.e., original oracle bone from our self-curated O2BR dataset and inked rubbings from the YinQiWen Yuandewstion (AYNU, 2020) dataset): (1) evaluating via the What query to measure the preciseness of the global perceptual descriptions on OBIs. Given the commonality of contents in sampled test sets, we predefined reference descriptions for each of the
5

---

Published as a conference paper at ICLR 2025
Table 3: Results on the OBI rejoining task. 'Yes-orNNo' and 'How' represent the absolute output and probability output, respectively. We report 5 for the 'Yes-or-No" query. Given that not all LMMs support multi-image inputs natively, we shrink the number of evaluated models.
How
LMM (variont)
Yes-or-Nof
Acc
Acc 70
GEMINI 1.5 PRO
28.53%
24.88%
46.37%
76.67%
GEMINI 1,5 FLASH
22.17%
19.33%
36.34%
66.76%
GPT-4v
27.63%
26.86%
43.75%
73.75%
GPT-4o (ver: 0806)
32.21
29.13%
48.43%
78.47%
QWEN-VI.-MN VL.-MAX (ver. 0809)
21.77%
13.14%
31.68%
56.67%
GLM-4V
5.33%
4.58%
12.19%
21.46%
XGen-MM (Instruct-interttare--)
12.08%
3.11%
9.32%
11.18%
mPLLIG-OW13 (Qwen2-7B)
14.48%
3.52%
12.63%
17.60%
MiniCPM-V 2.6 (Qwen2-7B)
13.23%
2.69%
11.18%
16.98%
MitenVL22-Liamm3-760 (Llama3-70B)
20.13%
5.18%
16.98%
31.68%
Intern VL2-40B Wouss-lermess2----k/ BB-34B
16.00%
3.31%
10.97%
24.84%
(nterLLM2.5-7B))
14.68%
3.73%
10.14%
17.81%
L.LVVA-NeTT (Qwenl.5-728)
17.45%
4.97%
14.29%
27.74%
LLaVA-NeXT (Laamu388))
11.41%
3.93%
9.52%
16.77%
IDEFICS-2-8F (Mistral-7B)
7.38%
3.52%
12.42%
14.91%
DeepScck-VV VL. (DeepSeek-LLMM-7))
9,40%
4.35%
8.90%
11.59%
WL (Wwn-VL(QQwen771 (Qwen-7B)
9.86%
4.02%
8,96%
14.08%
two datasets. The max_ tokens for all LMMs is set to 200. The similarity between text embeddings used as the evaluation metric. (2) evaluating via the Yes-or-No query after specifying the response direction (the words "OBI" appear explicitly in the query), designed to complement the evaluation on coarse-grained perception. The accuracy of answers adopted as the evaluation criterion. (3) evaluating via the How query to quantify the ability of LMMs in parsing irregularly structured character-level OBI contents. We use mean relative error (MRE) as the metric. (4) evaluating via the Where query to achieve finer-grained perception (i.e., dense oracle bone character locating), designed to more effectively uncover the practicability of LMMs in real OBI processing pipelines. Mean intersection over union (mloU) is used the metrie. More details are in App. C.2.
Results. In scenario (1), as shown in Tab. 2. among proprictary LMMs, the recently-released released GPT-4o reaches the best performance in terms of the relevance of answers on coarse- -gaained perception, followed by performance improvement, these models provide more precise answers than the open-source LMMs. Similar improvement can be observed for the same series of LMM (e.g., LLaVA and InterVVL2) as the number of parameters of the language backbone increases. In scenario (2), we add keyword (i.e., oracle bone) directly to a Yes-or-No question to further evaluate the global perception of LMMs. We find most LMMs perform nearly 100% correctness, indicating that these models are highly effective in handling queries that involve explicitly directed content. One exception is the GLM-4V series that only achieves an average accuracy of 3... We speculate it is due to the answer preference bias (Wu et al., 2024). A deeper analysis is in App. C.2.2. In scenario (3), transition to examining the fine-grae grained visual perceptual of LMMs, i.e., counting the number of oracle bone characters. Our results in Tab. 2 indicate that GLM-4V and (nteen//22-Llama--777
which shows rather close results. By achieving over 12.4% on average
exhibit the best performance within their respective division groups Additionally, we observe some abnormally large values (>50) or repeated values from Gemini 1.5 Flash, XGen-MM-4B, LLaVA-NeXT-72B. LLL.V--1..5-133 Qwen-V VL that result in their relatively large MRE (Zhou et al., 2024). In scenario (4), all evaluated LMMs consistently underperform. Gemini 1.5 Pro significantly outperforms other L.Mss (surpasses GPT-4y and GPT-4o two orders of magnitude) (Ueno & Lynn, 2024) with its exclusive coordinate return function. Overall, current LMMs are still not usable for character-level OBI locating and are far from the public-level human.
and
3.2 EVALUATION ON REJOINING
In this section, we delve into the rejoining performance of LMMS against various eroded, tiny, and irregularly structured fragments of OBI, focusing on the estimation of rejoinable fragments. This task is also major concern in the recovery of paper money, calligraphy, and painting files recovery.
Setup. Given the absence of publicly available rejoining datasets in the OBI domain, we create dataset consisting of 200 complete OBI pieces across two appearances (iee, original and inked

---

Published as a conference paper at ICLR 2025
Table 4: Classification accuracy (%) on HWOBC, Oracle-50k, and OBI125 datasets (delimited by slashes) under different queries. 'Yes-rrNNo' and 'How' represent the absolute output and probability output, respectively. We report Aco@ for the Yes-or-Noo query.
How?
LMM (variant)
Yes-or-Not
Acc
GEMINI 1.5 PRO
68.75%/66. 00%/55 55%
88, 08.159//88.59%7777...
30%%
100.0%
GEMINI 1.5 FLASH
02.50%%/60.50% 50%//50.555/
22.0099//2.500%771... 707.77
NM0.0%/99.500%:./
1.0%%
GPT-4v
69.50%
19.50% 00%/5..75%
86.75% 0010////8882557775555577
100.0%/100.0%%/.....
GPT-40 (ver: 0806)
72.75% 53..5%
10.5500%%22... /62.50% %
89.75%
2/775.5075 50%
100.0%/100.0%/9..
59%
QWEN-VL TWEN-VVL-MAX (ver. 0809)
64.25%%/65.755// 75%/55500%
85.00% 251.88855///999 2.5%
GLMM4V
35.50//29..7 755.775%//44450
02.5099/9..559//34
71.00% 555555455555444555555
83,50/4/85.
(0./8/577777777 75%//77.75%
XGen-MM (Instrucr-nteerfeve---
46.75% http:/488..000/444..3
37.75
009; (199//49.005 000%
mPLUG-Ow[3 (Qwen2-7B)
39.25%./39. 75%
44.75%
56.25%/55.500
MiniCPM-V 2.6 (Qwen2-7B)
40.75% 00442...00///.000/
45.75% 00%/42.75%%
56.75%/55. (11555575%% 75%
Intem_V.22LLanaal (7lama3-70B)
44.75% 500%/43...2. .5%
53.75% /55.00 %/50.75 %
69.75% 90.5%%/69.0000 % %
InlemVL.2-40B (Nows-Hermess-WV--5)
42.75%%/43.5005
49. 49.75%%/50 /505 70..5555...0%//..... 75%
63.75% /625 2,5 25% 35%
IntemVL2-8B (httmLLM2.577)
05225%/41.75%%/3 75%%
47.75% 00%//4.7554
59.75% 00% /56.50%
LLaVA-NeXT (@wenl.5-72B)
46.25% 541 (4.000%
51.75%//5 00%%/4900%%
66.75% 75% 26%%
LLaVA-NeXT (Lloma3-88)
44.00%%/44.000///888 00%.000.%%
46.75%
25%/44.755%
53.75%
75%
IDEFICS-2-8B (Mistrul-7B)
14.75%//44 44.55%/44.50%% 50%
46.75% 50%/44.00%%
58.75%/58. 75%
DcepSeck-VL (DeepSeel-LLM-77)
42.25% (55///440.77/3...////. 75%6/35.75%
47.00% (.00%/4..255/44353. 25%//4.2256
56.75%/58. 25% 2585
Qwen- VL (Qwen-7B)
14.25%%/42.00%%/38.7555
48.00% 48.00%/551.00%//4445559
61.25% 70.99//6..50//...... 50%6//1.255
oracle bone fragments) with 483 rejoinable fragments. More details are in App. B.2. To evaluate the rejoining performance of LMMs on oracle bone fragments with different marginal sealability concerns, we elaborate two evaluation scenarios: (1) evaluating via standard Yes-or-No question that represents the absolute judgments to tentatively explore the rejoining performance; (2) evaluating via How question that outputs rejoinable probabilities for pairs of fragments, allowing for a more accurate and quantitative measurement. Accuracy @k k indices are utilized as evaluation metrics.
Results. Tab. 3 presents the evaluation results of LMMs on the OBI rejoining task, from which we gain several valuable observations as follows. First, GPT-40 and Intern VL2-Llama3-766. achieve the best performance in their respective categories, although there is still a huge gap between them. Second, among all proprietary LMMs, the rejoining performance under 'How' queries is significantly better than that under 'Yes-or-No' queries. This indicates that the answer in probabilistic form better reflects the differences in visual perception compared to absolute form output. Third, we notice that GPT--4o and Gemini 1.5 Pro reach over 76% in Ace@ 10 metric, illustrating the possibility of of using LMMs to assist the OBI rejoining efforts. Fourth, the number of parameters is roughly proportional to the performance, and newer models tend exhibit superiority under the same scale of language backbone models. Fifth, we find that the overall performance of open-source LMMs is still far away from being truly usable (only 3.85% 11. 11.39% and 18.65% on average terms of Acc@ 1, 5, and @ce@/A 10, respectively), and their ranks vary greatly under different evaluation criteria.
3.3 EVALUATION ON CLASSIFICATION
In this section, we evaluate the performance of LMMs on OBI classification tasks and try to answer: (1) How robust are LMMs in classifying orucle bone characters? Since the intra-class and inter-class similarity of OBIs are sometimes numerically close, we then investigate the capabilities of LMMs in focusing on atypical characteristics and try to answer: (2) How well do LMMs distinguish between different categories of OBIs?
Setup. We sample 500 images across 100 categories from each of the HWOBC, Oracle-50k, and OB1125 datasets test set. One image is kept as the label in each category and the remaining four as query images. To answer the above questions, we construct three evaluation scenarios: (1) evaluation on oracle bone rubbings and handprinted images, aiming to assess the robustness of LMMs to content qualities. Due to the inadequacy of manual rubbing and the limitation of scanning equipment, there inevitably various noises in rubbing images, which could greatly affect the classification accuracy; (2) evaluation on different queries (i.e., Yes-or-No question and How question) designed to generate absolute and probability forms of responses, aiming to assess the robustness of LMMs under varied query prompts; (3) evaluation on {10, 20, 30, 50, 100) classes that encompass mixed image sources and subsets characterized by similar structures, aiming to study the conditions under which LMMs will fail in classification. We thereby reconstruct a variable-quaa quantity mixed test set by sampling an equal number of oracle bone character classes from three datasets. More details are in App. C.3.
7

---

Published as a conference paper at ICLR 2025
Table 5: Performance comparison of LMMs on the OBI retrieval task. Results OBC306 and
OBI-UDD datasets are delimited by slashes. We report the averaged Recall@1, 1, Recall@ and Recall @ 10 on 'How' question.
Yes-or-Nof mAP@@ Yes
Howt
LMM (variant)
Ricial
Reculle
MAPSS
GEMINI 1.5 PRO
0.4316/0.5734
0,225/0,250
0.671/0.688
1.000/1000
0,644/0.76
GEMINI 1.5 FLASH
0.3876/0.5344
0.19//0.225
0.635/0.66 644
0.975/1000
0.562/0.74
GPT-4v
0,4228/0.5680
0.205/0.225
0.650/0.676
1.0001.000
0.624/0.74
GPT-40 (ver, 0806)
0,4550/0.6122
0.235/0.250
0.686/0.706
1.000/1.000
0.688/0.80
QWEN-VL-MAX (ver. 0809)
0.4223/0.5716
0, 195
0.621/0.638
1,00011000
0, 0.664/0.78
GLM-4V
0.3018/0.3867
0.125/0.175
0.495/0.613
0.925/1.000
0,520/0.72
XGen-MM (Instrac--nterleav--B3
0,2668/0.3423
0,085/0.200 200
0.336/0.569
0.855/0.975
0,366/0.66
mPLUG-Owe O0l0 (@wen2-78)
0,2814/0.3555
0.075/0.22 225
0.371/0.588
0.870/0.950
0.384/0.62
MiniCPM-V V 2.6 (Qwen2-78)
0,286340,3575
0.070/0.225
00.51
0.885/0.925
0.368/0.66
InternVL2-Llamaa-700 (Llama3-70B)
0.3557/0.4266
0.150/0.250
0.460/0.675
1.000110000
0.522/0.72
InternVL2-400B (Nous-Hermes22 Yi-38B
0,3260/0.50
0.125/0.250
0.445/0.655
L,00/1.000
0.480/0.68
InternVL2-8B (IntemLMM..577))
0,2844/0. 3623
0.095/0.22 225
0.374/0.650
0.925/1000
0.420/0.68
LLaVA-NeXT (Qwen1.5-72B)
0,3478/0.414
0.155/0.250
0,468/0.669
1000/1.00
0.562/0.70
LLaVA-NeXT (Llama3-88)
0.2793/0.3600
0.075/0.225
0.358/0.606
0.925/10000
0.348/0.60
IDEFICS-2-8B (Mistral-7B)
0,2844/0.3660
0.090/0.200
0.354/0.619
0.875/1,000
0,402/0.66
DeepSeck-VL (DeepSeel-LLM-7B)
0,2867/0.3450
0.105/0.225
0,343/0.600
0.930/1.000
0.416/0.64
VL. (Qwen-7H)
0,288340,3528
0.080/0.225
0.345/0.588
0.920/0.90 925
0.422/0.66
Results. In scenario (D, from Tab. 4, we can observe that the classification results on HWOBC and Oracle-50k datasets are markedly better than those on the OBI125 dataset (surpass by 16.4% and 19.2% for GPT-4o under 'Yes-orNNo' queries). We believe that this can be attributed to the image quality of OBI in the dataset, which affects the LMM's perception of character structure. Indeed, the two handprinted OBI datasets, HWOBC and Oracle-5Ok, are cleaner than the contammnnteee OBI125 dataset (Sec. C.3). In scenario (2), we find significant performance dis- crepancies between 'Yes-orNNo" and 'How' queries, which is similar the other tasks. This highlights the necessity of designing more appropriate textual answer forms and ssecifpcciiic metrics. Moreover, the Gemini 1.5 series is on par with that of their best competitors (GPT-4v and GPT-4o), which even achieves nearly 90% and 100% accuracy in terms of Ace@ and Ace@5 metries respectively. This proves the feasibil- ity of using LMMs for preliminary classification of OBI images. In scenario (3), we select four representative LMMs, i.e., EnternVL2-88 Intern VL2-40B, Intern VeeenLL---amma3661 and GPT-4v, for evaluation, and obtain three observations from Fig. 3. First, we notice overall performance improvement compared to the previous experimental setup since the source- mixee test set increases the inter-class differences. Second, the accuracy of these models decreases the number of classes increases. This shows that current L.Ms struggle to perform structural comparisons across a large number of categories, potentially limited by the granularity of their outputs. Third, compared to models with smaller scales, the performance degradation of Intern -L2-LLama3-7776 and GPT-4v less pronounced as the number of classification categories increases, indicating that LMMs with larger parameter sizes tend to have better robustness.
Halee 2-08
100
3-452
80
Ass/it
60-
40
20
Number Classes
Figure 3: Effects of the number of char- acter categories on classification accu- racy.
3.4 EVALUATION ON RETRIEVAL
In addition to classification, the retrieval task is also critical for building large-scale OBI image databases. Considering that LMMs combine powerful visual perception and natural language understanding capabilities, they can serve as the demand processing core of OBI retrieval systems. Here, we evaluate the performance of LMMs on OBI retrieval tasks given different types of queries.
Setup. We adopt two query formats (i.e., Yes-or-No and How questions) akin to those employed in the classification tasks. A total of 600 images sampled from OBC306 and OBI-IJDH datasets are used distractors to simulate real
used for evaluation. Among each test set, half of the images OBI retrieval scenario. During experiments, OBI images in the same category are retrieved. We take the averaged Recall@k & and mean Average Precision (mAP) metrics to quantify the OBI retrieval
8

---

Published as a conference paper at ICLR 2025
BERTSeore:D 0.196
GPT-40
BERTS 0.235
What mesning Bis bono chutacla??
Chade Boow aracter imag ondort Chiness sal pc. This soript was usod duriig the Shong deratty and is one enriest
"Euc"
What the meaning of this orasoo bone sharadeer?
Drocie Bone charadte the Image represents
"moouhh" Chinase soript This chanatee is pictogng dopicting 3 human mouth whth hax been presened mocem
Golden Description "his dartoon "kng" sggtying Landent Chinese
Golddn Description [Tle bathe cienacee" deantts "dhai" tepeaall prnsorn.
witting.
wrihin
Qwen--L -L -ax
0,2829
yourd
Owen-VL Max
0.3143
The oracle bone chareeer you provides dedd reprosands the contapt of & one of the okissi Chisse dharatios
cracie bore characiee you pronaded gieo
OBI
largs"
OBI
Modemn
-
the cest "penun" "ceapli ore of the cidert Cherese charaders and is still used modern C Chinase wrtting.
Chinese writing
T-40
Scoce- 5885
GPT-40 BEK Score D,77
The Chade Hone charecter represents sain (Tfil top pottion the character vertioll below symbo tine tallinn to the gro.
The Chade Boxe thanatier image it visuely desik sorralligg branching out or ming whic comsisent with the modeen Chineur character
onacce bone charmes woar?
orace bone characler?
Description made donotes or The uppo doy whi the lover peet repareents saneirees..
Description The Grace lone character "cote" icomasisaate an 1815 etr, St so0tis.... Madern Chinese
Owen-VLL-Max
02295
importent
Qwen-VVL-Max
0,3335
Uhe orade bona character you provided E (sars, in cinnn Chisass this character recrasentt numberr "thrss"" one
The once bone charater you prov ded is AC mrth In angien Chinete, this charamer represents concegs "wre" nese characters and is stll modern Oh nese witting
OBI
Chinese
Moddam
FR
OBI
cued in moders Chinass wiiting.
Figure 4: Qualitative comparison of deciphering results between two state-of-the-ar LMMs, i.e., GPT-4o and Qwen-VL-Max, in single round of direct questioning. It is noted that neither GPT-4o nor
has fully deciphered these four oracle bone characters.
performance of LMMs in multi-round queries. Note that we use mAP@ Yes) as the evaluation metric in Yes-or-No scenarios, where |Yes| denotes the number of 'Yes' answers.
Results. As presented in Tab. 5, most proprictary LMMs can well retrieve relevant OBI contents from test pools except GL.M-vv, which is on average 40.48% worse than the other five in terms of Recall@ 1. Besides, when expanding the search scope to ten candidates, all proprietary LMMs as well as over 63.6% open-source LMMs perfectly retrieve all relevant OBI contents in OBI-IJDH. However, the retrieval performance fell off a cliff in the OBC306 dataset, which we attribute to the same image quality problems encountered with the classification task. As visualized in Fig. 1, such cropped oracle bone rubbings contain various distortion problems, such as stroke- bbokkn, edge-cracked, spindles, and dense white regions, making them harder to tell apart and showing great necessity to add image enhancement measures before these tasks. Furthermore, models with a large number of parameters still perform relatively better in this task and How queries receive more precise search results.
3.5 EVALUATION ON DECIPHERING
Deciphering the oracle bone inscriptions has long been the ultimate goal of this ancient language study. Large multi-modal models have strong cross-modal understanding capabilities, enabling the models to perform multilingual visual-text transformation with linguistic analysis. In this section, we examine the potential OBI deciphering abilities of LMMs by asking: (1) Can LMMs interpret oracle bone characters correctly? (2) How do models behave in the face of different character formation principles (c.g., pictograph, ideogram, and radical-bass based variants) and character frequencies? (3) What is the reasoning mechanism of LMMs during the OBI deciphering process?
Setup. To evaluate the decipherment performance of LMMs. we construct four evaluation scenarios: (1) deciphering oracle bone characters with different frequency levels F. Three test sets (i.e., Tier-l:
F2 500, 7ier-2: 100 F < 500, Tier-3: 10 < F < 100) with a total of 40 characters sampled from the HUST-OBS dataset according to the taxonomy in (Liu, 2010; Chen, 2010); (2) deciphering oracle bone characters in pictograph formation. We follow the category definitions in (Chou, 1979;
Da WEIH, 2021) and choose 28 representative OBI characters from the EVOBC dataset as test set; (3) deciphering oracle bone characters in ideogram formation. Similar to scenario (2). 22 OBI characters from the EVOBC dataset are selected; (4) deciphering oracle bone characters with structurally similar variants (i.e., radical). We pick out 10 different radicals from the OBI Component 20 dataset (Hu et al., 2024), each accompanied by 5 component-levee variants. Fig. 9, Fig. 10, and Fig. 11 in the appendix visualize the sampled test sets for the above scenarios. Different from other approaches (Chang et al., 2022; Guan et al., 2024b) that take the decipherment of OBI establishing correspondences with modern Chinese characters, we directly use the LMMs to generate a segment of interpretation, as illustrated in Fig. 4. Given the absence of dedicated evaluation criteria for OBI decipherment, produce text embeddings for all decipherment results and calculate the BERTScore (Zhang et al., 2019) with lapert-labelled golden descriptions (i.e., text references). More details are in App. B.3.
Results. For scenario (1), Tab. 6 presents the evaluation results across different character frequencies. We find that the evaluated LMMs achieve on average 0.1905, 0.1898, and 0. 1791 on Tierl, Tier2,

---

Published as a conference paper at ICLR 2025
Table 6: Results on OBI deciphering task. Given a reference deseription x = ( mg) and a candidate deciphered result å (fi,... F2k,, we use contextual embeddings to represent the tokens, and compute matching using cosine similarity.
Datasets
HUST-OBS
EVOBC
OBI Component 20
Average
LMM (variant)
T
7ier-2
Tier-3
pictogroph
idengrion
Rodical
HUMAN (public)
0.4507
0.3884
0.3437
0,4966
0.3627
0,2812
0,3872
GEMINI 1.5 PRO
0.1766
0.3834
0,3589
0.4226
0.3696
0,3750
0.3810
GEMINI 1.5 FLASH
0.3545
0.3829
0.3567
0.3661
0.3595
0.3648
0.3641
GPT-4V
0.3764
0.3808
0.3596
0,4424
0.3551
0.4066
0.3867
GPT-40 (ver: 0806)
0.3891
0,3510
0,3660
0.4535
0.3893
0.3767
0.3876
QWIN- QWIN-VL-MAX VL-MAX (ver. 0809)
0.2345
0.2273
0.2322
0.2565
0,2533
0.2785
0.2471
GLM-4V
0,2180
0,1638
0,2122
0,2353
0.2451
0.1683
0.2071
XGen-MM (Intriccsineeeeeeeee8))
0.1194
0,1352
00.1369
0.1526
0.1268
0,1031
0.1283
mPLUG-OW[3 (Qwen2-7B)
0.1609
0.1822
0.1897
0.2036
0.1751
0.0684
0.1633
MiniCPM-V 2.6 (Qwen2-78)
0,1411
0,1694
0,1584
0,1703
0.1529
0.0907
0,1471
Moondream2 (ver 0728)
0.1129
0,1171
0,1063
0.1297
0.1176
0,1155
0.1
1itemWL2-LLaana3-7000 (1lama3-708)
0.2324
0,2139
0.2156
0.2355
0,2123
0.1881
0.2163
Intem NNous-Hrrmes--1-3444
0.1966
0.1917
0,1883
0,2007
0.1902
0.1376
0.1842
IntemVVL2-88 UnternLMM..-77B)
0.1676
0.1739
0.1340
0.1746
0.1517
0,1670
0.1615
GLM-4V-9B (GIM-4-9B)
0.1291
0.0821
0.0478
0.0835
0.0909
0.0399
0.0789
VL 1.lama3-111111 19B (Ilama3 8B)
0.1096
0.0673
0.0852
0,1024
0.1079
0.1192
0.1019
LLaVA-NeXI (Qwvenl.5772
0.1501
0,1647
0.1372
0.1468
0.1123
0.0863
0.1329
LLaVA-NeXT (Lloma3-8B)
0.0893
0.0873
0.0892
0.0806
0.0813
0.0298
0.0763
IDFFICS-2.8D (Mistrul-7P) 7R)
0.1639
0,1252
0.0985
0.1601
0,2189
0.1517
0,1531
DeepSeck-VL VL (DeepSeek-LLM-75
0.0458
0.0628
0.0504
0.0324
0.0593
0.0931
0.0573
Intern XCompose22 V1. (Inteml
0.2963
0.3101
0.2660
0.2650
0.0626
0.2304
LLaVA-v1.5 Hicuma-v1.5-J330
0.1368
0,1669
0.1629
0.1128
0,1214
0,1168
0,1363
LLaVA-v1.5 (Vccma-v1.5-7B)
0.0677
0.1165
0.0906
0.0924
0.0887
0,1488
0.1008
Owen-VT.(Qwee--))
0.1145
0.0912
0.0757
0.1080
0.0635
0.0744
0.0879
and Tier3 subsets, respectively. indicating that the deciphering performance on common characters is better than that on rare characters. For scenario (2) and (3), the deciphering results on pictograph are more accurate than ideogram (average BERTScore: 0.2012> 0. 1837), which shows the characteristics of pictographs evolving from drawings and reveals the deciphering mechanism of LMMs, that is, by establishing visual associations between font structure and real objects. For scenario (4), we notice the worst performance (average BERTScore = 0.166) on the Rudical subset compared to other test sets, which means LMMs impeded in distinguishing and deciphering component- source LMMs by an average of 145.9% across
variants.
Furthermore, proprietary LMMs outperform all deciphering tasks, suggesting that the open-source LMMs still have a substantial gap to fill for practical application. It is worth noting that GPT-4o, GPT-4v, and Gemini 1.5 Pro have approached or exceeded public-level humans in some scenarios. Surprisingly, M-xerllMXXooppseer2 the Top-/ performance among all open-source LMMs (even surpassing Intern VL2-Llama3-766 and IntemVL2-400) which owns several times of parameters). We suspect that this might be attributed to the specific training datasets (Cai et al., 2024) since the match of data content sometimes outweighs the sheer quantity of parameters (Zhang et al., 2024a). Meanwhile, we observe some highly repetitive outputs (such as 'The image you'l 've provided appears to be a stylized representation of an oracle bone charucter.' or other meaningless answers) in the VALVV--Ne' series and its previous version LLLVAA- v1.5, which leads to inferior deciphering results in all scenarios. Fig. 4 provides an example of the deciphering results of GPT-4o and Qwen-VL-Ma. VL-Max on four high-frequency oracle bone characters. We also investigate the effect of adding pre- pre-prompt during querying in App. C.4.1. More deciphered results for currently undeciphered OBI are presented in App. C.4.2.
reaches
4 CONCLUSION
We provide comprehensive evaluations of the applicability of LMMs in OBI domain from different perspectives. Overall, there is still considerable room for LMMs to improve when addressing tasks that need domain-so -saacifc knowledge. However, taking LMM as a unique technique to assist studies on oracle bone inscriptions is a promising direction, which may significantly reduce learning costs for domain professionals and help them solve OBI-related problems more easily than learning or training traditional deep learning algorithms. Additionally, there are many properties of inputs that affect the performance of OBI processing based on our evaluations, which is worth further exploring. We hope these results can bring insights into the future improvements of LMMs for finer- mee-ggaineed and more robust visual perception. We address the ethical concerns of our research in App. A and discuss the limitations of this work in App. D.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by the National Key R&D Program of China under Grant 2021 YPE022060000 in part by the National Social Science Fund of China under Grant 24Z300404220,, and in part by the Shanghai Office of Philosophy and Social Science under Grant 2023BYY00S.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprini wrXiv:2303.08772 2023.
AYNU. Yingiwenyuan: Oracle bone character detection dataset. https: / //tp::://gww... aynu. edu. cn/ html ?sysid=3, 2020. Accessed: 2024-07-21.
home/ /oone//wwn// http:/detaaal/nnnnnnn
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint 2r0iv:2308. /2966, 2023.
Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Marcella Cornaa, and Rita Cucchiara. The revolution of multimodal large language models: A survey. XrXiv preprint A.KV::2402.1245111 2024.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint 2rXiv:2403.17297, 2024.
Xiang Chang, Fei Chao, Changjing Shang, and Qiang Shen. Sundial-gan: A cascade generative adversarial networks framework for deciphering oracle bone inscriptions. In Proceedings of the 30th ACM International Conference Multimedia, pp. 1195-1203, 2022.
Banghao Chen, Zhaofeng Zhang, Nicolas Langrené, and Shengxin Zhu. Unleashing the poten- tial of prompt engineering in large language models: a comprehensive review. arXiv preprint arXiv:2310. 14735, 2023.
Tingzhu Chen. Restudy of the structural system of the Yin Shang oracle bone inscriptions. Shanghai People's ' Publishing House, 2010.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual- linguiiic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185 24199, 2024a.
Zijian Chen, Wei Sun, Jun Jia, Fangfang Lu, Zicheng Zhang, Jing Liu, Ru Huang, Xiongkuo Min, and Guangtao Zhai. Band-2k: Banding artifact noticeable database for banding detection and quality assessment. IEEE Transactions on Circuits and Systems for Video Technology, 2024b.
Zijian Chen, Wei Sun, Yuan Tian, Jun Jia, Zicheng Zhang, Jiarui Wang, Ru Huang, Xiongkuo Min, Guangtao Zhai, and Wenjun Zhang. Gaia: Rethinking action quality assessment for ti-generated videos. arXiv preprint arXiv:2406.0660
2024c.
Zijian Chen, Wei Sun, Haoning Wu, Zicheng Zhang, Jun Jia, Ru Huang, Xiongkuo Min, Guangtao Zhai, and Wenjun Zhang. Study of subjective and objective naturalness assessment of ai-generated images. IEEE Transactions on Circuits and Systems for Video Technology, 2024d.
Zijian Chen, Wei Sun, Jun Jia, Ru Huang, Fangfang Lu, Ying Chen, Xiongkuo Min, Guangtao Zhai, and Wenjun Zhang, Joint luminance-chrominance learning for image debanding. IEEE Transactions on Circuits and Systems for Video Technology, pp. 1-1, 2025. doi: 10.1109TTCSVT. 2025.3544030.
Hung- Chou. Chinese oracle bones. Scientific American, 240(4): 1979.
Research Centre for Humanities Computing CUHK. Multi-function chinese character database.
https: /hhuannummmmmmmmmmmmmm arts. s. cuhk. edu, hwtp://w//eeieiieeee hatsss/eexx///i/m 2014. Accessed: 2024- 08-26.
11

---

Published as a conference paper at ICLR 2025
Dalu Da WEIH. Deciphering the oracle bone script and its derived ideograms. The International Journal of Chinese Character Studies, 4(2): 233, 2021.
Xiaolei Diao, Daqian Shi, Jian Li, Lida Shi, Mingzhe Yue, Ruihua Qi, Chuntao Li, and Hao Xu. Toward zero-shot character recognition: gold standard dataset with radical-Ievel annotations. In Proceedings of the 3 31st ACM International Conference on Multimedia, pp. 6869-6877, 2023.
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. Interolm-xcomposer2: Mastering free-form text-image composition and comprehension vision-language large model. arXiv preprint arXiv:2401.1642 16420, 2024.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
Yoshiyuki Fujikawa and Lin Meng. Obi dataset for ijdh and obi recognition application. http: h/www//io... ihpp..p....... se. wew.riituumme. ac htt.:jp/BBBBaa wBwddaaaaeeeDDa 2ip, 2020. Accessed: 2024-07-
21.
Yoshiyuki Fujikawa, Hengyi Li, Xuebin Yue, CV Aravinda, G Amar Prabhu, and Lin Meng. Recog- nition of oracle bone inscriptions by using two deep learning models. International Journal of Digital Humanities, 5(2):65-79, 2023.
Feng Gao, Xu Chen, Bang Li, Yongge Liu, Runhua Jiang, and Yahong Han. Linking unknown characters via oracle bone inscriptions retrieval. Multimedia Systems, 30(3):125, 2024.
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al. Chatglm: family of large language models from g3m-130b to glm-4 all tools. arXiv preprint arXiv:2406. 12793, 2024.
Haisu Guan, Jinpeng Wan, Yuliang Liu, Pengjie Wang, Kaile Zhang, Zhebin Kuang, Xinyu Wang, Xiang Bai, and Lianwen Jin. An open dataset for the evolution of oracle bone characters: Evobe. arXiv preprint arXiv2240.112667 2024a.
Haisu Guan, Huanxin Yang, Xinyu Wang, Shengwei Han, Yongge Liu, Lianwen Jin, Xiang Bai, and Yuliang Liu. Deciphering oracle bone language with diffusion models. arXiv preprint arXiv:2406. 00684, 2024b.
Jun Guo, Changhu Wang, Edgar Koman-Range Hongyang Chao, and Yong Rui. Building hierar- chical representations for oracle character and sketch recognition. IEEE Transactions on Image Processing. 25(1):104-11 118, 2015.
Moruo Guo. Collections of oracle-bonn inscriptions. The Zhong Hua Book Press, 1982.
Wenhui Han, Xinlin Ren, Hangyu Lin, Yanwei Fu, and Xiangyang Xue. Self-supervised learning of orc-bert augmentator for recognizing few-shot oracle characters. In Proceedings of the Asian Conference on Computer Vision, 2020.
Junwen He, Yifan Wang, Lijun Wang, Huchuan Lu, Jun-Yan He, Jin-Peng Lan, Bin Luo, and Xuansong Xie. Multi-modal instruction tuned lims with fine-grained visual perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13980-13990, 2024.
Zhikai Hu, Yiu-ming Cheung, Yonggang Zhang, Peiying Zhang, and Pui-ling Tang, Component- level oracle bone inscription retrieval. In Proceedings of the 2024 International Conference Multimedia Retrieval, pp. 647-656, 2024.
Shuangping Huang. Haobin Wang, Yongge Liu, Xiaosong Shi, and Lianwen Jin. Obc306: A large- scale oracle bone character recognition dataset. 20/9 International Conference on Document Analysis and Recognition (ICDAR), pp. 681-688. IEEF, 2019.
Tianshu Huang. The Fifth Collection of Oracle Bone Rejoinings. The Xue Yuan Book Press, 2019.
12

---

Published as a conference paper at ICLR 2025
Runhua Jiang, Yongge Liu, Boyuan Zhang, Xu Chen, Deng Li, and Yahong Han. Oraclepoints: A hybrid neural representation for oracle character. In Proceedings of the 31st ACM International Conference Multimedia, pp. 7901-7911, 2023.
Le Kang, Peng Ye, Yi Li, and David Doermann. Convolutional neural networks for no-reference image quality assessment. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1733-1740, 2014.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint 2xXi::4405.022466 2024.
Bang Li, Qianwen Dai, Feng Gao, Weiye Zhu, Qiang Li, and Yongge Liu. Hwobe-a handwriting oracle bone character recognition database. In Journal of Physics: Conference Series, volume 1651, pp. 012050. IOP Publishing, 2020.
Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger Ilms supercharge multimodal ca- pabilities in the wild, May 2024a. URL https://llava 2024- 05 10- 1lava next stronger 11ms/.
github. wtcchb..o//lll.. 1o/blog/
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13299-13308, 2024b.
Hongming Lin. Zui Gu: Research on Oracle Bone Rejoinings. The Taiwan Book Store Press, 2008.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. The Twelfth International Conference Learning Representations, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024a.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating Ilms agents. In The Twelfth International Conference on Learning Representations, 2024b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He. Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXhv:2307.0628/ 2023b.
Zhiji Liu. On the two concentration features of the character frequency of bone inscriptions. Studies in Language and Linguisties, 30(04): 14-1211-2222 2010.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, al. Deepseek-vl: towards real-world vision- Saniuall understanding. arXiv preprint arXiv:2403.05525, 2024.
Yi Men. Six groups of newly joined huang type oracle bone inscriptions. Cultural Relics of Central China, 2:88-91, 110-91,22 2008.
Moondream.ai. Moondream2. https: httpps://mondream.. ai, 2024. Accessed: 2024-08-24.
OpenAl. Hello gpt-4o. 2024-08-27.
20244 Accessed:
OpenAl. Learning to reason with llms. htttps::/openat.ccmm/// 2024b. 09-13.
2024b. Accessed: 2024-
Runqi Qiao, Lan Yang, Kaiyue Pang, and Honggang Zhang. Making visual sense of oracle bones for you and me. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12656-1 16656-16665, 12665, 2024.
13

---

Published as a conference paper at ICLR 2025
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, cran-baptiste Alayrac. Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055300 2024.
Hossein Talebi and Peyman Milanfar. Nima: Neural image assessment. IEEE transactions image processing, 27(8):3998-400 2018.
Yuan Tian, Guo Lu, Yichao Yan, Guangtao Zhai, Li Chen, and Zhiyong Gao. A coding framework and benchmark towards low-bitrate video understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9568-95889999 9578, 2024.
Chemao Tsai. Catalog of Oracle Bone Rejoinings. The Le Xue Book Press, 1999.
Tzutalin. Labeling. httppss//gttcabb.cmm/zztaaaaaaae 08-23.
Img, 2015. Accessed: 2024-
Leo Ueno and Trevor Lynn. Gpt-4o: The comprehensive guide and explanation. https: /blog. roboflow. com/ 4os--si-viiiiii weloonuss-ccssee 2024. Accessed: 2024-08-30.
UNESCO.
Chinese oracle-bone inscriptions.
https: : ///////n./ unesco.org
memoryo
2017. Accessed: 2024-09-12.
Mei Wang and Weihong Deng. A dataset of oracle characters for benchmarking machine learning algorithms. Scientific Data, 11(1):87, 2024.
Mei Wang, Weihong Deng, and Cheng-Lin Liu. Unsupervised structure-fextuure separation network for oracle character recognition. IEEE Transactions on Image Processing. 31:3137-3150, 2022.
Pengjie Wang, Kaile Zhang, Yuliang Liu, Jinpeng Wan, Haisu Guan, Zhebin Kuang, Xinyu Wang, Lianwen Jin, and Xiang Bai. An open dataset for oracle bone script recognition and decipherment. arXiv preprint arXiv:2401.15366 2024a.
Shibin Wang, Wenjie Guo, Yubo Xu, Dong Liu, and Xueshan Li. Coarse-to-fin fine generative model for oracle bone inscriptions inpainting. In Proceedings of the 1st Workshop on Machine Learning for Ancieni Languages (ML4AL 2024), pp. 107-114. Association for Computational Linguistics, 2024b.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi. Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.0300 2023.
Haoning Wu, Zicheng Zhang. Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, et al. Q-beuch: A benchmark for general-purposs. foundation models on low-level vision. In The Twelfth International Conference Learning Representations, 2024.
Jici Xing, Guoying Liu, and Jing Xiong. Oracle bone inscription detection: a survey of oracle bone inscription detection based on deep learning algorithm. Proceedings of the International Conference Artificial Intelligence, Information Processing and Cloud Computing, pp. 1-8, 2019.
Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael Ryoo, al. xgen-mm (blip-3): A family of open large multimodal models. arXiv preprint arXiv:2408.0888 2024.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianci Cai, Haoyu Li. Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone, arXiv preprint arXiv:2408. 01800, 2024.
14

---

Published as a conference paper at ICLR 2025
Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou, mplug-ow13: Towards long image-sequence understanding multi- large language models. arXiv preprint arXiv:2408.04488 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal under- standing and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024.
Xuebin Yue, Hengyi Li, Yoshiyuki Fujikawa, and Lin Meng. Dynamic dataset augmentation for deep learning-based oracle bone inscriptions recognition. ACM Journal on Computing and Cultural Heritage, 15(4):1-22)) 2022.
Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets Ilm finetuning: The effect of data, model and finetuning method. arXiv preprint 2tti:2402.1119933 2024a.
Chongsheng Zhang, Ruixing Zong, Shuang Cao, Yi Men, and Bofeng Mo. Ai-powered oracle bone inscriptions recognition and fragments rejoining. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences Artificial Intelligence, pp. 5309-3311 2021a.
Chongsheng Zhang, Bin Wang, Ke Chen, Ruixing Zong. Bo-feng Mo, Yi Men, George Almpanidis, Shanxiong Chen, and Xiangliang Zhang. Data-driven oracle bone rejoining: A dataset and practical self-supervised learning scheme. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4482-4492, 2022.
Gechuan Zhang, Dairui Liu, Barry Smyth, and Ruihai Dong. Deciphering ancient chinese oracle bone inscriptions using case-based reasoning. In International Conference on Case-Baser Reasoning, pp. 309-324. Springer, 2021b.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 2019.
Yuheng Zhang. Compilation of Oracle Bone Inscriptions: Reconstruction and Interpretation. The Wanjuanlou Book Press, 2020.
Zhan Zhang, Yun-Tian Wang, Bang Li, An Guo, and Cheng-Lin Liu. Deep rejoining model for oracle bone fragment image. In Asian Conference on Pattern Recognition, pp. 3-15. Springer, 2021c
Zicheng Zhang, Haoning Wu, Chunyi Li, Yingjie Zhou, Wei Sun, Xiongkuo Min, Zijian Chen, Xiaohong Liu, Weisi Lin, and Guangtao Zhai A-bench: Are Imms masters at evaluating ai- generated images? arXiv preprint arXiv:2406.03077 03000000007, 2024b.
Lexin Zhou, Wout Schellaert, Fernando Martínez-Plumen Yael Moros-Daval Cèsar Ferri, and José Gernández-Oralloo Larger and more instructable language models become less reliable. Nature, pp. 1-8, 2024.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-leve prompt engineers. In The Eleventh International Conference on Learning Representations, 2023.
15

---

Published as a conference paper at ICLR 2025
Begin
Previous imagee
unads ina iingge?
Next image
Yea
What is this image?
1cm
R R37777
Figure 5: Interface of subjective experiments for the recognition task.
APPENDIX
A ETHICAL DISCUSSIONS
A.1 ETHICAL DISCUSSIONS OF OUR RESEARCH
Our work holds the potential for significant social impact, both positive and negative. Firstly, the main motivation of this work is that the archaeological discoveries of oracle bone inscriptions (OBI) are normally considered as highly serendipitous findings while their interpretation relies heavily on the large volume of manual efforts (from excavation to digital archiving), yet most of their meanings are still unknown, hindering historians and sociologists to further understand the contents of production and life in ancient times. In the early stage of OBI research, the deciphering oracle bone characters is highly limited, either by referring to ancient books or by working backward from perspective font evolution, which was a fairly stagnant state for much time. Recently, with a large volume of large multi-me modil models (LMMs) claiming their powerful cross-modal information processing capability, it unknown for both computer vision experts and OBI scholars LMMs can identify and decipher them. In our work, find while most LMMss are still not completely applicable in some OBI processing tasks that need fine-grained perception and deeper cognition, exhibits trend that the performance of the model improves year by year and relies its design for specific tasks. Our findings offer promising direction that using LMMs to tackle the OBI problems has lower knowledge cost than the traditional model-task in one-to-one form certain occasions. Secondly, our research on the model applicability in OBI tasks provides a necessary understanding of the nature and potential causes of the inferior perception and cognition abilities of LMMs in the chowledge-specifff domain. The evaluation of model performance and the subsequent discoveries would spark broader discussion on the rational use of L.MMs to facilitate the study of ancient characters. Our work could serve as a reference point for discussions on developing next-generation LMMs and evaluation metrics for the OBI area. Thirdly, we acknowledge that unregulated and unlimited deciphering results of LMMs may present potential risks in generating misinformation and spreading false cultural values.
A.2 ETHICAL DISCUSSIONS OF DATA COLLECTION
In this section, we detail the ethical concerns that might arise during the experiments. Specifically, we invite five Chinese college students with no background in the ancient Chinese language represent public-level performance. They are instructed to perform two tasks that are of great interest to OBI scholars, Le., recognition and deciphering. Note that all participants clearly informed of the
16

---

Published as a conference paper at ICLR 2025
umto 12
Kowel
Figure 6: Sample images of the original oracle bone fragments in the O2BR dataset.
contents in experiments. Similar to (Chen et al., 2024b;e;d), we addressed the ethical challenges each subject a signed and informed agreement that they agreed their subjective
by obtaining from opinions would be used for non-commercial research, making it equipped with such legal and ethical characteristics.
For the recognition task, we divide the whole subjective experiment into two stages considering the workload and avoiding visual fatigue. In the first stage, participants are asked to describe the provided image contents and determine whether it is an OBI image, thus completing the What and Yes-or-No queries, as depicted in Fig. 5. In the second stage, participants are told to delineate the positions of individual oracle bone characters as they perceive them. We adopt the commonly used toolbox "Labellmg" (Tzutalin, 2015) for annotation. The resulting number of bounding boxes and the corresponding coordinates are used for evaluating the performance under How and Where queries. As for the deciphering task, participants are told to write down their understanding of each oracle bone character, which is then taken to compare with the golden description for alignment evaluation. The user interface is the same as the recognition task (Fig. 5). Overall, it took over a week to complete the whole subjective experiment, where each participant contributed an average of 1 14.8 hours to attend.
B MORE INFORMATION ON BENCHMARK DATASETS
B.1 THE PROPOSED O2BR DATASET
The objective of recognizing oracle bone inscription lies in accurately locating each single character on original oracle bones, rubbings, or fragments, thus assisting downstream classification or deciphering tasks (Xing et al., 2019). In the past decade, a variety of datasets (Guo et al., 2015; Huang et al., 2019; Han et al., 2020; AYNU, 2020; Zhang et al., 2021a; Yue et al., 2022; Wang et al., 2022) have been established for oracle bone character recognition. Among them, Oracle-20k (Guo et al., 2015) was proposed under a controllable lab environment with a total of 20,039 synthesized OBI images in 291 categories. Later, Oracle-50k (Han al., 2020) was released with more diverse character classes and images. In (Zhang et al., 2021a), a large-scale dataset with character-leved annotations, OracleBone- 8000, was collected, which contains 7,824 OBI rubbing images scanned from (Guo, 1982) and 128,770 annotated character crops. Similarly, YinQiWen Yuan-detection (AYNU, 2020) dataset contributed 9,306 annotated OBI rubbing images with character-level bounding boxes. However, the image contents of the above OBI recognition datasets are all handprinted rubbing form, rather than original oracle bone. More importantly, the majority of datasets either lack the location information for individual oracle characters do not release the data publicly. To address these problems, propose the Original Oracle Bone Recognition (O2BR) dataset, which consists of 800 carefully curated original oracle bone images and 4,211 bounding boxes.
17

---

Published as a conference paper at ICLR 2025
B.1.1 SOURCE CONTENT COLLECTION
To curate the O2BR dataset, we began with the initial filtering of available websites using hashtags commonly associated with oracle bone image, such as #oracle bone archaeology. #she and bone script, and # #ancient character. This process allowed us obtain set of candidate websites on oracle collection supported by civic organizations, museums, and research institutes around the world. Considering the type, quantity, and usage requirements of the images, we choose to download original oracle bone images from the website' of open museum led by the Institute of History& & Philology, Academia Sinica. Out of the potentially accessible retrieved results, around thirty thousand contained downloadable and visible OBI photos as of July 2024. We utilized a Python-based crawler to download a total of 21,453 raw images. Note that all images downloaded from the website are released under an appropriate Creative Commons (CC) license that allows further editing and redistribution. After that, two graduate students with oracle research backgrounds were involved in removing small fragments that do not contain complete oracle bone characters.
Finally, 800 original oracle bone images are selected as the annotation source. Fig. 6 presents several samples of this dataset. Tab. provides detailed statistics.
B.1.2 ANNOTATION PROCESS
Some studies (Li et al., 2020; Diao et al., 2023) suggested the use of automatic object detection methods to save manual annotation effort. However, considering the particularity of the image content in this dataset and the lack of relevant trainable datasets, we adopted the commonly used toolbox "LabelImg" (Tzutalin, 2015) to conduct the character-level annotation. Fig. 7(a) shows the annotation interface. Two OBI domain experts participated in this process. Besides, avoid duplication of evaluation tasks in OBI-Bench, we merely required the annotator to frame the location of a single oracle bone character. Afterward, a senior OBI domain expert was invited to proofread the annotations. If there a disagreement between two experts, the senior expert will re-annotate the image and determine the final annotation, which helps to guarantee the quality of our proposed dataset.
Table 7: Statistics of the pro- posed O2BR dataset.
Attribute
Value piseks)
Minimum
167
Maximum Width
Minimum Height
168
Maximam Height.
2664
Avenge Wilh
1579
Averuge Height
1192
Arg. sumber of bounding botes/unage 35 5,264
B.2 THE PROPOSED OBI-REJOIN DATASET
In the early days, the most important reference in OBI domain is the "Collections oracle bone inscriptions" " book (Guo, 1982), which contains nearly 2,000 groups of OBI rejoining results. Other repre- sentative books such as "Catalog of Oracle Bone Rejoinings" (Tsai, 1999). "Zui Gu: Research on Oracle Bone Rejoinings" (Lin, 2008), "The Fifth Collection of Oracle Bone Rejoinings" (Huang, 2019), and "Compilation of Oracle Bone Inscriptions: Reconstruction and Interpretation" (Zhang, 2020) involve the achievements of over 4,000 groups of OBI rejoining results in the past 30 years. However, there is no publicly available rejoining dataset for any type of OBIs in digital image form. Recently, researchers collected real-world dataset for rejoining Oracle Bone fragments (Zhang et al., 2022), which consists of 998 oracle bone rubbing images with manually outlined top and bottom marginal areas. Unfortunately, its proprietary nature limits its use by the community. This motivates us to build the OBI-rejoin, an OBI dataset comprising 200 complete OBI pieces with 483 rejoinable fragments, shown in Fig. 8. Tab. 8 lists the statistical information of OBI-rejoin dataset.
Table 8: Statistics of the pro-
posed OBI-rejoin dataset.
Value (in pixels)
Attribute
map
fragments
Minimum Width
64
46
Maximum Width
1268
909
Minimum Height
107
29
Maximum Height
2913
1273
Average Width
370
265
Average Height
465
313
Avg. nunber of fragments//mpp = 2.415
B.2.1 SOURCE CONTENT COLLECTION
We used Python-based crawler to scrape metadata from the official website2 of the Pre-Qin Research Office at the Institute of History, Chinese Academy of Social Sciences. This process covered 217
ptttps://opennuseum..o tw/objects
18

---

Published as a conference paper at ICLR 2025
LOL
(a) Image annotation interface for the O2BR dataset
(b) Image segmentation interface for the OBI-rejoin dataset
Figure The illustration of the annotation interfaces for the O2BR dataset (coordinate) on recognition task, and the OBI-rejoin dataset (adjacency) rejoining task.
Figure 8: Sample images of the original OBI rejoining results (left) well the separated fragments (right) in the OBI-rejoin dataset.
19

---

Published as a conference paper at ICLR 2025
T
7
Σ
+
OBI
Modern Chinese
#
J
H
2
F
X
T
+
m
X
4
-
3
OBI
0
+
X
8)
5p
3
E
Chinese
Ticr-2
Y
OBI
2
1
[
7
4
Chinese
Tier-3
A
D
Modern Chinese
H
3
Figure 9: Visualization of 40 oracle bone characters with different frequency levels.
pages and encompassed the time frame from December 2005 to July 2024, resulting in 4,249 OBI images'. To ensure data quality, conducted an expert examination that manually excluded hand- drawn contour images as well as extraneous contents, retaining only inked rubbings and original oracle bone forms the image sources of the OBI-rejoin dataset. As result, a total of 200 OBI images covering two forms of OBI (i.e., original and inked) with different character styles and resolutions are selected.
B.2.2
ANNOTATION PROCESS
Here, we refer to the OBI rejoining results as "map" to reflect its special role in the construction process OBI-rejoin dataset. Three domain experts in OBI research and one senior OBI scholar were involved in the annotation process of OBI-rejoin. Each annotator was equipped with PC with commercial photo editing software. Fig. 7(b) illustrates the annotation interface. To precisely reproduce the authentic conditions of oracle bone fractures, the annotator manually tears "maps" on appropriate borders utilizing their domain expertise to induce central or peripheral fractures. double-check procedure applied after the annotation to ensure quality and rationality. Overall, it took each annotator approximately 3 hours to finish annotation, and the whole construction process took over week complete. A lot of effort and discussion retrieve suitable rejoining results from the crawled metadata. To the best of our knowledge, OBI- rejoin is the first open-source oracle bone inscription rejoining dataset. We hope that OBI-rejoin can facilitate future development of advancing OBI rejoining algorithms and serve as benchmark dataset evaluate the performance of rejoining methods.
required in order to manually
B.3 COLLECTING GOLDEN DESCRIPTIONS FOR DECIPHERING TASK
The existing OBI deciphering works merely decipher oracle bone characters by linking them to modemn Chinese characters (Chang et al., 2022; Guan et al., 2024b), which does not constitute true decipherment. With the historical and cultural changes, many oracle bone characters can no longer find correspondence with modern Chinese characters. Thus, turn the most traditional dictionary form and extract golden descriptions from an authoritative Chinese character database (CUHK, 2014).
We extend our deepest gratitude the frontline rescarchers and scholars involved in the meticulous collation and proofreading of the oracle hone inscriptions.
20

---

Published as a conference paper at ICLR 2025
Pictograph
x
3
A
pd
OBI
Modern Chinese
f
+
4
7J
OBI
# A
$
D
Modern Chinese
3
D
Ideogram
9
-
OBI
Modern Chinese
F
+
#
7]
+
7
OBI
Modern Chinese
R
#
$
Figure 10: Visualization of 28 and 22 oracle bone characters in pictograph and ideogram formation, respectively.
3
1
o
' Radical
11
OBI
T
O
OBI
Modern Chinese
8
*
Modern Chinese
*
Radical
7
1u2
OBI
11
4x
g Rudicul
OBI
0
3/4 8
Radical
Modemn Chinese
E
Modern Chinese
iF=
H
3 2
$
A
イ +
Modern Chinese
Modern Chinese
Radical
Radical
g
OBI
$
g
f
OBI
2
i/
Modem Chinese
$
Modern Chinese
7
3)
Radical
Rudical
OBI
OBI
Modern Chinese
24
Modern Chinese
Radical
[o]
Radical
Figure 11: Visualization of 10 different radicals with their structural variants well as the corre- sponding modern Chinese characters.
21

---

Published as a conference paper at ICLR 2025
Table 9: A brief view of 17 open-source LMMs evaluated in the OBI-Bench in chronological order.
Vision Architectures (V)
V-1.
Language Architecteres (L)
Model Names
Backbone
ASize
Aligmment
Backbone
Sigi TPAM0M
364
Perceiver resimpler
Phi mini
decnder-anly
N "mPLUC-OWBBBB (Ye 2024)
SigLIP-400M
384
MLP projectur
Qwen2-7B
decoder only
MiiiiiCPM 2.6-8B (Yao oll, 2024)
Sigt JP-400M
224
Perceiver resampler
Qwee2-78
decndes-anly
MMondremm221.8 5B MMcodream..., 2024)
SigLIP-400M
378
MLP projector
Phi-1.5
decuder-anly
24 1 (Chon al, 20244)
InteraNIT-6RR
448
MI.P projector
Hermes-2-Theite
decuder-anly
420 (Chen et al, 2024a3
InteraViT-GBB
448
MLP projector
Nous Hermes 2- 34B
unnyder-anly
24.7 112ee1112-888 (Chen 2024c)
InterVVTTS00M
MLP projector
Internl-M2.5788
drcudes-auly
GLMM-VV99 (GLM 2024)
ELI-
224
MLP projector
GLM-4-9B
decoder-anly
(Wang et ol,, 2023)
EVA2-CLIP-
224
MLP projector
Laamo-33 3-----8- Instruct
decuder-anly
24 LLa al. 2024a)
336
MLP projector
Qwen1.5-7 5-720
only
2415 LLLVAA-NNXTTTBB (Li 201-4))
CLIP-WI-L/14
336
MLP MLP projector
Llama-3 In--trctt
decoder-anly
"DDEFICCS 2- Laurençon ct (202-)
SigLip 100M
384
MLP projecior
Mistral-7B
decoder only
24 DeeepSecck
al, 2024)
SAM-B&SigLIP-L
10248:384
MLP projector
DeepSeea-LLLM-22
decoder-only
Insern CCC.LLLMMXCCmmuer222/22 VL-7B (Dong et al,, 202()
CLIP-VIT-L/IE
336
Partial LoRA
M2-7raLM2-7B
decoder only
21 LLLaWA-v..5-11B (Liu at. 2024aJ
LLP
336
MLP projector
Vicuna
decuder-only
"LLaVA-v1 (Liu et 2026k)
CLIP-VIT-L/4
336
MLP projectur
Vieuma-v1.5-78
decoder-only
21"Qwen VL (Bai 2023)
CLI--NIT-G11
448
Cross-atcntion stentint
Qwen-7B
decoder only
C ADDITIONAL EXPERIMENTAL RESULTS
C.I BENCHMARK CANDIDATES
To ensure the comprehensiveness and timeliness of the results, we select 23 up-to-date and prevailing
LMMs for evaluation. The proprietary LMMs include Gemini 1.5 Pro (Reid et al., 2024), Gemini 1.5 Flash (Reid et al., 2024), GPT-4v (Achiam et al., 2023), GPT-4o (OpenAI, 2024a), Qwen-VL-Max VL-Max (Bai et al., 2023), and GL.M-4v (GLM et al., 2024). The open-source LMMs include XGen-MM- instruct-interleave (Xue et al., 2024), mPI.UG-OW13-77B (Ye et al., 2024), MiniCPM-V 2.6-8B (Yao et al, 2024). Moondream22 1-.6 (Moondream.ai, ai, 2024), IntemmL2-Llama3-700 (Chen ct al., 2024a), InternVL2-40B (Chen et al., 2024a), IntemVL2--BB (Chen et al., 2024a), GLM-4V-9B (GLM et al., ( (Wang et al., 2023), LLaVA-NeXT-7238 (Li et al., 2024a), LLaVA-NeXT-8B (Li et al., 2024a), IDEFICS-2-8H Laurençon et al. (2024), DeepSeek-VL-7B (Lu (Dong et al., 2024), ALaVA-v1.5-133 (Liu et al.,
2024), BogVLLM2-LLaMA3-COO
et al., 2024), 2024a), LLaVA-v1.5-7 (Liu et al., 2024a), and QWen-VL (Bai et al., 2023). Tab. 9 summarizes and compares the vision and language architectures of open-source LMMs.
Additionally, considering the cultural uniqueness of OBI, and to ensure the representativeness of human performance on the OBI-Bench, recruited five Chinese college students with no background in the ancient Chinese language to reflect public cognition. Initially, we instructed all participants to have clear and consistent understanding of all drestion-answar patterns by familiarizing themselves with the tasks through exposure to similar cases. Note that we conduct user studies in the same order of tasks we test LMMs avoid interference between tasks. Given the inherent variability of LMMs, identical prompts can yield responses. To address the impact of such situations on our evaluation, we implemented 5-round average strategy.
C.2 ADDITIONAL DETAILS OF EVALUATION RECOGNITION
C.2.1 EVALUATION CRITERIA
For What question, we calculate the cosine similarity between the embeddings of output answers and the golden descriptions. Specifically, we use the Sentence Trans python library and model, which maps sentences to 338-dimensional dense vector space.
the all-MiniLM-16--
For How question, we use the relative counting error metric to measure the performance of different LMMs:
N Chruth
Careed
MRE
Chruth
i=1
where N is the number of evaluated OBI images. Ctruth and Opred denote the actual number of oracle bone characters in the image and the predicted number of oracle bone characters respectively.
co/ ses/seetence trans former aaaeea/aaaa /a aall -Mini I-M v2
22

---

Published as a conference paper at ICLR 2025
Table 10: Impact of the judgment bias. Answer accuracy of LMMs Yes-or-No questions.
LMM
Yes-or-No
GEMINI 1.5 PRO
95%
GEMINI 1.5 FLASH
93.37%
GPT-4v
95.13%
GPT-40
98.68%
OWEN-VL-MAX
92.26%
GLM-4V
86.68%
XGen-MM (Instract-iteereeee--B)
83.55%
mPLUG-OWIB (@wen2-7B)
87.68%
MiniCPM-V 2.6 (Qwen2-7B)
82.2 %
Intern 1ntemWL224.lam33766 (Llama3-70B)
93.59%
Inter (Nous-HHermes----BBl) -2-3332222222233
92.24%
InternVL.2-88 (Intern/
86.55 %
GLM-4V-9B (GLM-4-98B)
81.62%
CogVLM22-Llam33-199 (Llama3-8B)
88.68 %
LLaVA-NeXT (Qwenl.5-72B)
94.77%
V..aVA-NeXT (1lama3-88)
85.50%
IDEFICS-2-8B (Mistral-7B)
87.55%
DeepSeek- VL (DrepSSek-LLMM-225
84.37% %
MoonnlL--CCooooser2--- 21mppsser2-V.... (IntermI.M2-7B)
86.88%
LLaVA-v1.5 (Vicuna-vl.5-138)
79.93%
LLaVA-v1.5 1.5 (Vicuma-vl.--7B)
75.58% %
Qwen (Qwen-7B)
84.57%
C.2.2 EXTENDED RESULTS ON YES-OR-NO QUESTION
Considering that the test set used in the recognition task is composed solely of authentic oracle bone rubbings or fragments, it may be affected by the response preferences of LMMs in Yes-or-No questions. this section, we take a deeper analysis of the Yes-or-No judgment ability of LMMs, and whether these models can get similar accuracy on questions that should be answered with 'Yes', as those should be replied to as 'No'. Specifically, we re-curate a content-mixed test set, which consists of 1,000 OBI images and other images containing ancient texts (such as the inscription the bronzes, characters engraved on bamboo slips, and characters written on silk). As a result, the proportion of OBI images and non-OBI images is 1:1. As reported in Tab. 10, the performance of GLM-4V and GPM-4V-9B improves greatly compared to the results in the O2BR and YinQiWen Yuan datasets, while the other LMMs perform similar level.
C.3 ADDITIONAL DETAILS OF EVALUATION ON CLASSIFICATION
To explore the effect of OBI image quality on classification accuracy, we further select two classical im- age quality assessment (IQA) metrics, namely NIMA (Talebi & Milanfar, 2018) and CNNIQA (Kang et al., 2014), to measure the average image quality within each test sets. As listed Tab. 11, the numerical results of both metrics show that the image quality of HWOBC and Oracle-50k datasets is better than that of OBI125. A similar conclusion to the main paper can be drawn that enhancing the image quality applying denoising operation can improve the OBI classification accuracy of LMMs (Chen et al., 2025).
C.4 ADDITIONAL DETAILS OF EVALUATION ON DECIPHERING
C.4.1 EFFECT OF PRE- PROMPT
Existing studies (Chen et al., 2023; Zhou et al., 2023) indicate that providing specific instructions, contextual information, or role settings allows the model to under- stand the user's needs and expectations and thus provide more accurate and relevant response. We thereby employ two schemes (i.e., role assignment and case instruction) to investigate the effect of pre-prompt.
Table 11: Image quality comparison be- tween two handprinted OBI datasets and one cropped from real rubbings.
Metrie
HWOBC
Oracle-50k
OB1125
NIMAt
5.3003
3,9973
3,6710
CNNIQAI
0.7165
0.6229
0.5612
Prompt Template for Role Assignment:
23

---

Published as a conference paper at ICLR 2025
Table 12: Effect of pre-prompt on the OBI deciphering task. We select Gemini 1.5 Pro, GPT4o,
Owen-VL-Maxy VL-Max, and GLM-4V for evaluation.
Datasets
HUST-ORS
EVORC
OBT Component 20 Radical
LMM (variant)
Tier-I
Tier-2
Tier-3
pictograph
ideogram
Average
GEMINI 1.5 PRO
0.3766
0.3834
0,3589
0.4226
0,3696
0.3810
GEMINI 1.5 PRO+Rolc
0.3824
0.3863
0,3603
0.4347
0.3819
0.3738
0.3866
GEMINI 1,5 PRo+RolesCase
0.3837
0.3867
0,3626
0.4386
0,3779
0.3722
0,3870
GPT-40
0.3891
0.3510
0,3660
0.4535
0,3893
0.3767
0.3876
GPT-4o+Role
0.3925
0.3525
0.3557
0,4585
0,3817
0.3729
0.3856
GPT-Ao+Rolo++asee
0.3922
0.3586
0,3688
0.4545
0.3829
0.3783
0.3892
QWEN- VL-MAX
0.2345
0.22773
"0.2322
0.2566
0.2535
0.2785
0.2471
QWEN- MA-NVVL-Mxx-Rol
0.2447
0.2535
0.2533
0.2585
0,2401
0.2796
0.2549
OWEN-VL-MAXE
0.2518
0.2778
0.2617
0.2830
0.2457
GLM-4V
0.2180
0.1638
00.2122
0.2353
0.2451
0.I683
0.2071
GLM-4V-Rolc
0.1977
0.1984
0.1989
0,2554
0.2448
0.1689
0.2107
GLM-4V+ Rooee+Cuuse
0.2187
0.1968
0.2137
0,2585
0.2487
0.1710
0.2179
<Imagel>
<Image2>
<magee>>>
Figure 12: Example of three oracle bone character images that are taken as case instructions along with their golden descriptions.
#System: You are a senior oracle bone researcher who excels in interpreting ancient texts. Please help me
predict the meaning of the undeciphered oracle bone characters reasonably and accurately.
#User: What is the meaning of this oracle bone character? <image>
Prompt Template for Case Instruction: We further randomly select three cases (OBI images with correct descriptions) as case instructions (Fig. 12) after role assignment.
#System: Here are three examples of deciphered oracle bone characters.
#System: (1) <imagel> The oracle bone character in this image means the blood, which also represents the blood of animals offered to the gods in sacrifices.
#System: (2) <image2> The oracle bone character in this image refers to the pattern and texture of the bamboo tubes merged together on musical instrument.
#System: (3) <image3> The oracle bone character in this image means the armpit, with the small dots indicating the armpits. It is like a person sweating in his armpits.
#System: Please refer the case form given above for the following answers.
#User: What is the meaning of this oracle bone character? <image>
In Tab. 12, compare the deciphering performance of LMMs with without pre-prompl operations. can be observed that adding role assignment and case instruction slightly improves the accuracy and reasonableness of decipherment results for Gemini 1.5 Pro, GPT-4o, Owen-VL-Max, VL-Max, and GL.M-VV by an average of 2.02%, 0.40%, 8.50%, and 5.21% % respectively. This demonstrates the effectiveness and necessity of prompt engineering in refining the answers of LMMs and avoiding meaningless outputs.
C.4.2 DECIPHERING RESULTS ON UNKNOWN OBIS
We provide the deciphering results LMMs for previously undeciphered five oracle bone characters from the HUST-OBS dataset (Wang al., 2024a), whose interpretation is central the core of
24

---

Published as a conference paper at ICLR 2025
Table 13: Taxonomy of LMMs categorized by the affiliation of the first author.
Model
Affiliation of First Author
Country
Proprietary LMMs:
GEMINI 1.5 PRO
Google Inc.
USA
GEMINI 1.5 FLASH
Google Inc.
USA
GPT-4v
OpenAl Inc.
USA
GPT-40
OpenAl Inc.
USA
QWEN-VL-MAX
Alibaba Group
China
GLM-4V
Zhipu Al Inc.
China
Open-source LMMs:
XGen-MM (nssructinterleaee41))
Sufesforce AI Research
USA
mPLUG-Ow, (Qwen2-7B)
Alibaba Group
China
MiniCPM-V 2.6 (Qwen2-7B)
ModelBest Inc.
China
Moondream2 (ver. 0728)
M87 Labs
USA
Intern NeerVL22-Llmm3-7000 (Llama3-70B)
Shanghai Al Lab
China
Interm VL2-400 NNuu--ermees-Y---3
Shanghai Al Lab
China
Interm V12-88 (IntemL.M2.5-7B)
Shanghai Al Lab
China
GLM-4V-9B (GLM-4-98)
Zhipu AI Inc.
China
CogVLM2-Llama3- 199 (Llamu3-88)
Tsinghua University
China
LLaVA-NeXT (Qwen/.5-72B)
University of Aisconsin-Madisse
USA
LLaVA-NeXT (Llama3-8B)
University of Wisconsin-Madison
USA
IDEFICS-2-BB (Mistral-7B)
Hugging Face Inc.
USA
DeepSeek VL (DeepSeck-LLMM-78)
DeepSeek-AI AT Inc.
China
VL (IntermLM2-7B)
Shanghai AI Laboratory
China
LLaVA-v1.5 (Vicmma-vl.5-13)
University of Wisconsin-Madison
Madison
USA
LLaVA-v1.5 (Vicuna-v1.5-7B)
University of Wisconsin Madonnin-MMaiioo Alibaba
USA
Qwen- (Qwen-7BB
China
14
China
USA
12
Average Runking
8-
11,300
11,700
12.152
6
9.940
9.990
7.060
7010
2
Recognition
Rejoining
Classification
Retrieval
Deciphering
Figure Comparison of the average ranking of LMMs under region-based based taxonomy.
Pre-Qin cultural and historical research, in Fig. 14, Fig. 15, Fig. 16, Fig. 17, and Fig. 18. Different from existing decipherment studies (Zhang et al., 2021b; Guan et al., 2024b) that decipher oracle meudo-modern Chinese characters,
bone characters characters in subsequent writing systems we directly leverage the multi-modal capabilities of LMMs to decipher them as description in English. Since the correctness of oracle decipherment result usually determined by peer review, we plan to make the code used in OBI-Bench and comprehensive collection encompassing more decipherment outcomes publicly available. We hope this contribution will assist OBI researchers in advancing the study of ancient languages.
C.5 IMPACT OF POTENTIAL CULTURE BIAS
We further investigate the impact of different cultural backgrounds OBI processing performance. Tab. 13 classifies LMMs by the affiliation the first author. Among all evaluated LMMs, there are 12 LMMs from China and 11 LMMs from the USA. To ensure the fairness of the results, exclude the earliest LMM (Qwen-V VL) to make the number of models the same. Given the differences in evaluation criteria and scales of different tasks, report the average ranking of LMMs according
25

---

Published as a conference paper at ICLR 2025
the region-bee based taxonomy, shown in Fig. 13. We find that cultural differences did not particularly affect the overall performance of the models. Besides, the LMMs originating from China still have some distance behind those from the United States.
D LIMITATIONS
Although our study provides comprehensive evaluation of LMMs in OBI tasks, there several potential limitations acknowledged below. First, the images in our test sets are limited scope with regard to size, content types for each task, and resolutions, which constrains the applicability of the OBI-Bench. We merely sample partial images from YinQiWen Yuandstection HWOBC, Oracle-50k, OB1125, OBC306, OBI-IJDH, EVOBC, OBI Component 20, and HUST-OBS datasets as test sets. Our celf-collected O2BR and OBI-rejoin datasets contain 800 and 483 images. Besides, in the deciphering task, we selected only 140 oracle bone characters for evaluation, which is considerably fewer than the number of deciphered oracle bone characters currently known. Another limitation of our work is that the query-answe no extra information that gives the LMMs clues to the intent of the image. The lack of such context hinders performance in cases such as OBI deciphering where context (role assignment and case instruction), has been shown to aid in image interpretation (App. C.4.1). Furthermore, coarse-grained perception perspectives such as What query in the recognition task and deciphering task involve subjectivity and should be based general consensus while defining golden descriptions.
forms used in our benchmark are relatively simple. There is
These limitations highlight the need for related future research. We encourage the community to view work starting point and extend the evaluations and analysis to further uncover potential capabilities of LMMs not only in the OBI domain but also in other ancient character processing domains and design possible task-oriented practical strategies accordingly.
E FUTURE POSSIBILITIES
Apart from the comments for possible next steps of research related to LMMs and OBI that have already been given, this section is devoted to the extension for some of them and then more topics with good potential based upon our understanding and rethinking for the field.
New possibilities of LMM in aiding OBI processing advancement can be tried in number of major directions:
Focus on Robust Preprocessing Pipelines: Developing preprocessing techniques that can enhance low-quality or noisy OBI data, such as denoising, fragment reconstruction, or super-resolution methods, may significantly improve LMM's performance on these tasks.
Domain-Specific Fine-Tuning with Interdisciplinary Collaboration: Fine-tuning models on domain-specific datasets can help bridge the gap in domain knowledge. Integrating visual and textual modalities can enrich the understanding of oracle bone inscription, thus enhancing the overall interpretative capability. Bridging the gap between computer science and humanities disciplines is crucial for effective application of LMMs in OBI studies. Collaborating with historians and linguists to create multi-modal annotated OBI datasets may benefit this effort.
Interactive Question Answering Systems: Future applications could include interactive question- -aweoirinnn systems where users can inquire about oracle bone inscription meanings and historical contexts in natural language, leveraging large multi-moda models for intuitive and accessible interactions.
is hoped that these can provide actionable ideas for researchers and would trigger further discussion, and more importantly, new exploration in this field.
26

---

Published as a conference paper at ICLR 2025
OBI researcher:
Prompt (Prefix):
You are senior oracle hone researcher who excels in interpreting ancient tents. Please help me prodict the meaning of the undeciphered ornele bone charneters rensembbly and accurately. Prompt (Main):
What the meaning of this orncle hone character? <image+>
GPT-40:
Structural Elements:
The character has a top part that resembles roof-like structure person standing, central part that looks like an enelosed or diamond shape, and lower section resembling container or mouth shage.
Possible Interpretations:
Enclosure or Containment: The diamond-like like structure could symbolize something contained or caclosed, often seen characters that imply boundaries, houses, objects held within.
Communication or Exchange: The moutb-like shape at the bottom might suggest communication, mouth, or something related to speech or verbal exchange.
Human Figure: The top part, somewhat resembling person, could indicate involvement of human or anthoopmorphic representation. Potential Meanings:
This corbination could imply concept related to ritall activities, such "speaking within boundaries" (potentially linked offirings or ceremonial actions).
specific within ritual contect, possibly a "messenger" "interpecter" who has function within might also indicate defined space.
Given the structure, character might symbolize ceremonial or commanicative role within a ritual.
Gemini 1.5 Pro:
The charncter appears to be comhination of three components:
Top: A vertical line with two short borizonial strokes branching out al the top. This element oflen represents a person or human figure. knotted pattern. This element might signify hinding, rying, connecting. Bottom: A U-shope, which could depict a container, vessel, or receptaele.
Middle: A criss-
Possible Interpretations:
Considering the combination of these elements, here are a few plausible interpretations:
Offering Sacrifice: The buman figure above the knotted pattern and container could represeut somcone presenting an offering or
sacrifice in vessel.
Binding or Confinement: The knotted pattem could suggest binding or restricting something within the container.
Stornge Preservation: The container with the figure above might indieste storing preserving something valuable important.
GLM-4V:
The charneter depicted bere, if it's an ornele bone inscription, appears to be composed of two elements: The top part looks like "=,"" which can mean "three" or be part of other words with different meanings depending on context. It also resembles the choracter for horse (95), but without the four legs, which might suggest stylistic variation different menning here. The bottom part is "Li," which means "mouth." Combining these clements doesn't that has lost its original meaning over time. Inseriptions of this nature often require additional context from similar inseriptions or historical records to accurately interpret.
known word in classical Chinese, so this character may be a unique combination
Qwen-V Beenn-L--WWW VL-Mxx:
The charneter you provided appears to be combination of several elements, which might represent speeific concept or objoct. The character on the left seems to depict a stylized form of plant or a tree, which is common in oracle bone script. The character on the right appears to concept related to agriculture or farming, such as "plowing" or "cultivating." " However, without more context or additional charncters, it's difficult to provide definitive interprelation. Orucle bone characters oflen require broader context understand their full meaning
simplified representation of hand
tool, Given these elements, the combined character might represent
MiniCPM- 2.6:
The oracle bone character shown in the image representation of the concept "mulual trust" or "reciprocal trust." This interpretation comes from common understanding characters, which often convey abstract ideas concepts. visual elements suggest ngreement mutual commitment between two parties, as indieated by the crossed lines that resemble arms or hands reaching out each other.
Moondream2:
The ocacle bone charneter in the image a symbol used in ancient Chinese cultre to represent the concept of knowledee ledge and guidance. often associated
idea of seeking wisdom and insight from past or the future. The character is typically depicted as a stylized representation of bone, which is a symhol of strength and durahility. In the context of the image, the oracle bone character could be interpeeted as a symbol of the importance of learning from history and seeking guidance from the past to make informed decisions in the present or anticipate the future.
LLaVA-NeXT-sa
The image you've provided appears be an oracle bone character, which is an ancient Chinese character carved on animal bones or turtle shells. Oracle hune characters are form of divination that used in uncieni China predict the future und to communicute with the gods. The specific menning of the charcter would depend on the context in which it was used, as well as the interpretation of the diviner. Oracle bone characters are longer used for divination moder lines, bui they are still studiod by scholars for their historical and cultural significance.
Figure 14: A qualitative comparison for the deciphered results of 7 representative LMMs on a genuine undeciphered oracle bone character (I).
27

---

Published as a conference paper at ICLR 2025
OBI researcher:
Prompt (Prefix):
You aze senior oracle bone rescarcher who excels in interpreting ancient texts. Please help me to prodict the meaning the undeciphered oracle bone characters reasonably accurutely. Prompt (Main):
What the mesming of this aruele bone character? <imagel>
GPT-40:
Structural Elements:
The character appears to depict person or human figure, often seen in Oracle Bone script to represent actions, roles, or identiries. The presence what looks like of authority.
specific bead shape might indicate a particular person or role, such as leader, shaman, or figure
Comparative Analysis:
Similar human-like forms in Oracio Hone inseriptions are oflen associated with roles in riuals, ancestrul worship, or specitic societal functions like divinens or warrions If the head shape suggesis crown or distinctive mark, could represent a sigmiBcanl person, such king noble.
Contextual Implications:
this character found alongside characters related divination, or sucrifice, could denote a person involved in ritual. possibly priest or king. The overall posture and design suggest a figure of importance, likelly standing or commanding, which can indicate leadership or an authorilative role.
Symbolic Meaning:
It is possible that this charneter significs "person" (A), "chicf" (K), or ritual participant, depeading on the context within which is found. specific headdress or symbol on lop might further differentiate figure from generic "person" symbol, emphasizing a specialized role identity.
Reasonable Prediction:
Based the visual analysis and typical expresentations of human figures in Oracle Bone script, this character likely represents person significance, possibly leader, ritual participant, or an important societal figure. If more context from surmaunding inseriptions is available, it would help refine the role meaning.
Gemini 1.5 Pro:
Potential Pictographic Interpretations:
Human Figure: The most apparent interpretation that it represents a stylized human figure. The lead is clearly depicted with distinct headdress hainstlle. The outstretched arms and legs suggest posture of either standing. walking, or possibly dancing.
Symbolic Possibilities:
Leadership or Authority: The prominent headdress could signify a person of high status, leader, or a shaman. The upright posture might further emphasize authority commanding presence.
Riual or Dance: The pase could depicr someone engaged in ritualistic dance or ceremony, perhaps invoking spirits or deities.
Journey or Movement:
outstretched limbs might symbolize jounscy, migration, transition from
another.
GLM-4V:
The character you've shown uppeans to be composed two elements:
The top part resembles Ir (yi), which his varions meanings including "to support," "to receive," " or send." " The bottom part looks like "Z" (yu), which can mean "again," "plus," ar have other significations depending on the context. However, without knowing
each other within larger context or having similar characters that character is associated with, we cannot ascertain its exact meaning.
these elements
Qwen-V Owen-VL-Max VL-Max
The cnccle bone chameter you provided appears to be pictographic representation. Ornele bone script one of the earliest forms of Chinese writing, and its characters ullen depict objects or actions in simplified form. The character you've 've shown seems lo represent a person. The top part the character resembles head with hat headdress, and the lower part shows the body and legs. This type character is likely to be piclogram of person, possibly indicating a specifie role or status, such as a noble, a warrior, or pricst. However, without additional context complete sentence, it's challenging to determine the exact mesning, Oracle characters often need to he read in context to understand their full significance. If you have more chaneters complete inseription. it would he helpfill in providing a more accurate interpretation.
MiniCPM 2.6:
The oracle bone character shown in the image **k" (fa), which means husband or married man. This character is derived from ancient Chinese script and been used for thousands of years to denote marital status, particularly reference male spouse within family context.
Moondream2:
The oracle bone character, also known as the Chinese zodiac symbol. represents the year of the year according to the Chinese zodiac. It symbol that combines the shape of bone with hird-like figure. often featuring pointed hat. The ornele bone symbal used to determine the year of a person's birth, and it is believed to bring good fortune and guidauce. The specific meaning of the omclee bone character varies depending on the zodiae sign. bui generally, it represents the year's characteristics life lessons for sign.
ALLVVA-NeT--B-
The image you've provided appears be oraele hone character, which form ancient Chinese writing, Oracle bone characters were used for divination and were carved onto animal bones, typically exen, to predict the fulure The specific meaning of this character would depend the context which used, well as the interpectation of diviner. Oracle hone characters are no longer used for divination in modemo times but are still stadied by scholars for their historical and linguistie signifieance.
Figure 15: A qualitative comparison for the deciphered results of 7 representative LMMs on a genuine undeciphered oracle bone character (II).
28

---

Published as a conference paper at ICLR 2025
OBI researcher:
Prompt (Prefix):
You are senior oracle bone reseurcher who excels in interpreting ancient texts. Please help me predict the meaning the undeciphered ornele bone characters reasonably accurntely. Prompt (Main):
What the meaning of this oraole bone character? <imaget>
GPT-440:
Structural Elements:
The charncter has prominent vertical line tunning tarough the center, which may suggest pole, spear, or some form of division. The
upper part appears to branching hom--like structure, which oflen symbolizes something with multiple points, branches, limbs. The circular rounded shupe in the side of the central vertical line could represent container, a shield, or something enclosed.
Possible Interpretations:
Ritual or Swerifiee Context: The vertical line and branching elements migght hint tool object used in nitual practices, possibly
related to sacrifices or offerings.
Symbol of Authority or Protection: The vertieal component could imply staff or seepter, indicating authocity, while the circular form might suggest protection or containment, such as a shiold or boundary.
Animal Representation:
upper brunching forms could resemble antlers hors,, suggesting a connection to animals, passibly
symbol or totem.
Potential Meanings:
This character may symbolize concept tied protection, authority, rixua. It might represent guardian figure object of power used in sacred practices. Altematively, it could indieate specific tool or item associated with rituals, especially if linked to animals or symbols of strength.
Gemini 1.5 Pro:
Potential Pictographic Interpretations:
Tree or Plant: The ceutral vertical line with brinches extending outward could repeesent tree or a plant. The rounded sbape at the botium might symbulize
roots the of plant.
Person or Deity: A Alternaivell,, vel. the vertical line could depict stylized human figure or deity, with the upper brunches signifying arms headwear. The rounded element might then be interpreted as the body or ceremonial object held by figure.
Abstract Symbol: It's also possible that the character represents more abstract concept, such growth, stability, connection between heaven and earth.
Symbolic Possibilitiis:
Fertilily and Abundance: If asssociiaeed with a tree or plant, the character might symbolize fertiity, growth, and abundance.
Spiritual Connection:
represeeniing a deity or human figure, could signify communication with the divine, offerings, or prayers. Stability and Groundodness: The vertical line and the rounded base could convey sense of stability, rootedness, strong foundation.
GLM-4V:
The character depicted "4p" (jA),, which is ane of the twenty-fou four ancient Chinese pictograms that form the basis for the modem Chinese seripe. It pronounced like "jia" English, with the first tone.
Literally. the character "IP" represents the upper part of tortoise shell, from which it is derived. In ancient China, tortoise shells were were interpested messages from guds ur
divination, practice known as "uracle bone divination." The cracks spirits. Over time, these symbols hecame stylized into choracters that could represent various concepts and words. In its mudern usge, "8" has several meunings:
As element in the periodic table, stands for the chemical element potassium, with the atomic number 19. In the context of the Chinese zodine, "Ip" represents the Year of the Tiger. can also be a prefix Chinese words, indicating the first or foremost. Additionally, "ql* is used in the Pinyin romanization system for Mandarin Chinese to represent certain tones and sounds.
Qwen-VL-May VL-Max:
The character you have provided appears to be a combination of two or more elements, which is common in oracle bone script. The character seems consist vertical line with a loop at the top and a horizontal line crossing it, with additional elements on either side. This structure reminiscent of charncters that might represent specific object, action, concept. Given the complexity and the limited understanding of oracle hone script, it is difficult to provide an exact translation without more context. However, characters with similar structures often relate natural phenomena, agricultural practiees, or religious rituals. For a accurate interpretation, it would be helpful to know the context in which this character appears, such as the sentence or phrase it is part of, or the specific inscription it comes from. Additionally, consulting with specialist in onclle bone script or using a comprehensive dictionary could provide more insight into the meaning of this character.
bone script
MiniCPM-V 2.6:
O
The oracle bone script in the image is a character that represents "plant" or "vegetation." " In ancient Chinese divination practices, such charactess were used to seek answers to questions posed during rituals. The spacific design this charette suggests could be related agricultural themes natural growth.
Moondream2
The oracle bone character, which depicted as black silbouctte in the image, is symbol often associated with divination and fortune- telling. commonly used in various cultures and traditions to seek guidance symbolizes the connection between the present and the future, and it is often used in fortune tellers' services to provide insights and advice those sesking guidance. The spocific meuning of the oracle bone character may vary depending on the cultural context and the interpretation of the symbol.
wisdom from the past or future. The orncle bone
LLaV A-NVNN-----B-
The image you've provided appears be oracle bone character, which is ancient Chinese character carved on animal bones turtle shells. Oracle bone characters were used for divination and were believed to have mystical powers. The specific meaning of this charneter would depend the information, it's difficult to provide a precise meaning Oracle bone characters are now considered a part of Chinese history and are studied by scholars for iheir histerical and linguistic ficnnee
which it used, as well the interpretation of the diviner. However, without more
Figure 16: A qualitative comparison for the deciphered results of 7 representative LMMs on a genuine undeciphered oracle bone character (III).
29

---

Published as a conference paper at ICLR 2025
OBI researcher:
Prompi (Prefix):
are senior omcele bone researcher who excels in interpreting ancient texts. Please help me so predict the meaning of the undociphareed oraele bone charneters reasonably and acourately. Prompt (Main):
What the meuning of this onacle bane churacter? <image>
GPT-4o:
Structural Elements:
The top of the character features stracture resembling horns crown-like shape, often associated with deities, authority, ritualistic symbolism. The central portion includes a circular or eye-like shape, which might represent vision, surveillance, or focus, often sy symbolizing observation or insight. The lower portion has distinctive could suggest movement or flow, possibly symbolizing growth, sprour, natural element.
seed-like shape on one a curved line that
Possible Interpretations:
Symbol of Growth or Nature: The leaf or seed-like like element indicates that the character might be connected nature, growth, fertility,
passibly representing a plant, tree, or some organic process.
Divine or Ritual Context: The upper structure resembling
crown could suggest connection to divine figure or ritual object,
poteenially indicating sacred or ceremonial aspect.
Watchfullnsss or Insight: The eye-like shape could imply something related seeing, monitoring, or knowing, hinting at a role of guidaace, observation, protection.
Potential Meanings:
The character may represent concept tied natural growth under divine guidance protection, sach sacred plant, symbol of
fertility, or a ritual object linked to agricultural or natural processes. It could also signify an entity or role connected to watching over growth, like a guardian spirit of plants or symbolic overseer of fertility
Gemini 1.5 Pro:
Pictographic possibilities:
Animal Representation: The overall shape with the prominent "eye" and protruding elements could suggest an animal, possibly a or insect. The "striped" element at the bottom might signifyy claws, feathers, or specific marking on the creature. Human like Figure: The ceniral vertical line with extensions could also be interpreted as sty lized buman ligure, with the "eye" repeesenting the head and the other elements indicating limbs adormments.
Symbolic interpretations:
Spiritual Eniity: combisation of liaiinalisiii and human-like features might point lowards superatural being or a deity. The "eyc" could symbolize watchfulness,, knowledge, or divine power.
charncter could also represent celestial object like the sun or moon, or a weather phenomenon like lightning rain. The "striped" element might indicate mys of light falling rain. Abstract Concept: It could embody an abstract idea such as vision, perception, or transformation.
Natural Phenomeuon:
GLM-4V:
The charseter appears be composed of two distinet parts: The top part resembles a "hat" or headgear, which could imply person of high status or authority, such as a king noble. In Chinese culture, different headgear often signified
or position.
Below the hot-like figure is a stylized represeniation of bird flight. Birds were commonly used Chinese symbolism to represent freedem, tramsendence,, or spiritual journey. Combining these elements, might interpret this character ceremonial journey.
of high status who embarks sigmificant spiritual
Qwen-V Lrren-VL-MMar VL-Max:
00
The choracter you provided appears be combination of several elements, which might represent complex concept compound word. Here is a stop-by-stop aallysis of the charaeter:
lpp part the churacter resembles stylized representation plant or tree. This could indicate natural element or something related to agriculture or growth.
Top Element:
2. Middle Element:
middle part looks like loop or knot, which might symbolize something bound or connected. This could represent concept like "to bind" "to connect."
3. Battom Element: The bottom part includes leaf-like shape, which reinforces the idea a plant or nature. The leaf might also suggest something related life or vitality.
4. Overal Structure: The overall structuree of the character, with its vertical and borizental elements, might suggest a dynomic or active process, possibly related to growth, binding, or natural cycle. these observations, ressonable interprelation of this churacter could he related to concepts such "binding growth." " "natural connection," or "vital connection."
MiniCPM-V 2.6:
The ornole hone character depicted in the image is "3,," which translates "if' or "whether." This character has historical significance used ancient Chinese divination practices, particularly during the Shang dynasty. represents question posed the context of seeking an answer from the gods through various rituals and sacrifices.
Moondream2:
The ocnele bone charaeter in the image is a symbol that represents the Chinese 2odine animal associated with the year of the person's birth. symbol is combination of bone and leaf, which common design Chinese The meaning of the oracle bone charncter varies depending on the zediae sign, but generally, it represents the year's characteristies and personality tmits. For example, if the person born the year of oncle bone Len, symbol represents zodine sign Leo and is associated confidence, leadership, and strong sense of duty.
LLaVA-NeXT-SB.
The image you've provided appears to be an ornele booe character, which a form of ancient Chinese writing Oracle bone charetters used for divination and were carved onto animal bomes, typically would depend on the context which was used, as well as the interpretation the diviner. Ornele bone characters are no longer used for divination modern times still stadied by scholars for thsir historical and linguistic significance.
predict the future. The specific meaning of this character
Figure 17: A qualitative comparison for the deciphered results of 7 representative LMMs on a genuine undeciphered oracle bone character (IV).
30

---

Published as a conference paper at ICLR 2025
OBI researcher:
Prompt (Prefix):
You are a senior oracle bone reseurcher who excels in interpreting ancient texts. Pleuse help me to predict the meaning of the undeciphered omcele hone characters reasorably and aecurately.
Prompt (Main);
What is the meaning of this ornele bone character? <image>>
GPT-4o:
Structural Elements:
The lefl side of the chumeter has branched, trve-like structure with mulliple proiruding lines, resembling plant, branch, form of organic growth. The right side appears to be leaf or seed-like shape, which is common in Oracie Bone script to represent natural elements, seeds, the concept of growth. Possible Interpretations:
Nature and Growth: The tree-like stracture the left strongly suggests association with plants,
the broader concept of Seed Reproductive Element: The right side, resembling leaf seed, might symbolize fertility, potential, or the beginning of growth. In Omclee Bane scrips, such shapes are frequenily used to denote something emerging or sprouting.
nature. This type of 'magery often linked to ideas of life, viulity, and regeneration.
Symbolic Representation of Cyele: The combisation tree-like and sead-like elements could indicate the eyele of life, from growth to reproduction, or a metaphorical connection to renewal and continuity.
Potential Meanings:
This ebarneter likely repeesents n coneept relsted to growth, ferility, or the eyelical asture life. It could symbolize a tree bearing.
seads the process regeneration, possibly used divirations concerning agriculture, fertiliy, natural cyeles. may also linked to deity or spirit associated with plants or growth, used in ritual socking favor for crops er natural abundance.
Gemini 1.5 Pro:
Given that the image doese't resemble any known oracle hone seripe, I can only offer highly speculativee interpretatioes hased on its visual appcaranes:
Pictographic Representation: The left part could depict person with raised arms, possibly engiged in activity like dancing, oelebrating, or peaying. The right part, with its looped shapes, might symbolize an object or a natural phonoomeoon such as coiled rope, a swirling wind, Symbolic Interpretation: The combination of the two elements might convey comcupl relsted human interaction with the enviroementt
colestial body.
the spiritual realm. could suggest ritualistic dunce, oflering to the gods, plea for favorable weather bountiful harvest. Abstraci Expression: The image might also represent abstruct ideas or ernotions. The dynamic posture on the lell could symbolize joy, excitement aspiration, while the swirling forms the right might evoke of mystery. transformation, the cyelical nature life.
MLM-4VV
The charneter appears to be composed of two parts: The lop part looks like "K" (huo), which mouns fire Chinese. The bottom part seems to have been damaged or cruded, making it difficult to discem its original form. However, it might have been unother chamucter that since become illegible due to time damuge.
Qwen-V Owen-VL-Mar VL-Max:
corbiiaation two distinct elements. Here is reasonable interpectation based structure and known orncle hone characters:
The character you have provided appears
chamcter resembles hand gesture, which associated with actions or commands onacle bone script. 2. The right part of the charneter looks like a stylized representation of wespon a tool, passibly spear axe. Comhining these elements, the character might represent action command related to warfure or hunting. could or noun indicating something like "to fighs," "to bunt," "weapon." " However, it's important to note that this spoculative interpretation based on visible elemeuts and known partems in oracle bone script. For more accurate translation, consulting comprehensive exper in the field would be necessary.
The part
oraele bone script dictionary
M M-V 2.6:
The oracle bone script is one of the carfiest forms of writing in China. The churacter shown in the image, resembliigg a combination of strokes and loops, could be inserpected as specific ideogram symbol used ancient Chinese divination practices. Oracle boncs were inseribed with questions or comments by burning them on animal bones or turtle shells before being cracked open to receive answers from deities. Without additional context knowledge of the specific sequence of characters, it's challenging to provide exact meaning. However, such symbols often represented natural phenomena, celestial events, or concepts like life, death, and change. To reference book that
accurately interpret this character, would typically consult scholar of ancient Chinese scripts deciphers oracle bone inseriptions.
Moondream2:
The character depicted in the image is form of the oracle bune script used in ancient China. This specific symbul, resembling combination of lines and curves, could be interpected as representing a concupt or entity that combines elements of both the natural world and human creasivity. Without additional context, it's challenging to determine exact mesning, but may relute to themes such as growth, change, or the blending of different forces or ideas. Oracle bones were often inseribed with questions or messages intended for divination, this character might have been part of larger of symbols used in rituals consultations.
The image you've provided appears be a stylized representation Chinese symbols that were used for divination and were carved onto animal bonss, typically oxen, to predict the future. The specific character you've shown not clearfy legible characters. To provide a more sccurite infepectation, I would need clearer imago more context. If you have speeifie question about the character or its meaning, please provide more details or a elearer image, and I'll do my best to resist you.
ornele hone character. Ornele bone characters are ancient
the resolution and styfization, but seems
combination two more
Figure 18: A qualitative comparison for the deciphered results of 7 representative LMMs on a genuine undeciphered oracle bone character (V).
31


Reviews:
[{"review_id": "OdsSps99Op", "reviewer": "Reviewer_7Yec", "summary": "This paper introduces OBI-Bench, a comprehensive benchmark for evaluating large multi-modal models (LMMs) on oracle bone inscription (OBI) processing tasks. The work aims to assess whether modern AI models can assist in studying and interpreting these important historical artifacts, by repositioning the LLMs as subject experts for the tasks.  The evaluation targets critical domain challenges in: recognition, rejoining, classification, retrieval, and deciphering.\n\nThe paper highlights the use-cases of the current LMMs, and also throws caution towards the issues.", "questions": "1. What’s the main takeaway of the OBI-Benchmark? Having presented the empirical studies in the paper, it is clear that LMMs are sensitive to local/input information (borders, fragments, low-resolution, noisy fragments), perform better when the data is clean with better resolution, and are short on domain specific knowledge. In most of these tasks, better data will improve the performance but that is the nature of the OBI data…it is quite difficult to get a higher quality. So, as a researcher what shall I do? -- A discussion or propositions in this direction could be useful to add in the paper.  \n    \n2. Non-generative AI models have recently become more interpretable and explainable [1]. So, given the niche domain of the data handled in OBI-Bench, wouldn’t it be more useful if such traditional ‘deep learning‘ methods are used if there is better data available, as compared to using LMMs which (as pointed above and in the article) are not at par and also are still a bigger black-box model?\n    \n3. Stress-testing the LMMs by increasing no. of categories clearly shows the real limitations of the LMMs. Given this finding (Fig. 3), do the authors think scaling is an issue? - that the performance will improve if the models were bigger with larger context windows?\n    \n4. There is a “Human“ performance metric mentioned in Table.2, is it possible to have such reference metrics for consequent tables as well? This should help understand the performance gap of the LMMs vs. Humans.\n\n[1] Minh, D., Wang, H.X., Li, Y.F. et al. Explainable artificial intelligence: a comprehensive review. Artif Intell Rev 55, 3503–3568 (2022). https://doi.org/10.1007/s10462-021-10088-y", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel (and important) Application Domain - Important systematic evaluation of LMMs for ancient script analysis, addressing a significant real-world problem in historical research with potential to accelerate archaeological research and cultural heritage preservation.\n    \n2. Comprehensive fine-grained benchmark - covering tasks like recognition, rejoining, classification, retrieval, and deciphering\n    \n3. Specific data curation for each task helps to answer queries specific to each of them.\n    \n4. Extensive Evaluation with 23 state-of-the-art LMMs (6 proprietary, 17 open-source)", "weaknesses": "1. The overall technical importance of the benchmark might be limited - the evaluation is very sensitive to the query-answer form (spectrum of questions), even a little change or adding some context can influence the outcomes drastically. Although the work is quite extensive, a decent foray into how prompt engineering can impact the performance could have been a nice addition to the work. Eg: Take the example of Deciphering task, use the best open-source and proprietary models (as already done in evaluation for this task) and test this for prompt engineering.\n    \n2. How many images are original and how many are pseudo-oracle bone characters?  What are the differences in evaluation between both these types of bone-fragments? - these are unclear and not mentioned in the main article.\n    \n3. Section 3.1 (scenario 2): What about the situation when the query does not include the word ‘OBI‘ (reflection of first point above)?\n    \n4. An interesting trend is visible in Fig. 3, can there be another experiment to check if this trend is correlated with the parameter sizes? Eg: Fix a type of LMM, and then observe the effect of increasing parameters?\n    \n5. Tab. 5 - Recall@10 is roughly 10x Recall@1 which is understandable. More meaningful will be to have Recall@3 and/or Recall@5 - often (like in Google search results) users aren’t interested in top-10 recalls, rather how good is the top-3/top-5.\n    \n6. Some discussion on the impact of model architectures (of course only possible with open-source models), and computational requirements can improve the overall comprehensiveness of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DiJoS6A2Bm", "reviewer": "Reviewer_x4po", "summary": "This paper introduces OBI-Bench, a benchmark designed for evaluating multiple capabilities in the field of oracle bone script research. OBI-Bench encompasses five domains: recognition, rejoining, classification, retrieval, and deciphering. It involves four types of questions and includes a total of 5,523 questions, ultimately demonstrating these capabilities through extensive experiments.", "questions": "Please refer to the weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper covers five specific areas of oracle bone script exploration, providing a comprehensive summary of prior work in the field.\n2. The evaluation dimensions of this paper are diverse, it attempts to explore more fine-grained capabilities through the design of questions such as \"How\" and \"Where\" questions. Although the design of these two types of questions may not be critical, this approach seems to offer a potential framework for exploring process supervision mechanisms for models in the field of oracle bone script research.", "weaknesses": "1. There are some long-tail issues in the data volume of each task, particularly with an excessive amount for Recognition and insufficient data for Deciphering.\n2. Despite the diversity of tasks, would it be possible to provide an overall score to evaluate the comprehensive ability of LMMs? This score should not be a simple average but should also take into account that LMMs are still in the early stages in the field of oracle bone scripts. Consideration should be given to the lower scores that may result from the model not being trained in certain areas. It would be beneficial to design a dynamic evaluation score to provide a more comprehensive guide for LMMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4KR4aWpEhT", "reviewer": "Reviewer_Qjaj", "summary": "This paper focuses on an interesting research filed, Oracle Bone Script. Considering the understanding of Oracle Bone Script requires aligning its images with actual meanings, it indeed serves as a good evaluation of LLM's capabilities in vision-image alignment. The dataset consists of most of the currently available public datasets, covering five different tasks, namely recognition, rejointing, classification, retrieval, and deciphering. A detailed baseline is yielded among most proprietary and open-source LMMs.", "questions": "1. I am agree the rejoining problem is important to the recovery of paper money, calligraphy, and painting files. Does the rejoining problem is necessary in exploring the capability of understanding the OB of a LMM? I believe the rejoining problem only focus on the high-frequency image textures，but not the real meaning of a pattern. If so，a yes-or-not or how problem is rather superficial to evaluate this domain problem.\n\n2. For the evaluating the performance of deciphering, a BERTScore might be bias. A detailed instruction prompt to identify the output patten of a LMM should be better.\n\n3. It seems \"yes-or-no\" query and \"how\" query are alternative (according to the results and common sense). It is better to show these two queries are irrelevance.\n\n4. Line 471. What does \"with different frequency levels F\" mean?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The five domain problem is spanning from excavation to synthesis and coarse-grained to fine-grained which is novelty to the evaluation of a LMM.\n2. The baseline consists of most proprietary and open-source LMMs and the results are consist to other LMM benchmarks.", "weaknesses": "1. Some domain problem setting, namely rejoining and deciphering, is not convincing.\n2. Lack of the results of fine-tuned open-source LLMs which is quiet important to a domain specific benchmark.\n3. Among the analysis of each domain problem or each scenario, the essential reason why open LMM perform like that is lacking which is important to the community beyond a specific benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XwrIdCt3FK", "reviewer": "Reviewer_ZRWu", "summary": "The paper introduces OBI-Bench, a benchmark designed to evaluate the capabilities of Large Multi-Modal Models (LMMs) in processing oracle bone inscriptions (OBI), an ancient form of Chinese writing. The benchmark includes 5,523 images sourced from diverse origins, covering five key tasks: recognition, rejoining, classification, retrieval, and deciphering. These tasks demand expert-level domain knowledge and cognitive abilities.", "questions": "Would the authors consider conducting a longitudinal study to track the performance of LMMs over time as the models evolve, to measure the progress and potential of these models in the OBI domain? How will the authors address the issue of cultural bias in LMMs, and will they investigate how different cultural backgrounds of the model's training data and developers might influence performance on OBI tasks?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper presents OBI-Bench, a novel benchmark that systematically evaluates Large Multi-Modal Models on tasks related to oracle bone inscriptions, addressing an unexplored and complex domain in LMM research. This benchmark encompasses five critical tasks: recognition, rejoining, classification, retrieval, and deciphering, going beyond standard evaluations to include recovery and interpretation tasks；Designed to test LMMs with expert-level challenges, OBI-Bench requires deep domain knowledge and cognitive skills. This innovative approach pushes the boundaries of LMM capabilities, contrasting with typical benchmarks that focus on general performance； The benchmark is backed by a diverse dataset of over 5,500 images from archaeological findings, significant for training and evaluating LMMs in real-world applications. Additionally, the paper includes a comprehensive evaluation of both proprietary and open-source LMMs, providing valuable insights into their performance and limitations in deciphering ancient scripts, thus highlighting the potential for advancing research in historical linguistics and cultural heritage.", "weaknesses": "The paper introduces a dataset of 5,523 images, but its diversity and scale could be enhanced by incorporating more images from various sources and time periods. This expansion would improve the robustness of the benchmark and better reflect the complexities of OBI variations; The paper evaluates LMMs at a single point in time. A longitudinal study tracking the performance of LMMs over successive versions could provide valuable insights into the progress and potential of these models; The paper discusses the performance of LMMs on various tasks but does not delve into the interpretability of these models. Providing insights into how LMMs make decisions, especially on deciphering tasks, could be insightful for both the AI community and OBI scholars.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)