Multimedia Retrieval of Historical Materials
Jieyong Zhu1 , Taichi Nishimura1 , Makoto Goto2 , Shinsuke Mori3
1
Graduate School of Informatics, Kyoto University
2
National Museum of Japanese History
3
Academic Center for Computing and Media Studies,
Kyoto University
April 2022

1

Introduction

Historical material is a collection of history, archaeology, and folklore materials. With the rapid advancement of digitization, large-scale multimedia data ofhistorical materials have become available on the web. As the data grows, it isdifficult for researchers to study the relationship between historical images andtext. Multimedia retrieval is a technique to perform retrieval tasks across multiple media. Recently, deep learning has accelerated research on natural language
understanding and computer vision, with remarkable performance reported in
multimedia retrieval tasks [6]. In this paper, we apply the state-of-the-art multimedia retrieval methods to Japanese historical materials and demonstrate theconstructed multimedia retrieval system. Figure 1 shows an example of multimodal retrieval tasks of historical materials.

2

Multimedia Retrieval

Multimedia retrieval takes one type of media (e.g., images and texts) as thequery to retrieve corresponding media of another type [4]. The key challenge ofmultimedia retrieval is how to convert different media data into a shared subspace, where semantically associated inputs are mapped to similar locations.
Various kinds of deep-learning-based approaches have been proposed in the literature [7]. We here employ one of them to realize our system.

3Proposal

This paper proposes a deep-learning-based approach to achieve multimedia retrieval of historical materials. Figure 2 shows an overview of our proposedmodel. The proposed method consists of two major processes.

1Query

Retrieval results
Image-to-text
retrieval

フカバチガタジヨウモンシキド
資料料名称
キ 深鉢形縄⽂文式⼟土器器

備考

資料料名 フカバチガタジヨウモンシ
称
キドキ 深鉢形縄⽂文式⼟土器器
コレク
テンマアトイセキシユツド
ション
ヒン 天満後遺跡出⼟土品
名

備考

資料料名称

ミカヨモチキンギンヌリドキ・
ハシ 三ヶ夜餅⾦金金銀塗⼟土器器・箸シジョウケデンライナシモトノ
コレク
ミヤコンギジュウキカンケイシ
ション名 リョウ 四條家伝来梨梨本宮婚儀什什
器器関係資料料

コレク
テンマアトイセキシユツドヒン
ション名 天満後遺跡出⼟土品
現状：破⽚片を接合し、割れ⽬目を泥泥⼟土で補修。「ＴＥ１１」の記号あり。法量量：⾼高さ２１．５ｃ
ｍ、⼝口径１２．５ｃｍ、底径８．８ｃｍ。時期・型式：縄⽂文
時代前期円筒下層ｄ１式。

備考

Text-to-image
retrieval

現状：破⽚片を接合し、割れ⽬目を泥泥⼟土で補修。「ＴＥ１１」の記号あり。法量量：⾼高
さ２１．５ｃｍ、⼝口径１２．５ｃｍ、底径８．８ｃｍ。時期・型式：縄⽂文時代
前期円筒下層ｄ１式。

Figure 1: Examples of image-to-text and text-to-image retrieval tasks. Retrieved results in the red boxes are the associated ones with the query in theleft.
Text-side

フカバチガタジヨウモンシキドキ
深鉢形縄⽂文式⼟土器器

テンマアトイセキシユツドヒン
天満後遺跡出⼟土品

現状：破⽚片を接合し、割れ⽬目を泥泥⼟土で補修。「ＴＥ１１」の記号あり。法
量量：⾼高さ２１．５ｃｍ、⼝口径１２．５ｃｍ、底径８．８ｃｍ。時期・型式：
縄⽂文時代前期円筒下層ｄ１式。

Joint embedding learning

Image-side

Text
Encoder

Concatenate
Text
Encoder

MLP

Text
Encoder

MLP

ResNet50(Encoder)

Triplet Margin Loss

Figure 2: An overview of our proposed model.
Text encoder. Recently, large-scale pre-trained model, such as BERT
(Bidirectional Encoder Representations from Transformers) [2], has achieved
great performance in NLP tasks. However, we don’t use BERT model becauseof the domain gap, since Japanese BERT is trained on Japanese Wikipedia
texts. The text data of historical materials include three main types: materialname, collection name, and notes. All the three types of data are in a tabularformat instead of a sentence format. Therefore, we use a word2vec model toconvert the texts into vectors, which is more simple and more reasonable.
Image encoder. The input of the image side is a single image of historicalmaterials. To convert images into vectors, we employ ResNet50 [3], a Convolutional Neural Network pre-trained on ImageNet.
Shared subspace learning. Finally, we convert text/image vectors intoshared subspace using symmetric multi-layer perceptrons with ReLU activation
functions. To train the model, we compute triplet margin loss [1], which makes
the vectors in the subspace for a given text-image pair close and otherwise long.

2

QueryTop 5 retrieval results
資料料名称

コレク
ション名

キヌキンシツヅレオリボタンモ
ンカミイレ 絹⾦金金⽷糸綴織牡丹丹⽂文紙⼊入
ノムラショウジロウイショウコ
レクション 野村正治郎⾐衣裳コレ
クション

コクシツボタンカラクサモンセ
資料料名称 ンゾウガンクシ 黒漆牡丹丹唐草⽂文
線象嵌櫛

ゾウゲキキョウカラクサモンウ
資料料名称 ルシエグシ（コガタ） 象⽛牙桔梗
唐草⽂文漆画櫛（⼩小形）

ホシノヘイジロウフクロモノコ
コレク
レクション 星野平次郎袋物コレ
ション名
クション

ノムラショウジロウイショウコ
コレク
レクション 野村正治郎⾐衣裳コレション名
クション

ノムラショウジロウイショウコ
コレク
レクション 野村正治郎⾐衣裳コレション名
クション

備考

備考

資料料名称

資料料名称 菊桐笙⽂文筥迫Ｘ⾹香袋・楊枝⼊入付
備考

越前国⻄西⽶米⽥田嶋之内永村御検地帳

コレク
ション名

ホンダケシリョウ 本多家資料料

備考

マ差出⼈人以真即⽇日閲覧：マイクロＲ９４８１３コマ※企画展「ＵＲＵＳＨＩふしぎ物語－⼈人と漆の１２０００年年史－」（２０１７．７）図録１３３⾴頁ＮＯ．３－１６７（ＮＯ．
３の３はローマ数字）

資料料名称 完両替依頼書状

資料料名称 請取⾠辰辰⾕谷地御年年貢⽶米之事

イワミノクニニマグンフクミツ
コレク
シモムラフクトミケモンジョ ⽯石
ション名⾒見見国迩摩郡福光下村福富家⽂文書

コレク
イノウケシリョウ 伊能家資料料
ション名

備考

⽬目録掲載⾴頁Ｐ．４１２年年代旧１０１８６９年年Ｘ

備考

備考

資料料名称

⽬目録４４６⾴頁分類類：村年年代：寛永五⾠辰辰極⽉月廿七⽇日差出：堀江半右衛⾨門宛名：左原村形状：―年年
貢Ａ（旧－９）。即⽇日閲覧：マイクロＲ１５６８１コマ

ショニュウヨウハライカタトリ
シラベチョウ 諸⼊入⽤用払⽅方取調帳

コレク
ホンダケシリョウ 本多家資料料
ション名

備考

差出⼈人⼤大久保久造即⽇日閲覧：マイクロＲ９４０１２コマ

資料料名称 菊桐唐草⽂文⾦金金襴筥迫
ホシノヘイジロウフクロモノコ
コレク
レクション 星野平次郎袋物コレ
ション名
クション
備考

⾹香袋・楊枝⼊入付

資料料名称

トクガワケソンゾクネガイ 徳川家存続願

コレク
ホンダケシリョウ 本多家資料料
ション名

備考

差出⼈人本多邦之助即⽇日閲覧：マイクロＲ９５６３コマ

Figure 3: Image-to-text retrieval examples. The ground truth in the retrievedresults is highlighted in the red box.
Table 1: Retrieval results on 1,000 samples.
R@1
R@5
R@10
medRmAP4

Image-to-Text
0.036
0.144
0.258
260.107

Text-to-Image
0.043
0.163
0.285
260.119

Random Ranking
0.001
0.005
0.01
500
0.002

DatasetThis is the first attempt to tackle multimedia retrieval of historical materials,
so no datasets exist in this field; thus we created the Japanese historical datasetof textual descriptions and corresponding images by crawling them from theNational Museum of Japanese History. The dataset contains 18,429 objects,including over 18k textual descriptions and over 79k corresponding images.

5Experiments

To measure the performance of the model, we perform multimedia retrieval
tasks. Figure 3 shows two examples of the image-to-text task. The query images
are on the left side while the top five retrieved texts are on the right side. Aswith previous studies, we compute three mainstream evaluation metrics, medianrank (MedR), Recall@K (R@K) [6], and mean average precision (mAP) [5] toevaluate the performance. Table 1 shows the results of 1,000 samples. The result
indicates that our system performs well in multimedia retrieval tasks comparedwith the random ranking baseline.

6Conclusion

This paper tackled the multimedia retrieval of historical materials using deeplearning-based multimedia retrieval methods. This work is the first attempt to3

tackle this problem, thus we constructed the dataset of Japanese historical texts
and images, and evaluated the model’s performance on it. The experimental results show that our constructed system performs well in the multimedia retrievalof historical materials. Future work will study a better method to represent thetextual data. We expect that our research will help researchers in gaining abetter understanding of Japanese historical materials, and will give a generalapproach to learning the shared subspace between textual and visual data.

References[1]

Vassileios Balntas et al. “Learning local feature descriptors with triplets
and shallow convolutional neural networks.” In: Bmvc. Vol. 1. 2. 2016, p. 3.

[2]

Jacob Devlin et al. “Bert: Pre-training of deep bidirectional transformers
for language understanding”. In: arXiv preprint arXiv:1810.04805 (2018).

[3]

Kaiming He et al. “Deep residual learning for image recognition”. In: Proceedings of the IEEE conference on computer vision and pattern recognition.
2016, pp. 770–778.

[4]

Jing Liu, Changsheng Xu, and Hanqing Lu. “Cross-media retrieval: stateof-the-art and open issues”. In: International Journal of Multimedia Intelligence and Security 1.1 (2010), pp. 33–52.

[5]

Nikhil Rasiwasia et al. “A new approach to cross-modal multimedia retrieval”. In: Proceedings of the 18th ACM international conference on Multimedia. 2010, pp. 251–260.

[6]

Amaia Salvador et al. “Learning cross-modal embeddings for cooking recipes
and food images”. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017, pp. 3020–3028.

[7]

Chairath Sirirattanapol et al. “Deep image retrieval applied on kotenseki
ancient japanese literature”. In: 2017 IEEE International Symposium on
Multimedia (ISM). IEEE. 2017, pp. 495–499.

4