# The HikarIA Project

Édouard de Saint-Ours, Christopher Kermorvant

► To cite this version:

Édouard de Saint-Ours, Christopher Kermorvant. The HikarIA Project. 2025. hal-05212174

HAL Id: hal-05212174

https://hal.science/hal-05212174v1

Preprint submitted on 8 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# The HikarIA Project: Using artificial intelligence to research and promote Japanese photographic heritage

Édouard de Saint-Ours (Guimet Museum, Curator of photography)
Christopher Kermorvant (TEKLIA, CEO)

## Introduction

Launched in October 2023 for three years, the HikarIA project takes its name from the association of the Japanese word for "light" (hikari 光) with the French abbreviation for artificial intelligence (IA). The project is led by the Guimet – National Museum of Asian Arts, in partnership with TEKLIA, a French company specializing in the development of AI software and especially computer vision applications. HikarIA is supported by the French State, through funding from the Caisse des Dépôts, as part of the national investment plan "France 2030" and has three main goals: 1) Preserve, study, and share the Guimet's photographic collections; 2) Develop new tools for art historical research by optimizing the automated description of historical photographs; 3) Enrich knowledge about photography in Japan during the Bakumatsu and Meiji eras (1853–1912).

## The Dubois collection

At the heart of the HikarIA project is an exceptional collection of photographs assembled by Dr. Joseph Dubois from the 1970s onwards, and purchased by the Guimet in 2007, thanks to the French Heritage Fund. Comprising approximately 19 000 photographs, most of them mounted on the pages of 284 albums, the Dubois collection offers a broad overview of commercial photographic productions in Japan before the 1920s, whether in material, technical, stylistic, or even economic terms. The collection is representative of all ranges of photographic products of the time: from cardboard binding to lacquer inlaid with gold and mother-of-pearl, from simple prints to hand-painted photographs, from rudimentary to the most refined colorization techniques (figures 1 and 2).
<figure><img src="image_2.png" /></figure>

Figure 1: SUZUKI Shin'ichi II, Nikkō, Tōshō-gū shrine, Omote-mon gate, 1880. Hand-coloured albumen print mounted on paper, painted silk, 33.5 x 40.8 cm (page). AP18231, Dubois collection, Guimet Museum, Paris. © Musée Guimet, Paris, Dist. GrandPalaisRmn/image musée Guimet

<figure><img src="image_3.png" /></figure>

Figure 2: Unidentified artist, album binding, c. 1890. Lacquer, gold leaf and ivory on wood, 32.5 x 41.5 x 6 cm. Album 069, Dubois collection, Guimet Museum, Paris. © Musée Guimet, Paris, Dist. GrandPalaisRmn/image musée Guimet
The Dubois collection has already been the subject of conservation, digitization, research, and publication campaigns in the past, notably thanks to a partnership with the University of Nagasaki and through an online publication.<fnref n="1" /> However, the volume of the collection has hindered its complete processing until today. The HikarIA project therefore began with the conservation and re-boxing of the entire Dubois collection, completed by the conservator Annabelle Simon in June 2025. The ongoing next step is the inventory, indexing, and digitization of all the early photographs of Japan preserved at the Guimet, which represent approximately 21 000 items. This second treatment, entrusted to an external service provider, will end in the Autumn of 2025.

The high-definition images and metadata produced during this phase will, on the one hand, replace a corpus of digital images created fifteen years ago and, on the other hand, enrich the metadata collected by the historian of photography Claude Estève, who has researched the Dubois collection for the Guimet over the course of several missions between 2014 and 2017. The existing reproductions are not of heritage preservation quality and the metadata is incomplete, but these elements constitute a valuable dataset nonetheless. They have also allowed TEKLIA to launch the second phase of the project from the outset.

## The HikarIA project

Since November 2023, TEKLIA has mobilized the Guimet's collections to identify and adapt existing AI models to the precise identification of all the elements visible in the photographs: places, architecture, decors, objects, clothing, professions, people, etc. These models will be used to produce detailed statistics on the iconography of the corpus. Crossed with the metadata already available, these elements will constitute a new and rich dataset on photographic productions in Japan before 1912.

All images and this dataset will then be published on an online platform developed by TEKLIA.<fnref n="2" /> It will be freely accessible and searchable according to different interfaces, including keyword and image search, statistical analysis, and spatial exploration in the form of clusters. The goal is to make this platform as appealing to the general public as to professional researchers. It will also be possible to identify and compare identical and similar photographs, map the colors used, and isolate different types of scenes like tea ceremonies or warlike compositions for instance (figure 3).

<footnote n="1">¹ URL: https://guimet-photo-japon.fr/.</footnote>

<footnote n="2">² available at this address: https://hikaria.org.</footnote>
<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /><figcaption>Figure 3: The HikarIA search results interface, which combines image content search and metadata facets. © Musée Guimet, Paris, Dist. GrandPalaisRmn/image musée Guimet</figcaption></figure>

## Accessibility

Nevertheless, ensuring accessibility to historical photographic collections goes beyond interface design and digital availability. It requires a broader commitment to sustainable dissemination, multilingual inclusivity and exploratory search paradigms that can accommodate unstructured and heterogeneous data. In the context of the HikarIA platform, which provides access to thousands of digitized photographs from 19th-century Japan, accessibility is approached as a multidimensional challenge encompassing the technological, linguistic and epistemological.

## Dissemination through IIIF

To guarantee long-term accessibility and interoperability, the HikarIA platform is built on the International Image Interoperability Framework (IIIF). This is an open standard that is increasingly being adopted by museums, libraries and digital archives to deliver high-resolution images and their associated metadata. This strategic choice is in line with recent national policy. In a note dated 21 May 2024, the French Ministry of Culture recommended the adoption of the IIIF protocol to improve the accessibility and visibility of cultural resources online. The Ministry highlighted IIIF as a standard for interoperable dissemination and digital sustainability, noting that its support for more efficient storage and transmission practices could help to reduce the carbon footprint associated with image dissemination.

In HikarIA's case, IIIF ensures the platform's sustainability and openness by adhering to a widely supported and evolving standard for disseminating digital heritage. It also enables the aggregation and cross-institutional visualization of photographic collections, making
it possible to process and compare items from different sources provided they are
exposed through IIIF-compatible endpoints. Furthermore, the framework facilitates the
integration of open-source visualization components offering advanced features such as
zooming, annotation and comparative analysis. IIIF also supports the distribution of
metadata alongside images, enabling multi-layered access to descriptive, technical and
interpretative information.

Multilingualism

Given the global historical significance of the collection and its interest to both general
audiences and photography specialists, multilingual accessibility was identified as a
core requirement. Although the photographs in the Dubois collection are housed in
France, they depict Japanese subjects and were widely circulated across the Western
world by 19th-century travelers and collectors. Consequently, the platform supports
navigation and metadata presentation in French, English and Japanese, thereby
reflecting the linguistic and cultural diversity of the collection and its potential users.

Although modern web frameworks such as Django, which was chosen for the HikariA
platform, support multilingual interfaces at a structural level, translating the content
itself—such as image descriptions, thematic tags and search filters—is far more
challenging. This requires expertise in translation, as well as in the socio-historical
context and technical vocabulary of 19th-century photography. Effective multilingualism
for cultural heritage therefore necessitates collaboration among historians, archivists
and photography specialists from different linguistic backgrounds.

Search paradigms

In addition to interface design and multilingual access, accessibility encompasses a
user's ability to find relevant content or make unexpected discoveries within a collection.
In this sense, the effectiveness of a search system is a core component of accessibility.
Whether users approach the archive with a specific question or a more exploratory intent,
the search functionality determines how effectively they can engage with the material.
This is particularly true of large photographic collections, where the visual content often
has more historical or aesthetic value than is described in the metadata.

Image retrieval in iconographical collections has traditionally followed two main
paradigms: Metadata-Based Image Retrieval (MBIR) and Content-Based Image Retrieval
(CBIR). MBIR relies on structured or unstructured metadata—titles, keywords, dates,
places—provided manually or through automated processes. CBIR, by contrast, seeks to
retrieve images based on visual similarity, using algorithmic analysis of color, texture,
shape, or spatial features.

Until recently, CBIR systems were limited in their capabilities, often failing to deliver
semantically meaningful results. As such, MBIR has remained the dominant model in
cultural heritage settings, where metadata allows for precise, controlled access to
catalogued images. However, MBIR comes with critical limitations when applied to large-
scale photographic corpora: the fine-grained annotation of individual photographs is
labor-intensive and requires significant domain expertise. For most historical collections, such metadata is available only for a small fraction of the total images.

Recent advances in multimodal Large Language Models (LLMs)—such as CLIP, SigLIP, BLIP, or LLaVA—have transformed the landscape of image search. These models are trained on massive datasets of image-text pairs and are capable of mapping both visual inputs and natural language queries into a shared semantic embedding space. This allows users to perform searches using free-form textual prompts (e.g., “a group of women standing under umbrellas in a studio”), even in the absence of any explicit metadata. In this model, retrieval is driven not by preassigned descriptors but by the semantic similarity between the user’s query and the embedded content of the image.

For photographic collections where image-level metadata is sparse or inconsistent, this new generation of CBIR systems offers an unprecedented opportunity. It enables scalable exploration of large, unannotated datasets, promotes serendipitous discovery, and allows users to interact with the collection in intuitive, expressive ways that reflect their own research or curatorial interests. It also rebalances the epistemology of search, moving away from cataloging constraints and toward interpretive access grounded in visual-linguistic reasoning. Table 1 presents a comparison of the metadata and content based approaches for image retrieval.

In this context, the HikarIA platform integrates a search interface that leverages multimodal embeddings for semantic retrieval, while maintaining compatibility with metadata-based filtering when available. This hybrid approach provides a flexible foundation for discovery, combining the reliability of structured metadata with the richness and fluidity of LLM-enabled visual interpretation.

<table><caption>Table 1: A comparison of metadata and content based image retrieval approaches.</caption><thead><tr><th>CRITERION</th><th>MBIR (METADATA-BASED IMAGE RETRIEVAL)</th><th>TRADITIONAL CBIR (PRE-LLM)</th><th>MODERN CBIR (WITH MULTIMODAL LLMS)</th></tr></thead><tbody><tr><th>PRINCIPLE</th><td>Search based on textual metadata (title, date, location, author, etc.).</td><td>Search based on visual features extracted from the image (color, texture, shape, SIFT, SURF, CNN, etc.).</td><td>Search based on semantic similarity between text and image in a shared latent space learned by a multimodal model (e.g., CLIP, SigLIP).</td></tr><tr><th>QUERY TYPE</th><td>Structured text (forms, keywords).</td><td>Example image or visual criteria (similarity search).</td><td>Natural language free-text or image as query.</td></tr></tbody></table>
<table><tr><td>SUPPORTED MODALITIES</td><td>Text only; multilingual depending on the translation of the metadata</td><td>Visual only.</td><td>Multimodal (text/image); multilingual depending on the model.</td></tr><tr><td>GENERALIZATION CAPACITY</td><td>Limited to available metadata.</td><td>Low: depends on the chosen descriptors, poorly robust to visual variations.</td><td>High: zero-shot possible, robust to visual and semantic diversity.</td></tr><tr><td>INTERPRETABILITY</td><td>High: result provenance is clear (title, author, date).</td><td>Low: hard to interpret visual descriptors.</td><td>Medium to high: explainable via semantic language concepts.</td></tr><tr><td>REQUIRED PREPROCESSING</td><td>Metadata normalization, multilingual encoding.</td><td>Extraction of visual descriptors (manual or automatic).</td><td>Indexing of images in the model's latent space (embeddings).</td></tr><tr><td>USE CASE EXAMPLES</td><td>- Find all photos taken in Yokohama in 1880. - Search for portraits tagged as "geisha".</td><td>- Find images visually similar to a given one. - Search for visual patterns (e.g., blue tones, round shapes).</td><td>- Search for "group of women with umbrellas" or "outdoor temple scene with cherry blossoms".</td></tr><tr><td>ADVANTAGES</td><td>Strong semantic control. Compatible with controlled vocabularies. Useful for archiving.</td><td>Usable without metadata. Based solely on visual data.</td><td>Intuitive search, no prior annotation required. Semantic relevance of results. Supports natural language.</td></tr><tr><td>DRAWBACKS</td><td>Depends on human annotations. Less useful for visual exploration.</td><td>Requires careful descriptor tuning. Results are hard to interpret.</td><td>Higher computational cost. May lack fine precision without adaptation. Errors sometimes difficult to understand.</td></tr></table>

## Addressing Al bias in historical photography

Although Al models offer new ways of exploring historical photographic collections, they are often trained using contemporary, digital images that lack material context. However, 19th-century photographs were physical objects embedded in albums alongside captions, borders and sequential narratives. Preserving this materiality is essential for historical interpretation and improving Al analysis. The following sections explore how
material context informs understanding and how preprocessing can overcome the limitations of existing models.

## Photographs within Albums

At the time of their creation and distribution, these photographs were bound into albums. Each album presents a sequence of images that reflects the editorial choices of the studio or visitor regarding themes, aesthetics or geographical journeys. Visitors aimed to create a souvenir of their travels in Japan. The views of the pages provide important contextual information such as the placement of the photograph within a narrative sequence, whether it has a caption or decorative border, the number of photographs per page, their size and position. All of these elements are integral to historical analysis and must be preserved in the digital interface. This is particularly important when studying patterns of album composition, such as thematic progression, the recurrence of particular views or motifs, and the juxtaposition of portraits and landscapes.

## Materiality as a prerequisite for AI-based content analysis

Most current image interpretation models, such as CLIP, SigLIP and GPT-based captioning systems, have been trained on large collections of images found on the web. These predominantly consist of contemporary, born-digital photographs that lack any meaningful material context. Consequently, such models tend to misinterpret or underperform when presented with historical photographs within their physical environment. For example, when a photo is rotated on an album page, or when multiple photos are presented on a page.

To overcome the limitations of semantic models that are trained using web-native images, the processing workflow incorporates a pre-processing step that separates each photograph from its physical surroundings before applying content analysis. A YOLO model custom-trained on the Dubois Collection is used to detect photographs within digitized album pages, identify their bounding boxes and estimate their orientation (e.g. 0°, 90°, 180°). This preprocessing step ensures that the extracted images more closely match the format and presentation context of the data used to train modern multimodal models. This reduces the risk of semantic confusion or captioning errors.
<figure><img src="image_6.png" /><figcaption>Figure 4: Automatic photograph detection with YOLO. © Musée Guimet, Paris, Dist. GrandPalaisRmn/image musée Guimet</figcaption></figure>

Due to the relatively small size of the corpus, the detection model has been integrated
into a semi-automated validation workflow, in which the prediction can be manually
corrected or confirmed. This interactive process balances automation with curatorial
oversight, ensuring that the resulting data is both accurate and suitable for indexing,
interpretation and semantic search. An example of automatic detection is presented in
figure 4.

Once a photograph has been extracted from the album page and correctly oriented, it is sent to a content analysis model (CLIP in the initial deployment) to generate a semantic embedding. These embeddings are stored in a vector database and form the basis for semantic searches. While the search and retrieval process is performed on the isolated photograph, the results are presented to the user within the original page view, thus preserving the visual and material context. This hybrid strategy enables computational efficiency and interpretive depth.

Furthermore, the platform facilitates album-based navigation, enabling users to explore
photographs as a narrative sequence, much as they would when viewing the album in
person. This facilitates research into patterns of visual storytelling, image reuse and
thematic structuring, while also restoring the cultural logic of the collection's historical
presentation.
Combining metadata and automatic person detection : a case study
for a quantitative analysis

Integrating descriptive metadata with automatic visual analysis opens up new possibilities for the quantitative study of historical photographic collections. To demonstrate this, we performed a large-scale analysis of 12 332 photographs from 276 albums, each of which contains a single photograph on every page. This consistent structure allowed us to establish a one-to-one correspondence between each digitized image and a single photographic subject, thus simplifying both metadata alignment and automated processing.

Each photograph was first categorized according to the nature of its setting and subject
matter using existing metadata. Five main categories were identified: in-studio
photographs (mostly costume portraits), religious architecture, natural landscapes,
urban landscapes and civil architecture. The 'in-studio' category comprises carefully
posed images taken in controlled environments, typically portraying individuals in
traditional attire. Often produced for foreign audiences, these photographs emphasize
costume and posture rather than spatial context. The other four categories refer to out-
of-studio views and document various aspects of Japan's built and natural environment,
ranging from temples and shrines to city streets and bridges.

To complement the semantic categorization, we used a YOLO-World object detection
model configured to recognize people in images (figure 5). This automatic detection
allowed us to quantify the number of visible individuals in each photograph. We then
grouped these results into five categories based on crowd size: images with no people;
images with one person; images with two people; images with three to five people; and
images with more than five people. This dual approach, which relied on curatorial
metadata and machine learning, made it possible to study how photographic
composition varies across different genres, as presented in figure 6.
<figure><img src="image_7.png" /><figcaption>Figure 5: Automatic person detection with Yolo-World. © Musée Guimet, Paris, Dist. GrandPalaisRmn/image musée Guimet</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Figure 6: Statistical analysis of the presence of persons in different types of photographs.</figcaption></figure>
The results show a clear correlation between the type of view and the number of people depicted. Religious architecture and natural landscapes are the most depopulated categories, with no individuals detected in over 48% of cases. Images in these categories often focus on the grandeur or serenity of their subject matter and were likely taken with long exposure times, making the presence of moving figures either undesirable or impractical. Urban landscapes and civil architecture, while still centered on built environments, feature human figures more frequently. Nearly 36% of images of civil architecture contain three or more people, suggesting an intention to document not only the structures themselves, but also the surrounding human activity.

By contrast, the 'in-studio' category reveals a completely different pattern. Only around 10% of these photographs lack a human presence, with the majority depicting one or two individuals, which is consistent with the genre's emphasis on posed portraiture. These images often offer stylized representations of Japanese identity intended for foreign audiences. They are meticulously composed in controlled indoor settings and feature elaborate costumes and painted backdrops. The prevalence of solitary or paired figures highlights the artificiality of these scenes, which serve more as carefully staged visual performances with ethnographic or aesthetic intent than as documentary records of everyday life.

This case study demonstrates the analytical richness that can be achieved by combining metadata-driven classification with modern computer vision techniques. The results reveal patterns that would be difficult to observe through metadata alone and open up avenues for further research into the visual strategies and cultural intentions embedded in the photographs. For instance, the correlation between genre and human presence sheds light on how photographers balanced documentation with staging and responded to the technical limitations of the medium. More broadly, such quantitative analyses support distant viewing strategies that complement close readings, offering new perspectives on historical photographic practices.

## Conclusion

Beyond the technical challenges, it is important to bear in mind that HikarIA also faces methodological challenges. The first one is the need to work with a corpus that is representative of photographic productions in Japan for over 60 years. The Guimet Museum collection is one of the largest in Europe, but it is not exhaustive. Working the AI models on a larger corpus would therefore allow for better results. The inclusion of similar international collections in the corpus would also connect them on the platform and provide a valuable global resource for historians. One additional goal of the project is therefore to enrich the corpus with other collections via the IIIF protocol.

The second challenge lies in the very nature of the iconography. Indeed, commercial photographs from the Meiji period are highly performative. The dominant style at that time, often called *Yokohama-Shashin*, presents a picturesque and artificial image of Japan intended for the foreign market (figures 7 and 8). Instead of illustrations, these photographs should be understood as artefacts from the dawn of globalization, an interface that staged Japanese culture for commercial purposes and strongly influenced foreign conceptions of the country.
<figure><img src="image_9.png" /><figcaption>Figure 7: Raimund von STILLFRIED-RATENICZ, Sumo fight: rikishi (wrestlers) and gyōji (referee) posing in the studio, 1877-80. Hand-coloured albumen print, 24.2 x 19.6 cm. AP11340, Guimet Museum, Paris. © Musée Guimet, Paris, Dist. GrandPalaisRmn/image musée Guimet</figcaption></figure>

<caption>Figure 8: TAMAMURA Kōzaburō, Geisha playing shamisen in the studio, early 1890s. Hand-coloured albumen print, 24.2 x 19.3 cm. AP15928, Guimet Museum, Paris. © Musée Guimet, Paris, Dist. GrandPalaisRmn/image musée Guimet</caption>

During the third and final phase of the project, the tools developed will be used at the Guimet Museum, with the help of associated scholars, to further research the history of photography in Japan. The previous steps will facilitate the attribution of photographs by association and research into the economy of the studios, their aesthetic evolutions, and the circulations of iconographies, images, sitters, props, etc. These elements will feed into a broader reflection on the dissemination of images, mutual influences between Europe and East Asia, the birth of stereotypes about Japan, and tourist practices at the dawn of globalization. The HikariA project will thus also open new horizons for research at the crossroads of historical, aesthetic, economic, and social issues.

## Bibliography

* Taylor ARNOLD and Lauren TILTON, *Distant Viewing: Computational Exploration of Digital Images*, Cambridge, MA: MIT Press, 2023.

* Tianheng CHENG et al., *YOLO-World: Real-Time Open-Vocabulary Object Detection*, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

* Sebastian DOBSON, *Japan on a Glass Plate: The Adventure of Photography in Yokohama and Beyond, 1853–1912*, Brussels: Ludion, 2023.
Claude ESTEBE, « Le premier âge d'or de la photographie au Japon (1848-1883): Constitution et analyse d'une base de données iconographiques de la fin de la période Edo (période Bakumatsu) au milieu de l'ère Meiji », PhD thesis, Letters and humanities, INALCO, Paris, 2006.

Luke GARTLAN and Roberta WUE (eds.), *Portraiture and early studio photography in China and Japan*, London: Routledge, 2017.

Glenn JOCHER and Jing QIU, *Ultralytics* YOLO11, 2024. URL: https://github.com/ultralytics/

Mio WAKITA, *Staging Desires: Japanese Femininity in Kusakabe Kimbei's Nineteenth Century Souvenir Photography*, Berlin: Reimer, 2013.