# Editing and Analysing Historical Astronomical Diagrams with Artificial Intelligence

Syrine Kalleli, Scott Trigg, Ségolène Albouy, Samuel Gessner, Mathieu Husson, Mathieu Aubry

► To cite this version:

Syrine Kalleli, Scott Trigg, Ségolène Albouy, Samuel Gessner, Mathieu Husson, et al. Editing and Analysing Historical Astronomical Diagrams with Artificial Intelligence. IAMAHA, Nov 2023, Nice, France. hal-04317291

HAL Id: hal-04317291

https://enpc.hal.science/hal-04317291v1

Submitted on 1 Dec 2023

**HAL** is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Editing and Analysing Historical Astronomical Diagrams with Artificial Intelligence

KALLELI Syrine<fnref n="1" />, TRIGG Scott<fnref n="2" />, ALBOUY Ségolène<fnref n="2" />, GESSNER Samuel<fnref n="3" />, HUSSON Mathieu<fnref n="2" />,
AUBRY Mathieu<fnref n="1" />

**Abstract.** The EIDA project explores the historical use of astronomical diagrams across Asia, Africa, and Europe. We aim to develop automatic image analysis tools to analyze and edit these diagrams without human annotation, gaining a refined understanding of their role in shaping and transmitting astronomy. In this paper, we present a baseline method to detect lines and circles in historical diagrams, based on text removal, edge detection and RANSAC. We compare this strong baseline to a deep learning approach based on LETR. This work contributes to the historical diagram vectorization, enabling novel methods of comparison and clustering, and offering fresh insights into the vast corpus of astronomical diagrams.

**Keywords.** Astronomy; Diagrams; Line/Circle detection; Manuscripts; Vectorization.

## Context

Astronomy flourished across premodern Eurasia as a means to understand the natural world and fulfill religious, ritualistic, and political needs. This knowledge primarily circulated in handwritten documents, which contained a rich diversity of diagrams and numerical tables alongside the texts. Building on the visual turn in the history of astronomy, we explore astronomical diagrams as visual heritage and as representations of, and support for, historical scientific reasoning. We aim to develop new image analysis tools that do not require previous human annotation to enhance the study and edition of diagrams [3]. We present a computer vision approach to examine diagrams on a large scale independent of linguistic or cultural origin, enabling a deeper understanding of their roles in shaping and transmitting astronomy.

## Methodology

Line segment detection is a fundamental task in computer vision, commonly tackled using traditional methods such as Hough transform and more recently deep learning approaches. We propose a baseline vectorization method that leverages RANSAC [2] for line and circle detection after initial edge detection and text removal steps. RANSAC fits models to subsets of data iteratively, selecting the most consistent ones as the final result. Our edge detection is based on Canny [1] and we use TESTR [5], a text-spotting transformer network, to effectively detect text, we validate the benefits of this step in Tab. 1. Additionally, we introduce a deep learning approach based on LETR [4], trained on synthetic data and extended to perform line and circle detection simultaneously. Our evaluation demonstrates the superior performance of the learning approach over the baseline method across a dataset of 15 annotated diagrams, which we distribute.

<footnote n="1">¹LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS, Marne-la-Vallee, France</footnote>

<footnote n="2">²SYRTE, Observatoire de Paris-PSL, CNRS, Paris, France</footnote>

<footnote n="3">³CIUHCT, Faculdade de Ciências, Universidade de Lisboa, Portugal
syrine.kalleli@enpc.fr</footnote>

<footnote>https://imagine.enpc.fr/~kallelis/iamaha2023/</footnote>
<figure><img src="image_2.png" /><figcaption>Fig. 1: **Fully automatic diagram vectorization pipeline.** We perform text and edge detection on a raster image, then apply RANSAC on the edge pixels to obtain the line and circle proposals.</figcaption></figure>

**Discussion and perspective**

While our baseline yields satisfactory results, it has limited generalization capabilities and several failure cases. The LETR extension resolves this, enabling faster inference, but faces challenges in diagrams with curves. We believe this network can be extended to perform arc and curve detection as well. Addressing these challenges will open new methods of comparison and clustering for diagrams, and prepare the stage for a fresh view of the huge corpus of astronomical diagrams.

<table><caption>Tab. 1: **Quantitative results.** We compare our RANSAC method with the LETR-based learning approach over our annotated set of 15 diagrams. Detected shapes are considered true positives when they are within a marginal distance of ground-truth annotations and false positives otherwise.</caption><thead><tr><th>Method</th><th>Fscore Lines ↑</th><th>Fscore Circles ↑</th></tr></thead><tbody><tr><td>RANSAC baseline</td><td>0.33</td><td>0.52</td></tr><tr><td>RANSAC baseline w/o text removal</td><td>0.24</td><td>0.35</td></tr><tr><td>LETR extension</td><td>0.38</td><td>0.70</td></tr></tbody></table>

**Acknowledgements**

This work is supported by ANR (project EIDA ANR-22-CE38-0014). The work of Scott Trigg is supported by the European Research Council (ERC project NORIA, grant 724175). Mathieu Aubry and Syrine Kalleli are supported by ERC project DISCOVER funded by the European Union's Horizon Europe Research and Innovation program under grant agreement No. 101076028. Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union. Neither the European Union nor the granting authority can be held responsible for them.

**References**

[1] John Canny. "A computational approach to edge detection". In: *IEEE Transactions on pattern analysis and machine intelligence* (1986).

[2] Martin A Fischler and Robert C Bolles. "Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography". In: *Communications of the ACM* 24 (1981).

[3] Boris Jardine and Nicholas Jardine. "Critical editing of early-modern astronomical diagrams". In: *Journal for the History of Astronomy* 41 (2010).

[4] Yifan Xu et al. "Line segment detection using transformers without edges". In: *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2021.

[5] Xiang Zhang et al. "Text spotting transformers". In: *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2022.