# IIIF as a Service for Researchers

Jean-Philippe Moreux

► To cite this version:

Jean-Philippe Moreux. IIIF as a Service for Researchers. 3rd cycle. France. 2021. hal-04675821

HAL Id: hal-04675821

https://hal.science/hal-04675821v1

Submitted on 22 Aug 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

# IIIF as a Service for Researchers

## The BnF Use Case

Jean-Philippe Moreux

Bibliothèque Nationale de France,
Département de la Coopération

**Time Machine Academy on IIIF, September 2021**

https://www.tim machine.eu/upcoming-time-machine-academy-on-iiif-international-image-interoperability-framework/
<figure><img src="image_3.png" /></figure>

## Outline

- IIIF as an essential part of digitization and enrichment pipelines

- IIIF for the dissemination of research results

- IIIF for scientific digital mediation
# Pipelines

<figure><img src="image_4.png" /><figcaption>A BAKOU. — Dune fontaine jaillissante de pétrole</figcaption></figure>

From IIIF as an interoperable format for image in heritage web portals to ... a standard for document exchange in the digital humanities ecosystem
# The GallicaPix PoC

## Motivations

* Hybrid retrieval PoC (2017) on iconographic material (text, metada, content-based image retrieval)

* Deep learning demonstrator for heritage collections: locally trained models, AI platforms and tools (commercial, open source)

* Internal and external users

* Use cases: information retrieval and digital humanities

* End to end IIIF support

<figure><img src="image_5.png" /></figure>
# The GallicaPix PoC

## The pipeline

<figure><img src="image_6.png" /></figure>
# IIIF pipelines

## The classics

<figure><img src="image_7.png" /></figure>

## IIIF 2021 Annual Conference

### Illumination Detection in IIIF Medieval Manuscripts using Deep Learning

Fouad AOUINTI – STIH Laboratory, Sorbonne Université
Victoria EYHARABIDE – STIH Laboratory, Sorbonne Université
Xavier FRESQUET – IReMus Laboratory, Sorbonne Université

2021 IIIF Annual Conference

June 22-24, Online

<figure><img src="image_8.png" /></figure>

SORBONNE
UNIVERSITE

## Our proposal

Using the latest version YOLO:

<figure><img src="image_9.png" /></figure>

1. Preprocessing: as the HBA images are not annotated with bounding boxes surrounding the illuminations, for each page, we randomly generated 10000 rectangles and labeled them

2. Training: we trained and validated YOLO on the HBA dataset

3. Analyze manifests: for each IIIF manuscript manifest, we extracted the URLs of the page's images

4. Detect illustrations: we evaluated each page's image on our model

5. Merge bounding boxes: we merged all the bounding boxes obtained to identify the illuminations

6. Generate IIIF URLs: for each illumination found on a page, we generate the corresponding IIIF URL

<figure><img src="image_10.png" /></figure>

... and the Time Machine Box!

<figure><img src="image_11.png" /></figure>

Time
Machine
# The GallicaPix PoC

## Advantages of using IIIF in a R&D activity

* API facilitates the development of prototypes: Gallica APIs + Gallica IIIF Image

* Interoperable standards like IIIF allowed us to work on multiple collections: Europeana APIs + The Welcome Collection IIIF repository

* Instant access to images: no more files!

* Digging in images with URLs

* Training datasets, GT... are stored as metadata, not image files

* Size of images needed for specific task can be tuned with a IIIF parameter

* Commercial APIs are directly feed with IIIF URLs

* Rendering of results (quality control) is very easy: rotating, sizing, cropping with URLs

```bash
curl -X POST -u "apikey:****" --form "url=https://gallica.bnf.fr/iiif/ark:/12148/bpt6k9604090x/f1/22,781,4334,4751/,700/0/native.jpg" "https://gateway.watsonplatform.net/visual-recognition/api/v3/classify?version=2018-03-19"
```

<figure><img src="image_12.png" /></figure>
# The GallicaPix PoC

## Drawbacks

* Not suitable for very large corpora (pressure on IIIF servers): a local copy is recommended

* A timeout may occur when calling service APIs with an IIIF URL (if the IIIF server is overloaded).

* The processing speed is reduced compared to local processing on a local file.

* If your IIIF servers are not fast enough, your partners will see it and suffer!

<figure><img src="image_13.png" /></figure>
# External collaborations

## Advantages of using IIIF in a R&D activity

<figure><img src="image_14.png" /></figure>

* GallicaSnoop PoC, 2020
(similarity search):

1.2M Gallica images downloaded and processed in 15 hours (one thread, waiting time between calls)

* No local storage of images

Project: https://snoop.inria.fr/bnf/

<figure><img src="image_15.png" /></figure>
## External collaborations

### Advantages of using IIIF in a R&D activity

* REMDEM project, 2020
  (identification of scribe on
  50k music scores):

* GT creation with an IIIF
    compliant annotation tool

* Each annotation is accessed
    by the computer scientists
    as a IIIF URL

Annotation tool:
https://www.dicen-idf.org/projet-recherche-opahh-iiif/

<figure><img src="image_16.png" /></figure>

https://gallica.bnf.fr/iiif/ark:/12148/btv1b52502403w/f2/1622,2755,145,118/pct:50/0-native.jpg

Tag: F clef

<figure><img src="image_17.png" /></figure>
<figure><img src="image_18.png" /></figure>

# Toolbox

## IIIF compliant production

## tools: input/output

## IIIF data

* **HTR:** Transkribus, eScriptorium

* **Annotation:** Mirador, [Annotate](https:// mirador.ens.psl.eu/), [labellmg](https://labellmg.org/), Simple Annotation Server...

* **Digital Repository:** Archipelago

* ...

<figure><img src="image_19.png" /></figure>

Project: ANR Michel Foucault reading notes (2018-2020)
[https://odhn.ens.psl.eu/article/foucault-fiches-de-lecture-ffl](https://odhn.ens.psl.eu/article/foucault-fiches-de-lecture-ffl)
<figure><img src="image_20.png" /></figure>

# DH Pipeline
Transkribus
+ BaseX

* INHA project: *Les Papiers Antoine-Louis Barye*

* From IIIF repositories (Gallica, INHA) to wep apps using Transkribus for transcription, TEI for data modeling and BaseX as a publishing tool

* At the end: a generic pipeline for DH transcription projects

<figure><img src="image_21.png" /></figure>

Project: https://skylab.inha.fr/PENSE/LesPapiersBarye/
# In-house workflow
## Mandragore-BnF

* **Mandragore v2 (2020-2022):** enlightened manuscripts database and its web portal

* **Mirador 3 + Annotation plugin** embedded in the production workflow

* **Validation of the visual concept** against the Mandragore ontology

<figure><img src="image_22.png" /></figure>

Project: http://mandragore.bnf.fr
# Service Provider Heritage Workflow
## Teklia

* Arkindex: automatic document processing with AI

* Platform based on IIIF

<figure><img src="image_23.png" /></figure>
Dissemination of
Research Results

* Heritage institutions can disseminate more metadata with IIIF

* Researchers can share annotations and transcriptions

* Institutions may wish to benefit from the work of researchers

DH Team #1

<figure><img src="image_24.png" /></figure>

Heritage
Institutions
Dissemination:
Datasets and corpora
as resources

<figure><img src="image_25.png" /></figure>
# Gallica R&D
## Training datasets

* Training datasets, GT... are stored as metadata: the formats that CS are used to handling (COCO, Pascal VOC...) but also rich formats like IIIF

* IIIF Collections and IIIF Annotations are used to describe and display documents in datasets

* IIIF Curation could be used to describe and display parts of images (segmentation /object detection datasets). Content State API is promising for opening documents at the right place/zoom

<figure><img src="image_26.png" /></figure>
# Visual Contagions
Corpora as lists of manifests

* Visual Contagion project: How circulation and dissemination of images take place on a global scale?

* Corpora's project as list of IIIF manifests

<figure><img src="image_27.png" /></figure>
# NewsEye European project

## Motivations

* OCR, HTR, article separation, NER...

* Support qualitative and quantitative analysis of newspaper data for the digital humanities.

More than 180,000 issues from 20 newspapers, from 1850 to 1950.

Project website: https://www.newseye.eu

Demonstrator: https://platform.newseye.eu

<figure><img src="image_28.png" /></figure>

This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 770299.

<figure><img src="image_29.png" /></figure>
# NewsEye
## Corpora as III Collections

National or thematic corpora used by the project are described as IIIF collections.

* Instant access to the data and reuse (parsing, scripting...)

* Browsing in any IIIF viewer

* IIIF Collections are well suited for periodicals

<figure><img src="image_30.png" /><figcaption>Le Matin (1936)</figcaption></figure>

<caption>NewsEye French dataset of periodicals (opened with Mirador 3)</caption>
Dissemination:

**IIIF Data in web apps**

<figure><img src="image_31.png" /></figure>
# The GallicaPix PoC

## Visual indexing as IIIF Annotations

* IIIF Gallica manifest enriched with GallicaPix annotations opened in Mirador

* Periodicals exported as IIIF Collections

* GallicaPix is now called from Gallica

<figure><img src="image_32.png" /></figure>

Vogue magazine: tennis
# NewsEye project

## IIIF Data: in/out

* As a consumer:

- Thrash in search results list

- Viewer in documents show page

* As a producer:

- Articles as annotations

- Named entities as tags

<figure><img src="image_33.png" /></figure>
<figure><img src="image_34.png" /></figure>

# NewsEye project
## Users datasets as IIIF Collection

* Export researchers datasets as IIIF Collections:

○ Exports documents as IIIF manifests

○ Two layers of annotations can be included in the manifests (All articles in the issue; Only articles that are part of the dataset)

<figure><img src="image_35.png" /><figcaption>NewsEye user datasets</figcaption></figure>
Digital Mediation

<figure><img src="image_36.png" /></figure>

Use cases:

* JADIS project

* NewsEye project: Exhibit

* Gallica: Storiiies
# Showcasing R&D results

## With IIIF

* JADIS project, 2019 (heritage maps segmentation and georeferencing):
  Maps of the city of Paris from BnF and BHVP collections

* Instant access to the Gallica corpora

* Basic but effective static web site
    (Github + IIIF images)

Project: https://bnf.hypotheses.org/9676

Rémi Petitpierre,
Master of Science

<figure><img src="image_37.png" /></figure>

ECOLE POLYTECHNIQUE
FEDERALE DE LAUSANNE

<figure><img src="image_38.png" /></figure>
# Storytelling for Digital Humanities

## How can researchers in humanities and social sciences showcase digital collections ?

* A challenge for any researcher who wishes to publicize research results to the general public

* Different ways to showcase: examples from the NewsEye project

<figure><img src="image_39.png" /></figure>

<figure><img src="image_40.png" /></figure>
# Storytelling for Digital Humanities

Exhibit: a IIIF storytelling tool for an interactive experience

* Shareable, image-based annotation and deep zoom

* A tool developed by [Mnemoscene](#) with support from the [Esmée Fairbairn Collections Fund](#)

* A project initiated during the Covid-19 pandemic to meet the needs of The University of St. Andrews in online and on-campus teaching

* Use of universal Viewer, an open source IIIF viewer

Other tools: [Storiiies](#) (Cogapp), [Tesselle](#) (Médialab SciencesPo)...

<figure><img src="image_41.png" /></figure>
# Storytelling for Digital Humanities

## An example of narration from a blogpost

https://exhibit.so/exhibits/LJBQVwAjXirDsniiWhl5

* For Newseye, links between the media treatment of the Coronavirus crisis and, 100 years earlier, that of the “spanish flu”

* Narration from blogpost about remedies claiming to work miracles against the 1918 flu, available in French on Gallica blog.

* Issues from the daily (Le Matin, Le Gaulois) and weekly press (Le Pêle-mêle, La Baïonnette, Le Régiment)

<figure><img src="image_42.png" /></figure>

Nejma Omari, université de Montpellier
# Storytelling in Gallica

## From documents to stories

Version 2 of the BnF-Cogapp [Storiies](https://www.bnf.fr/en/cognac/storiiies) (2022):

* The editor can be launched on any Gallica document

* Can mix multiple documents in a story

* Will save user's stories in user's account

<figure><img src="image_43.png" /></figure>
# Thanks!

jean-philippe.moreux@bnf.fr

henry.huguet@bnf.fr

## Resources:

* https://api.bnf.fr

* https://gallicapix.bnf.fr

* https://snoop.inria.fr/bnf/

* https://platform.newseye.eu

* https://github.com/altomator/IIIF/

<figure><img src="image_44.png" /></figure>

Some of the projects described
were presented at the IIIF
2021 Annual Conference