The NTCIR-18 SUSHI Pilot Task:
Searching Unseen Sources for Historical Information
Tokinori Suzuki,1 Douglas W. Oard,2 Emi Ishita,1 Yoichi Tomiura1
1Kyushu University, 2University of Maryland

ntcir-sushi@googlegroup.com
Kickoff Meeting: March 29, 2024
1

The SUSHI Pilot Task
• Goal:• Find undigitized documents in archival repositories• Subtasks:
A: Container Ranking
B: Archival Reference Detection
B1: Citation classification
B2: Reference boundary detection

Archival containers in the US National Archives* https://bit.ly/3tV3eNd2

QuerySubtask A: Container RankingContainer Ranking System

Evaluation measures are computed froma system’s ranking of containers
Average Precision: 0.5

Ranked Containers

3124

Relevant containers
contain
Relevant undigitized documents

Previously digitized documents can be used to learn a model of container content

3

Container Ranking: Document Collection
Such dense digitization is rare!

• ~23,000 fully digitized documents from the US National Archives• Each was originally stored in one of ~100 boxes• A small predefined sample is defined for use by teams to train systems• Relevance judgments are made using digitized documents not in the sample

Example document

Uncorrected OCR text

Metadata4

~25 Topics for Container Ranking Pilot Task
Title: Meetings with Brazil’s President
Description: Meetings between US diplomats and the President of Brazil

Narrative: Documents discussing in person meetings between
representatives of the United States government and the President ofBrazil are relevant. These may be past, present, or future meetings, anddocuments discussing a possible meetings would be relevant regardless ofwhether the meeting actually occurs. Meetings conducted by telephone, ormeetings that involve only Brazilian government representatives other thanthe President would not be relevant.
5

Container Ranking: Evaluation Measures• Mean Container-nDCG [Main measure]• Each relevant document is assigned a relevance grade• Highly relevant [score: 3]• Somewhat relevant [score: 1]• No relevant [score: 0]• Raw score (CRel) for a container is sum of scores for its undigitized
documents• Limited to the 10 most relevant documents per container𝑘• Mean Success@12𝐶𝑅𝑒𝑙𝑖 − 1
𝐶𝑜𝑛𝑡𝑎𝑖𝑛𝑒𝑟 − 𝐷𝐶𝐺 = ෍log 2 𝑖 + 1
𝑖=1• Count as a success if the most relevant container is at first of system’srank6

Subtask B: Archival Reference Detection
Footnotes, Endnotes, References(“Citations”)

8. Slayton DK. Gemini Extravehicular Activity Program:Mission National Archives at Fort Worth, TX 1964 (30January), RG 255, E.75, Box 383, file 2: Extravehicular
Planning, Crew Procedures and Training.

Wheeler, D., and R. Garcıa-Herrera, 2008: Ships’ logbooks inclimatological research: Reflections and prospects. Ann. New York Acad.
… Several archive sources have been used in the preparation of thispaper, including the following: Log-book of HMS Richmond. The U.K.
National Archives. ADM/51/3949

Archival Reference Detection SystemB1) Citation classification
Archival Reference

Not an Archival Reference

8. Slayton DK. Gemini Extravehicular Activity
Program: Mission National Archives at Fort Worth,
TX 1964 (30 January), RG 255, E.75, Box 383, file
2: Extravehicular Planning, Crew Procedures andTraining.

2. Thomas P. Hughes. The evolution of large
technological systems. In The social construction oftechnological systems. Bijker, W., Hughes, T. P.,
and Pinch, T. Eds. Cambridge, MA: MIT Press,
1987:50-82.

B2) Reference boundary detection

Part of Archival ReferenceKetting, De Amsterdamse chirurgijns, unpublished masters‘ thesis,39, 41, 44. After 1760, the pupils take about eleven to twelve years
to graduate their master’s examination. NA, VOC 14.204, 14.209,
14.217. Municipal archive of Schiedam: chirurgijns gilde OAA
3077 Date of birth was retrieved by J. Leenders in the municipal
archive of Rotterdam, 1-6-1742.

7

Evaluation of Archival Reference Detection• Citations:• ~1,000,000 citations automatically extracted from scholarly publications• Ground truth judgement created by trained annotators• Evaluation Measures:• Citation classification• Precision• Recall• F1 [Main measure]• Reference boundary detection• Jaccard coefficient, computed on characters [Main measure]8

How To Participate• Sign up with NTCIR!
• Join the mailing list

• Send an email to ntcir-sushi@googlegroup.com• Download the collections and baseline systems• Available May 1, 2024

• Try something!

• You can participate in any or all of the subtasks• Submit to the Dry Run for early feedback• Due August 31, 2024

• Submit your official runs
• Due January 31, 2025

• Come to Tokyo for NTCIR-18!

• June 10-13, 2025 (online participation will be possible)

9