Century: A Framework and Dataset for Evaluating Historical Contextualisation of Sensitive Images
Authors: Canfer Akbulut
Kevin Robinson
Maribeth Rauh
Isabela Albuquerque
Olivia Wiles
Laura Weidinger
Verena Rieser
Yana Hasson
Nahema Marchal
Iason Gabriel
William Isaac
Lisa Anne Hendricks
Venue: ICLR 2025 Spotlight (2025)
Keywords: historical
contextualisation
image
dataset
multimodal
VLM
evaluation
Primary area: datasets and benchmarks
TL;DR: A dataset of sensitive historical images is curated and used to demonstrate historical contextualisation capabilities of SOTA multi-modal models.

Abstract: How do multi-modal generative models describe images of recent historical events and figures, whose legacies may be nuanced, multifaceted, or contested? This task necessitates not only accurate visual recognition, but also socio-cultural knowledge and cross-modal reasoning.  To address this evaluation challenge, we introduce Century -- a novel dataset of sensitive historical images. This dataset consists of 1,500 images from recent history, created through an automated method combining knowledge graphs and language models with quality and diversity criteria created from the practices of museums and digital archives. We demonstrate through automated and human evaluation that this method produces a set of images that depict events and figures that are diverse across topics and represents all regions of the world.
We additionally propose an evaluation framework for evaluating the historical contextualisation capabilities along dimensions of accuracy, thoroughness, and objectivity. We demonstrate this approach by using Century to evaluate four foundation models, scoring performance using both automated and human evaluation. We find that historical contextualisation of sensitive images poses a significant challenge for modern multi-modal foundation models, and offer practical recommendations for how developers can use Century to evaluate improvements to models and applications.

Published as a conference paper at ICLR 2025
CENTURY: A FRAMEWORK AND DATASET FOR EVALUATING HISTORICAL CONTEXTUALISATION OF
SNSSTIIVE IMAGES
Canfer Akbulut'
Kevin Robinson'
Maribeth Rauh
Isabela Albuquerque
Olivia Wiles
Laura Weidinger
Verena Rieser
Yana Hasson Lisa Anne Hendricks
Nahema Marchal
lason Gabriel
William Isaac
Google DeepMind
ABSTRACT
How do multi-modal generative models describe images of recent historical events and figures, whose legacies may be nuanced, multifaceted, or contested? This task necessitates not only accurate visual recognition, but also socio-cultural knowl- edge and cross-modal reasoning. To address this evaluation challenge, we intro- duce Century novel dataset of sensitive historical images. This dataset con- sists of 1,500 images from recent history, created through an automated method combining knowledge graphs and language models with quality and diversity cri- teria created from the practices of museums and digital archives. We demonstrate through automated and human evaluation that this method produces set of im- ages that depict events and figures that are diverse across topics and represents all regions of the world. We additionally propose an evaluation framework for eval- uating the historical contextualisation capabilities along dimensions of accuracy. thoroughness, and objectivity. We demonstrate this approach by using Century evaluate four foundation models, scoring performance using both automated and human evaluation. We find that historical contextualisation of sensitive images poses significant challenge for modern multi-modal foundation models, and of- fer practical recommendations for how developers can use Century to evaluate improvements models and applications.
INTRODUCTION
Advancements in generative models that allow for input in multiple modalities have enabled many downstream applications, such as general-purpose chatbots that can generate fluent and accurate de- scriptions of images and answer questions about visual content (e.g. Vinyals et al., 2016 Anderson jet al. 2018: Alayrac et al., 2022 Chen et al., (2022). In recent years, advancements in model capa- bilities, such as cross-modal reasoning (Zhang al. 2021). have enabled new applications beyond literal and objective descriptions of an image and its contents. For example, users and developers can now request contextualized image descriptions from foundation models, requiring these models to integrate knowledge beyond what is immediately visible in the image.
Given these advancements in capabilities, how should AI developers and researchers evaluate more complex displays of image understanding? In cases where appropriate description demands nuanced reasoning across modalities and the consideration of multiple perspectives such as images with en- grained social, cultural, or historical meaning traditional evaluation methods and metrics may fall short. Methods developed for descriptive captioning, for example, place emphasis on accuracy and fluency as desirable outcomes. However, they are not designed to assess contextual "out of frame" knowledge, nor are they able to provide a signal for more holistic measures. such as interpretive validity and cultural sensitivity. Moreover, due to their reliance pretations, traditional methods may not generalise well to the open-ended, fong-format responses necessitated by more involved image description tasks.
set of rigid "golden" inter-
"Equal contribution.

---

Published as a conference paper at ICLR 2025
"Describe this image and include historical context about what is depicted in the image."
Comments by authors
This black and white photograph depicts bombing of Hiroshima, Japan August 1945.. This haunting of utter destruction captures the horrific, unprecedented devastation unleashed by this new weapon. The flattened cityscape testifies to the bomb's awesome explosive power leveled the city, Images like this helped reveal to the world the true consequences and human toll of nuclear warfare.
depicts the aftermath of the atomic
The photo is of the town
Hame in Syria,
Homa Massacre in 1982
pert of the Islamiitt
uprising in Syrios
This image depicts three well dr-ssed African American individuals... Given professional nature their were likely noteble leaders or public figures in the Africen American community... documenting the growing prominence. leadership and achievements African Americans society and Institutions during the decades following the Civil Rights movement of the 1950s-60s...
The photo depicts Tatiani and Pau! Rusesabagina Rwandan human rights activists with the US Ambessador to Sweden.
demeanor. these individuals
The image depicted shows farous photograph known as "Raising the American flag on Mount Suribachi taken during World War II.. on February 23, 1945, during the Battle of lvvo Jime.. In the photograph, group of U.S. Marines is reisinngggg the American flag stop Mount Suribachi.. This act symbolized the hard fought victory and the Marines' tenacity in cepturing this heavily fortified position from Jepanese forces. image also highllghts the unity and bravery of the American soldiers..
photo from same day, and also depicts Ire Nayes: however. this the "Gung Ho' photograph, rather image described.
iconic
The image depicts map of the Republic Texas 1836, which was independent sovereign nation at the time. The mep shows the claimed borders of Texas, along with the locations several key battles and events from the eerly 1800s, the area that would become Texas was part of Mexico. tensions
The map nernes and depicts Ethlopia. marking sites related to the 1974-1191 Ethlopian Civil War.
Revolution against Mexico. Historical context: In
Anglo American settlers began moving into the region, between them and the Mexican government over issues like slevery and self-rule.it 1835, the Texas Revolution began colonists took up arms against Mexican control..
The image shows the logo of the Women's National Basketball Association (WNBA). The logo consists of the letters "WNBA" white, with "W" stretched form a sihouette that resembles female basketball pleyer in motion... the WNBA logo, with its dynamic and powerful symbolism, represents the strength, athleticism, and determination of the leegue's 's players and serves as a recognizable emblem...
This a logo for the United State of Women Summit event 2016 focused on gender equaiity.
WOMEN
Highlights added by authors to ilustrate poternial fallures accuracy and objectivity Nistorical contertuelizetion by muti-mmddlllmmiiiiiiii models.
Figure 1: Century is diverse set of images that can be used to measure model capabilities such as historical contextualisation. Here, we demonstrate that state of the art multi-modal models (GPT4-o and Claude 3 Opus)- prompted with image from Century and instructions to describe its contents and historical context- demonstrate failures in accuracy and objectivity.
New evaluation methods are sorely needed to capture the full spectrum of capabilities. This need is particularly critical in the context of historically sensitive images, as system failures in this domain may lead to representational harms (Luccioni et al. 2023), erase marginalized perspectives (Sarhan & Hegelich 2023). or raise misinformation concerns (Dulour et al. (024). To close this gap, we create Century, a challenging dataset of historically sensitive images to assess how ttate-of-the-art models perform historical contextualisation on image description tasks that require cross-modal socio-cultural understanding.
Our contribution. We present three contributions to multi-modal generative Al evaluation. First, we describe a novel, scalable methodology for identifying sensitive historical images, based on qual- ity and diversity criteria synthesized from interdisciplinary theoretical and curatorial perspectives.
Second, we create Century, a dataset of 1,500 sensitive historical images depicting events figures, and locations from all regions of the world. We demonstrate the validity of our dataset through human evaluation and automated evaluation, and release all evaluation data of Century images for further analyses.
Third, we create reproducible, reference-fre frren evaluation protocol for measuring historical contex- uulisation, a complex task requiring cross-modal socio-cultural understanding. Using our protocol, analyze the historical contextualisation capabilities of four systems across dimensions of accu-
2

---

Published as a conference paper at ICLR 2025
racy, thoroughness, and objectivity. We find that Century poses significant challenge for modern multi-modal foundation models, and offer practical recommendations for developers on using Cen- tury to improve foundation models.
2
RELATED WORK
2.1 LANGUAGE AND VISION BENCHMARKS
Widely used captioning and visual question answering benchmarks, like Flickr30k (Young et al, 2014). MSCOCO (Chen et al. 2013), and VQA (Antol al.] 2013), evaluate the quality of Al gen- erated outputs by comparing to "ground truth" responses written by human annotators. As these benchmarks evaluate literal descriptions of large set of images, majority of which have in- herent cultural or historical significance, the associated human-written annotations are similarly considered be free of value judgments. Yet, as Van Miltenburg (2016) point out, human annota- tors frequently add socio-cultural knowledge and context to "ground truth" descriptions. This may indicate that prior evaluation tools likely capture socio-cultural context, even if this not explicitly evaluated.
Beyond literal description, some benchmarks explicitly test "out-of-frame" or contextual knowl- edge. OKVQA (Marino et al.] 2019), Visual Common Sense Reasoning (VCR) (Zellers et al. 2019) and A-OKVQA (Schwenk et al. 2022) evaluate whether models are able integrate outside knowl- edge in order to answer questions, covering a broad base of common-sense and world knowledge. Other image evaluation frameworks contain a narrow image composition to assess capabilities in specific domain area, such historical understanding: these evaluations may images of geo- graphical landmarks (Weyand et al.) 2020 Li et al. 2012) and historical figures (Lee et al.) (002a)) evaluate information retrieval, requiring models use external knowledge correctly identify image contents.
Though these evaluation frameworks test for knowledge beyond what can be gleaned from image alone, applications of modern multi-modal foundation models require capabilities beyond short, objective responses, as assumed by many such datasets. Similarly, evaluations that artificially constrain systems to multiple choice responses may not reflect the performance of that system in open-ended generative contexts (Lum et al., 2024 Zheng et al., (003a).
2.2 MUSEUMS AND DIGITAL ARCHIVES
How should generative AI system appropriately describe historical image of tragic event, con- troversial figure, or sacred location? The Al development and research communities may not have an established nor definitive answer to this question, which is fraught with cultural considerations, potential biases, and inherent subjectivity. However, museums, archives, and digital collections offer valuable insights to the description of such materials, stemming from their legacy of communicating the importance of historical artifacts to wider audiences (Ogden 2007 Sealy et al., 2021
Typically, museums and archives contextualize visual artefacts with additional didactic information, to provide further socio- historiitoiiaall context or offer interpretation of a piece through a contempo- rary lens (Parry et al., 2007). These curated perspectives are often perceived by viewers - without prior knowledge of the subject matter to be relatively objective and authoritative source of in- formation (Nashashibi 2003) Acknowledging the role they play in introducing new audiences to historical content, many of these institutions have developed robust guidelines around the discussion of morally charged or divisive history, offering a valuable starting point for approaches to evaluating Al-based historical description (Jo & Gebru 2020).
Recently, museums have been subject to a critical discourse highlighting colonial and violent legacies that surface in curation and description of artefacts, speaking to the inherently partial nature of historical commen- tary and interpretation (Aldrich) 2013). Many museums have acknowledged these criticisms by making public changes their curatorial practices and communication strategies (Pitt Rivers Museum).
3

---

Published as a conference paper at ICLR 2025
3 CENTURY IMAGE DATASET
3.1 DEFINING QUALITY AND DIVERSITY FOR SENSITIVE HISTORICAL IMAGES
For our purposes, we define a high-quality historical image set as one predominantly composed of sensitive images. Drawing upon established definitions from museums and archives, we identify three conceptual dimensions of sensitivity in visual media. These dimensions serve as the foundation for an evaluation task designed to assess the compositional quality of our image set.
In prior work that discusses the archival and museum approach to describing sensitive historical top- ies to new audiences (Savenije et al. 2016 Zembylas & Kambani 2012 Pabst 2018 [cchorch] 2020 Savenije & De Bruijn, 2017, Gagen| (021) sensitivity is often framed as an affective phenomenon: sensitive topics are those that need to be discussed with care, as doing so without appropriate consid- eration may cause offense or discomfort to the viewer, especially for viewers with prior emotional investment in a historical issue or cause.
We adopt the prevalent definition of sensitivity one of our evaluation dimensions, and also in- vestigate two related conceptual properties that often emerge in discussions of historical images: commemorativeness and controversiality. In discussions of curation and display standards, com- memorative image depicts or represents a tragedy or atrocity suffered by an individual or group, and therefore needs to be discussed with care to ensure it engages an appropriate "way of remem- bering" (Waters & Russell III 2013). A controversial image depicts divisive subject matter that may require additional nuance and care in how its context is conveyed, as neglecting to do may stoke tensions (Savenije et al. 2014) and risk erasing the perspective of an affected group (Schnei- der & Hayes 2020] Koggel| (020). Although the dimensions are likely to be correlated due their conceptual similarities, they are considered separately when labelling Century images through both human and automated evaluation. We share illustrative example images and justifications displayed to human raters in Appendix H
For diversity, we focus on coverage of the four themes from Section 3.2 (conflict, oppression, dis- crimination, and reform), the type of the image (e.g., photograph map), and the primary geo- graphic region depicted in the image. Geographic diversity is of particular importance, given that creating geographically diverse evaluations help better serve global users and developers using vision language models (Bhatt et al. 2022 Dev et al. 2023).
3.2 CONSTRUCTING A DATASET OF SENSITIVE HISTORICAL IMAGES
In this Section, we describe the novel methodology developed to source a large and diverse set of historically sensitive search terms. Century is created by mining 37.8 million records in the Wikipedia-Based Image Text Dataset (WIT) (Srinivasan et al. 2021) using the approach outlined in Figure 2 The WIT dataset consists of images taken from Wikipedia pages, with each record consisting of link to an image hosted Wikimedia Commons, alongside information about the Wikipedia page from which the image originates, the primary language of that page, a reference description of the image, and an indication of whether the image is the primary image for that page. We choose the WIT dataset because it contains large numbers of real-world images, but note that our method can be flexibly adapted to other image datasets in future work. When mining WIT, we include images from all splits including the training split, and discuss our rationale and address potential concerns in Appendix
Themes and types of images. In order to source a sizable collection of search terms on which to filter the WIT dataset, we must first establish conceptual "root nodes" that would allow us to identify historically significant entires. To do so, we conducted an interdisciplinary review and identified four themes that are commonly implicated in sensitive discussions of modern history: Anastasoll 2002), oppression [Teeger 2015 Mycock, discrim- Ladson- Ladson-Billings 2020 Peek et al., 2020), and reform (Vandeyar &
conflict (Warnasuriya 2017
ination Waaserstromm 1976;
Swart 2018 Brasted 2005). To improve diversity and generality. chose four types of images to
target: images that depict events, organizations, people, and locations. Theses specific themes and types are the input into our ceneral-purpont automated methods.
Automated mining with a knowledge graph. We use a knowledge graph built from multiple data sources, such Freebase and Wikipedia, to query for entities related to the four broad themes. Each
4

---

Published as a conference paper at ICLR 2025
knowledge graph
labels from automated eval
search phrases
quality filters
themes types
WIT
labels from human eval
foundation language model
Wikipedia
lists
Figure 2:
Overview of automated pipeline for creating Century, and methods for creating labels with automated and human evaluation. The released dataset includes both sets of labels for each image.
entity a discrete entry in the knowledge graph that is connected to other entries through hicrarchi- cal conceptual edges. In order to inerease the scope of the search, manually expanded each theme into semantically similar concepts (shown in Appendix Table We then query the knowl- edge graph for people, organizations, events and locations that are associated with these concepts, prioritising results towards entities tagged with labels associate with the 20th and s centuries. of 7,385 unique search terms, of which we release large sample to facilitate future research efforts. Finally, we use these search terms to mine WIT by matching Wikipedia page titles, with details in Appendix E
This produced
Automated mining with foundation language models. To improve coverage of sensitive and his- torical images, we generate additional search terms not covered by the knowledge graph approach by prompting foundation language models. Since WIT is composed of over 37M records, using a foundation model to label each image and then filter would be impractical. We use foundation models to generate diverse set of candidate Wikipedia page titles that may contain sensitive his- torical images, building on prior work on synthetic data generation (Radharapu et al.) 2023). We draw 50 responses and merge the results. This method produces 1,297 unique candidate page titles, which we use to mine WTT. Interestingly, find that only 15.1% of the page titles produced by these foundation models overlap. Instructions for reproducibility are in Appendix Gi with exact models versions in Appendix(A] A Final set of images. We used the search terms produced by Knowledge Graph and foundational language model mining mine WTT. We match any WIT record where the Wikipedia page is in English, and the page title contains the search term. We detail the results of each search strategy, as well as additional quality filters applied in mining WIT, in Appendix
instructions from two foundation models, GPT-4 Omni and Claude Opus,
For the final set of images in Century, we target size of 1,500 total images that is practical for use in system evaluations, downsampling images mined with the knowledge graph method. The final datasets consists of 1.156 images from automated mining with knowledge graph (77.1%) and 216 images from automated mining with foundation model (14.4%). For completeness, we additionally include 128 images (8.5% from a manual review of the Wikipedia lists that describe historically important photographs (see Appendix F.
3.3 MEASURING QUALITY AND DIVERSITY OF CENTURY IMAGES DATASET
In this section, we evaluate the quality and diversity of the image dataset produced by our method- ology, critical step in establishing conceptual validity (Jacobs & Wallach) 2021 Blodgett et al.) 2021 Since evaluating quality particularly challenging areas where human judgements vary [Zhang et al. 2023 Aroyo al. (202)), we introduce operational definitions of quality, describe au- tomated and human evaluation methods, and additionally release all quality ratings on the Century dataset enable further research.
Labelling images with automated methods. While using foundation models is a common ap- proach for evaluating ependended text (Schick al. 2021: Zheng al.] 2023b) and has recently been explored for image inputs (Hu et al., 2023; Cho et al. 2024 Chan et al. 22023] Wiles et al.) 2024), little work has examined the effectiveness of automated evaluation when involving complex
5

---

Published as a conference paper at ICLR 2025
Image type
images
Content type
images
Concept
images
Region
images
Photograph
64.6%
Event
46.1%
Conflict
36.8%
N. America
18.9%
Document
16.7%
Location
23.4%
Reform
21.0%
W. Europe
11.4%
Artistic depiction
12.6%
Person
16.1%
Oppression
5.1%
E. Europe
8.0%
Symbol
4.5%
Organisation
6.0%
Discrimination
3,4%
W. Asia
6.7%
None (other)
0.1 1%
None (other)
2.3%
None (other)
23.4%
E. Asia
6.1% %
(no majority)
1.2%
majority) 6.0%
(no majority) 10.2%
(no majority)
15.7%
Diversity of images: Human evaluation results showing that the concepts, content types and images types that our method targets all represented. Images received three distinct ratings per dimension per labeling method (human and automated). Each column represents the percentage of all images in Century that were considered to be of certain dimension by labeller when aggre- gating by majority choice. Every UN sub region is represented. Automated labeling results are in Appendix K
Table 1:
socio-cultural reasoning or judgments (Dillion et al. 2024 Lin et al. 2023), especially with cross- modal inputs. To advance research in such methods, we use automated methods to label images in Century across our dimensions of quality, report results with six different labeller models, and release automated labels alongside labels from human evaluation.
Quality results. Using the labelling approach outlined in Appendix 2 we find that 96.1% 1% of Century images are described as "somewhat sensitive" or higher by or more labeller models. Averaging across labellers, we find that 61.5% of images are "somewhat sensitive" or higher when taking the mean rating across labellers. We additionally discover differences in ratings of each dimension of quality across auto-labeller models, sometimes differences as large as 30 percentage points (eg. GPT- 4 Turbo and Gemini 1.5 Pro for sensitive, GPT-4 Turbo and Claude Haiku for controversial). We describe distributions of quality ratings broken down by labeller model in Appendix [ highlighting how choices of labeller model can greatly influence how automated evaluation results are interpreted and reported an area for future research building Century.
Diversity results. We find that Century represents all targeted concepts and image types, and includes images representing all United Nations sub-region sub-regions, with detailed breakdowns in Appendix K We discuss in Section how these labels can be used used for disaggregated evaluation along dimensions like geographic region and image type (Barocas et al. 2021).
Labelling images with human evaluation. We also perform human evaluations to label images in Century along the same dimensions of quality and diversity as with automated evaluation. We recruit 151 participants on a crowd-sourcing platform and ask them to annotate images in Century to create 3 ratings for each image (M 29.4, SD = 22.9 images rated per participant). Participants were required to be fluent in English and self-reported as having relevant research experience experience (e.g. undergraduate degree in history). The full annotation task, including compensation rates, was reviewed and approved by an internal ethics review committee. Participants were paid at least the living wage for their location, with average compensation of f16.50 per hour. Recruitment is described in Appendix and full instructions and task design are shown in Appendix X
Quality results. We find that 90.9% of images are described as "somewhat sensitive" higher by one or more human raters. Averaging across ratings, we find that 55.8% of images have mean rat- ing of (3) "somewhat sensitive" or higher. Interestingly, we find that aggregate metrics are broadly similar across automated and human evaluation methods, though we find human raters to be more conservative in their judgments, assigning lower scores on average than automated labellers. Mea- sures of inter-rater reliability in Appendix M fall in range that reflect the complex socio-cultual knowledge and judgement required in this task (Wong et al., 2022 Salminen et al., (018). Finally, review disagreement in human judgements of sensitivity using CrowdTruth metrics (Aroyo et al. 2023) in Appendix N and include examples from qualitative analysis in Appendix O
Dataset release. We release Century on Githubl three distinct datasets: the first is text file containing all search terms of historical events and figures collected through our mining method; the second contains all images in Century Wikipedia links, alongside other metadata attributes; the third dataset includes all image annotations collected through human and auto-rater methods, including both diversity and quality labels.

---

Published as a conference paper at ICLR 2025
Dimension of Quality
Labelling method
"Not at all"
"Somewhat"
"Somewhat" (3) or higher
(1)
(3) or higher
mean rating
mean rating
any rater
Sensitive
Human eval
1.7%
55.8%
90.9%
All six labellers
0.4%
61.5%
96.1%
GPT-4 Turbo
14.7%
48.8%
GPT- Omni
6.7%
70.9%
Gemini Pro
6.3%
91.7%
Gemini Flash
10.8%
66.0%
Claude Opus
4.0%
90.4%
Claude Haiku
0.7%
86.9%
Controversial
Human eval
2.3%
45.3%
85.0%
All six labellers
0.5%
54.9%
91.9%
GPT-4 Turbo
23,4%
50.2%
GPT-4 Omni
5.9%
65.9%
Gemini Pro
12.6%
70.1%
Gemini Flash
9.6%
60.4%
Claude Opus
4,9%
78.7%
Claude Haiku
1.3%
87.8%
Commemorative
Human eval
2.
57.6%
90.1%
All six labellers
3. %
50.3%
88.2%
GPT-4 Turbo
30.5%
45.5%
GPT-4 Omni
6.1 %
71.5%
Gemini Pro
13.4%
77.8%
Gemini Flash
25.8%
55.2%
Claude Opus
9.2%
76.7%
Claude Haiku
10.3%
65.4%
Table We present human and automated evaluation of our Century images dataset. Though results vary across evaluation methods, we find convergent evidence that Century contains sensitive images that are promising candidates for evaluating historical contextualisation capabilities. Differences in distributions of automated labeller ratings are shown in Appendix J and differences in human ratings image dimensions are summarised and discussed Appendix
4 HISTORICAL CONTEXTUALISATION IN MULTI-MODAL MODELS
With the quality and diversity of our image dataset established, we explore how Century can be used to assess foundation models. Specifically, we present an approach for evaluating historical contextualisation the ability to generate descriptions that accurately capture the historical context of an image based solely on the image itself. This is a complex task that builds upon foundational capabilities in vision language research, such as entity recognition and commonsense reasoning, while incorporating editorial considerations.
We describe definition of response quality, reproducible evaluation method, and apply evaluation method to four publicly accessible systems: GPT-4 Omni, Claude Opus, Gemini 1.5 Pro, and Gemini 1.5 Flash. We generated responses from each model using the method detailed in Appendix P creating a test set for evaluating their ability to provide historical context for images.
4.1 DEFINING QUALITY FOR HISTORICAL CONTEXTUALISATION
Historical contextualisation of sensitive images is a task with rich prior work in museums, archives, and collections. We reviewed over principles drawn from guidelines and recommendations across a diverse set of institutions (Appendix QQ. and from an inductive analysis of frequently occurring recommendations, synthesised a novel protocol for measuring the quality of historical contextual- isation. We focus on three dimensions: accuracy in identifying key elements and their context, thoroughness in providing concise yet comprehensive descriptions that enable understanding and further research, and objectivity by avoiding biased language and interpretations (more detail in Appendix R.. We adapt this protocol to evaluate multi-modal models with automated and human evaluation methods.

---

Published as a conference paper at ICLR 2025
GPT-4
Gemini
Gemini
Claude Opus
Omni
Pro
Flash
Dimension
Element
Method
"Agree" (4) or higher, mean rating
Accuracy
Correct identification
Automated
53.3%
44.0%
37.2%
21.9%
Human
45.5%
24.3%
20.2%
12.2%
Factuality errors (inverted)
Automated
41.8%
30.2%
20.0%
19.4%
Human
Thoroughness
Beginner friendly
Automated
63.7% 54.5% %
53.4%
39.2%
28.7%
Human
28.0%
27.1%
20.1% %
Appropriate summary
Automated
51.0%
43.7%
27.5%
17.9%
Human
20.6%
15.3%
20.2%
9.5%
Objectivity
Due weight
Automated
53.1 %
45.9%
25.1%
19.0%
Human
40.2 %
23.3%
16.5%
17.5%
No loaded language
Automated
76.5%
61.9%
44.3%
56.3%
Human
36.5%
34.4%
29.8%
29.1%
Opinions not stated as facts
Automated
69.8%
51.6%
34.3%
37.3%
Human
38.1%
31.7%
26.1%
33.3%
Table 3: Evaluation historical contextualisation capabilities for GPT-4 Omni, Gemini Pro 1.5, and Gemini Flash 1.5, and Claude Opus. Automated evaluation results are the percentage of responses scored as "agree" or higher (n=1,500), taking the mean rating across labeller models (details in Appendix (S). Results from the Human evaluation results percentage of responses scored "agree" or higher (n=378), when taking the mean rating from human evaluation ratings. Human evaluation for factuality errors (reported as free-text responses) discussed in Appendix U
4.2 RESULTS ACROSS FOUNDATION MODELS
Automated evaluation of historical contextualisation capabilities. We experimented with six la- beller models (GPT-4 Turbo, GPT-4 Omni, Gemini Pro, Gemini Flash, Claude Opus and Claude Haiku) to produce evaluation ratings for the accuracy, thoroughness, and objectivity of model re- sponses generated by four target models (GPT-4 Omni, Gemini Pro, Gemini Flash, Claude Opus) We report main results by ensembling across four of the best-performing labeller models Each re- sponse is scored by computing the mean rating for each response across labeller models, and re- porting the percentage of responses that scored "agree" or higher. Implementation details are in Appendix
In Table we find that Century is effective at discovering opportunities for all foundation models improve historical contextualisation. Additionally, in Appendix Figure we observe that GPT-4 Omni's responses consistently receive higher ratings (indicating they "agree" "strongly agree" with the quality statement), and that this pattern holds across all questions, regardless of the labeller model used.
While we do not observe systematic patterns of "selff-chhancmmeen"" bias (Zheng et al. 2023b), we do find aggregate differences in distributions of ratings by labeller model, particularly for GPT-4 Omni, and describe these in Appendix Figure IT Further, in Appendix Figure 12 we find indications of response length bias for some dimensions of quality. Discussions of failures in labeller models can be found in Appendix(S)
We additionally find in Appendix Table 12 that when querying with instructions that explicitly quest historical context as opposed to queries that target identification only, evaluation results differ along objectivity dimensions. Evaluations building Century could extend this analysis, measur- ing differences in capabilities as indexed by explicit contextualisation requests and other common prompt patterns.
2 All versions of model checkpoints used for labelling and response generation are detailed in Appendix More recent model checkpoints are available at the time of submission, but are not included in the evaluation results.
8

---

Published as a conference paper at ICLR 2025
Human evaluation of historical contextualisation capabilities. Human evaluation in open-ended generation is challenging, particularly for tasks that require cross-modal understanding and socio- cultural reasoning. To complement the automated results and showcase the opportunities for future work to build on Century, we conduct a small-scale human evaluation. Through a crowd-sourcing platform, we collect human annotations for model responses. We use sample of 63 images drawn from Century, and for each image we draw samples from each model with explicit instructions provide historical context. Each model response was assessed with distinct annotations. More details on human data collection are in Appendix T
In Table 3 we find that more responses from GPT-4 Omni has a mean rating of "agree" or higher as compared to Gemini Pro or Claude Opus. In all dimensions, we see Gemini Pro performing better than Gemini Flash (with the notable exception of "Appropriate summary"). While these human evaluation results are from a small evaluation set and may be underpowered (Howcroft & Rieser 202T), we enable further research from the community by detailing the instructions given to participants in Appendix X.1 and sharing all collected annotation results (including "incomplete" ratings with less than distinct annotations).
Qualitative review. Given complexity in historical contextualisation, and challenges with measure- ment validatity across all evaluation methods, we also conducted a qualitative review. We describe examples of failures in historical contextualisation in Appendix V
Considerations in measuring contextualisation. In interpreting the results of the evaluation task, must take into account that it relies context-free evaluation, meaning models are evaluated on historical contextualisation with images passed in with no additional textual information. However, in qualitative assessments of the Century dataset, we found that images may need different amounts of accompanying information in order to have a reasonable expectation that models perform well against our evaluation criteria. For example, some images are of iconic or highly identifiable events or figures (e.g. the fall of the Berlin Wall), while others contain fewer identifiable features.
Another challenge arises from differences in how directly images capture the target historical scenes. Some images in Century are intended to depict events, which are not tangible entities and can rep- resented variety of ways. As a result, the main Wikipedia images accompanying articles on historical events cannot be considered to unambiguously capture the target keyword. For exam- ple, an image of modern-day monastery, which was once a site of a historical battle, used as a representation the battle.
In future applications of evaluations based on Century, we recommend developers experiment with both context-free and context-driven protocols, and compare performance between both to iden- tify especially context-sensitive images. For images where performance on context-free evaluation is particularly lacking, researchers may experiment with augmenting performance with retrieval- augmented generation architectures (Chen et al.] 2024). As with automated historical artifact tag- ging (Wu et al.] 2023), limited training examples for specific historical events figures may impact multi-modal model performance on Century. Comparing system performance on Century with other benchmarks context-sensitive reasoning vision tasks (Wadhawan al. 2024) can reveal if sys- tem weaknesses are specific to the particular challenges posed by historical image data.
5 DISCUSSION
We provide recommendations for developers and discuss limitations, providing entry points for the research community to begin adopting or building upon the Century dataset.
5.1 RECOMMENDATIONS FOR DEVELOPERS
For developers of foundation models, we recommend using Century to evaluate fundamental capa- bilities and cross-modal understanding and reasoning. We provide evaluation protocol for assess- ing normative dimensions of system responses that can be adapted to specific application contexts. We recommend developers begin their evaluations with our "starter" set (n = 80 images), which includes all images with a mean rating of "sensitive" > 3.0 across human and automated evaluation, with each geographical sub-region downsampled more than six images.

---

Published as a conference paper at ICLR 2025
Human evaluation continues to be critical for evaluating complex responses along socio-cultural and normative dimensions, and incorporating insights from diverse rater pools remains critical for mea-
surement quality (Aroyo et al.] 2023; Parrish et al. 2024). While foundation models show promise as tractable evaluation tools (Amodei et al. 2016; Bai et al. 2022), the variance in performance across and within model families demonstrated the need for incremental adoption of automated evaluation. We recommend validating with human evaluation, qualitative review, and stakeholder feedback when implementing the benchmark. To this end, we demonstrate how system develop- ers can use Century metadata to perform disaggregated analysis (Barocas et al. 2021) and evaluate specific areas of strength and weakness by geographic region in Appendix W
5.2 LIMITATIONS
Distribution of dataset. There are over- and under-representation biases in the dataset: as expected, automated search term sourcing yields more images from North America and Western Europe. Us- ing the labels we release geographic and context diversity of the dataset, future work may ex- pand upon the representation of certain groups, languages, and locales by conducting more targeted searches (e.g. focusing on sourcing images relevant to Pacific Islander history). In applying the current version of Century during development, developers should be mindful that observed im- provements in historical contextualisation may not be evenly distributed across all global regions. To assess performance variations, we recommend leveraging the geographic labels included in the open-source release and conducting qualitative reviews of model outputs, particularly for regions with less representation in the dataset.
Lack of targeted inclusion of affected communities. Quality criteria for images were derived from guidelines established by museums, archives, as described in Section 3.1] These norms may not al- ways reflect the views of communities whose histories the images in Century may reflect. In some instances, guidelines drawn from institutions may contradict the wishes expectations of commu- nity. For example, a persecuted community may not agree that Al-written descriptions of an instance of persecution should be evaluated for loaded language, arguing instead that an appropriate caption ought to explicitly condemn the pictured event. Future work may apply participatory approaches achieve more inclusive and representative guidelines and image annotation (e.g. [Bergman al.) 2024 Qadri et al., 2023 Weidinger et al.) 2024). In the meantime, developers may wish to use the Century framework obtain frequent signals of Accuracy and Thoroughness, while calibrating signals of Objectivity to insights drawn from relevant communities and stakeholders.
Generative labelling for non-majoritarian normative perspectives. Though generative models are increasingly used for annotation (and Al-based annotation results are presented for Century), there are well-known pitfalls for using such models to reflect subjective or normative perspec- tives, especially when the "normatively correct" perspective can vary across group and identity lines (Pavlovic Poesio 2024 Abid et al. 2021 Venkit et al., 2023 Kamruzzaman al. 2023). Failure modes may arise if models are unable to represent or calibrate to pluralistic views, and, as a re- sult, preferentially endorse majoritarian views or erase minority perspectives (c.g.AAbid Abid et al.) [02T]] Venkit et al.] 2023; Kamruzzaman al. 2023). Future work may seek to represent the influence of non-majoritarin perspectives on voting outcomes through approaches like jury learning (Gordon et al. 2022). They may also explore simulating more diverse rater perspectives (Lee al. (024b) by eliciting in-group perspectives through techniques like demographic matching (Weidinger et al, 2024). In the short term, developers should seek to calibrate automated labeling against diverse (e.g. census-represeetttivv))) human signals, especially in high-stakes evaluations like those intended to inform deployment decisions.
CONCLUSION
We introduce Century the research and AI development communities. We demonstrate several evaluation protocols using the images in the dataset, including automated labelling methods and human evaluation with crowd-workers. We release the main Century dataset to enable further anal- yses, applications, and expansions upon our set of historically-releaant sensitive images. We hope Century will serve as a useful foundation for testing the historical contextualisation capabilities of cutting edge multi-modal generative models.
10

---

Published as a conference paper at ICLR 2025
7 ETHICS STATEMENT
We address several ethical concerns relevant our work.
Human subjects. Before recruiting any participants to annotate images in our dataset, the annota- tion task reviewed and approved by an internal ethics review board. All participants were paid at or above living wage for their location. While there was a risk that participants could be exposed to sensitive graphic content during the annotation task due to the nature of the dataset, provided them with mechanisms to limit exposure (e.g. voluntary blurring of images, unlimited skipping to the next image). Content identified as too graphic for annotation at any point was filtered out of all future tasks. Generally, participants reported a positive experience with the annotation task.
Dataset. We enable developers access images in the dataset by sharing links the original To avoid stripping images of their metadata and other originally available information, all images included in are attributable to the original source through the links provided. To prevent misuse or training for unwanted purposes such as generating harmful responses to sensitive historical images we do not release model outputs or associated rating signals, but instead release the image dataset and the evaluation protocol. More detail on the content included in the open-source release can be found in Section 3, under Dataset release.
Bias. Information surfaced through knowledge graph and large language model curation methods may be biased. We demonstrate early indications of bias and lack of representativeness Appendix K To allow future researchers to account for the lack of representation for certain regions in fu- ture analyses and implementations of, as well as enabling improvement upon representation in the dataset, we release geographic labels associated with all images.
8 REPRODUCIBILITY STATEMENT
We share the Century dataset in full, alongside human and automated labels for quality and diversity labels described in Section 3.3 We also share all permissible raw search terms used to curate Cen- tury from the Wikipedia Image-Text Text Dataset enable their application other text- or embedding- based image datasets. Further details on reproducing the search term collection and dataset curation steps are available in Appendices B through G
Further, we provide details to ensure reproducibility of our model evaluation method in Section 4.2 and Appendix [ All models used for evaluation purposes (listed in Appendix A are proprietary and accessible through API, so we cannot guarantee that future calls will produce the same outputs.
We make all human annotation instructions available as screenshots of the pages participants saw during the task in Appendix X onwards.
We refrain from sharing model response annotation dataset reported in Section 3.3 due to data leak- age concerns.
REFERENCES
Abubakar Abid, Maheen Farooqi, and James Zou. Persistent anti-muslim bias in large language models. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 298- 306, 2021.
lean-Baptiss Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716- 23736, 2022.
Robert Aldrich. Colonial museums in postcolonial europe. In Museums in postcolonial Europe, pp. 12-31. Routledge, 2013.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in ai safety, 2016.
11

---

Published as a conference paper at ICLR 2025
Harry Anastasiou. Communication across conflict lines: The case of ethnically divided eyprus. Journal of peace research, 39(5):581-596 2002.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6077-6086, 2018.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2223-,433, 2015.
Lora Aroyo, Alex S. Taylor, Mark Diaz, Christopher M. Homan, Alicia Parrish, Greg Serapio- Garcia, Vinodkumar Prabhakaran, and Ding Wang. Dices dataset: Diversity in conversational ai evaluation for safety, 2023.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Ols- son, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran- Johnson, Ethan Perez, Jamie KeTI, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mer- cado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawtod Tom Con- erly, Tom Henighan, Tristan Hume. Samuel R. Bowman, Zac Hatfield-oddds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback. 2022.
Solon Barocas, Anhong Guo, Fiee Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wort- man Vaughan, Duncan Wadsworth, and Hanna Wallach. Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs, 2021.
Stevie Bergman, Nahema Marchal, John Mellor, Shakir Mohamed, Iason Gabriel, and William Isaac. Stela: a community-centre approach to norm elicitation for ai alignment. Scientific Re- ports, 14(1):6616, 2024.
Shaily Bhatt, Sunipa Dev, Partha Talukdar, Shachi Dave, and Vinodkumar Prabhakaran. Cultural re-contextualization of fairness research in language technologies in india, 2022.
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. Stereotyp- ing Norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Chengqing Zong. Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume Long Papers). pp. 1004-1015, Online, Au- gust 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2022 acl- URL org.orgg//........o 2021 202l.acl-loog.
https: TalE::://aclanthologo
Monica Brasted. Protest in the media. Peace Review: A Journal of Social Justice, 17(4):383-388 2005.
David Chan, Suzanne Petryk, Joseph E Gonzalez, Trevor Darrell, and John Canny. Clair: Evaluating image captions with large language models. arXiv preprint 23X1:23110.19971. 2023.
Dawei Chen, Zhixu Li, Binbin Gu, and Zhigang Chen. Multimodal named entity recognition with image attributes and image knowledge. In Database Systems for Advanced Applications: 26th International Conference, DASFAA 2021, Taipei, Taiwan, April 1 11-14, 2021, Proceedings, Part II 26, pp. 186-201. Springer, 2021.
Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language models in retrieval-augmentee generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 17754-17762, 2024.
Xi Chen, Xiao Wang. Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv: 27X7792209.06794 2022.
12

---

Published as a conference paper at ICLR 2025
Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint
2015.
Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. Davidsonian Scene Graph: Improving Reliability in Fine-Grained Evaluation for Text-to-lmage Generation. In ICLR, 2024.
Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. Dealing with Disagree- ments: Looking Beyond the Majority Vote in Subjective Annotations. Transactions of the Association for Computational Linguistics, 10:92- 10, 01 2022. ISSN 2307-387X. doi: 116./..111..1. a_ 00449
10. N0.1162/aacla..04449 htttpsss::///oi.
Sunipa Dev, Akshita Jha, Jaya Goyal, Dinesh Tewari, Shachi Dave, and Vinodkumar Prab- hakaran. Building stereotype repositories with complementary approaches for scale and depth. In Sunipa Dev, Vinodkumar Prabhakaran, David Adelani, Dirk Hovy, and Luciana Benotti (eds.). Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pp. 84-90, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi: URLpttps: peittps://aclaathoooogg wttp:o/2222....n-11.....
10. 10.18865/v1//022..3nnp.
Danica Dillion, Debanjan Mondal, Niket Tandon, and Kurt Gray. Large language models as moral experts? gpt-4o outperforms expert ethicist in providing moral guidance, May 2024. os..
Nicholas Dufour, Arkanath Pathak, Pouya Samangouei, Nikki Hariri, Shashi Deshetti, Andrew Dudfield, Christopher Guess, Pablo Hernández Escayola, Bobby Tran, Mevan Babakar, et al. Ammeba: A large-scale survey and dataset of media-bar based misinformation in-the-wild. arXiv preprint arXiv:2405.1 /697, 2024.
Elizabeth Gagen. Facing madness: The ethics of exhibiting sensitive historical photographs. Journal of Historical Geography, 71:39-50, 2021.
Mitchell I. Gordon, Michelle Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S Bernstein. Jury learning: Integrating dissenting voices into machine learning models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1-19, 2022.
David M. Howcroft and Verena Rieser. What happens if you treat ordinal ratings as interval data? human evaluations in NLP
inder-powered than you think. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 202/ Conference on Empirical Methods Natural Language Processing, pp. 8932-8939, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguis- tics. doi: 10.18663/112221.mmmppp emm.. comnlp-mat 1al1pppmaa in. 703
703. UBL/attpss//adllanttcoooo.. org/2011
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A. Accurate and interpretable text-to-image faithfulness evaluation with question swering. In ICCV, 2023.
Smith. Tifa:
Abigail Z. Jacobs and Hanna Wallach. Measurement and fairness. Proceedings of the 2021 ACM Conference on Fairness, Accountabiliry, and Transparency, FAceT 221. ACM, March 2021. doi: 000.1145/3442188.4455555 4ttp:4440....../................
URL http::tt://ttt doi wttp:ooo.rrgg/// 1145:/1...11///3 011444/44428888 11441//4/541888444991 344500
Eun Seo Jo and Timnit Gebru. Lessons from archives: Strategies for collecting sociocultural data in machine learning- In Proceedings of the 2020 conference on fairness, accountability, and transparency, pp. 306-316, 2020.
Mahammed Kamruzzaman, Md Minul Islam Shovon, and Gene Louis Kim. Investigating subtler biases in Ilms: Ageism, beauty, institutional, and nationality bias in generative models. arXiv preprint arXiv:2309.0880 2023.
Christine M Koggel. Epistemic injustice in settler nation: Canada's 's history of erasing, silencing, marginalizing. In Reconciliation, Transitional and Indigenous Justice, pp. 118-129. Routledge, 2020.
13

---

Published as a conference paper at ICLR 2025
Terry K Koo and Mae Y Li. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of chiropractic medicine, 15(2):155-16) 2016.
Gloria Ladson-Biin Biilinggs.. Just what critical race theory and what's it doing in a nice field like education? In Critical race theory in education, pp. 9-26. Routledge, 2020.
Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Teufel, Marco Bellagente, et al. Holistic evaluation of text- to-image models. Advances Neural Information Processing Systems, 36, 2024a.
Yoonjoo Lee, Tae Soo Kim, and Juho Kim. How reflect diverse people's perspectives in large- scale llm-based evaluations? HEAL Workshop at CHI Conference Human Factors in Comput- ing Systems, 2024b.
Yunpeng Li, Noah Snavely, Dan Huttenlocher, and Pascal Fua. Worldwide pose estimation using 3d point clouds. In European conference on computer vision, pp- 15-29. Springer, 2012.
Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base llms: Rethinking alignment via in-context learning, 2023.
Alexandra Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jernite. Stable Bias: Analyzing Societal Representations in Diffusion Models, March 2023. URL http::/arxixv org/ hog//bs//e 2303. 11408 arXiv:2303.1 1408 [cs].
Kristian Lum, Jacy Reese Anthis, Chirag Nagpal, and Alexander D' A'mour. Bias in language mod- els: Beyond trick tests and toward ruted evaluation, 2024.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
MediaWiki
contributors. Help:images. Images Accessed on [insert date).
Andrew Mycock. After empire:
The polities of history education in post-colonial world. Pulgrave handbook of research historical culture and education, pp. 391-410, 2017.
Salwa Mikdadi Nashashibi. Visitor voices in art museums: The visitor-written label. Journal of Museum Education, 28(3):21-25, 2003.
Sakari Nieminen and Lauri Rapeli. Fighting misperceptions and doubting journalists' objectivity:
A review of fact-checking literature. Political studies review, 17(3):296-309, 2019.
Sherelyn Ogden. Understanding, respect, and collaboration in cultural heritage preservation:
conservator's 's developing perspective. Library Trends, 56(1):275-287, 2007.
Kathrin Pabst. Considerations to make, needs to balance: Two moral challenges museum employees face when working with contested, sensitive histories. Museum International, 70(3-4):84-97, 2018.
Arjun Panickssery, Samuel R. Bowman, and Shi Feng. Lim evaluators recognize and favor their own generations, 2024.
Alicia Parrish, Vinodkumar Prabhakaran, Lora Aroyo, Mark Díaz, Christopher M. Homan, Greg Garcia Alex S. Taylor, and Ding Wang. Diversity-aware annotation for conversational AI safety. In Tanvi Dinkar, Giuseppe Attanasio, Amanda Cereas Curry, Ioannis Konstas, Dirk Hovy, and Verena Rieser (eds.), Proceedings of Safely4ComvAAI: The Third Workshop on Safety for Conversational
I.RE-COLING 2024, pp. 8-15, Torino, Italia, May 2024. ELRA and ICCL. URL http:/:::///aaaahhhaaa/s/ org/ 2202 safety4convai-1..
Ross Parry, Mayra Ortiz- Williams, and Andrew Sawyer. How shall we label exhibit today? applying the principles of on-line publishing to an on-site exhibition. Museums and the Web 2007, 2007.
14

---

Published as a conference paper at ICLR 2025
Maja Pavlovic and Massimo Poesio. The effectiveness of Ilms annotators: A comparative overview and empirical analysis of direct representation. arXiv preprint arXiv:2405.01 2024.
Monica E Peek, Monica B Vela, and Marshall H Chin. Practical lessons for teaching about race and racism: successfully leading free, frank, and fearless discussions, Academic medicine, 95(12S): S139-S144, 2020.
Pitt Rivers Museum. Critical changes to displays as part of the decolonisation process. URL https://
ox.ac. uk/ Seritical--hangesscco.. changeesccllllps22255555
Rida Qadri, Renee Shelby, Cynthia L Bennett, and Emily Denton. Ai's regimes of representation:
A community-centered study of text-to-image models in south asia. In Proceedings of the 2023 ACM Conference on Fairness, Accountabilityy and Transparency, pp. 506-517, 2023.
Bhaktipriya Radharapu, Kevin Robinson, Lora Aroyo, and Preethi Lahoti. Aart: Ai-assisted red- teaming with diverse data generation for new llm-powered applications, 2023.
Joni O Salminen, Hind A Al-Merekhi, Partha Dey, and Bernard Jansen. Inter-rater agreement for social computing studies. In 2018 fifth international conference on social networks analysis, management and security (snams), pp. 80-87. IEEE, 2018.
Habiba Sarhan and Simon Hegelich. Understanding and evaluating harms of ai-generated image captions in political images. Frontiers in Political Science, 5, 2023.
Geerte Savenije, Carla Van Boxtel, and Maria Grever. Sensitive 'heritage' of slavery in a multicul- tural classroom: Pupils' ideas regarding significance. British Journal of Educational Studies, 62 (2):127-148, 2014.
Geerte M Savenije and Pieter De Bruijn. Historical empathy in museum: uniting contextualisation and emotional engagement. International Journal of Heritage Studies, 23(9):332-84) 2017.
Geerte M Savenije al. An intriguing historical trace heritage? learning about another person's heritage in an exhibition addressing wwii Sensitive Pasts. Questioning Heritage in Education, pp. 218-239, 2016.
Timo Schick, Sahana Udupa, and Hinrich Schütze. Self-diagnosis and self-debiasing: A proposal for reducing corpus- corpus-based based bias in nlp. 2021.
Tsim D Schneider and Katherine Hayes. Epistemic colonialism: is it possible to decolonize archae- ology? The American Indian Quarterly, 44(2):127-145: 2020.
Philipp Schorch. Sensitive heritage: Ethnographic museums, provenance research and the potential-
ities of restitutions. 2020.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi.
A-okvqa: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision, pp. 146-162. Springer, 2022.
Mark Sealy, Makeda Best, Ilisa Barbash, and David Odo. Troubling images: Curating collections of historical photographs, Feb 2021.
Patrick E Shrout and Joseph Fleiss. Intraclass correlations: uses assessing rater reliability. Psychological bulletin, 86(2):420, 1979.
Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. In Pro- ceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21. ACM, July 2021. doi: 10.1145//40043353446355 URL
http:// doi. www.oogg.1.... 344455554443333632555 3463257
Miriah Steiger, Timir Bharucha, Sukrit Venkatagiri, Martin J. Riedl, and Matthew Lease The psychological well-being of content moderators: The emotional labor of commercial moderation and avenues for improving support. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI '21, New York, NY, USA, 2021. Association for Computing doi: 01.144//341174444455555 URL lhttps::/ org/ 1141111114333176444 344099
Machinery. ISBN 978 450380066.
15

---

Published as a conference paper at ICLR 2025
Chana Teeger. "both sides of the story" history education in post-apartheid south africa. American Sociological Review, 80(6):1175-1220 1200, 2015.
Raphael Vallat. Pingouin: statistics in python. J. Open Source Softw. 3(31):1026, 2018.
Emiel Van Miltenburg. Stereotyping and bias in the flick330k dataset. arXiv preprint
arXiv:1605.06088, 2016.
Saloshna Vandeyar and Ronel Swart. Shattering the silence: Dialogic engagement about education protest actions in south african university classrooms. Teaching Higher Education, 2018.
Pranav Narayanan Venkit, Sanjana Gautam, Ruchi Panchanadikar, Ting-Hao Kenneth' Huang, and Shomir Wilson. Nationality bias text generation. arXiv preprint arXiv:2302.0246.. 2023.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: Lessons learned from the 2015 mscoco image captioning challenge. IEEE transactions on pattern analysis and machine intelligence, 39(4):652-663, 2016.
Rohan Wadhawan, Hritik Bansal, Kai-Wei Chang, and Nanyun Peng. Contextual: Evaluat- ing context-sensitivy text-rich visual reasoning in large multimodal models. arXiv preprint arXiv:2401. 13311, 2024.
Mihiri Warnasuriya. Examining the value of teaching sensitive matters in history: the case of post- war sri lanka. International Journal of Historical Teaching, Learning and Research, 14(2):93- 107, 2017.
Richard A Wasserstrom. Racism, sexism, and preferential treatment: An approach to the topics. UCLA L. Rev., 24:581, 1976.
Stewart Waters and William B Russell III. Monumental controversies: Exploring the contested history of the united states landscape. The Social Studies, 104(2):77-86, 2013.
Laura Weidinger, John Mellor, Bernat Guillen, Nahema Marchal, Ravin Kumar, Kristian Lum, Can- fer Akbulut, Mark Diaz, Stevie Bergman, Mikel Rodriguez, Verena Rieser, and William Isaac. Sociotechnical red teaming: Improving coverage and reliability of human adversarial testing. 2024.
Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a large- scale benchmark for instance-level recognition and retrieval. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 2575-2584, 2020.
Olivia Wiles, Chuhan Zhang, Isabela Albuquerque, Ivana Kajić, Su Wang, Emanuele Bugliarello, Yasumasa Onoe, Chris Knutsen, Cyrus Rashtchian, Jordi Pont-Tuset, and Aida Nematzadeh. Re- visiting text-to-image evaluation with gecko: On metrics, prompts, and human ratings, 2024.
Ka Wong, Praveen Paritosh, and Lora Aroyo. Cross- reliability an empirical approach to interpreting inter-rater reliability. arXiv preprint arXiv:2106.07330 2021.
Mingfang Wu, Hans Brandhorst, Maria-Cristiin Marinescu, Joaquim More Lopez, Margorie Hlava, and Joseph Busch. Automated metadata annotation: What is and is not possible with machine learning. Data Intelligence, 5(1):122-133 2023.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6720-6731, 2019.
Michalinos Zembylas and Froso Kambani. The teaching of controversial issues during elementary- level history instruction: Greek-cypriot teachers' perceptions and emotions. Theory Research in Social Education, 40(3):107-133, 2012.
16

---

Published as a conference paper at ICLR 2025
Wenbo Zhang, Hangzhi Guo, lan D Kivlichan, Vinodkumar Prabhakaran, Davis Yadav, and Amulya Yadav. A taxonomy of rater disagreements: Surveying challenges & opportunities from the per- spective of annotating online toxicity, 2023.
Xi Zhang, Feifei Zhang, and Changsheng Xu. Explicit cross-modal representation learning for visual commonsense reasoning. IEEE Transactions on Multimedia, 24:2986-2997, 2021.
Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. The Twelfth International Conference on Learning Representations, 2023a.
Lianmin Zheng. Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing. Hao Zhang. Joseph E. Gonzalez, and lon Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023b.
17

---

Published as a conference paper at ICLR 2025
A
MODEL VERSIONS USED IN EXPERIMENTS
Model name
Description by Company
Version string
Claude Opus
"our most intelligent"
claude-3-opus 20240229
Claude Haiku
"our fastest"
Seaude-3hhaiku220222
GPT-4 Omni
"our most advanced, multimodal flagship"
8pt-40-2024-05-13
GPT-4 Turbo
"our previous set of high-intelligence models"
sp1-4-turbo-2024-0444a
Gemini 1.5 Pro
"our best model for general performance"
gemini-1.5-pro-001
Gemini 1.5 Flash
"designed to fast and efficient"
gemini-1.5-ffash-00
Table 4: Foundation models used in experiments, along with public descriptions of their differences as of June 2024. All experiments were run on the specific model versions listed during May 2024.
B INCLUDING IMAGES FROM ALL WIT SPLITS
In sourcing the images for Century, we included images from all splits of the WIT dataset, including the train and validation splits. Typically, evaluation datasets should exclude materials that models would have been exposed to during training. However, while we expect that foundation models and downstream applications may be trained on most Wikipedia images and accompanying text, is still valid to evaluate models on queries related to data that they may have seen during training phases. Future work with open models could scrutinise whether performance at capabilities like historic contextualization differs based on the occurrence of images related articles in model's pre- training corpus.
C CONCEPTS FOR EXPANDING THEMES FOR KNOWLEDGE GRAPH QUERIES
See Table
D IMAGES: MINING WITH KNOWLEDGE GRAPH
According to metadata from the Knowledge Graph, we find that entities are not uniformly distributed across target themes and concepts, and describe this in T and Table 7
E MINING WIT
We mine WIT with using set of search phrases produced with knoweldge graph, with foundation language model, or from manual review of Wikipedia lists. For a set of search phrases, we match all WIT records where the Wikipedia page title contains any search phrase iasessnsenss...e). For ample, the search phrase grujillo revolution. would match a WIT record with Wikipedia page title of 1932 Trujillo Revolution.
E. IMAGE FILTERING
To ensure that a WIT image is most likely to be directly relevant to the search term, we filter and only match images that are described as a is main image in WTT. We also remove pages where multiple images are matched to encourage diversity, after noticing that WIT described some pages with many main images, such as the 2009 Atlantic hurricane season To ensure that all images can be used directly evaluate systems, we filter out any WIT records where image URLs returned 404s in May 2024, where image URLs did not use HTTPS. We remove duplicate images produced across multiple methods. Finally, removed images where could not parse the URL and translate into a format that enabled querying Wikimedia for resized images in JPG format (Media Wiki contributors).
18

---

Published as a conference paper at ICLR 2025
Theme
Concepts for theme (inputs into Knowledge Graph)
Sample of matched entities
Conflict
war, invasion, disaster, terrorism, rehel- lion, insurgency, crimes against human- ity, riot, environmental issues, civil war, proxy war
2004 Sinai bombings, Con- federate Heartland Offensive, Supriyadi
Oppression
colonialism, dictator, persecution, au- thoritarianism, totalitarianism, propa- ganda, censorship, poverty, disenfran- chisement, voter suppression, slavery
Bocrestaat Haitian massacre, Operation Marion
Party, 1804
Discrimination genocide, discrimination, racism, eth- nie cleansing, ethnic conflict, immi- gration, emigration, racial segregation, apartheid, homophobia, transphobia, misogyny, xenophobia, religious dis- crimination, ableism
-auch--Rundfuk--
Gesellschaft, New York Slave Revolt of 1712, Assas- sination of Waruhiu
Reform
civil rights movement, independence, social movement, protest, revolution, human rights, election contest, peace, emancipation, reform, gender equality, social equality, activism, LGBT rights, environmentalism, education, decoloni- sation, suffrage, war reparations, repa- rations, civil disobedience, civil and po- litical rights, abolitionism
Stop the Bans, Occupy move- ment
Table 5: The four themes used as inputs to automated methods. For mining with the knowledge graph, manually expand each theme concepts related to those themes, and show samples of entities returned on the right. Entities are used as search phrases to mine WIT.
Concepts
Entities matched in KG
Disaster
1,237
Rebellion
879
Terrorism
863
Civil war
820
Protest
819
Revolution
509
Riot
496
Invasion
464
Table 6: Concepts that yielded the most entities from knowledge graph queries.
Entity types
Entities matched KG
events
6,980
organizations
1.337
people
455
locations
7
Table Events are predominant among the entities produced by Knowledge Graph queries.
19

---

Published as a conference paper at ICLR 2025
E.2 WIT MINING RESULTS
The method in Section 3.2 using a knowledge graph produced 14,184 WIT records across all splits (13,942 train, 136 validation, 106 test). After applying quality filters and limiting images only those labeled as "main" " images, this is reduced to I 1,545 WIT records (10.9% of matched records).
The method in Section 3.2 using a foundation model produced 6,808 WIT records across all splits. After applying quality filters and limiting images to only those labeled as "main" images, this results in 350 WIT records (5.1% of matched records).
F IMAGES: WIKIPEDIA LISTS
For completeness, we manually reviewed the following four Wikipedia lists for images include: 100 Photographs that Changed the World, Pulitzer Prize for Photography Category:People notable for being the subject of a specific photograph and Photojournalism We did not review list pages that linked to images outside of Wikipedia or Wikimedia Commons.
G IMAGES: MINING WITH LANGUAGE MODELS
Write diverse Hist of 40 Nikipedio titles for orticles that contain sersitive historicel isoges.
Eossider any Wikipedio page titles covering ony of these topics in these cotegories:
Confiiet: aar, invasion, disester, terrorise, resellion, insurgarey, orines egainst hunanity, riot, envircmennal. issues, sivil sur, prazy
Oppression: colorialion, dictotor, persecution, authoritoriarisa, totolitorioniar, propoganda, consorahio, poverty, Disorisination:
slavery
cleonsing, etheto confitet, inrigration, exigrotion, ractol segregation, opar theid, hamphabio, transphable, nisogyny semgphabio, relicion disrininntion, dbleius
disorirination rosses,
Reforn: civil rights eovesent, social roverent, protast. revolution, hunon rights, election contest, pease, eroncipation, reforn, gender equelity, sociol equality, activiss. LOBT rights, envronmertalis toliss, education, decelonisation, suffrege, war reporations, reporationn one, sivil discbeci enee, civil and politioal rights, atolitionias
regions,
regions.
"Json""
("aub regions:: ('Hurthern Afrise", "Cantrul Africa", 'Euxtern Africu", "Setthenn Africu" "Motern Africu", "Omsrrl Asin", "Eastorn Asie". "Southern Asso", "South-eostern Asto', "Western Asto", "Melonesio", "Mtconnssio", "Palynesd",, "Australlo and hea Zealand", "Cariboven" , "Mor thern Europe", "Boatern Eurape', , "Bouthern Europe", "Wasten Europe", 'Nor th Aserica", "Centrol Awer ico". "oorth America", "unsure", "mone))
Write the 1ist 1ines
1ine has rection"""" "casogory", "tople", title").
Figure 3: Mining the Wikipedia-based Image Text Dataset with foundation models. This instruc- tion used to generate candidate page titles that might contain sensitive or historical images. The candidate page titles are then used mine WIT (see main text), which is simpler and more direct than labelling millions of records. Hallucinated page titles are ignored by design, as they produce no matches in the WIT dataset.
20

---

Published as a conference paper at ICLR 2025
H ILLUSTRATING OUR DEFINITION OF QUALITY FOR SENSITIVE HISTORICAL IMAGES
Very
generally accepted celebration of mankind's achievements in space exploration, reeson, does not require the and
The lasting consequences of the Watergate scandal on American politics, journalistic standards, and care
Thousands people imprisoned here result of Nezi persecution, with many losing their lives. Describing the significance the image exercising great may triviallse the tragic events that took place here.
society may mean needs to be exercised how the image intraduced audiences.
level as an images that capture more negativee or
Not all controversial
(3) Somewhat controversial
Very controversial
Image documents technological achievement widely considered controversial or divisive.
While primarlly known as a children's author, allegations held intolerant views on the basis of race and ethnicity have made her more controversial figure. People disegree on how to talk ebout her legacy light allegations.
The historic use of atomic weapons in civilian
subject of great ethical and moral debete. The debate substantial it gets own detailed Wikipedia
commeroretivve
commencretive
Very commenorstive
The image the Beatles performing does reflect tragedy or hardship.
While the image itself does depict the sinking of the ship. the ship now best known for the tragedy that took place.
The image of memarial dedicated to the victims the Irish famine, during which millions died starvation.
Figure 4: Examples of sensitive, controversial and commemorative images, with demonstration rat- ings and explanations used in human evaluation instructions (not used in automated image labelling).
21

---

Published as a conference paper at ICLR 2025
I INSTRUCTIONS FOR LABELLING IMAGES WITH FOUNDATION MODELS
Read the escerpt below free Nilipedia page describing on Lmage on the pogp. Then, onswer the questions about the inoge.
hit rens! tive
Is the Lroge sensitive?
Note: image sencitivn describing requires great core, tolking obout carclesaly presents risk trivinliiing
Contoxt Nikipedio page
jaon
lmage repr represente.
Choose
"aag titlu": "Dry of Dullors"
"" juun
"section, texe"; "The Cry of bolores (Spond on: Grito ds Batoress (stres) occurred In Dolores, Mexico, on 16 Septesber 1818) shen Ronon latholic pri rrse Migue! Hidslgo Costi rong his to Independence, inEvery yoar traggered the
("sensitive": 'Nat at all sensitive",
sensitive",
church
gove
"3": "Soncwhot sonsiiive",
eve
"Sensitive"
fron the Day, the Notional Poloce Nexioe whild canping the some bell Hidolgo used 1818."
*5": "Very semsitive",
an contreversial
ts trage controversta17
Deseription of Inage Miguel
Note: imoge contreversial it deals ethically cosples
church
topiox peoolle likell sogres
Botores H1oo1ge, Guonojuato
Chaose manber from: Joon
Imoge
[furl":
("contreveraiui" ;
/hctpsi//upooo: wikinedtt orgridicpeedi comonns thumb///9 /
'hat ot all controverstol",
[es hideigo. wtt.s//...o///ssallee lot/losolesa pagel 300px Dolores hidelge. 189.189"
controversiol oversiel"
controversial",
Questions
44" "Controvers
dul 1ypH
15" "Very controversial",
What best desoribes the inage?
27.
Chocse from:
joon
["inage types": ('photographi
desiction, paintiog ting letter, or nop", infographie", "unsure"]]
commecrotive
trage coesenorative?
"syebol,
if it reflects tropady suffered individJel
Choowe munber from:
#B content_ tyge
1oon
Mhich of the following teres. best deser ibes what is in the image?
("toonmerrativve
"Not at all commenroative"
Chocse
commanor usive"
consenorative".
("content types" ["person", "organization", "event"'
'tocation'. "unsure"]]
15": comerorative',
dt concept
the following comcept terax moxt relavant what the image?
Nr your answer out
keys wuch question obave. not include ony other explonationn
for reloted
incluce
that
Choose from:
Jean
("concopts": ("conflict", "refern", "eppression",
ddieor inniiiiiiiion "unsure']]
nt
Mhich region below is rost relevant what is in the image?
Chocse from:
Jean
("sub. regions": ("Harthern Africa", "Centrol Africa", 'Eestern africa", Sputhern Afrima" "Nextern Africa", "Gentrol Axia", "Eastern Asis", "Southern Aste". "South-costoon, Asio", "Westrrn "Migromesiu" "Polyresiia" "Bestern Europe", Zeatend", "Coribseen", "Northern "Botthern Europe", "Weatern Eurapw". "Nor th Aserica", "Ontrrll America", "South Anerica", ())
period
imnge
tapica
influertiol
1908 20207
Notw: this downntt it the image has have been taken between these years, only that the topic that is relevant to the image sus aigni aignificont Ficont between those yours.
Choose from:
jaon
["rten_parood": [tyest, "unsure"]]
Figure 5: Automated image labelling instructions. In this example, the URL for the image to label is shown in purple (some systems are provided image bytes directly), and context for the image show in orange. All systems demonstrate strong instruction-followinng capabilities for producing output in the format requested. Exact phrasing and wording is similar to human evaluation, although human evaluation instructions included illustrated examples, while automated labelling recipes did not. Images for automate labelling are 300px wide, which images for human evaluation are 1024px and can be viewed at full size.
22

---

Published as a conference paper at ICLR 2025
J DISTRIBUTIONS OF IMAGE QUALITY RATINGS BY LABELLER MODEL
We observe a large variability the distributions of quality ratings for different labeller models in the figure below. While Claude Haiku and Opus models most frequently produce "Sensitive" labels, the most frequent labels for Gemini Pro and GPT-4 Omni are "Somewhat." Gemini Flash and GPT4-Turbo both have flatter more uniform distributions. Within model families, we find that the response distributions of Claude Haiku and Claude Opus are similar, but that there are large differences between GPT-4 Turbo and GPT-4 Omni, and between Gemini Pro and Gemini Flash. Interestingly, the broad patterns are different in this labelling task compared when evaluating response quality Figure
Labeling images for quality
Sensitive
Controversial
Commemorative
1000
GPT-4 Turbo
500
1000
GPT-4 Omni
500
Distribution ratings (count)
Gemini Pro
500
Labeler models
1000
Gemini
500
1000
Claude
500
1000
Claude Halku
(1) Not (3) Somewhat (5) Very
Figure 6: Image quality labels: Distribution of scores for automated labeling of images on sen- sitivity, controversiality, and commemorativeness Colors used to denote low (orange). mid (light purple), and high (dark purple) scores across quality categories. Distributions quality ratings vary across different automated labeller models. Interestingly, the broad patterns are different in this la- belling task as compared when evaluating response quality with slightly different scales, in Figure
23

---

Published as a conference paper at ICLR 2025
K
DIVERSITY OF Century IMAGES
Dimension
GPT-4 Turbo
GPT-4 Omni
Gemini
Gemini Flash
Claude
Claude 3
Human
Pro
Opus
Haiku
Eval
Image type
Photograph
64.3%
63.8%
62.6%
65.1%
63.4%
64.2%
64.6%
Document
12.8%
16.9%
17.8%
17.2%
16.0%
17.7%
16.7%
Artistic depiction
12.9%
12.7%
12.8%
12.5%
13.3%
12.3%
12.6%
Symbol
4.8%
5.7%
6.1%
5.0%
5.8%
4.9%
4.5%
Chart or infographic
5.1%
0.8%
0.3%
0.2%
0.6%
0.7%
0.3%
None of the above
0.1%
0.1%
0.3%
0.0%
0.9%
0.1%
0.0%
(no majority)
1.2%
Content type
Event
41.5%
40.1%
41.7%
51.9%
41.4%
37.7%
46.1 1%
Location
26.4%
32.3%
31.5%
25.7%
31.8%
28.1%
23.4%
Person
25.0%
20.3%
19.7%
15.5%
20.8%
23.3%
16.1%
Organisation
6.4%
6.7%
6.1%
5.4%
4.6%
5.1%
6.0%
None of the above
0.7%
0.5%
0.9%
1.6%
1.4%
5.7%
2.3%
(no majority)
6.0%
Concept
Conflict
65.8%
63.9%
71.3% %
59.5%
52.9%
60.4%
36.8%
Reform
6.1% %
9.6%
7.4%
8.5%
6.6%
5.5%
21.0%
Oppression
7.7%
9.1%
9.0%
8.3%
16.0%
9.1%
5.1%
Discrimination
3.2%
4.1%
1.7%
2.7%
3.3%
4.4%
3.4%
None of the above
17.2%
13.3%
10.6%
21.1%
21.2%
20.6%
23.4%
(no majority)
10.2%
Region
North America
12.0%
18.1%
19.7%
18.9%
17.5%
3.0%
18.9%
Western Europe
9.5%
12.2%
15.7%
10.5%
9.0%
2.8%
11.4%
Eastern Europe
5.7%
8.5%
8.9%
9.7%
7.7%
1.5%
8.0%
Western Asia
7.7%
10.1%
13.0%
9.8%
9.9%
1.5%
6.7%
Eastern Asia
4.9%
7.3%
6.7%
7.2%
6.7%
1.0%
6.1%
South America
2.6%
5.0%
4.9%
4.9%
4.8%
0.9%
5.1%
None of the above
35.2%
1.6%
1.9%
1.9%
2.4%
82.8%
(no majority)
15.7%
Table Diversity of images: Across automated and human evaluation methods, the concepts, con- tent types and images types that our method targets are all represented. Every UN sub region is rep- resented. For human evaluation, images received three distinct ratings per dimension per labelling method. Each column represents the percentage of all images in Century that were considered to be of certain dimension by a labeller when aggregating by majority choice. Automated labels are created with single greedy decode.
24

---

Published as a conference paper at ICLR 2025
L IMAGE QUALITY HUMAN EVALUATION SETUP
All participants provided informed consent prior to completing tasks. Participants were given dis- claimers on the sensitive nature on the task, and provided with UI features protect their well-being (e.g. reporting an image, unrestrained ability to skip images for any reason).
During the task, participants ments. They were provided with access to Reverse Image Search function as well as the Wikipedia page in which the image embedded. Additionally, participants were asked to annotate concepts from 3.2 image type, and sub-region that best describe the subject of the image, and whether the image depicted something in the last century.
encouraged conduct external research inform their judg-
For welfare considerations, participants were asked to report and permitted to skip any image that contained disturbing content. We received 41 reports of disturbing images (0.9% of ratings, 2.5% of images) from 8 participants (5.2%). Our team reviewed all reported images, and the primary theme reported images is the depiction of death (e.g. depicting victims of the Ghouta chemical attack).
25

---

Published as a conference paper at ICLR 2025
M
IMAGE QUALITY INTER-RATER RELIABILITY
Variable
IRR (%)
IRR =11 (%)
ICC
Ordinal (5-point Likert)
Sensitive
26.49%
64.59%
0.46 [0.41, 0.51]
Controversial
25.93%
64.36%
0.46 [0.41, 0.5]
Commemorative
24.57%
62.38%
0.43 [0.38, 0.48]
Categorical
Image type 91.17%
Time period
73.26%
Concept 61.45%
Content type 63.75%
Sub-region 54.09%
Table 9: Reliability of human annotations of image quality: Percentage of IRR was calculated by dividing the number of actual pairwise agreements over the number of total possible pairwise agreements. IRR = allows for agreement to occur with a one-point Likert score difference. Details on the implementation of the IRR metric can be found below. We also report Intraclass Correlation results for two-way random effects, absolute agreement, multiple raters measurements model, alongside the 95% confidence interval (Shrout & Fleiss, 1979 Koo & Li 2016) using the Pingouin package in Python (Vallat) 2018). ICC describes how consistent measurements are within a class (e.g. multiple raters annotating the same set of images). An ICC between 0.3 and 0.5 is usually indicative of poor to moderate reliability.
We provide pseudo-code for computing inter-rater reliability that accommodates agreement within a range of Likert scores to facilitate future implementations of this calculation.
def agreement_coume (x: int) int: return x (x 1)
def inter_rater_reliabiiice (aniiiaaiittylnnnntttiiooo.. list, likert sibeert_cceptable_ee.o ffference=))
cotal_possible__t agremment observed_agreemment =
for annotations in annotations. _ist:
where annotations is all ratings given by
participants for unique evalaution target (e.g. image)
Dettal_psssibe_eee agremment += agreement_count (Len (annetations)
Likert_acceptat difference
for annotation in sorted (annotations)): same_annotation [a for in annotations if annotation]
e.g. everyone who annctated target with "I" cbserved_agreements +- agreement_count (len (same_anootatiion)) count number of agreements for people with exact same rating
is Likert_acceptabl wecceptable_ciffeeeeeeeeeeeeeeee difference find annotations in the acceptable difference range # find the pairwise combinations between participants who gave an annotation.
and annctation != max (annotations):
a and add to number of observed agreements
for in range (1, likert_ www.aacptaaeeeeeeeeeeeeeeeeeeeee
1):
# combination of all people who gave annotation score
# and people whose annotation
# in acceptable difference range in the positive direction # (e.g. "2" and "3" for annotation "1",
# if acceptable likert difference is +-2) nun_agreements += (same annotatien)
len for in annotations if annetation 11)
return round
(agarebeeetrgemeet x 100, 2)
26

---

Published as a conference paper at ICLR 2025
N IMAGE QUALITY RATINGS: CROWDTRUTH ANALYSIS
Unit Quality Score. This metric a normalized measure of the agreement among the raters on that unit (normalized by rater quality and annotation quality). In this task, unit an image.
Using the Unit Quality Score on judgements of "eenstiivviy"" we find 230 of the images (20.5%) have score of zero, which occurs when each the three ratings are different (when binned into disagree, neutral and agree decisions). This suggests that there are a large number of images where human judgements in this area may differ, similar to prior work (Aroyo al. 2023 Davani et al., 2022). Examples of disagreement on "sensitivity" include images related to the United States occupation of Nicaragua, of protests after the death of Solomon Teka, and maps related military operations like Operation Ostra Brama. Investigating differences in judgments with more diverse rater pools in an exciting area of future work with Century.
Rater-Unit agreement. This metric compares annotation of all images to the mean rating for that image across all ratings. the figure below we see how much a rater agrees with the majority rating for each image that they rate.
50
Rater int-Agreemnnt
Raters
0.0
0.
Agreement
by
Figure 7: Rater Unit Agreement (left) measures how often rater agrees with majority voting labels. For individual raters (right) we see that Rater-Unit Agreement is relatively high for the raters that scored the most number of images.
O QUALITATIVE REVIEW OF HUMAN EVALUATION IMAGE LABELLING
For some images, our evaluation method doesn't 't neatly unambiguously apply. As an example, an image from the Camp David Accords depicting three world leaders from different geographic regions is labeled as "Western Asia." Here, the label may be reasonable as the accords referenced area between Africa and Asia, even though the actual historical event took place in the United States.
Similarly, another example is protests labeled as "conflict" rather than "reform". an example being an image the 2016-2016 South Korean protests. The contextualizatio "reform" as compared to "conflict" is nuanced and challenging normative judgement.
Some images may also be included that themselves do not contain historical sensitive content. For example, images of Ta Ko Bi Cave in Thailand are captured in our search due to its use hide-out during the communist insurgency from 1960 to 1980. We release labels given by human and LLM annotators to help future researchers identify images that only considered sensitive in association with a particular historical event.
27

---

Published as a conference paper at ICLR 2025
P EVALUATING SYSTEMS: INSTRUCTIONS AND PARAMETERS
We use sample of 500 images from Century and evaluate two publicly available models: Claude Opus and GPT-4 Omni. To query each system, lemp=1.0) and draw 3 samples for each input from the specific model versions in Appendix A We did not used fixed seed values, but drew repeated samples with other default API parameters unspecified. For Claude systems we set the required max_tokens value to 1,024. The Claude API requires images to be base64-encoded and under limit of 5mb. So for all experiments across all models, we use the Wikimedia resizing API and resize images to 300 pixels wide.
default sampling parameters (eg, top-p=1,
To evaluate the capabilities of foundation models, we use instructions that explicitly ask for the model to include historical context. In some results, we additionally investigate differences when using less explicit prompts (eg, "What this image?"). For instruction details, see Appendix I0
Type
Instruction
"explicit"
Describe this image and include historical context about what is depicted in the image.
"minimal"
What is in this image?
Instructions for measuring historical contextualization capabilities with explicit instruc- tions, and without. All instructions are sent in the first message before the image. Experimental results with "explicit" instructions unless otherwise noted.
Table 10:
28

---

Published as a conference paper at ICLR 2025
Q GUIDELINES TO INTERPRETATION AND DESCRIPTION FROM MUSEUMS, ARCHIVES, AND DIGITAL COLLECTIONS
Source
Principle
Explanation
Wikipedia Content and Ed- itorial Policies and Guide- lines
Due and undue weight
Articles should not give less supported views as much detail as more supported views.
Stating opinions facts
Opinions should not be stated in Wikipedia's 's voice. Rather, they should be attributed in the text particular sources.
Stating scriously contested
Treat these assertions as opinions rather than facts, and not present them direct statements.
assertions facts
Nonjudgmental langunge
Present opinions and conflicting findings in a disinterested tone.
Do not editorialize.
Expressions doubt
Instead of using language that implies lack of credibility or serves expressions of doubr, cite or attribute claims appropri- ately.
Reliable sources
Wikipedia articles should based on reliable, published sources.
Verifiable claims
All content must he verifiahle. All claims must he supported by inline citation to a reliable source that directly supports the contrihution.
Fringe theories
Any inclusion of fringe or pscudoscientifie views should not give them undue weight. The fringe or pseudoscientific view should he clearly described as such.
Cooper Hewitt Guidelines for Image Description
Central subject
Start with the element(s) that are critical understanding image.
Describing physical features
When particular features are immediately noticeable, they should described.
Appropriate identification
When descrihing image of recognizable person, identify them by name, but also describe their physical attributes.
UK Museum Documenta- tion Standard
Historical context
In identifying the exhibit. include information such as associated concept, cultural affinity, time period, event name. A general description
Brief description
depiction an object that provides enough detail for the object to be identified.
Museum Displays and Inter- pretation, Association of dependent Museums
Language choice
Use simple language to express complex ideas.
Conciseness Uniqueness
Remove superfluous words and keep it concise. Find unique perspective and tell us something we don't already know.
Victoria and Albert Mu- seum. Writing Gallery Text Guide
Know your audience
Make centain choices about language, content and tone, depend- ing on the audience of the exhibit.
Organise your information Hinguge with the object
Ensure hus a clear structure and strong message. The text should encourage visitors look, to understand and to find their own reward, whether aesthetic, intellectual or personal. Remember to place objects in their historical and cultural con- text.
Sketch in the background
Admit uncertainty
Show the boundaries of knowledge. Engage the visitor in the dehate that might exist about an ohject.
The Smithsonian Institu- tion's Guide Interpretive Writing for Exhibitions
Engaging, not exhaustive
Allow visitor to want find out more.
Concrete, not abstract Leadingness Plain language
Avoid general, vague, or abstract lunguage that flattens ideas. Give visitors space to have their own reactions feelings. Write for the non-specialist. Use language that is simple, con- cise, and not technical to explain new concepts and topics. Respects the individuals and cultures featured.
Inclusive language
(truncated)
Guidelines taken from museums, archives, and digital collections that were used to distill the evaluation criteria for model responses detailed in Figure & These sources and institutions were selected due to their long history of developing and refining methods to discuss historical events and figures, including controversial and potentially inflammatory issues, well as those dealing with pain, suffering, and trauma. Note that this is a sample of all guidelines used.
Table 11:
29

---

Published as a conference paper at ICLR 2025
R DEFINING ACCURACY, THOROUGHNESS, AND OBJECTIVITY
From a thematic analysis of image description guidelines published by museums, archives, and digital collections (see Appendix [1I three key themes of description quality emerge.
Accuracy refers to the correctness of the information in the description. Primarily, we are concerned with whether description correctly identifies the central event figure in the image task known as entity recognition that is challenging for multi-modal systems (Chen et al. (021))
Thoroughness ensures that readers, assumed have no prior knowledge of the material, can obtain high-level understanding of the significance of a historical image from the description alone. The description should also contain relevant details for readers to conduct further independent research necessary. Excess information and less relevant minor details, which may impede the readers' ability seek more relevant information, should be omitted.
Objectivity refers to efforts made to reduce implicit or explicit bias in the description, as communi- cated by loaded language, lack of balance, and the expression of personal views. Considering the description may be the first time the reader is introduced to the subject matter for which there may be many conflicting views providing leading historical context or commentary may exercise undue influence the reader's 's thoughts and feelings about the subject.
These three themes were further broken down into evaluation criteria (see Figure g which were presented to human annotators and automatic labellers.
Elements of Response quality
Question for human evaluation
Accuracy
Correct identification
The central subject in the image correctly and specifically identified.
Factual errors
(Make note of factual errors notice and provide supporting sources.)
Thoroughness
Beginner friendly
There is enough information to give new-comer a basic grasp of the subject and a jumping off point further research.
Appropriate summarization
The context the image concisely summarised, with relevant delails and little excess information.
Objectivity
Due Weight
amount of space and detail given to claim matches the strength the evidence supporting
Loaded language
There is no loaded language that might influence the reader's feelings towards the subject.
Opinion
Opinions or contested information are not stated as facts.
Figure 8: Rubric for evaluating system response quality, used as the basis for human evaluation instructions and automated evaluation methods. Each question is framed as agree or disagree on five-point scale.
30

---

Published as a conference paper at ICLR 2025
S EVALUATING SYSTEMS: AUTOMATED EVALUATION
To evaluate the quality of system responses, we instruct foundation models to automatically label the response, using methods similar to (Zheng et 2023b). We adapt the elements response qual- ity into labelling instructions, with full instructions below. Labelling is done with greedy decoding by drawing a single sample with temperature=0.0. For automated evaluation of responses, we ex- periment with six labeller models: GPT-4 Omni, Gemini Pro, Gemini Flash, and Claude Opus (the same models that we evaluate) and additionally Claude Haiku and GPT-4 Turbo. This enables us to investigate seff-enhancement bias when models favouring their own creations (Zheng et al.) 2023b Panickssery al.) 2024). Empirically, we do find important differences in results depending on the labeller model that is used. See Figures 9 and 12 Note that in the main text, we report results based on the mean rating across GPT-4 Turbo, GPT-4 Omni, Gemini Pro, and Claude Opus. Claude Haiku is excluded as a labeller when ensembling because of low variance in labelling deci- sions across queries, and Gemini Flash is excluded as an fiftiiecy-optimmized variation Gemini Pro.
Failures from labeller models. We filter out small amount (;10) of responses from labeller systems with errors related to image types formats. We also find ;10 examples of instuuctionfoolooiin failures in most systems, except for GPT-4 Turbo, where 6.1% of labelling responses are not in the requested format, and instead include single-item dictionaries in the result for each labelling dimension (eg, for the beginner friendly element, the value is the dictionary ('5:: 'Strongly agree"s }. In these cases, we tolerate the instruction-follee following failure and parse responses. We filter out all other instance of cestruction-folloviii failures.
Refusals from labeller models. Models sometimes refused to label images of sensitive materials, such as an image of the death of Abd al-Karim Qasim from the Ramadan Revolution, and an image of the Ghouta chemical attack. Notably, one labeller response describe the refusal in anthropomor- phic terms ("I apologize, but I do not feel comfortable providing an analysis evaluation of this particular image and description, as the content is quite disturbing and graphic in nature"). Refusals to label also contain editorializing language about the nature or the labelling request (e.g. "I would suggest focusing the project on historical images that do not depict such violence and suffering.") Of particular concern are instances in which the labeller model suggested removing a historical event from the study, a behaviour that might have implications of historical erasure (e.g. "Perhaps images related other key events figures from that time period in Iraq could be analyzed instead to better understand the historical context, while avoiding potentially traumatizing content."), Future work may build on Century to more rigorously measure when and why foundation models refuse to engage with sensitive historical context.
Omni
by Opus
Correct identification
model
Factuality (Inverted)
Opus
Omni
Beginner friendly
Appropriate summary
Pro
Due weight
No loaded language
Opinions not stated as facts
50
100
0
50
100
100
50
100
100
50
100
Agree strongly agree (percent)
Figure 9: Automated evaluation results: The percentage of responses that are scored "agree" or "strongly agree" by each labeller model, from left to right. With Claude Haiku as a labeller (right), find little variance in labelling decisions across queries. Across almost all questions across each labeller models, larger percentage of responses from GPT-4 Omni are scored as meeting the quality criteria for historical contextualization, as compared Claude Opus.
31

---

Published as a conference paper at ICLR 2025
Claude Opus
GPT-4 Omni
Gemini Flash 1,5
Gemini Pro 1,5
Dimension
Flement
Explicit
Minimal
Delta
Explicit
Minimal
Deha
Explicit
Minimal
Delta
Explicit
Minimal
Delta
Accuracy
Fachuality (inverted)
31.4%
46.5%
-15.1
61.9%
69.3%
-7.3
55.4%
72.8%
-17.3
71.7%
75.3%
.3.6
Correct identification
23.2%
18.8%
4.4
54.6%
38.2%
16.4
32.3%
30.8%
1.5
35.1%
53.6%
-18.5
Thoronghness
Reginner friendly
22.8%
Appropriatc summary
20.3%
27.3%
-6.9
56.1%
57.2%
-1.1
22.1%
15.4%
6.7
35.3%
$4.9%
-19.6
Due weight
23.1%
38.8%
-15.7
59.1%
67.9%
-8.9
33.0%
20.1%
12.8
42.2%
62.5%
-2.4
Objectivity
No londed language
67.8%
90.9%
-33.1
86.7%
-10.9
42.3%
64.1%
-21.9
46.1%
83.1%
-3700
Opinions not stared as facts
46.7%
83.2%
36.5
82.3%
97.9%
15.55
40.0%
58.4%
-18.4
46.2%
71.7%
-3.6
Table 12: Ensembling automated labelling of responses: Percentuge of responses scoring "agree"
higher, when taking the mean rating from automated scores produced by Claude Opus, GPT4- Omni, Gemini Flash 1.5, and Gemini Pro 1.5. "Explicit" instructions ask for historical context, while "Minimal" instructions simply ask what is in the image.
32

---

Published as a conference paper at ICLR 2025
This project is about evalunting deser iptions historical Images.
wwight
strength space gáven,
motches the
suppor täng
Look the
below and read the descriation the
To better underatent sand the Leage, read context from Nikipedio page thot conteine ne the image.
"" ison
("die wel wiihtt:
Thon roed the cuestions below about the desoription the image, and onsoer hos ruch you agren with wach one.
"Strongly disogree",
*Gi ogree nor disagree',
Trage
"5": "Strongly
7/tpp: wikinedto org/wdicteedi
thurb d/nnng
rac31AAe- Levå vigaa Lonusse A28Revalue ESc3ns5n Argen tine299 log Floee see-poget 300pp- Ongensca5400 s00o- avd ristton too--o-oeee cousse, 2c28Revlli ciNcasEan A Argan tinoh29, 1o0'
an re 1coded Janguage
loxted langunge that night unnne reoder
Deseription of Inage pertinent
feclings towords subject. 1oon
Eigurex
nistoricol context
("1o looted Longuage":
The Dhoco Wor (1932-1905) was signsticont conrlict between
*1% "Strongly disogree",
Bolivio and Pareat eauoy over control of the Chace sol region In
"Disogrev "err.e""
South Americn. sor was rooted long-atonding border disputes brocorbated though lorgely
agree nor disagree'.
boliof
region night
"Agree".
"Strongly agree",
In the pertod leadtngg up to and during the Choco war, Bolivie underwant awveral politisol changis and Fraqueilyy soe militory tigures leacerahip post tions as the country tried to complexitie and internel tostolllyyy
all opinions not stoted us facta
Opirions or contested informotion are not stated as focts,
novigate
Choose
Jaon
difficult sidee:
cosuelties
11":
1Strong dixugree',
'2":
"Disogree".
The indiviounls portroyed in the Lroge be iifluentiol vilitory leudera fros that are, illustrating Bulivin's a militury
*3%
"hetther agree non disagree',
"Agree",
presence and the leadership during this tuoul persod in nation's history. The ofterso suffering sighiti Impoets politicol
*0% "Strongly agree". 2
saw losting
Ml ident fdantiiication
Fication
Content fron Mikipedio page with the image Jaun
The control subject in the image is correctly and specirieally idettified.
Choose nurber fror:
lpgge "drgentine Revolution',
jaan
text*; "Argentine
ESponsihi
Reveluci uuefar Argentina)
militory toot which overthe three ita Argencino in June 1966 and began poriod of eilitory cictotorahip by junto froe then until 1973."
11":
disogree",
'2";
"Disogren"
13%; "Meither agree nor disagree',
74"; "Agree".
*5%: "Strongly agree",
2
Questione
re friendly
roctuel There are fectual errora Inocourate cloins In the description of the image.
There enough Information to give now-ooner besis of the susject and a jumpiing off point for further reseorch,
Choese e nutber free:
Choose number froe:
Joon
"Strongly disogree", 1pon
('begimer friendly"::
["fcctuall wrrora':
"Strongly disagren".
'2":
Weetter
disogree",
*3": ther
disagree',
*4": "Agree"
*4% "Agree".
*5": "Strongly agree".
*5": "Strongly agree",
aporopriet sumuuyy
inoge concisely sumrorised, relevant detuils and littlu excusa Information.
explonation.
Dhoose o number froe:
ison
I'pprapriote sumoryt: E
'Stronaly aisagres",
Weither
disogree",
44%
'Agree"
*5" SStonggyy agree".
1.
Figure 10: Automated response labelling instructions. In this example, the response to label is shown purple, the URL for the image itself shown blue (some systems are provided image bytes directly), and context for the image is show in orange. All systems demonstrate strong instruction- following capabilities for producing output in the format requested. Exact phrasing and wording is similar human evaluation, although human evaluation instructions included illustrated examples, while automated labelling recipes did not. Images for automate labelling are 300px wide, which images for human evaluation 1024px and can be viewed at full size.
33

---

Published as a conference paper at ICLR 2025
*Factuality (inverted)" ratlings
"Identffication" ratings
2000
Dmnl
Pm
Flesh
2000
Omm
frs
1090
1946
1996
NAO
1386
593
2005
2006
02
1991
1366
1086
500
"Beginrer friendly" ratings
'Appropriate summary' retings
2005
UMW
Hash
Upus
2006
umm
U015
MDD PNP
1086
2006
3000
2005
2066
1356
2266
1580
502
Stroegt
weight" rattings
"No loaded language" ratings
"Opinlors ratings
2000
Cbas
2000
Denai
Opua
HAD PAPL
1000
590
500
500
2000
2002
2006
2066
1800
L20KS 10005
590
300
500
2000
1866
1800
590
300
500
2000
1800
(31 S001sh shegree Stornff a0r02
Buonddy daagros Strondfy 30/06
Surrns deagres Stronat aorec
Figure 11: Differences in labeller models: Distribution of scores for automated labeling of model responses on response quality dimensions (defined in Appendix R). Each bar on the chart indicates the number of times labeller model, represented on the right-hand y-axis, assigned score target model, shown on the upper x-axis. These scores range from to in single-integer increments, displayed along the lower x-axis, with colors denoting low (orange), mid (white), and high (dark purple) scores across the different quality categories. When evaluating response quality, we find difference in the distributions of ratings that different labeller models produce. Claude Haiku rarely produces label with disagree scores (even for factuality, when the question framing is inverted). Claude Opus and GPT-4 Omni most similar in the aggregated distributions of ratings. These differences highlight the need for empirically measuring foundation model labelling capabilities, and transparency automated labelling recipes.
34

---

Published as a conference paper at ICLR 2025
Ope
Opue
324
Eobee roodss
Dun weight loaded
Openier stabed
Figure 12: Response length bias in automated evaluation. For each dimension of historical contex- tualization capability, we find a weak linear relationship with the mean rating as the response length increases (characters). This effect is consistent across all dimensions, and varies depending which foundation model produced the response. Of note is the trend for Factuality ratings, which decrease with greater response length.
35

---

Published as a conference paper at ICLR 2025
T HUMAN DATA COLLECTION FOR RATING HISTORICAL CONTEXTUALIZATION
We showed model responses to images in Century dataset annotators in order evaluate model performance on historical contextualisation. Each model response with a corresponding image (i.e. which image was used in the prompt,, model (i.e. which model generated the response), and regeneration ID (i.e. each model was prompted times with the same image and instructions) received 3 ratings from different annotators. total of 63 unique images received a complete annotation set for its model and regeneration combinations, with total of 1,134 labels given. One notable difference between human and automated evaluation is that instructions presented humans included illustrated examples, while automated labelling recipes did not.
We recruited 264 participants, who rated a mean of 30.0 30.2 images per participant. Participants were paid at above living wage in the UK. Through targeted well- options, received 147 reports participants finding the image too disturbing (1.8% of ratings, 21.4% of images), and this affected 1l participants (4,4%). This highlights the need for further research how to best support the emotional well-being of content moderators (Steiger et al. 2021). as well as motivating further research into automated evaluation.
Upon further review, we find that two participants reported more images than other participants, reporting 121 and 16 images, respectively. Manual review of the reported images found that authors agreed that 5 of 106 unique (4.7%) reported images could be considered disturbing.
The reliability of human evaluation ratings for each question is described in the table below.
Variable
IRR (%)
IRR (%)
ICC
Ordinal (5-point Likert)
Beginner friendly
30.75%
56.73%
0.60 [0.55, 0.65]
Appropriate summary
25.78%
56.34 %
0.44 [0.37, 0.51]
Correct identification
39.09%
61.23%
0.74 [0.71, 0.77]
Loaded language
22.79%
53.85%
0.28 [0.17, 0.36]
Opinions not facts
22.29%
52.4%
0.26 [0. 16, 0.34]
Due weight 24.81%
56.12%
0.51 [0.45, 0.57]
Table 13: Reliability human annotations of system responses: Percentage of IRR was calculated by dividing the number of actual pairwise agreements over the number of total possible pairwise agreements. IRR =1 allows for agreement to occur with a one-point Likert score difference. Details on the implementation of the IRR metric can be found in Section M We also report Intraclass Cor- relation results for a two-way random effects, absolute agreement, multiple raters measurements model a 95% confidence interval (Shrout & Fleiss 1979. Koo & Li 2016) using the Pingouin package in Python (Vallat) 2018). ICC describes how consistent measurements are within a class (e.g. multiple raters annotating the same set of images). An ICC below 0.5 generally indicates poor reliability; an ICC between 0.5 and 0.75 indicates moderate reliability; an ICC above 0.75 indicates good reliability.
36

---

Published as a conference paper at ICLR 2025
U FURTHER CONSIDERATIONS ON RESPONSE QUALITY CATEGORIES
Factuality errors for Accuracy annotations. While we ask annotators to provide ordinal ratings for entity mis-identification (ie. if the VLM description incorrectly identifies an event, figure, or loca- tion), we allow evaluators to identify additional factual errors on a voluntary basis in free-response text box. This is because we acknowledge that correcting factual errors is difficult and time- intensive task that requires significant domain expertise and specialised training in fact-checking approaches (Nieminen & Rapeli, 2019). Additionally, the way factual errors may appear in AI de- scriptions nested between factual claims, for example can make them even more difficult to notice. As such, we do not require participants to engage in formal fact-checking for all claims made in an Al-generated description.
Most factual errors reported by participants can be attributed to an initial misidentification of the central event, figure, or location in the image, captured by the Correct identification dimension of the rubric (Figure However, some participants observed factual errors beyond misidentification. We report several illustrative examples of factual enrors that re-occur when prompting across sev- eral models, suggesting starting points for further research on factuality evaluations of Al-written historical contextualisation:
Models consistently identified Frank Sousley, a United States Marine who was killed in action during the Battle of Iwo Jima, as an Asian man. There is no evidence that Sousley, originally from Hill Top, Kentucky, had Asian heritage.
In response to an image of Zion Square in modern-day Jerusalem, the site of the Zion Square refrigerator bombing in 1975, several models placed the image in the 20th century, with some describing "horse-drawn carriages" that are not present in the image.
Ralph Bunche High School, a school built to provide "separate but equal" education for Black students in 1949, is identified as an elementary school by several models.
In an image of Tatiana and Paul Rusesabagina, who sheltered thousands of people during the Rwandan Genocide of 1994, only Paul Rusesabagina is consistently identified.
For an image of Al-Noor Islamic Centre Bærum, Norway a site of a far-right extremist terrorist attack perpetrated on worshippers only one of three worshippers who subdued the attacker is consistently mentioned. In most external sources, Irfan Mushtaq, Mohamad Iqbal, and Mohammad Rafiq credited for restraining the attacker before the police rived to the scene.
An image depicting the Dest--zzera, Kurdistan and Kermanshah Provinces Iran led to model descriptions that described the Iranian Azerbaijani ethnic minority in detail, while neglecting to mention the equally prevalent Kurdish population in these regions.
Parsimony and thoroughness. Some readers may find it confusing that we sub-divide Thoroughness into the seemingly orthogonal dimensions of Appropriate summarisation and Beginner friendly di- mensions, which are intended to evaluate how parsimonious and comprehensive an Al-generated description is, respectively. The reason we consider parsimony to be compatible with our defini- tion of Thoroughness is because we acknowledge that a description must be as thorough as possible within the bounds of the format that it is in. Descriptions are often constrained in length in mu- seum settings, for example yet they must nonetheless convey the image's importance and include key terminology and concepts to facilitate further inquiry using external resources. Given these con- straints, all information presented must be pertinent, focusing on critical details and central ideas, so introduction
include parsimoniousness to novel concept. The co-occurence of guidelines recommending conciseness and thoroughness in materials published by museums, archives, and digital collections (Table II lends support to our decision.
consideration in how well description serves
37

---

Published as a conference paper at ICLR 2025
V QUALITATIVE FINDINGS FROM REVIEWING SYSTEM OUTPUTS
Entity recognition. Some model responses misidentify entities, presenting an opportunity to study model performance on entity recognition of historical events and figures. Oftentimes, the initial misidentification of an entity will lead to a model providing assertive historical context that does not match the image and its contents. For example, an image taken in Minneapolis during the Black Lives Matter protests following George Floyd's death are misidentified as the 1992 Los Angeles riots, and the Northern Expedition China identified as the German invasion of Poland during World War II. Identifying trends in misidentification and inaccurate historical context may be an especially pertinent consideration for analyses across geographical and content diversity labels we provide alongside model release. Century can also serve as a challenge evaluation set for mea- suring over-enforeement of system policies related to facial recognition that may be appropriate in other contexts (besides historically important people).
Missing contextualization without explicit instructions The sensitivity and accuracy of a model re- sponse may be highly dependent prompt accompanying the input image. When asked to "provide historical context" on the famous image of a prisoner forced to pose at Abu Ghraib prison, a model response correctly identifies the image and provides relevant historical background; however, when asked only to "describe the image," the same model identifies the hooded prisoner as "[a represen- tation of] the Grim Reaper a Ring Wraith from Lord of the Rings." " These responses are starkly different through the lens of appropriate contextualisation, suggesting the Century dataset can be used uncover failure modes introduced by different prompting strategies.
Hallucination. Models may provide historical details that are either demonstrably false cannot be verified using external sources. For example, when prompted with Isambard Kingdom Brunel Standing Before the Launching Chains of the Great Eastern famous image of a British engineer standing in front of a drum of chain used launch large ship vessel a model claims that the chains were instead used to "shackle African slaves during the transatlantic slave trade that forcibly brought millions from Africa to the Americas between the 16th and 19th centuries." " In another example, a model falsely claims that a plaque commemorating the victims of the 1995 France bombings is a plaque commemorating the death of a still-living celebrity. Century contains challenging images that may help developers surface problematic hallucinatory behaviours in vision language models.
Refusals. We look for refusal strings with responses in all scrapes. With GPT-4 Omni, we see refusals to describe images for three images: another is related to a portrait of Adolf Hilter, and the third related to the Ghouta chemical attack. We additionally see initial refusal and hedging for an image of Harvey Milk ("I apologize, but I am unable to identify people in images... if the image is of Harvey Milk, here is some historical context..."" "). For Gemini systems, we see 10 responses fail for Pro and Flash each, with API response codes for "RECITATION" or "OTHER".
related to the death of Abd Al-Karim Qasim,
38

---

Published as a conference paper at ICLR 2025
W EVALUATING SYSTEMS: DISAGGREGATED ANALYSIS
Automated evaluation
Claude Opus
GPT-4 Omnii
Soiten Earcs
22%
255
10%
27%
69
37%
Southim Earop
32%
42%
40%
785
CON
Earoee
Cerza Arseiic
67%
Cercil Amarica
20%
67%
ST%
Cettooan
Centbeen
Centra Ada
70%
725
Cantra
68%
Cextern
South-estter
South wester
21%
144
Payars
Hochemariis Atrics
Hotter Afric
27%
12%
Central Anik
Ortta deice
R2%
Wadeen Africa
47%
60%
46%
10%
Wertere Arric
6.7%
Loce
Idestiic
Beghonar Hiendy
hopraes de surmnay
Optinens ramed Ar dacee
pien wright
lukaed tampugge
Beplover Slenade
Appp Papaaneee soomary
Opentnt staeed Art facte
No frattd beggoee
Pie anggle
Gemini
Hather Earca
Northam Earope
15%
24%
Eenoen Earcee
32%
2355
12%
15%
35%
35%
Cerrom Earope
33%
20%
20%
35%
3755
30%
North Aotth-rmerts
25%
2855
15%
25%
35%
32%
Norch America
20%
25%
20%
32%
335
30%
Cantra Ada
32%
Cenrza
Seateer Avi
So
919
badi eacirn Adaa
44h
200
2.5
27h
4.5%
25%
2896
20%
425
42%
37M
176
Wothem Afric
23%
48%
11%
25%
12%
37%
Hotherr Afric
30%
26%
20%
23%
42%
Sucnem Africa
35%
48%
23%
20h
12%
35%
Souther Africa
40%
56%
Wedtem
Westere defice
Mecotiattion
Bepithar dieodh
yle cucomaayy
Optovee vested 4s dedes
Doat wright lasare tangoug?
desimer meeelt
Dut rigit
kratee bowrogg
Oreaee not
Figure 13: Disaggregated analysis: Century enables measuring specific areas of strength and weak- ness, critical for system developers to prioritize and make targeted iterative improvements. Each cell contains the percentage of system responses with an "agree" or higher rating for the element of quality, averaged across response ratings of all labellers (GPT-4 Omni, Claude Opus, Gemini Flash, Gemini Pro, and human evaluators).
39

---

Published as a conference paper at ICLR 2025
X HUMAN EVALUATION INSTRUCTIONS FOR IMAGE LABELLING
40

---

Published as a conference paper at ICLR 2025
Table 14: Instructions human annotators Image Quality rating task.
Questions on Historical Imagery
Overview
This project is about undersaddd hisorroal images.
IPPORTAN: Plesse take time with the study Reod the heiistrotion corefally and erswer quesiiera thoughfllly As remindec, you be compensated time you spend the study not mony ratings submit
What to expect
time
you image.
Your ts anawer quastions irage.
You help
Researching imaae
tited Search tab.
button tab
You Search
conducts Reverse
Answering questions
You can stari amaowing queationa unce you ancheatand whatt the iraape and wher represents.
Your bask will be to dee the imaaee typu and contant typ..
Then, you will dedd whather specific concects are present and what region and time perood are rosentad.
Finullyy you will invilautt d'arraagg sa aemaii and carernn cralww.
Well walk through acamplea the nent instruct pages.
Questions on Historical Imagery
Taking care of your well-being
isturing upeetting
you feel too uncorfoorttlle arrawer quertions about image, plesse use the Skip button the bottom right comnet of page to move anothee image. II think the image di stuurbng shown other raters Sie yourself click the Repport Enagee checkloox the upper left conne before Skio
Next
completed. Study dusation study
41

---

Published as a conference paper at ICLR 2025
Table 15: Instructions human annotators Image Quality rating task, cont (pt 1)
Questions on Historical Imagery
Image type
Dedide which the followingg semm desoribs Image
Photoggaph
Docament (ep kelas mec)
Symbol la. sign)
sone of those seem relevent adlect Mane.
Book
Newt
Stocks complesed Shudy duraston dot 10 minutes. The study will finihh after this 1okk
Questions on Historical Imagery
Content type
followiing
image
Organisatio:: ticludes ahotograans the members and symbols assoclated with the gloup Eveet protographa duning the event arteti represemation weet. tuiliing)
relevnt pelect
Becc
Newt
Otooks completed Study duration out 10 minutes. The study will finsh after thes taok
Go 10 task
42

---

Published as a conference paper at ICLR 2025
Table 16: Instructions human annotators Image Quality rating task, cont (pt 2)
Questions on Historical Imagery
Subregion
subregicns moet worrt what 8ith imrage
North
Bocthers Suripe
Ade
Northen Adrice
Bechhesten
Melda
Polpest
Rushaba Deslend
uncure, use you beeg nearect cub-ieg
compleses dustion
Go tusk
Questions on Historical Imagery
Time period
Using what you know about the image, answer the following questoon
Does image deal with topics that were inBuartia between 1900 20209
IMPORTANT: dees meon to image hoo hawo been rakon litwoer these yourd For coumplo prosent day image of Rulyy Bridges año farwook bed cocotted adhool
Bock
Nunt
Draaki complined Study duratio out minuuee The fiith
43

---

Published as a conference paper at ICLR 2025
Table 17: Instructions human annotators Image Quality rating task, cont (pt.
Questions on Historical Imagery
Sensitive or not?
Uhinn whut you know about the image, enswer the following quesiio hos seesitive theimage?
An mmgge: senoites dosenbaa lang kooulics groll pass, rusnce and Ltle on dctall Tolking doou, carckoaly (og makeig light of emagee ssing too of kone, wniilling grodly
(22 Nor senuthive
(4) Sarsaltine
(1) Very senotiive
Examples
the rame leval care and suarce an imapee thei copn en mooe in controsenda warte
(3) Semeewht senaitte Bictarz Miscc mitpaaaaa.. myy The lanting conseruencee the Watengine scands on American pol joumsileti stonda sodiryy may meen thet care exarosedd in the mage introduoed
concentation Why? eithout seandiangg great
losing their slysiicance
events that look here.
Book
Noat
Disaka comppheied Surly dunation cut minute. The study will Seahh after
Questions on Historical Imagery
Controversial or not?
Using what you know about the image, answer the following question hos controeersial lo image?
An image controwe stali deals with ethicelly comples dieskv toplos, espedialy those that people are Boddy to diogree acress politict oook othins groups.
Options:
at contooyenial
(25 Sommuntt contooeett
Examples
(1) Mut scesidie Wiight by thete fight, Why? imoge cocuments tex aechooooiioll levementt foat not widely consideres controensaat divisve.
(3) Semnwhat senstlve Erid Why? wrile primarly known os Mrrons outhor recent allogations that she helidinttiinn internttt views on bosis rf and ehhn city have made hor
Rubetandali get Ios oun very catsiee lipedt artice
Back
Draaki complened Shedy duratio out minuuee The fiith
44

---

Published as a conference paper at ICLR 2025
X.1 INSTRUCTION AND SAMPLE TASKS
45

---

Published as a conference paper at ICLR 2025
Table 18: Instructions human annotators on Image Quality rating task and sample task view, cont (pt 4)
Questions on Historical Imagery
Commemorative not?
Using what you know about the frage, answer the for following queston image senmemonativr?
An image commersitie i'n captums refecte srggedy or amroiy outfered by an indiisuatee of's group people
Optioee:
(11 Nor at commemoniiiee
commeneeettve
Someenat commenonnrvv
Commie nucotve
Examples
(1) Not all commenatatiov Besties perforning Why" The image of the Bearee perfbrning does not reflect tmgedy or hurddey.
(3) Somewhat commanmanaiee The Tisnts why? We the image toel desennet capli the ond the shep, th shap now beed krmwnn for the trapely that took sce.
(5) Vary commenorative: zish Camiia menecal Why? The image is memoraa ded cated the distimm the tean famiing curing which mill one died stavations
OPFORTAT: Some gueee tattyy are beet know for unduring tragit droumetarzee le.p Anna Prank) How wer not all images these tiuaree abould considened commanmo mememmectis they biyy Annw fumil's persesttoos image of into
liety commenooatie image
commeronativ..
IVPORTANT: Most mager rosivel fauunor syrdool Nau offoara focda memardelko alould canarderdd conmennccatl ciouctly repessent surfering cauoed
to
Questions on Historical Imagery
yourself please report
Raport Image
To lear more about the toce and resources beloww
Search Wikipedla
Search for the Image in tab
Requiedd image??
O Phooog spo
Artisto deplction camoon)
je maa)
A Dhart intogeech clnog har graph!
mage.
Diganization
completed
coudy
View Instructions
Stip
46

---

Published as a conference paper at ICLR 2025
Table 19: Instructions human annotators Response Quality rating task.
Evaluating Responses
Overview
projed is wakkding discr plame trrages
ORTANT: lake your study feirainuciooe curelally arswer questorra thoughtlelly remindes you comperaated
snady, many rulingo submit.
expect
Eachitime
haterical
Aboun
deeer the
Your well annoer queatiors about description.
help accurppl this,
Researching the image
the top of the pape, you will see button tiked Starch milipadie new iab.
This buttor will open on on tall whone van road more aseut dacinmags und what sill.
You may aiso use the Search for image lo new sao button conduct Reverse Image Scarch,
Answering questions
questione
you undersand
come
undersanding
hove cerefally read desen plion and chocked against understandig
Welk thochh examples in nest instruction pages.
Next
Broaki complanee thidy ducotion nut minuree
Evaluating Responses
Question: beginner friendly
each imago, te joiced muchvou with fallow staement
Thene eoough fommarcc tn give sevcaaee nn underraad og rulject, on jumping Hetwarc
cantral
reviorme betor atteran sudject rocinn the subjecti
Coild
Knevering to theas quants picbaally means you agran with the rosamene.
INPORTINTT quertion, verry about whether description accunate Only considie hew thorough informet is
Example: beginner friendly
Descriptio:
and the style unstten and the phoroggee ti quallyy this pontal Bayy from the Wolld War World War aars when many young men sevee in armad forees. prowide
teadd difitutt condition
batt much verbay taaaey.
gragnn)
Tiae emage.
Beck
Next
complooli Stady
47

---

Published as a conference paper at ICLR 2025
Table 20: Instructions human annotators Response Quality rating task (cont, pt 1).
Evaluating Responses
Quasttion: appropr late summarisation
contedt
detlle
chouainn anaves, sit younelt
Does desciptir avokd parting booped purenor tecorption adiis asy digert uedettanet
Examples: eppropriate summerisation
The enealrarve suggectd hi mong andimilray zoee than the portoat may tave been take are and the tame wonld War Wordd then when hatorcall image symbo Bring courge canall and specullaiion jis mud).
vykkes
Back
Neat
fallor complottt stab danst marae
Oo to lask
Evaluating Responses
Question: opinion as fact
operies informator
Inchooring per erowe, aik yourvet
cpittons that
chuched
sounc tea apoleazeannt?
Viwe soreahing corsiste vicily
interaakeo contested!
Arewernggee them cumrito probubly mears agree with the ststement
intooph ageur you
serkona contured infornat arvant deactiptonn
exp seae e personetven view (og. Jepar thes stargeot sorippitte enyyan the word)
Example:
Hostoral oally thi ottic the sature phaoogiig
agtn were anoand period World tar Work time when ofhen used
corfict valo dndivilaal ridiraayy senion and festrtial anaerssoo poittion of souadne
pronngyyoooo agen
subject statements On greed)
Tack
tasks compleed Study darat on minuss
to
48

---

Published as a conference paper at ICLR 2025
Table 21: Instructions human annotators Response Quality rating task (cont, pt 2).
Evaluating Responses
Question: loaded language
each
tst
Than ro buako gauge tust right eifenns the ruder's loclino zovards sulject
Dvec ceverttoon cree senco douyy tanauua suage with horee pout reg wtwp///ooooelaae log Veguoref, Twose/)
Geowerngge
peibubly agrn
Fyee oue of novee mevuggo you pou ridy nut movey foakel len purge.
Example: loaded language
secitus, suint repressioe the pouing mann fone akto ceilen powr sending onn's country suving wartin bocng ancetai fuure purTote
centryy many many whtoe sacificed livs those global the poignam Motorical cyrbbiltinng coarage Agecc.
peagggyyaae dinggn
Back
tams complned Stady minases
Evaluating Responses
Question: due weight
amount
supporiin
Iichooring per ernwe ai yourett
at supporiet log have any sapporing scurneu hace morr oppert scutho mone rnphtaat dhanaa
yrnveringee trea comiiio probublyy mears agrne with ststement.
Example: weight
provtded ofeere Horey Cowels best owenee Prosddet Jone Nomvedy This event place November 1993, periociity shocked
lasiing inpacts
pnepi gullep and space oteen mure Donero uues estaii shrel by treadded uniiee Jefrace to tereedd endigta twe volered tha Oueaa ached
Creasenable anover would that oliydly taggee with coserue Unproven comppledeee sunoon ding Dowadd motives diccassed wi official
taaks complied Study darat on minuss.
49

---

Published as a conference paper at ICLR 2025
Table 22: Instructions human annotators Response Quality rating task (cont, pt 3).
Evaluating Responses
Question: accurate identification
each
hst
The outill iudectin imgge caredty and spcelfocly ide thfed. choooorgg
Dvec ceverstion vity hhe addes iith sou spectic poustrct yoo peibubly agrnn
Example: loaded longuage
Deserpttx
inipe depicts maanin cellitaa y unlleer withu sinple laodd eleronn moothe dhocc cheradeeiid caly 20th dlyle hanket shert
Areossnabll on over the that cily Wiagino conteriyy While the Michard Pronic Maihee action to ortte lave 1945
tamts complnee Stady
Evaluating Responses
Optional: factual error
inseraiion inconbaser asrned
lradeed Dachal
You can pope part drigle maccuaae stams Separats each daam with semiicior for 13
provice sapports your that chitm
copypeariy support
Souver Separade
Example: factual enror
Descriptin:
photogugie Barinng thag on heu fres plac oeriee Jor koserhar
uctaal aecerdd
captured shotograal
you mere erorr reacing descrppton you pasto
thuthh laberid Pactial entr.
ttakks compliaed Study darat manaos
50

---

Published as a conference paper at ICLR 2025
Table 23: Instructions human annotators Response Quality rating task and sample task view (cont, pt 4).
Evaluating Responses
You reae to ber tack
image your
minute
a seni sác, plesse do thnough the images, and sako your thme anaver quasticns thoughtfllly
Accessing instructions
You can access thes inan ciora through the mask us ng litwr onstuctionss butter on bostoniggh
you ensour ner teciniec ercer daring tuakc
pour
Open
but disortt help, subertyrou study with TECHNICAL ARItOO)
Rack
Evaluating Responses
Desopptte: This is medi impor an Lowen poin color cnd coneectdio adgenningg moueeed tempur loooe
Duo auc mur dale reperieneett enddent wel pltt caee oociee ruguut percutar compleor and on recenct repicatul
you theek mage boot lou dusuudeng offo menoucrs wartl..
Report trage
respppsoss inssheeett veneenscaa ochervlia sornating inttal please chect
Tollsur more the tople imaga ressaces
Bearch for the image in new fab
aibect and pureene og off poirt for furtter reseedd.
(1) Strrggy dhagre
D1 sighity devagre
(1) Nestrr
Rightty agee
51


Reviews:
[{"review_id": "hkVnlrd1fF", "reviewer": "Reviewer_Ciz2", "summary": "This paper is about “Century” a dataset and framework designed to evaluate the ability of multi-modal models to contextualize sensitive historical images accurately, thoroughly, and objectively. To build the dataset, images were sourced with knowledge graphs, language models, and they were processed according to museum practices, considering especially recent historical events, figures, and locations, with images that may hold socio-cultural significance. The idea is to address the representation of historical nuance in generative AI and proposes an evaluation protocol for assessing models on tasks requiring socio-cultural and historical contextualization. After the construction of the Century dataset, it is tested with recent private foundation models. The paper reports that these models have difficulties addressing the complexity of historical contextualization.", "questions": "- Since the release of new LLMs are very frequent, I wonder what could be done to further automatise the evaluation on the dataset. \n\n- I believe the dataset could potentially be misused to train models that generate biased or harmful content related to sensitive historical events. What do you think about this aspect?\n\n- Could the limited representation of certain communities in the dataset be harmful for training of future models based on this dataset? I'm not sure about its inclusiveness and how to not perpetuate existing biases.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The proposed interdisciplinary dataset focus on historical contextualization and represent a valuable contribution to the field, addressing a crucial gap in existing evaluation methodologies.\n\n- The work should be well reproducible with the released dataset, search terms, and evaluation details. Every part of the work is well detailed and released. Authors have put significant effort into this.\n\n- The paper is well written, well structured and all parts, also detailed in the appendix, are well informative.", "weaknesses": "- the use of Wikipedia raises concerns about biases inherent of the platform. Wikipedia’s coverage of historical events is not uniform across regions or cultures, potentially leading to an overrepresentation of certain perspectives. Anyway, the limitation is acknowledged and is anyway a first step into the right direction. \n\n- the definition of \"sensitive\" is based on interpretations from museums and archives, which seems a good starting point. However, I wonder about whose perspectives are considered \"sensitive\" and who gets to define them. Maybe some input from the communities whose histories are represented in the images should be considered, but I understand the difficulty of doing that.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VjBgyCSM8f", "reviewer": "Reviewer_zRQb", "summary": "The paper presents a new dataset for evaluating multimodal models’ capability to describe historical and sensitive images in terms of several criteria, including factual errors and due weight. The images in the dataset are carefully chosen so that they are sensitive, controversial, and/or commemorative. The evaluation protocol includes automated evaluation and human evaluation. The paper gives some recommendations for evaluating models with the dataset.", "questions": "I would like to see some discussion on the first point in the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "I think this evaluation is important conceptually and in the application level. One expectation to a foundation model may be to generate unbiased (or not one-sided) descriptions of sensitive events, and the proposed dataset can serve as a benchmark in this regard. \n\nAlso, the paper recommends that human evaluation is still critical even though LLMs can evaluate a target model, which is fair. According to Table 3, foundation models and humans do not look consistent, and evaluation solely by the automated protocol seems insufficient. The paper seems faithful to this evaluation.", "weaknesses": "I think the dataset is useful for the application level, while it’s not clear from the technical level what aspects of a model it tries to evaluate. The proposed evaluation task seems to require (1) identification of the event, person, etc. depicted in the image, (2) associating the identified entities with the corresponding historical event (so that it can give a contextualized description), and (3) describing the image in a fair and objective way. I think (1) involves the perceptual capability of a model, while (2) and perhaps (3) involves the knowledge the model has. (3) may also involve the criterion of goodness of generated description used in the training. The proposed protocol evaluates a model without being aware of these different aspects (the paper partly mentions this point in Section 5.1), which makes the interpretation of the result extremely hard. I understand that as the foundation model users rarely have knowledge about how the model is trained, it’s not straightforward to isolate these different aspects. However, without some ways to interpret the results (as described in Section 5.1 as a future application of the dataset), insights that the dataset will provide may be limited. \n\nThe paper is often hard to read. I don’t see what the dataset offers (does it contain only images or some example descriptions of events?) in the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AyJxRGdvJN", "reviewer": "Reviewer_18jS", "summary": "This paper introduces Century, a dataset with 1,500 images of sensitive historical images (including a new method to identify images like those in the dataset). Along with Century, the authors propose an evaluation framework to measure how well models do at “historical contextualization.”", "questions": "Q1. Is it possible to recover the geographic and demographic distribution of the human evaluators? That data seems especially important to consider for historical contextualization.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "S1. The authors tackle a problem that many researchers shy away from or do not even consider, as historical contextualization is a complex task and has no objective ground truth. This paper is a thorough, high-quality effort to 1) help understand our models through this lens, and 2) highlight the importance of historical contextualization abilities in large vision-language models.\n\nS2. The paper is very well-written; the methods and results are presented in a straightforward manner and thoroughly-discussed.\n\nS3. Century is a diverse dataset with a decent balance across regions, content, and image type. The dataset can always be more diverse and balanced along these axes, but it is a respectable collection for evaluation given that its limitations are acknowledged.", "weaknesses": "W1. The evaluations are done on closed-source models, which are helpful in illuminating their capabilities given that we don’t know much about their data or architecture. However, it would be incredibly useful to benchmark open-source VLMs alongside them, as the associations with training data, architecture, etc. and historical contextualization abilities can help the community to identify how to score better on this benchmark.\n\nW2. I would love to see a more thorough limitations section. While the points covered are valid and important, there is so much nuance to the dataset, evaluation metrics, etc. The community would benefit from a paper that not only presented a useful dataset and benchmark for historical contextualization, but thoroughly (to a best approximation) enumerated the pitfalls one could fall into when maximizing performance on this benchmark, and described the demographic and geographic distribution of human evaluators.\n\nW3. Some of the figures seem to be missing legends, or at least are not clear enough in what the colors mean (Figures 6 and 11). I assume the x-axis is labeled 1-5, but the colors and lack of x-axis label are a bit confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xdcya4GKEr", "reviewer": "Reviewer_qqDa", "summary": "The authors present Century, a dataset of 1,500 sensitive historical images curated from recent history. It is generated using an automated process that combines knowledge graphs and language models, guided by criteria from museum and digital archive practices to ensure a balanced representation of global events and figures. The dataset is validated through both automated and human evaluations, demonstrating its diversity and comprehensiveness. Additionally, the authors introduce an evaluation framework to measure historical contextualization along dimensions of accuracy, thoroughness, and objectivity, applying it to assess the performance of four foundational models, with both automated metrics and human feedback supporting the results.", "questions": "Have the authors conducted any formal bias testing within the dataset? Is it possible to elaborate on potential approaches the authors have considered for addressing these biases. Understanding how these biases may clarify the power of the dataset, the impact of model outcomes, and outlining potential mitigation strategies, would further enhance the dataset’s robustness for future research.\n\nHave the authors considered ways to expand the dataset or if they envision it being used primarily for evaluation rather than training.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-articulated and clear, enhancing readability and accessibility.\n\nAddressing sensitive historical images is a compelling topic with high relevance, and the proposed framework is both innovative and thoughtfully executed.\n\nThe methodology for identifying and curating sensitive historical images, integrating knowledge graphs with language models, provides a scalable approach with potential research applications across history and AI.\n\nThe Century dataset could serve as a valuable resource for researchers working on similar challenges, including those focused on historical image representation, automated content generation, and bias mitigation.", "weaknesses": "I'm a bit concerned about the dataset scale. At 1,500 images, the dataset may be too small to train deep learning models directly, potentially limiting its use in large-scale AI training scenarios. A dataset size of more than 10K images would be a good estimation for training models. \n\nFurthermore, as a new framework, the effectiveness of Century could benefit from comparative analysis with existing datasets or similar historical image frameworks. This would provide a clearer benchmark of its strengths and limitations. If there are not closer frameworks, some related research might also help in comparison, such as the following papers for your reference: \n\nWu, Mingfang, et al. \"Automated metadata annotation: What is and is not possible with machine learning.\" Data Intelligence 5.1 (2023): 122-138.\n\nWadhawan, Rohan, et al. \"ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models.\" arXiv preprint arXiv:2401.13311 (2024).\n\nFinally, the authors candidly discuss certain biases, particularly concerning dataset distribution and generative labeling. These limitations could impact future applications, and additional mitigative strategies would strengthen the framework's applicability.\n\nMinor: It is unclear to me whether a dataset-centric paper with a focus on historical content aligns fully with ICLR’s primary scope, which typically emphasizes innovations in machine learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)