Enhancing Textbooks with Visuals from the Web for Improved Learning
Authors: Janvijay Singh
Vilém Zouhar
Mrinmaya Sachan
Venue: EMNLP 2023 Main (2023)
Keywords: textbooks
learning
education
images
TL;DR: Images are automatically assigned to textbooks which improve their quality.

Abstract: Textbooks are one of the main mediums for delivering high-quality education to students.
In particular, explanatory and illustrative visuals play a key role in retention, comprehension and general transfer of knowledge.
However, many textbooks lack these interesting visuals to support student learning.
In this paper, we investigate the effectiveness of vision-language models to automatically enhance textbooks with images from the web.
We collect a dataset of e-textbooks in the math, science, social science and business domains.
We then set up a text-image matching task that involves retrieving and appropriately assigning web images to textbooks, which we frame as a matching optimization problem.
Through a crowd-sourced evaluation, we verify that (1) while the original textbook images are rated higher, automatically assigned ones are not far behind, and (2) the precise formulation of the optimization problem matters.
We release the dataset of textbooks with an associated image bank to inspire further research in this intersectional area of computer vision and NLP for education.

Enhancing Textbooks with Visuals from the Web for Improved Learning
Janvijay Singh* G
Vilém Zouhar Mrinmaya Sachan E
School of Interactive Computing, Georgia Institute of Technology
ETH Zürich, Department of Computer Science i iamjanvijay@gatech. edu
{vzouhar, m ch
Abstract
Textbook Section Photosynthesis
Good Image Assignment Image Assignment
Bad
Textbooks are one of the main mediums for de- livering high-quality education to students. In particular, explanatory and illustrative visuals play a key role in retention, comprehension and general transfer of knowledge. However, many textbooks lack these interesting visuals sup- port student learning. In this paper, we investi- gate the effectiveness of vision-language mod- els to automatically enhance textbooks with images from the weh. We collect a dataset of e-textbooks in the math, science, social sci- ence and business domains. We then set up text-image matching task that involves retriev- ing and appropriately assigning web images to textbooks, which we frame as a matching op- timization problem. Through a crowd-sourced evaluation, we verify that (I) while the original textbook images are rated higher, automatically assigned ones are not far behind, and (2) the precise formulation of the optimization prob- lem matters. We release the dataset of text- books with an associated image bank to inspire further research in this intersectional area of computer vision and NLP for education.
Introduction
crapgar sil trau 4 Inrset tfe plants
X
Cycle
Paokaythouu hew plint
vecetennnn chamical procees which uose eunlight to fimn carton
dioxide into sugars The sugan or byd
Chemical Reaction
the bniieeaiee for "larct invaapees and it pricess calnc the Calvih contired neviiie the
sycle caee th oncy fommed NACHH
The
Illustraiion of good and bad image assign- ments. The first subsection does not require an image. The second subsection in the bad assignment has lated image but without a strong connection. Likewise, the last subsection has picture of Calvin, which is lated but does not have high pedagogical value.
Figure I:
as an ideal illustrative visual should not only be related to the textbook material, but also have ped- agogical value (see Figure/ 1).
Agrawal et al. (2010, 201 already enhance text- books with images from the Internet. They use sim- ilarity scores of textual captions from images and the text present in the textbooks for image assign- ment. This approach requires the image captions to be available. Beyond the problem of availabil- ity, the captions also highly context dependent which reduces their utility in our setting. In this work, we aim to reinvigorate this area, and present and analyse a real dataset of textbooks. We do so by setting up the problem of image assignment using textbook and Wikipedia images.
1 Introduction
Students use textbooks as one of the primary medi- ums of learning. It is thus imperative that text- books are designed to provide a rich learning and engaging environment. Visuals enhance learning through number of means, including the ability to retain information, as well as its ability to promote comprehension and knowledge transfer (Carney and Levin, 2002; Dimopoulos et al., 2003; Katsi- 2008;
oloudis, 2007; Hibbing and panjwani et al., 2009; Mayer, 2019). Automatic ap- proaches for retrieving and assigning images from the web to textbook chapters can therefore assist textbook designers in the creation of better text- books. However, this is a very challenging task
In contrast to previous work, we rely on the re- cent advances in vision-language models, such CLIP (Radford et al., 2021) and DALL-E (Ramesh et al., 2021). We analyze our dataset to gain in- sights into the organization of concepts and illus- trative images within the textbooks. This analy- sis inspires the formulation of a new optimization
Work done during an internship at ETH Zürich "Code & data: htt..h////ch-lppceetttttoooooooo

---

problem focused on modeling of illustrations in textbooks. The solution to this problem maximizes the coverage of the illustrations while minimizing redundancy during image-to to- co-paragraph assigments. Our approach uses CLIP to retrieve and appropri- ately assign images to long-form text, particularly a textbook section. Then, the overall assignment is obtained in following stages:
et al., 2014), that contain natural images and their captions, which are rather short. Additionally, Schneider et (2021) show that current multi- modal models perform poorly at retrieving relevant images for longer and more complex textual in- puts. The reason for this poor performance is the pre-training on shorter and very specific image cap- tions. This is strong requirement to our work. which is focused on even longer text inputs. We explore the problem of assigning images to lengthy text, which highlights issues such as ensuring com- prehensive coverage of concepts and avoiding re- dundant image illustrations. To move even closer to a real-world setting, we perform this task with actual textbooks and human study.
1) Given a piece of text, produce set of concepts that should be addressed by visualization,
2) Given an image and a concept, determine their mutual relevance, and
3) Given a piece of text (c.g., textbook section), produce an adequate assignment of images.
Each of the three sub-problems listed above can be solved with variety of approaches indeed we explore several variants, which we describe in Section Because of the modularity of this ap- proach, each of the sub-problems can be improved independently and adapted to the dataset in future work. Overall, we contribute the following:
The task of text- book enrichment first explored by Agrawal et al. (2010, 2011), I), who assume that web images have associated relevant captions. We note that the caption of images are largely dependent on the context where the image was originally assigned.' The language of the caption may not even match the textbook's 's language. To alleviate all this, we do not assume that the images have associated relevant captions. Seo et al. (2015); Kembhavi et al. (2017); Lee et al. (2022) also studied associating images with textual information. However, their primary goal has largely been to comprehend the image con- tent, and thus differs from our objective. Finally, there has been more past work on NLP applied to textbooks (Sachan and Xing, 2017: Sachan et al., 2017, 2020). However, the goal of these works also differ significantly from ours.
Enriching Text with Images:
A dataset that contains text and images drawn from 35 textbooks covering math, business, so- cial sciences, and science in addition sec- ondary image bank of ~312 images taken from Wikipedia (Section 3).
Formalization of multiple textbook enrichment optimization goals (Section 4).
Human evaluation and an in-depth examination of the possible failure modes and challenges of the proposed methods (Section 5).
2 Related Works
Vision Language Models: Alignment between texts and images has seen rapid progress recently with models such as CLIP (Radford et al., 2021) and DALL-E (Ramesh al., 2021). However, their usages are limited to a short and specific text prompt to which the performance is usually quite sensitive. We focus on the problem of retrieving images for very long textual inputs, specifically a textbook section, where it is unclear which part of the text specifically describes the relevant image.
Dataset
We now present the process of curation and struc- ture of our dataset with an analysis.
3.1 Data Collection
OpenStax Books. We source both the text and assigned images from 35 textbooks from an on- line textbook publisher openstax.ong .org covering four subjects: business, social sciences, sciences, and maths. Each textbook organized into chapters, sections, subsections, and paragraphs. See their distribution in our dataset in Table For each sub-
Image Text Matching for Long Texts: Recently, Wang et al. (2022a.b); Zeng et al. (2022) trained better language-vision representations with more nuanced associations. such as multiple vision tasks or finer image-text alignments. However, this progress is mainly confined to datasets, like MS- COCO (Lin et al., 2014) and Flickr-30K (Young
For example, the same image of poster from Wikipedia has different captions on different pages: "The munitions industry heavily recruited women workers, as represented by the U.S. government's 's Rosie the Riveter propaganda campaign" and "J. Howard Miller's 's We Can Do It! poster from 1943s".

---

section of the textbook, we identify the following key clements:
QL. How are concepis distributed? The pat- terns in concept mentions are similar across sub- jects (Figure 2b). The distribution of concepts within subsections (Figure 2a) reveals an average of 5.6 concepts per subsection. An average con- cept is mentioned 2.7 times within a subsection (Figure 2b) that is, concepts are infrequently mentioned in the subsection. Notably, each section concept is mentioned in only 1.7 subsections on average (Figure 2c), emphasizing the high localiza- tion of concepts to specific subsections.
Text: Raw text from the subsection.
Phrases: Raw text from the subsection decom- posed with overlapping sliding windows.
Concepts: Key concepts laught in the subsection are the bolded words phrases, headings, index terms, and key terms (both marked explicitly in the book such).
Images: Image(s) assigned within subsection.
Subject
Sections
Subsections
Images
Q2. What influences the number of assigned images in a subsection? On average each section consists of 5.5 subsections and 3.3 assigned images (Table 1). To answer the question at hand, we conduct a regression analysis with the number of images in subsection as the predicted variable, and the following as features:
Science
1431
6775
5973
Math
461
4314
1177
Social Science
517
2463
1514
Business
553
2669
962
All
2962
16221
9626
Table I: Summary of the number of sections, subsec- tions, and assigned images in our dataset. We partition the chapters (~9.66 images) into tain (~7.2K images),
http::/ss/wwwdd//p/arrrpppp their total # in the subsection.
concepts _uiin:qq # of unique concepts men- tioned in the subsection.
dev (~1.2K images), and test (~1.2K images) splits. While partitioning, we assign each book to a single split we ensure that all subjects are well represented differ-
asec__coneppe % of unique concepts from the section in the subsection.
ent splits and that there is minimal concept overlap.
%sec_ coce__co:ept:: % of total concept mentions from the section in the subsection.
Wikipedia images. To mimic the task of text- book enrichment, we use dataset of images from Wikipedia that are relevant to the concepts in the OpenStax Books dataset. This dataset serves as a proxy for images from the web. We search for relevant Wikipedia articles for each concept, with a maximum of 20 articles retrieved per concept. From these articles, we extract images and their captions by searching the article in the WIT dataset (Srinivasan et al., 2021) or directly from the article. The final dataset has approximately ~312K unique images included in the relevant articles.
asse_wordss % of total words in the section which are in the subsection.
%sec_par saccc_prarrpphhi tion in the subsection.
% of paragraphs in the sec-
posi tion of the subsection in the section from 0
(beginning) to (end).
subject of the book.
Based on the results in Table 3, the number of assigned images to a subsection can be best pre- dicted from total number of concepts, words, and paragraphs of the subsection. Unexpectedly, this is not true for the number of unique concepts. Fur- thermore, the position of the subsection within the section is negatively correlated the image count that is, the subsections located later in the sec- tion have fewer images. The subject of the book also impacts the subsection's 's image count, with differing coefficients for each subject. Overall, the regression model yields Pearson correlation of 0.59 with p<10-- a high degree of predictability.
Image Bank. We combine OpenStax Books im- ages and the Wikipedia images form the Image Bank. Our objective to retrieve and assign rele- vant images from the Image Bank to each section that is present in the OpenStax Books dataset.
3.2 Dataset Analysis
We profile the dataset according to a series of ques- tions, which will inform the problem formulation.
Q3. Are images exclusive the assigned subsec- tion? We use CLIP's similarity scores for image- phrase relevance (detailed in Section 4). For each image in the textbook we distinguish between the
Example of subsection: openstax.org// org/books/cooccppss boooogg/pueesss-oovvvveeeaaaa ooverview-d--phootsytteees
'A window-size of 75 tokens and an overlap-ratio of 1/s.

---

Math
Science
Social Science
Business
20
20% Math subsections have concepts
No difference between subjects
No difference
Sutsections
Concepts
06 Concepis
60
between subjects
40
1.0
40
20
20
%
10
20
30
Concepts
# of Mentions
# of Subsections
(a) concepts across subsections.
(b) mentions of across concepts. (c) subsections with a concept mention.
Figure 2: Distribution concept-mentions. On average: subsection has 5.6 distinct concepts; (b) concept is
mentioned 2.7 times within subsection; and (c) concept from section is mentioned in 1.7 of its subsections.
Position of Most Similar Phrase
Before
Present
After
% of Concepts Mentioned
% of Images
60
50
Math
40
Science
Business
Overall Science
Math Business Social Science
Social Science
20
0
10 of Top-Similar Phrases
20
30
Figure 3: Proportion of images with the highest similar- ity scores phrases from the subsection before, present, and after the gold subsection. Across all subjects, nearly 57% of the gold images have the highest CLIP similarity with at least phrase the subsection before after.
Figure 4: Proportion of concepts mentioned in top-k most similar text-phrases to the gold-set of images in subsection. On average for each subsection, 50% concepts are mentioned the top-3 most similar text- phrases with the gold images.
present subsection and the one before and after. Then, we assign images to the subsection with the mathestg phrase. The percentages of most- similar images in the before and after subsections is nearly equal (Figure 3), which is contrary to the intuition that the subsections after the current one would refer to the concepts in the image. The dif- ference between the before/after and present sub- sections the greatest in the business category, indicating uniqueness of images assigned to a par- ticular subsection compared. Such uniqueness, as determined by the CLIP scores, is most absent in mathematics books. Overall, the images do not exclusively best-match the phrases from the gold- assigned subsection.
phrase with concept mentions (Figure 4). This war- rants further usage of CLIP scores as a measure for matching concepts to images.
Image Retrieval and Assignment
We first describe an image retrieval model and then formalize our task and the optimization ap- proach. CLIP (Radford et al., 2021) is a state- of-the-art vision-language model trained on many image-caption pairs from the web by maximizing the dot-product similarity the image and caption en- codings. We further fine-tune CLIP image-text pairs from the OpenStax Books dataset.
Q4. Are concept mentions associated with as- signed images? We rank subsection phrases us- ing CLIP similarity scores with subsection images. We use this ranking to calculate the percentage of concepts that were mentioned in the top-similar phrases associated with the gold images in the subsection. This way, we evaluate whether text- phrases with higher association to the gold-image also had more concept mentions. Indeed, there is a correspondence between the gold images and
4.1 Image Assignment Formulation
Our formulation focuses on the assignment of im- ages subsections." We begin with notation:
subsections u in section = (ul, Uol (1) concepts in subsection
[C1, cr} (2)
"We can assign images to the entire book by concatenating all chapters into one long "section". One can also assign images to paragraphs by treating them as a subsection each with single paragraph.

---

We decompose the text of subsection into phrases using a sliding window approach. These phrases may mention a particular concept:
c). Informally, cov(c, i) is iff concept e covered by image i, otherwise 0. Next, we formalize the coverage and redundancy.
cov(c, i) =
phrases in a subsection (41....,tt)
(3)
Coverage. Coverage for a section s and a subset of images I' is the number of unique concepts in section which are covered by images in I';
Let m(t), ci) denote mention of concept in if Cę mentioned in 4 otherwise
{
m(4,, cx)
C(I' s)= | E' cov(c, i) = 1 (6)
(4)
Redundancy. Redundancy of a section s and set of images I' is the total number of times concepts in s are multiply covered:
For fair comparison, assign the same num- ber of images to each subsection in the gold assignment. This can also be automated with the image count prediction (Section 3.2/Q2).
R(I'ss) -E cov(c, -((',, (7)
4.2 Local Assignment
OCS SEZ
The most straightforward solution is to select an image for each subsection independently by maxi- mizing the subsection text-image similarity, Specif- ically, we assign each subsection u, with an image E I, which maximises the following function:
We now introduce the concept of set submodu- larity which will be necessary for proving approxi- mation bounds of the optimization.
Definition 4.1. A function f is said to be set sub- modular if and only if C. B Va [x)) f(A) 2 (() f(B). Informally, the func- tion yields diminishing returns for item x.
S(E)), u)
>>>Em sim(i,t)
(5)
Theorem 4.2. The coverage function C is set sub- modular. That is for I" such that I" C I' CI and E I it holds that C(I" U (()) C(I") C(I' U {i}) C(I').
Here, I denotes set of all the images in the image bank. Moreover, sim(i, t) denotes probability (nor- malised dot-product similarity across images) of any image i and certain phrase t as given by the fine-tuned CLIP model.
Proof. Let C' and C" be sets of concepts from a subsection covered by images in I' and I" respec- tively. As per the definition of C, we have C" C Let C, be the concepts covered by image i.
While local assignment is fast and simple, our qualitative analysis reveals that it lacks global co- herence and may assign images depicting overlap- ping concepts to the same section. For example, every subsection mentions the concept "molecule", then all subsections can be assigned the same im- age of a molecule. This finding aligns with our previous results (Section 3.2/Q3) and is supported by the redundancy metrics in Section 5.
U (
C(I's))
(8)
c''uc -|'ll
(from def.) (9)
C uGg -cc"
(from C" C C') (10)
C(I" U (i), -C(I",)) (from def.) (11)
4.3 Global Assignment
Theorem 4.3. The negative redundancy function
The analysis in Section 3.2 revealed that the most- relevant phrases for gold images are not restricted to the assigned subsection and that the concepts are localized within their respective sections. There- fore, for better global coherence in our assignments, we assign images based on concepts rather than select subset of im- ages that covers most of the concepts (coverage) while avoiding overlaps (redundancy). To define coverage and redundancy functions for concepts in a section, we first define a boolean function for image, i, covering a concept, c, as follows:
is set submodular. That is for T" such that I" c I' I and E I it holds that RR(I" (1) + R(I") -R(I' U (i)) + R(I').
Proof. Similarly to Theorem 4.2, for redundancy function R we observe that:
phrases. Specifically,
R(I U (igh.s) R(T',s)=
(12)
c'nC,
(from def.) (13)
C" nc,
(from C" C C') (14)
R(I" U (g). - -R(I", s) (from def.) (15)

---

Observation 4.4. Both C and R are monotone. For the global assignment, we choose images I' I such that the following is maximised:
concepts and has most similar text. Note that the local and global assignments are specific cases of this formulation. This formulation achieves our desideratum - images are assigned to specific sub- sections also with global context consideration.
= -(('ss) R(I',s) s)
(16)
G (Equation 13) is a submodular function because it is sum of two submodular functions. Find- ing the optimal solution to G' is NP-hard (Lovász, 1983). However, since G' is submodular func- tion, greedy algorithm can lead to fairly good /e R 68%-approximation of optimisation of under cardinality constraints [Z'] < B. where is the budget (Nemhauser et al., 1978). Once images I' are greedily computed for a section, we assign an image i E I' to the subsection, u, which maximises C((i), w); to the subsection in which the image covers the most concepts.
Human Evaluation
The goal of improving textbooks is to help students learn. Testing a wide range of models directly by monitoring learning progress would require a very expensive long-term evaluation. Instead we turn to an intrinsic crowd-sourced evaluation where we ask teachers what they think about the qualities of the assignment.
5.1 Setup
We selected 32 croke-workers from Prolific who are native English speakers and work in educa- tion. We compare different assignments: the gold one by human and three automatic ones (Section 4). Each participant is assigned a single section and evaluates all systems on this section. This methodology may cause an unwanted prim- ing effect, which we address in Section 5.2. We chose this setup deliberately because lot of the annotation time is spent on reading the section text and we wanted to share this cost by annotating multiple assignments at once. Our evaluation con- sists of close-ended (limited number of answers) questions that pertain to both the local (subsection) and global (section) textual context (full annotation guidelines are in Appendix A):
4.4 Joint Assignment
The local assignment captures relevance while the global one also captures redundancy. To optimize both of them, we formulate the following objective with trade-off hyyper-prramete 3 2 0:
J(I', S(('', + 3 ()) sim(1 t) + B G(I',
(17)
= =ΣΣ
iCI' tCs
Note that a submodular function, since is the sum of two other submodular functions: 3 G and S. The former is submodular due to the non- negativity of 3 and previously proven submodular- ity of G. The submodularity of S is proven below.
Local evaluation:
Is this image relevant to educational concepts described in this subsection text?
Theorem 4.5. The local assignment function S is set submodular. That is, for a set of images I'
this image redundant compared to previous images in this subscction?
and I" such that I" C I' C I and any image I' it holds that S(I" U (t)) (03)-5(00 S(I")
I I
What is the type of this image?
S(T' (i})) S(I'). Proof.
Global evaluation:
Is this image relevant to educational concepts described in this section?
S(I' U (i),s) -((''s)) Σ sim(i,t) (18)
ICs
Is this image redundant compared to previous images in this section?
< S(I" (1),s) S(I"
(19)
Is this image didactically useful for explaining this section text?
Considering the submodularity of J, we select images greedily, similarly to optimizing G. Once set of images, I', is greedily computed for a sec- tion, we assign an image i I' the subsection, u, which maximises S(I', u) + 3 C(I', u); to the subsection in which the image covers the most
The annotators first answer the local questions for all images and then the global questions. This way we made sure that they scanned the entire section and had some overview of all images and how they relate to each other before answering the

---

Rolling Motion with Slipping.
the case of roling motion with slisping, we must use the coefficent of kineti friction, which ghee rise to the kinete fricion force since stati: fricion is present The sturton is in the case of sipping. VoM Ruw O, because point P on the wheel is not at rest the surface. and Vp O Thus. =
image relexcnnt edurational concepts described this subsection texd?
Iridovant
Somreiia ralosiin
undicdddd
Ivolic. lane)
Is image redundant compared to previous images this subsection? ded
(ellet
What the type of this image??
retura image
Blagram
graph/ plet
readecle
multille
other
rchect
image relevaant
concepts descrried section?
Indevant
someeshet relearti
undecided
Inllcct
Is this image redundant compared to previous images in this section?
Undecieed
(solction)
Is this image didactically for expleining this section texd?
vis
undec ded
Figure 5: Annotation window for a single image within subsection (there may be multiple images in subsection). Note that, the image featured is not a suitable choice to illustrate this particular subsection.
global questions. The annotation pipeline (for assignments, local/global) is shown in Figure 6 and the uscr-intedd ince-interface in Figure 5.
We then focus on two most important evalua- tion criteria: relevancy and redundancy. The re- sults for those, shown in Figure 8, clearly show preference for the Gold image assignment, suggest- ing that automatic assignment is still inferior to humans. From the automatic methods, the Local performs the best relevancy-wiss, However, it is outperformed by Joint with respect to redundancy. We also note that there is very little difference be- tween Local and Global evaluation category. This may be caused by evaluation bias (i.e. annotators are likely to give similar score for both local and global questions).
Insination
foint ossignment
gold ossignmmntt
locul assignment
glotol ossignment
local
global
local
plobal
local
global
local
global
Figure 6: The annotator is presented with the same
section but with randomized image assignment order.
Bette
Global usefulness
Glabal relevancy
Local relevancy
Relevant
redundancy
Worse
redundancy
Somewhat relevant
Position
Local
Global
Figure 7: Average score for all categories as dependency of the position evaluation. Color bands show 90% I-lest confidence intervals.
Ineeevant
Gold
Global
Joint
Local
Not redundant
5.2 Evaluation Results
We first verify the evaluation setup validity by checking whether the position in the evaluation queue has effect on the evaluation scores. Recall that the annotators were shown the same textbook section with 4 alternate images assignments. While Figure/ shows some variance along the indepen- dent variable of evaluation position, the differences are not significant, justifying our evaluation setup.
Redundant
Gold
Global
Joint
Local
Figure 8: Local & global relevancy and redundancy across 4 image assignment modes. Gold is overall the best, though Local is better at relevancy.
Next, we examine all the remaining evaluation categories Table 2. While Gold is the best across

---

all, the significance of the difference varies. The Joint assignment is never the worst, suggesting to be a robust choice.
resent abstract concepts, relationships, and events which cannot be well modeled by models like CLIP that are majorly trained on natural images.
Evaluation Category
Ordering
Long textual description of concepts. Another source of error was that some concepts had long tex- tual descriptions. For example, the description of Stokes' theorem spans multiple paragraphs. Learn- ing to associate image with a part of text may lead to loose and spurious associations, resulting in poor downstream assignment performance. This high- lights the need for vasion-language representations that can effectively model long text descriptions and establish better image-text associations.
Local relevancy
Go 20.001
L >70075
J 12.163 GI
Local redundancy
Go 20.008
J >0.258
Gl 2,,299 L
Global relevancy
Go 20.001
L >0.179
P0.106
Gl
Global redundancy
Go 20.011
J 20.337
Gl
20.307
Global usefulness
Go 20.00 J >0.383 L >0.211 GI
Table 2: Ordering of assignment modes with numbers showing one-sided Welsch's 's t-test p-value. Gold- Go,
I. Local, Joint, - Global.
Local
relevancy
2
2
4
3
Global
6
7
4
fix) cot
relevancy
20-
Local
9
6
7
4
:::
jo
redundancy
6
3
5
1
Global
6
7
redundancy
5
6
5
1
6
2
Global
8
usefulness
8
Gold
Readable
Natural
Graph
Other
Toble
Multiple images
Auto.
flowhaart
text
image
plot
Figure 10: Our approach assigns left and right images Stokes' Theorem and Factors Affecting Engagement in Democracy. In both cases, image is not specific enough and only very loosely relevant to the subsection text. This underscores the weakness of our model, which learns relevance with limited textual context.
Figure 9: Category-wise gold and automatic image as- signment scores (0-9); higher scores indicate better per- formance. In each cell, the tep-left corner displays the score for the gold assignment, while the bottom-right displays the score for the automatic assignment (aggre- gated across all methods).
Conclusion and Future Work
5.3 Qualitative Analysis
We presented a dataset and a new task of enriching textbooks with visuals from the web. We proposed several technical solutions for this problem using neural image retrievers combined with a new as- signment optimization setup. Annotations by work- ers in the education industry verified that, even though the human assignment is still of the highest quality, the automatic assignments are not far be- hind. There are multiple venues for making further progress on this problem. First, individual con- cept importances and text-image relevance models could be improved and plugged into the existing algorithms. The varying domains of images and lengthy textual descriptions of concepts present challenges that could pave the way for exploring new approaches for learning image-text associa- tions. Professional textbook designers can be in- cluded to further refine the assignment optimization objective and pose this as a human AI collaboration problem.
The Joint optimization method aims to reduce the reader's 's cognitive load compared to the Local method that aggregates scores from all text-phrases. The Local method results in repetitive covering of single concept from the section with top-images. In contrast, Joint and Global assign images cover- ing wider range and variety of concepts in the section, enabling a greater level of text enrichment, Appendix B shows examples of assignments by these approaches. We remark that structured image types, such as graphs, multiple images, or those that are less identifiable, receive lower ratings sys- tematically (Figure 9). We now elaborate on the two major limitations in our models.
Varied domain of images. One limitation of our approach is demonstrated in Figure 10 where struggles to model non-natural images such as dia- grams, graphs, and plots. These images often rep-

---

Acknowledgments
science textbooks and press articles about science and technology. Research in Science Education, 33:189- 2 216.
We thank the anonymous reviewers for their feed- back on our paper. MS acknowledges support from the Swiss National Science Foundation (Project No. 197155), a Responsible Al grant by the Hasler- stiftung; and an ETH Grant (ETH-19 21-1) for this work.
Anne Nielsen Hibbing and Joan L. -rinksnn 2008. picture is worth thousand words: Using visual images to improve comprehension for middle school struggling readers author(s). In The Reading Teacher.
Petros J. Katsioloudis. 2007. Identification of quality indicators of visual-based learning material in tech- nology education programs for grades 7-12.
Limitations and Ethics Statement
While automatically enhancing textbooks with im- ages holds promise, we point out:
Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Ha- jishirzi. 2017. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. 2017 IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 5376-5384.
Image selection bias: Images from the web are at risk of being biased because they do not nec- essarily come from the same distribution as text- book graphics. However, images from Wikipedia are possibly more suitable for this purpose be- cause they are of encyclopedic nature.
Dong Won Lee, Chaitanya Ahuja, Paul Pu Liang, Sanika Natu, and Louis-Philippe Morency. 2022. Multi- modal lecture presentations dataset: Understand- ing multimodality in educational slides. ArXiv, abs/2208. 08080.
Intellectual property: Practitioners who use our automatic image assignment method for text- books should take care always follow the asso- ciated copyrights and attributions.
Pedagogical usefulness: While we employed workers to intrinsically judge the quality of the assignments, the results should be replicated with an extrinsic evaluation (beyond the scope of our study) which also considers the impact on student learning and information retention.
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common objects in context. In European Conference on Computer Vision.
I László Lovász. 1983. Submodular functions and con- vexity. In Mathematical programming the state of the art, pages 235-257. Springer.
Quality control: The target audience of text- books are students, who are sensitive group. In the current formulation, the optimization will always produce some assignment but there is no mechanism for quality assurance. This could re- sult in inappropriate images being assigned and expert human scrutiny should be employed.
Richard E. Mayer. 2019. Multimedia learning. Visible Learning Guide to Student Achievement.
George L. Nemhauser; Laurence A. Wolsey, and Mar- shall L. Fisher. 1978. An analysis of approximations for maximizing submodular set functions i. Mathe- matical Programming, 14:265-299
References
Saurabh Panjwani, Luana Micallef, Karl Fenech, and Kentaro Toyama. 2009. Effects of integrating digital visual materials with textbook scans the classroom. International Journal of Education and Development using ICT, 5:55-71.
Rakesh Agrawal, Sreenivas Gollapudi. Anitha Kannan, and Krishnaram Kenthapadi. 2011. Enriching text- books with images. In Proceedings of the 20th ACM International Conference on Information and Knowl- edge Management, CIKM '11, page 1847-1856, New York, NY, USA.
Alee Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In ICML.
Rakesh Agrawal, Sreenivas Gollapudi, Krishnaram Ken- thapadi, Nitish Srivastava, and Raja Velu. 2010. En- riching textbooks through data mining. In Proceed- ings of the First ACM Symposium on Computing for Development, ACM DEV '10, New York, NY, USA.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-shot text-to-image gen- eration. In Proceedings of the 38th International Conference Machine Learning. volume 139 of Proceedings of Machine Learning Research, pages 8821-8831.
Russell N. Carney and Joel R. Levin. 2002. Pictorial illustrations still improve students' learning from text. Educational Psychology Review, 14:5-26.
Kostas Dimopoulos, Vasilis Koulaidis, and Sklaveniti. 2003. Towards an analysis of visual images in school

---

Mrinmaya Sachan, Avinava Dubey, Eduard H. Hovy, Tom M. Mitchell, Dan Roth, and Eric P. Xing. 2020. Discourse in Multimedia: A Case Study in Extract- ing Geometry Knowledge from Textbooks. Compu- lational Linguistics, 45(4))627-665.
Coef.
SE
T-Val.
p
intercept
-0.279
0.020
-13.753
concepts
-0.003
0.002
-1.179
concept_mentions
0.005
0.001
7.519
words
0.001
0.000
27.405
paragraphs
0.011
0.003
3,879
Mrinmaya Sachan, Kumar Dubey, and Eric Xing. 2017. From textbooks to knowledge: A case study in har- vesting axiomatic knowledge from textbooks to solve geometry problems. In Proceedings of the 2017 Con- ference on Empirical Methods in Natural Language Processing, pages 773-784, Copenhagen, Denmark. Association for Computational Linguistics.
Xsec_concepts
-0.001
0.001
-1.065
meetoos
0.002
0.001
2.417
Xsec_words
0.002
0.002
1.591
Xsec_ Asec_paragraahh
0.002
0.002
1,438
posi
-0.001
0.000
-5.630
subject_math
0.260
0.021
12.201
subject_science
0.520
0.018
28.817
Mrinmaya Sachan and Eric Xing. 2017. Learning solve geometry problems from natural language demonstrations in textbooks. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics ( (*SEM 2017), pages 251-261, Vancouver, Canada, Association for Computational Linguisties.
subject_social
0.152
0.022
6.83
**
Table 3: Regression analysis for predicting the number of images in a subsection. The following variables are significant predictors: concept_mentione paragraphs, position, and subject of a subsection. SE is standard error, ** is < 10 and * is p < 0.05.
words,
Florian Schneider, Özge Alaçam, Xintong Wang. and
Chris Biemann. 2021. Towards multi-modal text-
A Annotation Guidelines
image retrieval to improve human reading. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Studenl Research Workshop, Online. As- sociation for Computational Linguistics.
The goal of your task is to help us evaluate research Artificial Intelligence (AI) based methods for creating more engaging and effective textbooks for students. You will be evaluating the performance of various Al-based methods that use images obtained from the internet to assign images to sections of a children's 's textbook.
Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm. 2015. Solving geome- try problems: Combining text and diagram interpre- tation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1466-1476, Lisbon, Portugal. Association for Computational Linguistics.
In the first stage your evaluation, you will be provided with text from section of the textbook that has been divided into multiple subsections. Each subsection will be enclosed within a bounding box and will include images at the end. Your task to evaluate the relevance, usefulness, and type of each image with respect to the text and images only within the same subsection.
After you finish answering questions in the first stage, new questions will appear with all the images (before and after shown in below and above image). these will be about second stage of evaluation. In the second stage of the evaluation, you will be reviewing the same subsections and images as before. Your task is to evaluate the relevance, redundancy and usefulness of cach image in relation to the text and images within all the subsections in this section. Here, you will he asked answer the new questions. Note: Please make sure to read the subsections that do not have any images during the first stage, as they will be useful for answering the questions the second stage.
Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Mare Najork. 2021. WIT: Wikipedia- based image text dataset for multimodal multilingual machine learning. Proceedings of the 44th International ACM SIGIR Conference on Re- search and Development in Information Retrieval.
Junke Wang. Dongdong Chen, Zaxuan Wu, Chong Luo,
Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu, Yu-
Choices concerning relevancy questions:
Gang Jiang, and Lu Yuan. 2022a. OmniVL: One
relevant: some concept deserihed in the subscction/section
foundation model for image-language and video- language tasks.
text is picturised in the image;
somew hemehhtt-releantt image is loosely on the same sub- ject/topic domain as the subsection/scctiinn text, but not exactly about the same concepts;
Wenhui Wang, Hangbo Bao, Li Dong, Johan
Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal,
irrelevant: not at all related to the subsection/section text.
undecided: unable to make a decision, maybe because of my low understanding of the text or image.
Owais Khan Mohammed, Saksham Singhal, Subho-
jit Som, and Furu Wei. 2022b. Image as a foreign
Choices for redundancy questions:
language: BEiT pretraining for all vision and vision-
yes: this image illustrates exactly the same concept as one of the previously shown images in the section/subsection: no: this image illustrates atleast some unique concepts as compared the previously shown images the sec-
language tasks.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hock-
enmaier. 2014. From image descriptions to visual
tion/subsection:
undecided: unable to make a decision, maybe because I do not understand the concepts illustrated by this image.
denotations: New similarity metrics for semantic in- ference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78.
Choices for didactic sisiicuuuuueesss questions:
yes: image will be appropriate for teaching the section to student - I would like it here (with an appropriate caption) if I was lcarning from this textbook;
Yan Zeng, Xinsong Zhang, and Hang Li. 2022. Multi-
grained vision language mreitraning: Aligning texts with visual concepts. ArXiv, abs/2111.000 10.0776.
no: image is not appropriate for teaching the scetion text; undecided: unable make the decision, maybe because of my less understanding of the text or image."

---

B Images Assigned with Joint
Section "Engagement in Democracy" from text-
book "American Government 3e"
Subsection "Why Get Involved?" - -Fig. 11
Sub. "Pathways to Engagement" - -Fig. 13
Subsection "Factors of Engagement" Fig. 14
Section "Systems of Gas Exchange" taken from textbook "Biology 2e"
Yang 2000
Subsection "Learning Objectives" - Fig. 16.
Subsection "Direct Diffusion" - Fig. 19.
Subsection "Skin and Gills" Fig. 20.
Subsection "Tracheal Systems" - -Fig. 23.
Subsection "Mammalian Systems" - -Fig, 17.
Figure 13: Image assignments for the "Pathways to En- gagement" subsection in "Engagement in a Democracy" section of textbook "American Government 3e".
Sub. "Lungs: Bronchi and Alveoli" Fig. 24.
Section "Stokes' Theorem" taken from textbook
"Calculus Volume 3"
Subsection "Stokes" T - Fig. 18.
Tumout U.S. Midtern
Subsection "Stokes" Theorem Proof" Fig. 25.
Subsection "Interpretation of Curl" -Fg.. 15. Section "Corporate Law and Corporate Respon- sibility" taken from textbook "Business Ethics" Subsection "The Advantages of Corporate Sta- - 12.
350 195 1527 150 1901 1951 2013 2015
Subsection "Balancing the Many Responsibili- ties of a Corporation"
Figure 14: Image assignments for the "Factors of En- gagement" subsection "Engagement a Democracy" section of the textbook "American Government 3e".
Fig. - 21.
Subsection "The Two Sides of the Corporate
Responsibility Debate" Fig. 22.
E
T, ('))
fangrath Floeing the Semecetic and Repbizen Darcideas
F
tee ind
FB
To
B7
t
t
60% 18C
Figure 15: Image assignments for "Interpretation of Curl" subsection in the "Stokes' Theorem" section of the textbook "Calculus Volume 3".
Figure 11: Image assignments for the "Why Get In- volved?" subsection in "Engagement in a Democracy" section of the textbook "American Govermment 3e".
Regiinatrry Syaters
WATCH THE FORDS BY
WRIGLEY FIELD HOME OF CHICAGO CUBS
Allstatee
BUSELL MOTOR CAR CO.
Figure 12: Image assignments for the "The Advantages of Corporate Status" subsection the "Corporate Law and Corporate Responsibility" section of the textbook "Business Ethics".
Figure 16: Image assignment for the "Learning Ob- jectives" subsection in the "Systems of Gas Exchange" section of the textbook "Biology 2e".

---

HOBBY LOBBY
Figure 21: Image assignments for the "Balancing the Many Responsibilities of Corporation" subsection in "Corporate Law and Corporate Responsibility" sec- tion of the textbook "Business Ethies".
Figure 17: Image assignment for the "Mammalian Sys- tem" subsection in the "Systems of Gas Exchange" sec- tion of the textbook "Biology 2c".
aho Iroffcdd by Soikily Pecerndf wi- MatusTardd
Hellings
4/2
Figure 18: Image assignments for the "Stokes' Theo- rem" subsection in the "Stokes" Theorem" section of the textbook "Calculus Volume 3".
Figure 22: Image assignments "The Two Sides the Corporate Responsibility Debate" subsection in the "Corporate Law and Corporate Responsibility" section of the textbook "Business Ethics".
(windpipe)
Pleura
(lung lining)
Lung
Pleural
Figure 19: Image assignments for the "Direct Diffusion" subsection in the "Systems of Gas Exchange" section the textbook "Biology 2e".
Figure 23: Image assignments for the "Tracheal Sys- tems" subsection in the "Systems of Gas Exchange" section of the textbook "Biology 2e".
POTOTY Royh
Bronchisc
Figure 20: Image assignments for the "Skin and Gills" subsection in the "Systems of Gas Exchange" section the textbook "Biology 2e".
Figure 24: Image assignments the "Lungs: Bronchi and Alveoli" subsection in the "Systems of Gas Ex- change" section of the textbook "Biology 2e"..

---

retrieved in assigned in the same subsection as gold-images for that particular subsection. Therefore, for each sub- section, the Image Bank categorized into gold- images and not gold-images. To gauge the retrieval quality of a subsection from a given retrieval ap- proach, we use these metrics. Ultimately we use the average of all of them.
fix cotx
A
R
10
ORCC path: AAA.SRAAEYYAA
Recall@ K: fraction of
top-K retrievals.
Recall@ R: fraction of gold-images retrieved top-R retrievals; where R is the number of gold- images in the subsection.
Precision @ K: fraction of retrieved images (K in total) that are gold-images.
Figure 25: Image assignments for the "Stokes" Theorem Proof" subsection in the "Stokes" Theorem" section of the textbook "Calculus Volume 3".
Precision @ R: fraction of retrieved images (R total) which are gold-images; where is the
number of gold-images in the subsection.
C CLIP Fine-tuning
average rank of each gold image given relevancy sorting.
Mean Gold Rank:
The CLIP model is composed of two parts: (1) the Image Encoder responsible for encoding the pt input image into a 512-dimensional vector Ip: and (2) the Text Encoder that encodes the 9th input text into 512-dimensional vector Tg. The model is trained with contrastive loss a dataset of 400 million image-text pairs from the web. During the training, relevant image-caption pairs maximize T, while Ip minimize for unrelated In and Tę. During our tneettuning we create hini-b image-text pairs extracted from the subsections in OpenStax Books Dataset.
C.2 Zero-Shot CLIP
In the experiments discussed of this section, we use a pre-trac tre-trained CLIP (without fine-tuning) to fetch relevant images for each subsection. The experi- ment results are presented in Table 4. For inference, adopt similar approach to the one described earlier. Below, we elaborate on the main differ- ences in experimental settings for these studies:
1. Concepts: We use all the concepts from a sub- section as its texi-queries (each concept is a separate text-query) and use mean for AGG.
During inference, we encode all images in the Image Bank as (2,, In) and all text- queries belonging
2. Cuustered-Coneepte We use the concepts from a subsection as its text- queries (each con- cept is a separate text-query). Next, we form clusters of text queries using k-means (k- 10) clustering on text encodings. For inference, we apply AGG the relevance scores P to text- queries belonging each cluster. Using the cluster's aggregated relevance score, we retrieve one image at time from each cluster a round- robin fashion. This experiment tests giving equal importance to various "concepts-clusters" leads to increased variation in retrieved results and better performance.
particular subsection as (Ti, Tas)- Next, we get similarity scores for all images in the Image Bank with respect to the thth text-query by calculating s = (UITT.. Iz. Tk...., IN Tk). We then normalize Sk to get pk = SOFTMAX(S*) Finally, for each subsec- tion, we compute the relevance scores of all images in the Image Bank by aggregating the pk values, resulting in P- AGG(PI (Pl, p2 ..... PM,, where AGG is an aggregate function such as the mean of N- N-dimensional vectors.
In search for the best image retrieval model, we provide details for various methods of creating image-text pairs from a subsection.
Saraateennted-Conepttsss 3. Concatenated- Concepts: We concatenate dif- ferent concepts from subsection together and use these concatenated phrases as the text- queries for each section and use mean as the ogg function. This experiment tests giving more context (multiple terms provide better con-
C.1 Evaluation Metries
We now explain the techniques for the retrieval model evaluation. We consider the images initially

---

text about the section content) with input text- queries improves the performance.
Precision
Recall
Gold
Exp #
Rank
@11 685 6920 100
eR
61
885 6:20 dr 100
oRR
4. Phrases: We follow the same setting as Exp C.2.1, except with overlapping phrases (each phrase a separate text-query) drawn from sub- section's is raw-text instead of the concepts.
0.64
9.68
18.53
22529
C.2.2 2.07 1.88
1.22 0.54 1,96 0.75
3.35
7.83
15.76 1.96 22973
C.2.3 6.34
3.36
1.79
0.67
3.76
2.23
6.52
12.48
20.99
3.76 19236
C:24 2.52
1:35 0.60 2,99
1.50 8.66 17:47
17325
Table 4: Zero-shot CLIP results. Precision and recall scores (within the range of 0 to 1) are normalized scale of O to 100 for better readability. The worst possible gold rank approximately 312K.
C.3 Fine-tuned CLIP
We now finetune CLIP with the experiment results are shown in Table 5.
Concepts: Fine-tuning: Let (i1, 12,-..., in) and (41.4.....,4m) respectively be the set of images and concepts included in sub-
Precision
Gold
Eap
Split
G1
65
@20
@000
QR
61
85
@20
1:00
GR
Prain
4.35
2,58
1.50
0.65
3.08
1.58
4.38
9.27
18.89
3.08
22720
17.98
section from OpenStax Books Dataset. We fine-tune CLIP using the following such 'cor- reet' image-text pairs from all the sections in the train split of OpenStax Books Dataset: ((1,)1) (11, 12),..., ... (1., fm), ... (in, ft),.................... . (Ias t. Inference: We follow the exact same setting as Exp C.2.1 for inference; except that we compute scores across different data splits.
fest
6.34
3.36
1.79
0.67
3.76
2.23
6.52
12:48
20.99
3,76
19236
2,81
1.01
14,85
16526
C.3.1 der
3.89
2,40
1.27
0.48
2.47
0.86
2.94
744
13.31
2.47
13786
3.03
1.05
1,00
16.21
2.57
14248
Aroin
7.05
2.41
5.15
2.61
15.15
29.12
5.15
C.3. dev
2,46
2,044
1.56
0.772
1.94
0883
3.70
9:00
20.45
1.54
89066
.,.,, (11, fm), (12, (1); (12, 12),
20.07
8.21
4,46
2.43
1.02
5.74
8.19
1.77
15.47
29.30
5.74
5169
C333 dev
5.26
3.72
2.14
1,46
12.26
25.47
3.82
Jevt
4.96
3.97
2.00
0.85
3.80
1.69
7,41
13.79
26.02
3.50
6126
prale
3.96
2.57
1.00
0.79
3.11
1.53
4.55
10.34
22.78
3.11
0071
0.64
21.85
3.01
1:70
1.21
0.64
1.71
0.95
2.82
8.02
19.22
1.71
5095
2. Concatenated Concepts: We follow the same setting Exp C.3.1 except that, during both fine-tuning and inference, we concatenate con- cepts together form text-queries for encoding (like in Exp C.2.3).
train
4.72
3,06
1.87
0.85
3.59
1,64
11.31
24.04
3.59
6315
C3355 dev
4.56
2:25
1.58
0.72
3:23
1.79
3.57
9.89
20.12
3.23
8509
test
3.29
1.97
1:26
0.63
2.65
1,35
3
7.95
18.84
2.65
8539
train
C.3.6 der
2,81
2,46
1.51
0.73
2.20
0.64
345
8:25
20.21
2.50
8000
2.47
1.14
18.49
2.23
Table 5: Results of experiment using Fine-tuned CLIP. Metrics shown here follow same format as in Table 4. The first multirow shows the zero-shot CLIP results from Exp C.2.1 and serves comparison baseline.
3. Phrases: We follow the same setting as Exp C.3.1, except that, both during fine-tuning and inference we use overlapping phrases drawn from subsection's 's raw-text instead of the con- cepts.
4. Pre-fine-tune: We pre-fine-tune the CLIP model on image-caption pairs from the Wikipedia images dataset. Following this, we perform fine-tuning and inference the same as that described in Exp C.3.2.
information, rather than using cach concept as a separate text-query; (c) pre-finetuning on image- caption from the Wikipedia images; (d) fine-tuning all layers of CLIP, opposed to only fine-tuning the last few layers. Ultimately, we select the model from Exp C.3.3 as our retrieval model, even though Exp C.3.5 achieved a slightly better gold-mean rank. This because Exp C.3.5 relies on Wikipedia image caption pairs, which may not be readily avail- able for all web images.
5. Frozen Encoders: We freeze the parameters of both text and image encoders except the last linear layers. Next, we carry out fine-tuning and inference same Exp C.3.2.
6. Frozen Encoders: We freeze the parameters of both text and image encoders except the last linear and self atffnattention layers. Next, we carry out tuning and inference same as Exp C.3.2.
Based on our experimentation, using results on the test split, we determine that the gold-mean rank can be improved through the following means: (a) fine-tume fine-tuning CLIP, as it outperforms zero-shot; (b) concatenating concepts to provide more contextual


Reviews:
[{"review_id": "oAuAk8ocMw", "reviewer": "Reviewer_XsLy", "summary": "", "questions": "Did you consider doing comparison between using finetuned clip vs pretrained clip?\n\nAlso, please elaborate sliding window approach in line 303 e.g. size of window etc.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and for acknowledging that the contributions of this paper will help researchers jumpstart work in this area.\n\n**Q1**: Yes, we provide a comparison between pre-trained and fine-tuned CLIP in sections 8.2 and 8.3 of the appendix. The results clearly suggested that the mean gold-rank (retrieval rank of gold-images for a subsection) with the fine-tuned CLIP is consistently better, even across various ways in which the input text is formulated.\n\n**Q2**: CLIP is pre-trained on image-caption pairs with a text limit of 77 tokens. We split the subsections' text into overlapping windows of 75 tokens, as mentioned in Footnote 3. This approach maintains the text granularity consistent with CLIP's pre-training. Moreover, overlapping windows ensure concepts are presented in diverse contexts. For instance, _\"photosynthesis involves plants absorbing sunlight\"_ offers more visual cues than _\"this process is called photosynthesis\"_. Because of this, instead of splitting the original text ABCDEFHI into non-overlapping chunks (ABC, DEF, GHI) we use overlapping chunks (ABC, BCD, CDE, …). We will highlight these details in the updated version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1: The paper is well written, well organized, easy to follow (except the math in lines 365 to 390 which was very dry but it may appeal to other audience). \n\n2: The neatly curated dataset containing texts, phrases, intelligently derived key concepts and images. \n\n3: Invaluable Indepth analysis of data asking key questions that help design a working solution for anyone that uses this data. \n\n4: A fair assessment of their performance. I was not expecting their approach to match or beat human performance, but the performance of their 'joint' optimization was not too far from it either. I can see this work be easily extended in the setup where \"Accept images for sections where automated setup is highly confident, and use human to rate or identify images for subsections where the automated setup fails or is not as confident\" . This would still save valuable human time by (semi)automating the process.\n\nOverall, this paper is a good introduction to the task, shares a useful dataset with its own analysis that will help other researchers jumpstart work in this area. I recommend we publish this.", "reasons_to_reject": "N.A.", "questions_to_authors": "Did you consider doing comparison between using finetuned clip vs pretrained clip?\n\nAlso, please elaborate sliding window approach in line 303 e.g. size of window etc.", "paper_topic_and_main_contributions": "This paper formalizes the problem of enhancing educational text with images from the web. The key contributions of the paper are as follows: \n\n1: Formalizes the task beyond simply calling it similarity matching. Gives it a local and global perspective. \n2: Creates a dataset of various educational topics. \n3: Examines points of failures and areas of improvement in this work.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N.A.", "Typos_Grammar_Style_and_Presentation_Improvements": "N.A.", "Justification_for_Ethical_Concerns": "N.A."}, "extra_text": {}}, {"review_id": "hgyiyjXmgw", "reviewer": "Reviewer_Ymez", "summary": "", "questions": "How did the authors made sure about the quality of annotation ? It will make the paper stronger.\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for highlighting the contributions of this work and for your constructive feedback.\n\n**Q1**: We measured the micro-averaged inter-annotator agreement between 32 annotators across all quantitative answers (12,900 pairs) in two ways: (a) partial match - answers within 1 point of the annotation scale; and (b) exact answer match. Further, we annotated a portion of the assignments ourselves, covering five sections, each on a different subject. The annotators' agreement (82% for partial and 52% for exact) closely matched the agreement from our annotations (81% for partial and 57% for exact), suggesting the annotators' annotations were of good quality. We will include these details in the updated version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Authors provided a detailed explanation of their datasets through some good insights about concepts and image distribution\n\nThey also tried to understand if image assignment has a correlation with different features (length of subsection, concepts, etc) \n\nThe output shows that the results are interesting and the error analysis provides insights on where the improvement can be made for future. \n", "reasons_to_reject": "I did not find any concrete reasons to reject the paper", "questions_to_authors": "How did the authors made sure about the quality of annotation ? It will make the paper stronger.\n", "paper_topic_and_main_contributions": "The paper creates a database for textbook section to image selection. Authors used openstax as well as wikipedia images for creating the image dataset. Authors started with dataset insights and then proceeded to image assignment problem and error analysis. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fN3BoTiT63", "reviewer": "Reviewer_r6d1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating the design of our study and for acknowledging potential benefits of our work in educational AI and multimodal learning!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed dataset is carefully collected and curated, which benefits our community.\n\n2. The analyses of the datasets are thorough, validating the usefulness of the datasets.\n\n3. The image assignment procedure is well-designed, which can probably assign semantic-relevant images to texts.\n\n4. Extensive evaluation also validates the contribution of the dataset and assignment algorithms.\n\n5. If this dataset is open-sourced as the authors claim, the research areas, including educational AI and multimodal learning, could benefit.", "reasons_to_reject": "I see no reason to reject this paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper studies an interesting topic of how to leverage visual images from the web to enrich the information of textbooks and help people develop a comprehensive understanding of them. The authors 1) curate a diverse dataset from textbooks and various web-source images, 2) conduct a thorough analysis of the constructed dataset, 3) sophisticated design an assignment procedure to assign collected images to the most relevant texts, 4) conduct comprehensive evaluations and analyses on the constructed datasets and assignment algorithms. Each step is of reasonable and constructive design supported by thorough analyses and statistics. The full paper presents a comprehensive and convincing study of \"Enhancing Textbooks with Visuals from the Web\".", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In general, all reviewers agreed that this research is both sound and exciting.

**Summary of Reviewer Feedback and Discussion:**
- **Reviewer XsLy** appreciated the neatly-curated dataset and in-depth analysis thereof, as well as the paper's fair performance assessment.  They were curious about a few finer details in the paper, which the authors provided in the rebuttal and promised to include in the revised manuscript.  The reviewer then thanked them for providing these details.
- **Reviewer Ymez** liked the detailed explanation of the dataset and the authors' careful investigation of the underlying task.  They also appreciated the thorough evaluation, and were curious about a few details pertaining to annotation quality.  The authors provided these details in the rebuttal, and promised to include them in the updated manuscript.
- **Reviewer r6d1** liked the carefully-curated dataset and thorough analysis thereof.  They also thought that the authors' methods were well-designed, and that the authors had extensively evaluated their work.  They felt that if the data is made publicly available, it may benefit numerous research subfields.  In the rebuttal, the authors thanked them for their appreciation of the study and acknowledgement of these potential benefits.
Decision: Accept-Main