A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding
Authors: Andrea Burns
Krishna Srinivasan
Joshua Ainslie
Geoff Brown
Bryan A. Plummer
Kate Saenko
Jianmo Ni
Mandy Guo
Venue: EMNLP 2023 Main (2023)
Keywords: Webpage Understanding
Multimodal Data
Text Generation
TL;DR: We present a multimodal webpage understanding dataset with a suite of generative tasks and a novel attention mechanism which utilizes the available modalities.

Abstract: Webpages have been a rich, scalable resource for vision-language and language only tasks. Yet only pieces of webpages are kept in existing datasets: image-caption pairs, long text articles, or raw HTML, never all in one place. Webpage tasks have resultingly received little attention and structured image-text data left underused. To study multimodal webpage understanding, we introduce the Wikipedia Webpage suite (WikiWeb2M) containing 2M pages with all of the associated image, text, and structure data. We verify its utility on three generative tasks: page description generation, section summarization, and contextual image captioning. We design a novel attention mechanism Prefix Global, which selects the most relevant image and text content as global tokens to attend to the rest of the webpage for context. By using page structure to separate such tokens, it performs better than full attention with lower computational complexity. Extensive experiments show that the new data in WikiWeb2M improves task performance compared to prior work.

A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding
Andrea Burns
Krishna Srinivasan
Joshua Ainslie?
Geoff Brown? Mandy Guo?
Bryan A. Plummer' Kate Saenko 1.3
Jianmo Ni
Boston University, Google, $FAIR
{aburns4, bplum, baenkoobbu edu, (krishnaps, jainslie, geaffbrown, jianmon, xyguo)@google.com
Abstract
WEBPAGE
SUMMARIZATION
Bl definiionn suczulent plants one diought-mmitttn plants in which the leaves, stem, or roces have become more than ususlly flsshy by the development of water storing..
Webpages have been a rich, scalable resource for vision-language and language only tasks. Yet only pieces of webpages are kept in ex- isting datasets:
CONTEXTUAL IMAGE CAPTIONING
image-caption pairs, long text articles, or HTML,, never all in one place. Webpage tasks have resultingly received little attention and structured image-text data left un- derused. To study multimodal webpage under- standing, we introduce the Wikipedia Webpage suite (WikiWeb2M) containing 2M pages with all of the associated image, text, and structure data'. We verify its utility on three generative tasks: page description generation, section sum- marization, and contextual image captioning. We design a novel attention mechanism Prefix Global, which selects the most relevant image and text content as global tokens to attend to the rest of the webpage for context. By using page structure to separate such tokens, it performs better than full attention with lower computà- tional complexity. Extensive experiments show that the new data in WikiWeb2M improves task performance compared to prior work.
A collection of suco ulett plants. including cact, from the Jardin bbotarlque di Ezo. France
PAGE DESCRIPTION GENERATION
In boteryy succulant plants also known mucculents, plants with parts that (hickered, flaslig and engorged, usuilly retain water anid climates asil conditions. The word succuient cerees from the Latin word sucus, meaning "Juioe" "sop". Sucoutent plants store water in various structures, such lcaves and (temss, The water content. of some succulent organs can up to 90 80%...
Tasks study with WikiWeb2M. Our dataset provides a unified webpage sample that contains all text, image, and structure, enabling new tasks like page de- scription generation. For image captioning and section summarization, remaining page text and images provide useful context, aiding task performance.
Figure I:
Yet limited prior work has studied tasks to evaluate multimodal webpage understanding itself.
1 Introduction
Many classification and generation problems can be studied with webpages: taxonomic webpage classification, webpage retrieval, web image cap- tioning, and page summarization. However, to date there is no open source, multimodal dataset that retains all webpage content. F.g., the Wikipedia Image Text (WIT) dataset (Srinivasan al., 2021) does not retain HTML structure and misses out on text sections. Thus, we propose the new Wikipedia Webpage (WikiWeb2MM) dataset of over 2M pages, which unifies webpage content to include all text, images, and their location (e.g., section index) single sample. Table 1 compares the statisties of WikiWeb2M to the existing English WIT dataset. Figure| shows an example of our WikiWeb2M benchmark suite; we design a set of tasks that re- quire webpage understanding at varying degrees of
Webpages are source of multimodal, structured content which have been used for both pretraining and finetuning purposes. Large scale noisy text or multimodal datasets scraped from the web have been used to pretrain large language or contrastive models (Raffel et al., 2020; Jia et al., 2021; Rad- ford et al., 2021; Aghajanyan et al., 2022). Down- stream tasks built from webpages have included instruction following, image captioning, news cap- tioning, image-sentence retrieval, and image-article article retrieval (Shi et al., 2015; Li et al., 2020; Gur et al., 2022; Sharma et al., 2018; Biten et al., 2019; Liu al., 2021; Srinivasan et al., 1021; Tan et al., 2022).
WWrk done during an internship at Google.
'Data can be downloaded at https:// thbbbhggtttttutt///// com/ www.eeece-daaasseeeeeeeeeeeeeeee http:////tttttttt//toottst / iiwieem...
md.

---

# Webpage Sections
# Images
Dataset
Structural Heading
Text
Image
Both
Total
Unique
Total
WIT (En)
3,660.211 4,955,835 6,817,950 221,523 3,236,254 11,693,497 4,438,642 5,940,431
199,872 2,847,929 3,047,801
WikiWeh2M
731,394 686,376
Table I: WikiWeb2M versus WTT (Srinivasan et al., 2021). WikiWeb2M re-introduces millions of text and multimodal webpage sections. We report counts over all splits; train, validation, and test are reported separately in Appendix A. WikiWeh2M and WIT (English subset) come from the same webpages.
granularity. Specifically, we use page description generation, section summarization, and contextual image captioning to evaluate model's ability to un- derstand a webpage at global, regional, and local level, respectively. For page description genera- tion, the goal is to generate an overarching global description of the webpage. The task of section summarization generates sentence that captures the key content of one section. Finally, contextual image captioning generates caption for one image within the webpage.
which uses our structured webpage data to select the most salient text and images global tokens in the prefix of our input sequence. Prefix Global ultimately more efficient, meaning longer input sequences can be used to reach better task perfor- mance. Our results can be beneficial to the many structured image-text domains outside of webpages such as mobile apps, figures, posters, infographics, and documents.
We include ablations across multiple axes: the pretrained checkpoint initialize from, the in- put sequence length, the feature inputs, and the attention mechanism. We importantly find that im- ages improve performance for all tasks, while prior work on contextual image captioning claimed oth- erwise (Nguyen et al., 2022). We are also able to improve task performance now that we have access to the entire page's 's content. Still, there is plenty of room to improve upon benchmark suite. We summarize our contributions below:
WikiWeb2M's tasks will allow for general study of multimodal content understanding with many- to-many text and image relationships and can also specifically improve interaction with web content. For example, webpage description may provide user who is blind more agency by allowing them to preview content before listening the entire body of image and text with screen reader (Vtyurina et al., 2019). addition to contextual captioning and section summarization aiding assistive technol- ogy, these tasks can be used for modern content generation, as there is growing interest in providing multimodal web snippets (Nkemelu et al., 2023).
A new open source multimodal webpage dataset, WikiWeb2M, containing 2M pages curated from English Wikipedia articles. Each sample contains all text, images, and structure present per page.
The study of webpages in multimodal context has even been motivated from sociological and anthropological perspective (Pauwels, 2012).
suite of multimodal generation webpage tasks that reflect webpuge understanding at three granularities: page description, section summarization, contextual image captioning.
While we curate a new dataset with Wikipedia, we note it is just one of many domains that could be used to study multimodal webpage understand- ing. Instructional websites, news articles, recipes, blogs, and more have bodies of text and images interleaved by layout or HTML structure.
A attention mechanism, Prefix Global, which mixture of local-global attention that separates prefix of global tokens. By defining more salient content from structured pages, it can outperform full attention while requiring fewer attention computations.
We utilize the T5 (Raffel et al., 2020) framework to address the WikiWeb2M tasks. One challenge in modeling webpage tasks is the length of the input data (i.c., long sequence results from flattening webpage text and images). While the full attention originally used in T5 is performant, it results in quadratic computational complexity with respect to the input sequence length. Thus, we define a new mixture of local- locallglobal attention, Prefix Global,
Ablations on attention, sequence length, input features, and model size. Images can help all tasks, notably by over 15% on contextual captioning, and page context boosts average performance by over 4% and 3% for section summarization and captioning, respectively.

---

Wiki
Train
Val
Test
Task
Train
Val
Test
# Pages
1,803,225
100,475
100,833
Page Desc. 1,435,263
80,103
80,339
# Sections
10,519,294
585,651
588,552
Section Summ.
3,082,031
172,984
173,591
Total Images
5,340,708
299,057
300,666
Image Caption. 2,222,814
124,703
124, 188
Table 2: Breakdown of the number of pages, sections, and images contained in each WikiWeb2M dataset split.
Table 3: Number of samples for page description gen- eration, section summarization, and image captioning task datasets after additional filtering of WikiWeh2M.
2 The WikiWeb2M Dataset
show the number of pages. sections, and images per split in Table 2. Note that Table 2 reflects statistics of WikiWeb2M, which is later refined to build our downstream tasks datasets. can be repurposed for other webpage understanding tasks or reprocessed with different data filters.
We create the Wikipedia Webpage (WikiWeb2M) dataset to have an all-in-one multimodal webpage dataset where all text, image, and structure content is retained. WikiWeh2M is built by starting with the Wikipedia Image Text (WIT: Srinivasan et al., 2021) English pages?. We re-scrape webpage sam- ples and keep all text, image, and structure avail- able, providing more contextual data which can be used to model existing tasks like contextual image captioning, as well as enable new webpage un- derstanding tasks like page description generation. We start with WIT URLs to create a high quality multimodal webpage dataset that has already gone through extensive content and topic filtering.
2.1 The WikiWeb2M Tasks
We apply WikiWeb2M to three tasks which reflect different granularities of webpage understanding: the page, section, or element level. Table 3 contains task sample counts which are achieved by further task-specific filtering; these data processing steps are included in Appendix A.I, with a discussion of potential dataset noise in Appendix A.2. We describe the tasks below.
Each webpage sample includes the page URL, page title, section titles, section text, images and their captions, and indices for each section, their parent section, their children sections, and more. This differs from WIT, which defined individual samples as image-caption pairs with metadata (e.g., originating section title.. Appendix A.3 includes a comparison of fields available in WikiWeb2M versus WIT. In Table 1, we report the number of sections and images compared to the English sub- set of WIT. We add nearly 1M total images to the dataset by keeping the images on a webpage regard- less of whether they have image captions available.
Generation. In the task of page description generation, the goal is to generate a de- scription of a page given the rest of the webpage's image, text, and structure. We use the Wikipedia- provided page description (not collecting annota- tions) and generate summaries from multimodal inputs, which differs from existing text-only article summarization work; this matters when we want to create a multimodal snippet from a webpage.
Page Description
Section Summarization. The goal of section sum- marization is to generate single sentence that high- lights particular section's 's content. The summary is generated given all images and (non-summary) text present in the target and context sections; see Figure 3 for a task example. Following the lead- ing sentence bias, we use the first sentence of a section as a pseudo summary (which is removed from the model inputs). We also found that a major- ity of human annotators deemed the first sentence as a reasonable summary; these findings are later discussed in Appendix F.
We provide section counts by type: structural, heading, text only, image only, and both text and image. Structural and heading sections do not con- tain immediate text. The former has subsections. For heading sections, section title available, while the content linked different article, was empty, or only had tables. We only retain sections if they are content sections (e.g., not the "See Also" section). A significant 6.8M text sections are in WikiWeb2M, none of which were available in WIT. For image quality control, we keep JPEG and PNG image types?. We make random 90/5/5 split and
Contextual Image Captioning. Nguyen et al. (2022) proposed contextual image captioning with WTT as the task of captioning an image given the image and its webpuge context. Target images are those available in WIT to ensure they have quality
'WIT held a CC BY-SA we recover WikiWeb2M is publicly available on Wikipedia, "Wo release image URLs, where they can be fetched.
3.0 license, and additional data

---

Prior Work: Transient Global Attention
Ours: Prefix Global Attention
Ateention keys.
Atterticn keys
glufoal
bokons
atlend thersselves plus fecal
Amerroo queeits
Artestinn qualtss
Each to its locai neighhehhood plas all todoms.
Haci inpui token attends focu! neighborthood plus to all giohal tohens.
sequence length
longts
Global
Remsining Token
Each global toker the resui everaging every appt tokens
&
Sau
Globsls lokett are the finst & inpat tnkens whichh we cenvider our prefisg k
LaverNorm
Figure 2: Local-global attention schemes. On the left we show Transient Global (TGlobal), which has local to local and local to global attention (Guo al., 2022). We propose the new Prefix Global attention which additionally has global global and global to local attention compared TGlobal. We define global tokens fixed-length prefix of the input, unlike TGlobal which defines additional global tokens that are aggregates over the full input sequence.
captions that be reconstructed. Wikipedia im- age can have three caption types (not all are always available): the alt-text, reference, and attribution descriptions. Alt-text serves as a text description for accessibility purposes, the reference description comes directly below the image in the rendered webpage, and the attribution description contains captions unique to the image across all webpages it appears in. Prior work only input the image, attribu- tion description and associated section text because that was all that was available.
input sequence, while others only have local atten- tion radius of tokens to the left and right. Not only is it desirable to prioritize more salient image and text content from the input data, but it can also reduce the computational complexity of the attention mechanism. While full attention is performant by allowing all input tokens to attend to each other, it results in quadratic computational complexity (O((²) for sequence of length I). Figure illustrates our Prefix Global and prior work's ' Transient Global attention schemes, where in each the ith row represents what the ith token can attend to. Guo et al. (2022) introduced LongT5 as an adaptation of the T5 model with Transient Global (TGlobal) attention to balance the efficiency of local attention, which allows for much longer put sequences be held memory, with the higher performance of full attention. TGlobal resulted similar or better performance than full attention with much longer sequence lengths, while having complexity of O(I(r + k)) for a variety of text summarization tasks (where
3 Prefix Global Attention
When structured image-text data available, we need not treat all images and text equally. With webpages, it may be more sensible to isolate cer- tain parts as more important. E.g., in contextual image captioning, the model should focus on the target image and section it came from, while us- ing the rest of the page as additional context. We can now isolate these inputs with the WikiWeh2M dataset because we have structural metadata signal- ing where each image and text element are located, as opposed to a bag of images and single long body of text. Le.,, the new structure available in our dataset can both serve as new inputs to the model and enable new attention mechanisms.
1/)
In addition to the local attention of TGlobal (see the blue attention diagonal in Figure 2 (left)), "tran- sient" global tokens are defined on the fly per layer. TGlobal defines & globals as the average of every 16 input tokens, which are additionally attended to by all other inputs. As a result, TGlobal has more global tokens the sequence length increases. In contrast, shown in Figure/ (right), Prefix Global uses a constant number global tokens. Specifi- cally, it takes a prefix of the input sequence. This is
We thus propose Prefix Global, local-global attention, capitalize on this intuition. A mixture local and global attention weights provides the means to designate certain inputs as "global" to- kens which can specially attend to the rest of the

---

CLOBAL TOKENS
REMAINING INFUT TOKENS
SECTION TEXT: Assanding ug the Bouth around a dird sus0 suoz.lert...
SECTION CAPTIONS: Succurente.
SECTION TEXT: In bakenr susculant pantts suscufants, plants with parts Fiat are.
SECTION CAPTIONS plants have thickenod stems...
SECTION INDEX
SECTION TITLE
SECTION INDEX
SECTION TITLLE Succurent Phant
these Adromischus mar fariee
Muttmodal Encocar Pnoffx Giobal Attantion
Dacsder
Other than in Antarctica, succu ents can be found within each continent.
Figure 3: WikiWeb2M section summarization with Prefix Global. With WikiWeb2M,, we can now use section structure separate the most relevant webpage content. The global tokens of Prefix Global (in green) are the first 512 tokens of the target section to be summarized: the first x images of the section, the section index, title, body text, and captions. Then the remaining sections (in blue) from the webpage are input; these have local attention, while the prefix global tokens attend every other token. We decode the summary (in orange) given the page inputs.
inspired by the leading sentence bias (Kedzic et al., 2018; Xing et al., 2021; Zhu et al., 2021), which shows that earlier content in body of text is often of greater importance. We define different prefixes for each task in Section 4. While we use section structure to define prefixes, Prefix Global can use structure from other sources: HTML/the Doc- ument Object Model, rendered webpage regions, PDF document layouts, or simply knowing a priori what task inputs are most salient.
and we embed images in our input sequence using a frozen ViT model (Dosovitskiy et al., 2021). We note that finetuning ViT may further improve per- formance. We compare three models defined by different encoder attention schemes: the original T5 which uses full attention, LongT5 with TGlobal attention by Guo et al. (2022) (checkpoints are pub- licly available), and our Prefix Global attention. We finetune all models from a T5 checkpoint pretrained with full attention on the text-only C4 dataset, and a ViT pretrained on either ImageNet (Deng et al., 2009) or JFT (Hinton et al., 2015).
Prefix Global has computational complexity of O((I k) - + k 1) for l global tokens, similar to local-global attention schemes ETC (Ainslie et al..
Parameter Settings. We finetune each model for 28 steps as done by Raffel et al. (2020) with a 128 batch size. Each model is trained on 16 TPUs, with the base model taking between 24-32 hours to run't (varies by task) with input sequence length of 1024. We do not perform hyperparameter tuning: all models use the Adafactor optimizer with a constant learning rate of 1e-3, an Adafactor offset of IM to account for pretraining steps, and loss normalizing factor of 218, For Prefix Global experiments, the default prefix size k is 512. For both Transient Global and Prefix Global, the local attention neighborhood is set to 127, done in Long (Guo et al., 2022).
2020), Longformer (Beltagy al., 2020), and Big- Bird (Zaheer et al., 2020). However, Prefix Global does not require any special pretraining and instead finetunes directly from full attention checkpoints (T5 in our case). This is distinct from L5ngT5, which also required pretraining with TGlobal atten- tion to be effective. Thus, as we show in Section 5 with Prefix Global's higher performance, it is both more flexible and performant attention. We also are the first to demonstrate using a local-global at- tention with multimodal inputs, and further show Prefix Global's s ability to be performant in multi- modal finetuning from text-only checkpoint.
Metrics. For quantitative results, report BLEU- 4 (Papineni et al., 2002). ROUGE-L (Lin, 2004). and CIDEr (Vedantam et al., 2015) metrics from a single run. BLEURT (Pu et al., 2021), CLIP- Score and RefCLIPScore (Hessel et al., 2021) are additionally reported in Appendix C for all results in the main text. We include qualitative results in Appendix B; we perform two qualitative studies
4 Experiments
We now detail the model variants used for experi- ments, parameter settings for reproducing our set up. the metrics used for evaluation, and key abla- tions perform.
Model Architectures. We henchmark with the T5 (Raffel et al., 2020) encoder frame- work. T5 takes a sequence of image and text inputs
'Example packing can further improve model efficiency.

---

(1) inspect the quality of generated text for all finetuning tasks and (2) discuss when and why im- ages may help the more text-based tasks of page description generation and section summarization.
URL, title, and remaining sections follow in order, Figure illustrates how an input sequence is de- fined with Prefix Global for section summarization.
Contextual Image Captioning, Similar section summarization, the target image and its originating scction's ' content contribute to the prefix tokens (the index, title, text, and non-target captions), followed by the URL, page title, and context sections.
Ablations. We compare each attention at differ- ent input lengths. Our sequence length ablations also include experiments where Prefix Global and TGlobal have the same number of global tokens to strictly compare how they define global tokens. Then we ablate webpage inputs (section text, ti- tles, structure, images, image captions) and use the best feature combinations for any remaining exper- iments. We run experiments with different model sizes (B16 or L16 T5 + ViT$) for Prefix Global at Ik input sequence length. Lastly, verify that WikiWeb2M's new annotations improve per- formance over prior work. Specifically, we ablate if the target, description, or context sections are input and if sections only from WIT vs. WikiWeb2M are input (since many text and multimodal context sec- tions were not originally kept in the WIT dataset).
5 Results
We now detail experimental results, first evaluating performance and efficiency of cach attention type at different sequence lengths. Then, we report input feature, model size, and annotation ablations.
5.1 Attention and Sequence Length
Performance Comparison. We begin by evaluat- ing performance for each task (page description, section summarization, and contextual image cap- tioning) when training T5 encoders with different attention types and input sequence lengths in Fig- ure 4. Prefix Global always performs better than TGlobal. We include two Prefix Global settings: a fixed Prefixs12 which sets 512 input tokens the prefix (default used for all other experiments), as well as a PrefixTGiolal which assigns the same num- ber of global tokens as TGlobal. PrefiXTGlobal uses 16 globals, where / is the input sequence length (TGlobal aggregates every 16 input tokens as a global token). This allows to compare the way both attention mechanisms define global tokens.
4.1 Defining Prefix Global Attention Inputs
Each sample's images are always included as part of the input's prefix tokens. We ablated the number of images that contribute to each task's prefix and include ablations in Appendix D.3. We use six images for page description and one image input for section summarization and image captioning.
We describe each task's prefix below. Note that we remove the text that serves as the target sum- mary caption from our inputs to the model for each task; this ensures there is model "cheating." E.g., for section summarization, since we utilize the first sentence of section as its pseudo target sum- mary, we remove from the inputs to the model.
Despite TGlobal defining additional side inputs as global tokens, it consistently underperforms Pre- fix Global even with the same number of globals. This confirms that defining a special prefix from the input sequence is better than taking aggregates over the full sequence. In Appendix D.1. we also show that just using the prefix of the in-order page inputs for page description (as opposed pulling out the section titles and first sentences) performs better than TGlobal. These results collectively show Pre- fix Global to be preferable to TGlobal. One key takeaway is that separating out more relevant inputs (via structure or other known biases like leading sentence bias) is a good idea.
Page Description. We input the images, page URL, page title, and all sections (index, title, text, cap- tions) in their structured page order. In addition to the images, URL, and page title participating in the prefix, we also include all section titles and section first sentences (up 512 tokens). This outperformed keeping the section titles and text concatenated in order; see Appendix D.
Section Summarization. The target section to be summarized is prepended to each sample's 's input se- quence. This means the target section's 's index, title, non-summary text, images, and captions contribute to the global tokens of Prefix Global. Then the page
Full attention and Prefix Global generally have higher performance at longer sequence lengths. It is impressive that Prefix Global scales or maintains performance with larger sequences even when its number globals is fixed to 512 (i.e., the num- ber of globals is not scaled with respect to input
"The basc/large T5 model used 220M/770M parameters.

---

Image Captioning
Page Description
38.0
29.8-
37.8
29.0
RONGEL Score
29.4
29.2-
he
ROUGE
30.5
29.0
37.2
28.0
30.4
37.0
28.6
38.2
1024
2048
4096
1024
2040
4096
1024
2043
4096
Input Seeg pence Length
Input Sequuence Length
Inpul Sequence el lenghh
Figure 4: Encoder attention and sequence length experiments. We use Prefix Global, TGlobal, and full attention at 1k, 2k, and 4k sequence lengths. Our experiments verify that Prefix Global is more performant than prior local-global attention TGlobal, and can even be more performant than full attention at long sequence lengths. Note that full attention at the 4k sequence length does not fit into memory. ROUGE-L is plotted.
Input Length
Attention Mechanism
page description, Prefix Global outperforms full attention at the 4k sequence length, while full at- tention cannot fit in memory. Given that the entire page's content can be useful for generating a page level description, it is sensible that full attention may perform better for smaller sequence lengths it allows for attention between all input tokens.
TGlobal
Prefix Global
Full
1024
325,632
916,480
1,048,576
2048
782,336
2,225,152
4,194,304
4096
2,088,960
4,842,496
16,777,216
Table 4: The approximate number FLOPs for each at- tention ignoring the # of attention heads and embedding dimension (both are the same for each attention). As sequence length increases, Prefix Global requires much fewer computations than full attention.
Efficiency Comparison. Prefix Global can outper- form full attention, while only requiring O(( k) . rr+ + k 1) attention complexity for global tokens. When implementing the Prefix Global at- tention, we manually created tensors representing block sparsity to avoid computing the full cross attention. We provide the approximate number of FLOPs for each attention mechanism in Table 4 when ignoring the number of attention heads and embedding dimension. At the 2k input sequence length Prefix Global requires about half the FLOPPs of full attention, and experimentally takes about half the time to complete the same experiment with all other settings fixed. The number of FLOPs of Prefix Global at 4k is just over those of full atten- tion at the 2k input length, and is able to fit into memory and maximize performance for each task. Lastly, the full attention and Prefix Global FLOP difference grows with sequence length. This can sometimes be seen experimentally: performance gaps are larger between full and Prefix Global for page description at 2k vs. 1k (0.20 vs. 0.09).
length). On the other hand, while TGlobal scales the number of globals to sequence length, its per- formance does not consistently scale. E.g., perfor- mance plateaus even drops at 4k input sequence length for page description and section summariza- tion, respectively. This may be because TGlobal defines globals as aggregates over the full input se- quence, which could introduce more noise or less semantically rich text at longer sequence lengths. One anomalous result occurs for image caption- ing: Prefix Global with 256 globals (PrefiXTGlobel at 4k input length) outperforms the 512 variant; as we did not exhaustively ablate the number of global tokens, further performance gains could be reached by optimizing the number of globals per task.
Prefix Global outperforms full attention at all se- quence lengths image captioning, which may be due to the global tokens including the target image and most relevant section content. This should ease the process of learning the most relevant tokens by allowing full attention between the first l target section tokens with the rest of the input sequence, while contextual information from other sections has local attention. For section summarization and
5.2 Feature Ablations
We investigate the role of each input feature with Prefix Global attention and fix sequence length 1k. Starting with just the text available from web-
"Structure features are kept if they were helpful in text- only experiments. Lee, they are included for page description and image captioning, but not for section summarization.
Full attention will OOM at 4k input length.

---

Feature Inputs
Page Desc.
Section Summ.
Image Caption.
Text
Title
Struct Caption Image
B
R
C
B
R
C
B
R
C
13.60
37.75
77.12
9.48
28.35
65.75
9.83
33.00
133.70
13.63
37.88
77.97
9.78
29.14
68.90
9.84
33.40
135.30
14.07
37.96
77.88
8.70
29.24
69.19
10.15
33.38
135.10
13.12
38.43
81.19
10.08
29.23
69.45
9.90
33.57
136.03
13.22
38.38
81.38
9.51
29.22
69.24
10.03
33.69
137.07
13.16
37.96
78.39
9.31
29.20
69.19
11.74
37.46
156.34
14.00
38.50
81.49
10.12
29.43
69.89
11.84
37.69 158.19
Table 5: Feature ablations with WikiWeh2M. We ablate the section body text, title, structure, captions, and images. Utilizing multimodal inputs results in the best performance for all tasks, We report BLEU-4 (B), ROUGE-L (R) and CIDEr (C) metrics.
ViT
Metric
when we include captions for the image caption- ing task, it refers to context captions from other images in the page that never serve target im- ages. Interestingly, this boosts performance. We suspect contextual captions help the model learn the style of captions we aim to generate.
Task
Model
Data
B
C
im21k
14.00
38.50
81.49
Base
Page
JFT
13.25
38.49
82.02
Desc.
im2 lk
14.67
39.63
88.90
Large
JFT
14.56
39.56
88.48
im21k lk
10.12
29.43
69.89
Base
5.3
Pretrained Checkpoint and Model Size
Section
JFT
10.15
29.40
70.03
Summ.
im21k
11.10
30.61
76.87
In Table 6, we perform additional experiments with VIT pretrained on JFT and large T5/VIT models. Unsurprisingly, larger models result in better per- formance. For page description and section sum- marization, scaling the model size results in larger performance gains than the impact of any individ- ual feature we ablated. On the other hand, model size has smaller gains for image captioning com- pared the impact of our feature ablations; the worst to best performance gap changed by an av- erage of 17.66% for feature ablations and only by 2.43% for model size, where we average the perfor- mance delta of BLEU-4, ROUGE-L, and CIDEr.
Large
JFT
11.24
30.54
76.92
im21k
11.84
37.69
158.19
Base
Image
JFT
11.66
37.35
156.01
Cap.
im2 lk
12.51
38.05
162.31
Large
JFT
12.08
37.33
158.81
Table 6: Pretrained model checkpoint ablations. vary the size of the T5 and VIT models (Base means both T5 and ViT are base-sized models, Large means both are large models) and which image dataset the VIT model was pretrained with (ImageNet or JFT-300M).
page sections, incrementally add section titles, indices and special tokens defining section struc- ture (the struct column of Table 5), the captions of images within each section, and the images. Each input boosts performance except section structure which has mixed results; for multimodal experi- ments we include these extra tokens if they helped in the text-only experiments. This may be due to these extra tokens consuming global tokens in the prefix that otherwise could have been more useful.
Preference to VIT representations pretrained on JFT ImageNet varies by task: section summa- rization tends prefer JFT, while page description and image captioning consistently perform best with large ImageNet trained representations.
5.4 Comparison to WIT Annotations
The proposed WikiWeb2M is a superset of WIT. For the same set of webpages, we unify all sections into webpage sample and reintroduce millions of sections and images that were not kept in WIT. Table 7 contains runs when using the original WIT data, the WIT data reprocessed to join the page sec- tions it originally contained, and our WikiWeb2M. For section summarization, the page description is more important than the other context sections. The page description may be more generally rele-
Images and their captions both improve perfor- mance, but result in the highest performance for each task when used in tandem. This illustrates that even when text captions are available, having their visual counterpart is beneficial. In Table
"BLEU-4 is less consistent than ROUGE-L and CIDEr.

---

Input Section Type Context
Section Source
Metric
Task
Target Description
BLEU-4
ROUGE-L
CIDEr
8.90
27.82
60.20
Section Summarization
WikiWeb2M
9.46
28.86
66.67
10.12
29.43
69.89
WIT
10.92
36.21
148.53
WIT 11.21
36.63
150.98
Image Captioning
WIT 11.45
36.88
152.69
WikiWeb2M
11.84
37.69
158.19
Table 7: Section input ablations. We vary using the target section, page description, and context sections. For image captioning, we also vary whether the sections come from the smaller WIT or our WikiWeh2M superset we do not run this ablation for section summarization, show that using all section types and the annotations made newly available with WikiWeb2M improve performance.
would result in different number train/valitest samples. Results
vant to all sections, while each section to be sum- marized contains distinct topic compared to the context sections from other parts of the webpage. Lastly, we find WikiWeb2M's s additional context sections improve captioning performance the most compared to those already available in WIT (com- paring the last two rows of Table 7). This confirms the importance of the annotations in Wiki- Web2M compared to those available in prior work.
from the web containing the images and HTML. of a webpage, this dataset was not open sourced. Their results illustrated that rich HTML data could be used to learn representations for tasks such as image generation, image in-filling and entity dis- ambiguation and linking. This demonstrates that webpage data can generalize to non- non-webpage tasks, but leaves webpage specific problems unexplored.
To our knowledge there is no open souree mul- timodal webpage data that captures all modalities. C4 was recently extended a multimodal version, MMC4 (Zhu et al., 2023). However, MMC4 does not retain structure, and instead uses CLIP scores to noisily match images to chunks of text that it could be aligned with. MMC4 has not yet been used for pretraining downstream upplications. In mobile apps, the closest domain webpages, there are two open source datasets that contain all modalities (text, image, and structure): Rico (Deka et al., 2017) and MoTIF (Burns et al., 2022).
6 Related Work
Webpage tasks have been studied with text only HTML for web element classification, HTML de- scription generation, and web navigation. Gur et al. (2022) proposed finetuning Large Language Mod- els for these tasks. Reinforcement Learning meth- ods have also trained agents to perform language commands in handcrafted web environments (Gur et al., 2019; Liu et al., 2018; Jia et al., 2019).
Wikipedia has previously been used to develop downstream tasks. For example, WIT (Srinivasan et al., 2021) released image-caption pairs from Wikipedia, in addition to some contextual section text. While WIT does not contain all of the page content, Nguyen et al. (2022) studied contextual image captioning with the available annotations. This is a webpage task and not strictly an image- text problem, as additional section text is included aid in Wikipedia image captioning, where cap- tions often contain finer-graned, knowledge based information. AToMiC also studied ways to improve multimodal retrieval with Wikipedia by defining more realistic evaluation sets (Yang et al., 2023). Aghajanyan et al. (2022) proposed CM3, Transformer with a causally masked pretraining objective. While CM3 relied on pretraining data
Conclusion
In this paper we study three generative tasks for multimodal webpage understanding: page descrip- tion generation, section summarization, and con- textual image captioning. To do so, we present the WikiWeb2M dataset, which retains all of the text, images, and structure from more than 2M pages. We propose a new attention, Prefix Global, which outperforms full attention by allowing the most salient text and images to specially attend all in- puts. Extensive ablations on attention mechanism. sequence length, model size and checkpoint, input features and section type reveal the most impactful factors on our benchmark suite and verify using WikiWeb22 to study webpage understanding.

---

Limitations
Acknowledgements
This work is supported, in part, by the Google Ph.D. Fellowship program.
The WikiWeb2M dataset reprocessed the webpages available in WIT. We begin with only the English subset of WIT, while it originally contained 108 lan- guages. Our dataset is limited to English and does not cover the vast multilingual data on Wikipedia. We can extend our dataset to cover all languages in WIT, but acknowledge it is monolingual date.
References
Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, and Luke Zettlemoyer. 2022. Cm3: A causal masked multimodal model of the internet. arXiv:200.075200
For page description generation and section sum- marization, we use pseudo summaries that are read- ily available from Wikipedia pages. While this is desirable from a scalability perspective and is practiced in other works, it can limit the evalua- tion quality of these tasks. However, did per- form small scale pilot collect human annota- tions for the section summarization task in which we asked the annotators if the first sentence suf- ficed; 94% of the time the majority vote out of five was yes. Pseudo summaries have also been used for other tasks like summarizing instructional videos (Narasimhan et al., 2022).
Joshua Ainslie, Santiago Ontanon, Chris Alberti, Va- clav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020. ETC: Encoding long and structured inputs in transformers. In Empirical Methods in Natural Language Processing (EMNLP).
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. 20X1v:200..05150.
Ali Furkan Biten, Lluís Gómez, Marçal Rusiñol, and Dimosthenis Karatzas. 2019. Good news, everyone! context driven entity-aware captioning for news im- ages. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
For the model settings we explore, we did not try all exhaustive combinations of features, attention mechanism, model configuration, and input length. We also only use T5 variants, but note T5 is state- of-the-art for generation style problems. Lastly, we design our set of fine-tuning tasks for generative tasks. Our work currently does not include tasks like webpage taxonomy classification webpage retrieval, but additional tasks like topic classifica- tion could be performed with WikiWeb2M.
Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A. Plummer. 2022. A dataset for interactive vision language navigation with unknown command feasibility. In European Conference on Computer Vision (ECCV).
Biplab Deka, Zifeng Huang. Chad Franzen, Joshua Hi- bschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017. Rico: A mobile app dataset for building data-driven design applications. In User Interface Software and Technology (UIST).
Ethics Statement
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In The IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR).
While the Internet provides a vast and rich do- main to collect data from, it also has potential risks. Wikipedia is a highly curated and monitored knowl- edge base of articles, but can be edited by the public, which can create potential quality risks. Ad- ditionally, Wikipedia is a largely fact-based domain, where incorrectly summarizing an article could re- sult in misinformation. We hope our dataset can be used as a new resource to improve the accuracy and factual correctness of text generation machine learning models. As we use Wikipedia data, there is no user data nor P.I.I. in the proposed WikiWeb2M dataset. Additionally, we ran analysis to remove a small subset of pages with potentially sensitive topics (e.g., natural disasters, funeral, blood).
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition scale. International Conference on Learning Representations (ICLR).
Mandy Guo, Joshua Ainslie, David Uthus, Santiago On- tanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2022. LongT5: Efficient text-to-text transformer for long sequences. In Findings of the Association for Computational Linguistics: NAACL.

---

Izzeddin Gur, Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin Huang, Aakanksha Chowdhery, Sha- Narang, Noah Fiedel, and Aleksandra Faust. 2022. Understanding html with large language mod- els. arXiv:2210.03945.
Medhini Narasimhan, Arsha Nagrani, Chen Sun, Michael Rubinstein, Trevor Darrell, Anna Rohrbach, and Cordelia Schmid. 2022. TI; dw? summarizing instructional videos with task relevance and cross- modal saliency. In European Conference on Com- puter Vision (ECCV).
Izzeddin Gur, Ulrich Rucckert, Aleksandra Faust, and Dilek Hakkani- Tur. 2019. Learning to navigate the web. In International Conference on Learning Rep- resentations (ICLR).
Khanh Nguyen, Ali Furkan Biten, Andres Maila, Lluis Gomez, and Dimosthenis Karatzas. 2022, Show, in- terpret and tell: Entity-aware contextualised image captioning in wikipedia. arXiv:2209.104 10474.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recogni- tion. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Daniel Nkemelu, Peggy Chi, Daniel Castro Chin, Kr- ishna Srinivasan, and Irfan Essa. 2023. Automatic multi-path web story creation from a structural article. arXiv:2310.02383.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021, CLIPScore: reference- free evaluation metric for image captioning. In Em- pirical Methods in Natural Language Processing (EMNLP).
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. Association for Computational Linguistics (ACL).
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in neural network. arXiv:1503.02531.
Luc Pauwels. 2012. A Multimodal Framework for Ana-
lyzing Websites as Cultural Expressions. In Journal of Computer-
Mennputer-Mediated Mediated Communication (JCMC).
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoe V. Lc, Yunhsuan Sung. Zhen Li, and Tom Duerig. 2021. Scaling up vi- sual and vision-language representation learning with noisy text supervision. International Conference on Machine Learning (ICML).
Amy Pu, Hyung Won Chung. Ankur P Parikh, Sebastian Gehrmann, and Thibault Sellam. 2021. Learning Compact Metries for MT. In Empirical Methods of Natural Language Processing (EMNLP).
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. 20Xiv:203300002
Sheng Jia, Jamie Kiros, and Jimmy Ba. 2019. Dom- q-net: Grounded rl on structured language. In In- ternational Conference on Learning Representations (ICI.R).
Chris Kedzie, Kathleen McKeown, and Hal Daumé II 2018. Content selection in deep learning models of summarization. In Empirical Methods Natural Language Processing (EMNLP).
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020, Exploring the limits of transfer learning with unified text-to-text trans- former. In Journal of Machine Learning Research (JMLR).
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. 2020. Mapping natural language instruc- tions mobile UI action sequences. In Association for Computational Linguistics (ACL).
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypemymee, image alt-text dataset for automatic im- age captioning. In Association for Computational Linguistics (ACL).
Chin-Yew Lin. 2004. ROUGE:
package for auto- matic evaluation of summaries. In Association for Computational Linguistics (ACL...
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Her- nandez, and Perey Liang. 2015. World of bits: An open-domain platform for web-based agents. 34th International Conference on Machine Learning (ICML).
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tian- lin Shi, and Perey Liang. 2018 Reinforcement learn- ing on web interfaces using workflow-guided explo- ration. In International Conference Learning Rep- resentations (ICLR).
Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Mare Najork. 2021. Wit: Wikipedia-baase based image text dataset for multimodal multilingual machine learning. In International ACM Conference on Special Interest Group on Information Retrieval (SIGIR).
Fuxiao Liu, Yinghan Wang, Tianlu Wang, and Vicente Ordonez. 2021. Visualnews: Benchmark and chal- lenges in entety-aware image captioning. In Em- pirical Methods in Natural Language Processing (EMNLP).

---

Reuben Tan, Bryan Plummer, Kate Saenko, P. Lewis, Avneesh Sud, and Thomas Leung. 2022. Newsstories: Illustrating articles with visual sum- maries. In European Conference on Computer Vision (ECCV).
explicitly have "fist_of" in their URL to be list heavy-pages and remove them. We also remove pages with fewer than two rich sections to ensure there is enough content for a page description task to be appropriate.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image de- scription evaluation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
For page section serve a target section for the task of section summarization, we require to have at least five sentences, contain neither a table nor list, and not be the root section. We filter out the root because the root (first) section is often the page description, which we choose to keep as a distinct task.
Alexandra Vtyurina, Adam Fourney, Meredith Ringel Morris, Leah Findlater, and Ryen W. White. 2019. Bridging screen readers and voice assistants for en- hanced eyes-free web search. In International ACM Conference on Computers and Accessibility (AS-
Lastly for image captioning, we follow Nguyen et al. (2022) and use the reference description as the ground truth caption to be generated. However, unlike Nguyen et al. (2022), we do not input the attribution description for the target image to be captioned because it often heavily overlaps with the reference description (the reference description is used as the target text generate). We further discuss this design choice in Appendix E. Again, we only consider images that were originally in WIT as target images to be captioned to ensure quality captions. We also only keep target images to be images which have reference description of at least three words.
SETS).
Linzi Xing, Wen Xiao, and Giuseppe Carenini. 2021. Demoting the lead bias in news summarization via alternating adversarial learning. In Association for Computational Linguistics (ACL).
Jheng-Hony Yang, Carlos Lassance, Rafael Sampaio de Rezende, Krishna Srinivasan, Miriam Redi, Stéphane Clinchant, and Jimmy Lin. 2023. Atomic: An image/text retrieval test collection to support mul- timedia content creation. In International ACM Con- ference on Special Interest Group on Information Retrieval (SIGIR).
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tanon, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, et al. 2020. Big bird: Transformers for longer sequences. In Advances in Neural Informa- tion Processing Systems (NeurIPS).
We additionally note that in WikiWeb2M we re- lease all three types of captions for each image (the alternative-text, reference description, and attribu- tion description), although not all three are always available for each image. The alternative text cap- tion for images is used for accessibility purposes, and future work can focus on generating these de- scriptions, as opposed to the reference description.
Chenguang Zhu, Ziyi Yang, Robert Gmyr, Michael Zeng, and Xuedong Huang. 2021. Leveraging lead bias for zero-shot abstractive news summarization. In International ACM Conference on Special Interest Group Information Retrieval (SIGIR).
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. 2023. Multimodal c4: An open, billion-scale corpus of images interleaved with text.
A.2 Dataset Noise
Our dataset is built from the weh, being processed from raw HTML. Noise may exist in our dataset in the formatting of text, e.g., mathematical formulas may have additional formatting text around them. In building our task datasets, the only noise we may introduce to the best of our knowledge is the processing of first sentences. The first sentence is separated from each section for the section summa- rization pscudo summary. It is also used as part of the global inputs for page description generation. Specifically, the code used to parse the first sen- tence may prematurely split sentence from pe- riod that does not signal the end of the sentence. We did manually inspect samples carly on and found this to be rare (e.g., 96/100 random samples were
A Additional Dataset Details
A.1 Dataset Processing
We now provide additional details on the filters and data processing steps used to convert WikiWeh2M into our three downstream task datasets.
For page description, we retain a page from Wiki- Web2M if it is not list-heavy and contains at least two sections with image or text content that do not contain a list or table. A small subset of Wikipedia pages essentially lists"; consider pages that
*For example, https: //en, www.wwiiiddda.. www.ipedaa.og/vvikki.. org/wiki// List_of www.aaf_mmmassstttmmmmm the Uni ted States

---

split correctly). Additionally, the sentences which were split prematurely could still be valid stan- dalone sentences. Our released dataset does in- clude all raw text, so others can reprocess it as they see fit. Figure 5 includes a code snippet for the sentence preprocessor we use and Table 8 illus- trates the 4/100 prematurely split sentences from our random sample.
B Qualitative Examples
To qualitatively evaluate model's ability our suite of webpage understanding tasks, we include two qualitative analyses. First, report several random output samples for page description gener- ation, section summarization, and contextual image captioning in Tables 15-17. In Appendix B.I, we discuss our findings from these randomly sampled oulputs. Next, in Appendix B.2, we include sam- ple outputs whose metrics improved the most from the text-only model to the multimodal model, explore why images understanding tasks. In this second setting we in- vestigate samples for page description and section summarization, since these tasks do not obviously require images in the same manner as contextual image captioning.
A.3 Dataset Analysis
We provide side by side comparison of the fields we open source with the WikiWeb2M dataset com- pared to those pre-existing in WIT in Table 9. Note that in addition to the new fields we introduce, WikiWeb2M has different data processing which allows for a great number of sections and images to be retained, as seen in Tables 10-12. In the main text, Table provided the aggregate counts over all splits for each section type and the number of images in our dataset versus prior work WIT. Ta- bles 10, 11, and 12 provide the same statistics but now broken down for train, validation, and test sets, respectively.
be helpful for webpage
B.1 General Qualitative Examples
We start with our first analysis of randomly sam- pled outputs for all three fine-tuning tasks. Samples are selected from the test set.
B.1.1 Page Description Generation
Beginning with page description in Table 15, the target and predicted output text are provided for three random pages on the topics of the Horahora Power Station, Hedevig Lund, and Cape Nome.
In Figure 6, two WikiWeb2M dataset samples are visually illustrated. Specifically, the webpages the topics of Succulent Plant and the Aguas Livres Aqueduct shown on the left of the figure visualize the original webpage for illustration purposes, and on the right we show a subset of the fields available in our WikiWeb2M samples for these same pages.
For the first article on the Horahora Power Sta- tion, the predicted output text is quite coherent and well formed, despite containing some inaccurate details that conflict with the content of the web- page. Our model correctly references the date it was opened (1913) and the date the power station was flooded (1947). It also correctly references Lake Karapiro, which was formed and ultimately led to the submerging of the power station. On the other hand, the name of the "Waikato" River was swapped with "Waihi." The model also referred to Horahora as coal-fired station when it is actually hydroelectric power station.
For additional data analysis provide sequence length details. We include the median, average, 90th percentile, and maximum sequence length val- ues for all input fields that make up a samplc's 's in- put sequence in the train split. We define sequence length as the number of tokens after preprocess- ing and SentencePiece tokenization. See Table 13 for the sequence length of the page URL, title, de- scription, section title and text, and image captions available (the alt-text, attribution description, and reference description).
Next, for the shorter article on Hedevig Lund, we find the model prediction to be very close to the target page description, although the painter's last names are slightly incorrect. Upon inspecting the page text, it appears the model included additional last names from the painter's parents' names (Ole Wilhelm Erichsen and Abel Marie née Isaachsen).
Lastly, we provide aggregate high level statistics over the train split of WikiWeb2M in Table 14. This includes statisties on the number of sections per page (the number of total sections well the number of content sections, with the latter referring to sections which contain image or text content) and the number of images per page or section.
In future work, methods that use pointer networks or direct copying from input text can be used to ameliorate these named entity failures.

---

alphabets -'([A-2a-2))"
prefixes "(Mr|SSt |r MMMaDDr|Poot |
suffizes [[Inc]Ltd]] 3r |Sr (Ga)
"[Mr --A-)][----- (1:--]]0000---22111100 [A)2]
(It's (Teyyss (TeirsTsssssrssrrrrarar BBt'ss Howeeerr | That's This
acranyns "[AA-]I--EA-<JI
websites "C.I lnweo//eetttttteeeeeeoe lorg sov/ineleee (edu)
digits "(L0-91)"
PreprocessText(com text)::
text ag_taxt
replace(" text re sub(prefixes, "Wirard>", text)
text sub(rebsites, "<ardbW""" text)
sub(dtgits "[.3" digits, "WAspddbl12", text)
text ".sub(""s" alphabets [IJ ". Wisare text) sub(acronyns
starters, "Wistopp 112",
text re. s..baalphabeets "t.]" alphabets + "[.]" alphabets *[.J', "Atrprebllzprrhhaeapeee text)
s..abbaalpaaeess "[.]" alphabets '[.]', "W/<pra>\1<ppa>> text)
text re r.s(""" suffixes "[.] starters, Westoo 112", text) text se.s surtixes "L.J", "Wisppd>" text)
ru. text re. re.su((r" (vee).]((we)) "Wspprd1122 text) decirel nunbers
*[.J", Wrordd' text)
22
if text:
text rexxt.rpplace"" "*, "7.")
if "\"" text:
27
text tept.rrplann replace (""",, 'V"."
if
text:
29
text rett..celacee"VV "V"1")
if "?"
31
text text. replaceeeeeeeee... ("?V"", (V*?")
1f "e.g." text:
text text.replacee Peplace("e.... "esprd>g<prd=")
if "i.c." in text:
rext..epl ("i.e." "isprdheeprdb")
1f "..." text:
text replacee "ppro><prdoppr>>))
if "Ph.D" text:
text text, replacce(PPP.... "Ph<prdbd<prd>""
41 text text. replace(" "isstops")
replace("?", "Pestopo")
text text. replace("!" "lestap?")
text replaceee ("orrdd", ".")
return text 46
irstiesttenteecassegg..... texx))
28
""""plits body of text into its first sentence and the rensining text.
Cade nedified from https://staccoverfllos
53 Args:
text: ingut stringg
55 Returns:
first: The first sentence body of text.
57 rest: The renaining, rejuined sentences fron a body of text that fullow the first sentence.
text PreprocessTextCop.teee
sentences split("<stopp")
62 sentences santences[:-1] sentences [s.strip() for sentences]
first sentences[:]]
rest og, [
67
return first, rest
Figure 5: The code used sentence splitting. This preprocessing used to separate the first sentence the rest of the section body for section summarization targets and for page description generation global input tokens.

---

Full First Sentence
Processed First Sentence In 2007, Saade founding member of What's Up!
In 2007, Saade was founding member of What's Up!. Swedish boy band which included Robin Stjernberg, Ludwig "hejLudde" Keijser and Johan Yngvesson.
Unicamp offers over one thousand extension pro- grams to the community, with different levels of minimum requirements (high school degree, un- dergraduate degree, etc.) and across all areas of study, focusing mainly on specialization courses and community outreach.
Unicamp offers over one thousand extension pro- grams to the community, with different levels of minimum requirements (high school degree, under- graduate degree, etc.
The proposed biosynthesis of ascofuranone was reported by Kita et al., as well as by Abe et al. 1988 The choir sang the German premiere of Joseph Jongen's 's Mass for choir, brass ensemble and organ, Op. 130, which not yet in print then, both in the Stiftskirche of Aschaffenburg and in St. Bonifatius.
The proposed biosynthesis ascofuranone was reported by Kita et al. 1988 The choir sang the German premiere of Joseph Jongen's Mass for choir, brass ensemble and organ, Op.
Table Failures of our sentence processor. We found 4/100 randomly sampled sections had first sentences which were prematurely split. We share these captions here and find they still are fairly reasonable sentences. The "full" first sentence on the left is determined via manual inspection.
The Cape Nome article is another example with slightly longer page description (four sentences). This sample strongly illustrates the model's ability convey a factually accurate and topically rele- vant page description even when the output text does not entirely contain the same content as the target description. Both the target and predicted descriptions begin with an overall summary of the topic, followed by geographical information. Our model's generated text also provides some histori- cal context that is accurately summarized from the article, which the ground truth description does not. It seems our model attempts to summarize each of the sections on the page to form a coherent page description, which may differ from the target page description on Wikipedia (i.e., the Wikipedia page description need not cover topics from the entire webpage and can vary in style page to page).
the historical section. That being said, the histor- ical section does discuss that Futallaby is freely available, making the model predictions sensible, relevant, and factually correct.
In the second section summarization example on the topic of Jamaica's 's pirate economy, the target summary discusses Spanish resistance to English occupancy to provide context for the growing pirate cconomy. However, neither the target nor predicted section summary directly address the pirate econ- omy. The model prediction is mostly accurate with correct references to English occupancy in 1655, but implicitly refers Port Royal as a fort the foot of the Blue Mountains, which geographieally, slightly questionable.
The third section summarization sample con- cerns the science fantasy novel "Thuvia, Maid of Mars," written by Edgar Rice Burroughs. Our trained model correctly references Burroughs fin- ishing a novel by June 1914, but it was Thuvia, Maid of Mars he finished, not the book "Tarzan." The model seems have confused multiple facts relating this book: Thuvia, Maid of Mars was the fourth, not third, novel in the Barsoom series and Tarzan was not a part this novel series (although Burroughs did also write Tarzan).
B.1.2 Section Summarization
For the task of section summarization, we include links to the webpage and the target section to be summarized from the article, and the target and predicted text in Table 16. Starting with the histori- cal section imageboards, we find that the target section summary is slightly more section specific than the predicted summary. Iee., the model gener- ated summary "Futallaby is free and open-souree imageboard script" could be sections other than

---

WikiWeb2M Field
WTT Field
page_title
page_title title
page_url
page_url
raw_page_
raw_page_descripiiee description
context_page-e page_description
section_title
dection_title
section_text
section_image
image_url
iaatlon_maae_mmimgg type
@etage_mime_yee _yteeeeyeeee_e
section_image_width
original_width
section_image_herr
original_heigh
section_image_raw_ lartage___w_reee_ddee desc
lectoo___ffeene__deeeee
section_image_rrwe desc
section_image__alt_exxx
clcan_page_descrippe desescicccriitinnn
section_iiagge_clenn_ee___L
section_image__lcan_ _ess
aectionimmaee_cattiin
section_inde index
section_ras _aterrraw_____ Batti__aw____________ ist_sentence
Sectio__clann |st_teneeece
section_rest _ee_rr_seseteeeeeeeeeeeeeeeeeee_
section_depth
Section_heading
dection_subsectii index
section_paaie Bacioon__aent____ _iddex
page_contains_imee
section_ conttons images
split
is_page__dddee scctions_without without_tabe list
page_contentteee wet.ooonntttccctinsss.....
is sanrriion_summmaiittinnnnsaaa is laange__cattion_anaa samppl
Table 9: Dataset fields in our WikiWeb2M dataset versus the WIT dataset.
B.1.3 Contextual Image Captioning
woman. This image has more generic target cap- tion, and it appears that our model tends to prefer generating detailed captions. As result, contains factually inaccurate information, stating the paint- ing itself is of Rabindranath, the son of Maharshi Devendranath Tagore, who developed Santiniketan. Additionally, the generated caption states the mural is at the Ashram Complex in Santiniketan. While Santiniketan, it is not confirmed to be the Ashram Complex given the content of the article; while the article states "It [the Ashram Complex] has beautiful frescoes by Nandalal Bose," remains ambiguous.
Lastly, Table 17, we provide qualitative exam- ples for contextual image captioning. We include links to the webpage and image (as well as illus- trate the image within the table), plus the target and predicted image captions. In the first image from the Longpré-le-See commune in France, while the target caption describes the main road in the im- age, a church also present the end of the road. This is confirmed by another image on the web- page which shows the same church. Thus, while our model did not predict the same exact target caption, it is still visually and factually accurate.
the painting
The second image is a photo painting of a
Then, for the third randomly selected image from

---

# Webpage Sections
# Images
Dataset
Structural
Heading
Text
Image
Both
Total
Unique
Total
WIT (En)
179,769 199,165
2,562,275
2.742.044 10,519,294
3,183,132 3,867,277
4,456,169
WikiWeh2M
658,241
616,534
6, 134,0886
2,911,268
5,340,708
Table 10: Comparison of WikiWeb2M and the WIT (Srinivasan al., 2021) dataset. We report counts here for the train split of WikiWeh2M.
# Webpage Sections
# Images Unique Total
Dataset
Structural
Heading
Text
Image
Both
Total
WIT (En)
10,198
142.737
152,935 162,381 585,651
238,155 284,975
249,313 299,057
WikiWeb2M
36,410
34,274
341,310 11,276
Table 11: Comparison of WikiWeb2M and the WIT (Srinivasan al., 2021) dataset. We report counts here for the val split of WikiWeb2M.
the article on WWOR-TV, the model's generated caption quite accurate the image and also over- laps heavily with the content of the ground truth caption. The only subtle inaccuracy the pre- dicted text is that it states the TV logo was in use from the early 1970s to early 1980s, when it was actually used until the year 1987, which should be considered the late 1980s.
B.2.1 Page Description
Starting with page description generation, we in- clude three examples Table 18. For each page we link to the Wikipedia article and include the target page description, the page description generated by the text-only model (noted as text under the type column), and the page description generated by the multimodal model (noted as multi under the type column) for comparison.
B.2 Unimodal to Multimodal Qualitative Examples
Across these examples, we find one trend: im- ages in the webpuge can improve the generated description's ' specificity. In the page description task, we allow up to six images present on the page to be included. Starting with the first exam- ple from the webpage Joan Carling, we see that the page description output from the multimodal model touches upon more topics and specifies de- tails beyond a high-level sentence summary (the latter being more similar to the output from the text- only model). The multimodal model's generated text includes references to the Champions of the Earth Lifetime Achievement Award that was given to Joan Carling as well as greater detail about her relationship with the Philippines and being of the Igorot people. These events and concepts are both captured in the images in the page, which seem to help specify more detail from the page in the generated description.
We now select a random subset of test set outputs for page description and section summarization for the best performing text-only model and the best performing multimodal (image and text) model. These models are base size T5/ViT models, as that was the model size used to perform feature abla- tions. Specifically, we select samples which have a higher ROUGE-L score with the multimodal model. For random subset of 1000 samples, we reverse sort by the change in ROUGE-L between the uni- modal and multimodal models, looking to inspect the samples most positively impacted by the inclu- sion of images. We hope to understand the settings under which images can aid these tasks.
As noted previously, we include this analysis for page description and section summarization, since these tasks may not require images, while image captioning inherently uses the input image. These examples are included in Tables 18 and 19. We did find that some of the most improved samples were an artifact of the text-only model repeating text tokens many times (a common failure of text generation models) and do not include those in our examples.
Similarly, for the second example from the page on Alice Bemis Taylor, the text-only model gener- ated brief and high level summary about her. On the other hand, the images on the webpage portray numerous important locations (either her childhood home or community spaces she actively partici- pated in or founded such as the Colorado Springs Arts Center and Day Nursery). As a result of in-

---

# Webpage Sections
# Images
Dataset
Structural
Heading
Text
Image
Both
Total
Unique
Total
WIT (En)
9,905
142.917
152,822
238.924 286,390
250,353
WikiWeb2M
36,743
35,568
342,554
11,082
162,605
588,552
300,666
Table 12: Comparison of WikiWeb2M and the WIT (Srinivasan al., 2021) dataset. We report counts here for the test split of WikiWeb2M.
Sequence Length
Med
Avg
Max
Poo
detail (unlike the prior examples which tended to- wards too little). The two images on this page are centered around the respective meeting locations of the parliamentary in France and Germany, which helps focus the topic of the page description.
Page URL
16
17.40
89
23
Page Title
5.20
42
8
Page Description
110
122.27
695
289
Section Title
3.64
176
7
Section Text
119
212.87
62,418
523
B.2.2 Section Summarization
Image Alt-Text
4
5.58
1,433
14
Image Attribution
17
29.76
30,121
59
In Table 19 we now include section summarization examples for several pages and include links to the webpage and specific section to be summarized. Slightly different from the trend found for page description generation, we see that images can im- prove the topical relevance of the generated section summary.
Image Reference
8.20
3,640
27
Sequence length statistics for various Wiki- Web2M data fields. Sequence length is defined by the number of SentencePiece Tokens. We report the median, average, maximum, and 90th percentile sequence length values for the train split.
Table 13:
For example, with the webpage Johann Joachim Quantz, the target section summary for the section on the Court of Frederick discusses how Quantz joined the court as a flute teacher to Frederick II. The image in this section illustrates Frederick the Great playing flute, which directs the model to focus on this instrument. The text-only model fails to mention anything about instruments in the summary, only discussing interpersonal rela- tionships. On the other hand, the multimodal model mentions both the flute and specifically Frederick playing the flute, which depicted the included image and also explains the reason why Quantz joined the court.
Statistic
Med
Avg
Max
Poo
Total Sections / Page
4
5.83
987
12
Content Sections / Page
4
5.13
939
11
Images/ / Page
2.96
1,679
Images / Section
0
0.51
653
Table 14: Analysis for the number of sections and im- ages per page section in the train split of the Wiki- Web2M dataset. Here we consider content sections as section with text or image content within (as not only containing lists, tables, or heading only).
cluding these images, the generated description now included more specific information regarding these places. By including these images, their cor- responding captions and related textual concepts in the input sequence are more greatly attended to. This may be a byproduct of images always serving as global tokens with our Prefix Global attention mechanism.
Next, the section on the 1800s history with re- spect to the St. Peter Catholic Church (in Mont- gomery, Alabama) continues illustrate how im- ages improve topic relevance. While the text model references the year the church was founded, the multimodal model references the lo- cation, which the target summary also describes (although they reference the location of the church at different degrees of granularity). If a specific section does not have images, an image is included from the images in the chronological order they appear the webpage. This means that for this section, the included image is the first image on the page map of Alahama showing the location of the St. Peter Catholic Church.
Lastly, we include the descriptions generated for the webpage on the topic of the Franco-German Par- liamentary Assembly. Again this example demon- strates that images can help maintain the correct level of specificity for the page description. In this case, the images help refocus the description to be more high-level than the description gener- ated from the text model, which includes too much
A third example can be found with the P&T

---

Webpage
Target Text
Predicted Text
Horahora Power Station was an early hydro- electric power station the Waikato River in New Zealand. was the country's first large- scale power station, completed in 1913. Ini- tially built to service gold mine, the power station was expanded to supply a significant part of the North Island. The power station remained in use until was submerged by Lake Karapiro, which was formed to supply the larger Karapiro Power Station.
Horahora Power Station was coal-fired power station located the banks of the Waihi River in the Waikato region of New Zealand. was built 1913 and flooded in 1947 when Lake Karapiro was formed behind the Karapiro Power Station dam.
Horahora Power Station
Hedevig Lund
Hedevig Thorine Christine Erichsen Lund was a Norwegian painter.
Hedevig Lund née Erichsen was a Norwegian painter.
Cape Nome is a headland on the Seward Peninsula in the U.S. state of Alaska. It is sit- uated on the northern shore of Norton Sound, 15 miles the east of Nome also on Norton Sound. It is delimited by the Norton Sound the south, Hastings Creek on the west, lagoon on the east and an estuary formed by the Flambeau River and the Eldorado River. From the sea shore, Cape Nome extends in- land by about 4 miles, connected by road with Nome.
Cape Nome is a headland on the Seward Peninsula of Alaska, United States. It is lo- cated 129 miles to the south cast of the Bering Strait. The Cape Nome mining precinct was established in 1898 immediately after the dis- covery of gold at Golovnin Bay.
Cape Nome
Table 15: Page description qualitative examples. We include three random samples from the page description test set and compare the target page description and predicted model output text.
Group article and 1900s-19500 Growth and Expan- sion section. The target section summary discusses the opening of the Shanghai office. The summary generated from the text-only model discusses the P&T Group expanding its business, which while correct, does not discuss the connection between opening new offices and expanding business. On the other hand, with the help of the HSBC Shang- hai office image input to the multimodal, the multi- modal model generated description which explic- itly mentioned the office opening. However, due both Shanghai and Hong Kong locations being mentioned the section, the model confuses the two in the generated section summary.
generation.
D Additional Model Ablations
We now provide additional experiments that could not fit into the main text, which include data pro- cessing ablations, additional feature ablations, and further comparisons to prior work.
D.1 Page Description
We first performed ablations on the data processing of WikiWeb2M for the page description genera- tion task dataset. Specifically, tried varying the number of content sections required for a particular page to be retained. See Table for comparison of when we required two vs. three vs. four sections to contain image or text content without table or list. We found the added samples improve performance consistently (i.e., the most performant setting is when the number of content sections required per page is set two).
C Additional Metrics
For results reported in the main paper, we addition- ally report BLEURT, CLIPScore, and RefCLIP- Score (the latter two are only relevant for contex- tual image captioning). See Tables 20, 21, 22. All results trends stay the same except that BLEURT is insensitive to most of the feature ablation results for section summarization and page description
10 SSrrucure features are kept if they were helpful in text- only experiments. Lee,, they are included for page description and image captioning, but not for section summarization.

---

Webpage
Background Target Text
Section
Predicted Text
Futallaby is PHP script based the Futaba script from Futaba Chan- nel.
Futallaby is free and open-source imageboard script.
Imageboard
Historical
Spanish resistance continued for some years after the English con- quest, in some cases with the help of the Jamaican Maroons, but Spain never succeeded in retaking the is- land.
The English occupied the island of Jamaica in 1655, establishing fort at the foot of the Blue Mountains.
Jamaica's Pirate Economy
Colony of Jamaica
Burroughs began writing Thuvia, Maid Mars, in April 1914, at the time describing as a 'Carthoris' story.
By June 1914, Burroughs had com- pleted the third novel in the Bar- soom series, Tarzan
Thuvia, Maid of Mars
Table 16:
Section summarization qualitative examples. We include three random samples from the section summarization test set and compare the target section summary and predicted model output text.
We also allow for text only pages to be kept in the dataset, as there are a small subset (roughly 2% of pages) that do not have any images after our processing. This could be due the Wikipedia pages changing since the original WIT dataset was released, or because we only allow JPEG and PNG images while WIT contained some other image types like SVG. We include additional ablations in Table 24 showing the effect of including or not including these unimodal pages; their effect is min- imal given how few there are in the dataset.
rows 5-9 in Table 26). We verify in row 5 that when inputting only the image and no contextual text, it is incredibly difficult to generate strong captions for these images which contain a lot of fine-grained information. However, in support of the impor- tance of having both images and text inputs, we find that for every text-only to multimodal compar- ison (where all features are the same except images are included in the latter), the multimodal setting al- ways results in substantial performance gains. For example, quantitatively comparing row and row in Table 26, where either only the section text is input versus the section text and image to be captioned are input, the performance differences are: BLEU-4 9.83 vs. 11.27, ROUGE-L 33.00 vs. 36,90, and CIDEr 133.70 vs. 153.44.
In Table 25, we show ablations for our prefix design with the page description generation task. Including the section titles and first sentences of each section in the prefix as global tokens improved performance for majority of metrics, and we kept this set up for the rest of our experiments. We note that even when not using a specially designed pre- fix (i.e., flattening the section inputs and allowing the first 512 tokens to serve the prefix, not sepa- rating out section titles or first sentences), the Pre- fix Global attention mechanism still outperforms Transient Global. This follows the principal from leading sentence bias that earlier information in the input text is important. Thus, if you have priori knowledge that a particular part of the input is more important than others, separating it into the prefix of our attention mechanism can effective.
Again, this differs from the findings of Nguyen et al. (2022); their experimental design likely min- imized the impact of images because they also feed in the attribution description as a textual input, which often is quite similar to the target caption. As a result, the model can "cheat" and utilize the attribution description while not relying on the vi- sual input. We have more discussion regarding this in Appendix E.
D.3 All Tasks
For each task in our WikiWeb2M suite, also ablated the number of images input to the model. These additional ablations are shown in Table 27. Results in the main text use the 90th percentile num- ber of images per page, six, for page description generation, and only one for section summarization
D.2 Contextual Image Captioning
As image captioning inherently requires images, we performed additional feature ablations on the text features while always including the image (see

---

and image captioning. Here we also try the average value for number of images per page which is three. We include these ablations for contextual image captioning as well, as we were curious whether having contextual (non-target) images input to the model would help at all. Ultimately it sometimes hurt performance, likely adding noise and making it more challenging for the model to discern which input image was the target image to be captioned. We only used the single target image as input for the rest of our experiments.
marization task is a subpart of this problem and we proposed an improvement over the News, text- only Mall PEGASUS model used to generate summaries in the prior Wiki2Story work by Nkemelu et al. (2023). Specifically, our for- mulation of multimodal section summarization is desirable so that we can also take images as con- textual input, as the goal is to generate multimodal content for ticular webpage (in this case, Wikipedia article). Originally, we attempted to collect human writ- ten section summarics ourselves. But when running an initial data collection pilot we found that when explaining the intended application of webpage sto- ries, a majority of the annotators deemed the first sentence to almost always (94% of the time) be good enough pseudo summary. In the other cases when a majority of annotators voted otherwise, we found the annotation quality was too poor to use the collected written summaries. It proved very dif- ficult to collect free form summaries of Wikipedia content. For both of these reasons we continued modeling section summarization with our pseudo summaries (the first sentence of the section).
consume the topic of par-
E Contextual Image Captioning Task Design
We now provide additional discussion on the task of contextual image captioning and how our input de- sign differs from prior work. Nguyen et al. (2022) recently introduced the task of contextual image captioning. We found our metrics were lower than those they reported for the task and investigated the causes. We ran additional experiments for contex- tual image captioning with the exact same sample inputs as Nguyen et al. (2022). Specifically, we tried only using the page description, target im- age, target image's section text, and the attribution description as a sample's inputs.
To perform this data collection pilot, we used internal crowd sourcing platform to hire seven crowd workers. They were located in India and paid hourly wages competitive for their locale. They have standard rights as contractors. The last four pages of our paper include PDF of our in- structions to annotators. We also tried to collect labels for well suited images for each section but ultimately did not use these annotations.
By including the attribution description (which often heavily overlaps with the target caption be generated), our performance is much higher, nearly matching prior work even when using different data splits (the prior work's 's dataset splits are not released). We report these reproduced results for our splits in Table 28. As discussed earlier, for our contextual image captioning task, we chose not to input the attribution description of an image given how much overlap it has with the target caption (the reference description). terms of other exper- imental differences, we also use VIT (Dosovitskiy et al., 2021) image representations while prior work used ResNet-155 152 (He et al., 2016), although both were pretrained on ImageNet (Deng et al., 2009).
F Section Summarization Pseudo Summaries
We were motivated to study the task of section summarization subproblem of Webpage Story Generation, which the task of converting web- page to an Instagram Story-like format. It con- sists of one multimodal Story page or slide per section, containing a section summary and paired image (from the same webpage). Our section sum-

---

Rendered Webpage
WikIWeb2M Sample Fields
Succulent plant
[ "apllt's
'aage, uzzll "bitneelien wkioedia wroww''S eoculent plant".
"page_tittll biille: Suocsdent Plant'
de contored with costuct reavy came renoters out recuteer are cirs. tncureed
"paaa_ aaaaa__caten__aee wes_icetttta..... thout tatte 1ist::
inntaiins inagess
alenta, and conation The
anuuly
caa___aae_dddcc_____e dcscriptiodd
bolany, sussulent plants, knownn as sucousents, ara planta with parts ara thiskaned, fieshy, and ongorged..."
Rassulet
santi shadues
"ia paae ssacriplion, sumple': 1,
comtays 11 Soems oxtetions inclsl
"section_ indea"ddddeee [0, 2.
enfecrable parodk by dring brck to undegrrand corage argans be regoosed auctalens
'seationn estlle r'uucuuurent
'Defiition; 'Appeerancsi, 'Habitat, "Consevvaion" /Familes genera: "Cutivation) ssection beach botary, succulemt plants, also known as eueoulents, are plants with.. By definiionn sucpulent plants are drought-resstent..... .:, Thee storage of water ottan givea.... "Other than Antarctica, succulents
ani Sucaioes shaachrrsal and by abilyy meris litted yaur dusees, euch mitzard incovsters comurre voarr
makes
Sacculets a tecllciicc calegoy ahice the leme delcibes ony urii der of pariictl
aevites spretcs perss cn Chpterree cos femily in Appheteoree
found within....
lannd surve them prears sdenttndeen toneeeer ih same lurk
"In South Mrica, several species.... "hhee approximately siaty different... "Sucaulents are favored as houseplants for their attradiveness... ..].
Ceroenss Cranulccmes speciss are
merttaatann torn
sant Suss
plo the abling
autiyy to briv rolevvllyy
'seation contains table Lixt [0, 0, 1.
the
'ix
www.eri suapiaa 1, 1,0,0,
Definition
'scotion_ inago_ _ettt
By colationn eutouse parts are chsugh resittat plats in vhich saves, dist, 'oll tame became mos ussaty barly the anvelprenn
Miiis ilen aisipeeia Busosdd clerthimee
ro aberniia aftets enlatonlii
esciude
plev ferny and nedden undoosstile
senee duyl
[I_ ing)
undergann groat cigas The andageuane argans much bulos, caree and taben, fer will wati-tia stking Starks Thut catt are inclased
[bitausteon witinedia orgh arg.w clantalmed
detivitrn many peccitvies would clasedt
Parro ahepae hingin anvmrnmeet acrournt,
soulent San Francisoo JPGG....
senseet leevs Nor recovee
plants
retroll loth
1s s___inage_oopp coppttoottoooo posaaappee [11). (1, D.....
...
en bonkly borm len muciseet applcyy e eccline exampe, incthserr thes volare Alacbook of Secocler flome
Águas Livres Aqueduct
'epl iin: train'
'Puge uzl;:
"Hiteelllen whinndia C2 Livrer Aquedued"
Contreee or
'page citle' Agues Livres Aqueducr.
(Pongguee Aduetuo Agui innorsee firs
"page. containe ne. inagee'' 1,
Hou of The Waten's a hiticic iq the du Libon Phetooal tik the most rum aeranaale
"page content, Sections without table l1at': 8,
of fiivenkery Pertegano reghrsenng af
canas
"rav, page. deseziytion's The Aguas Livres Aqueduct is a
historic equeduct in the of Liebon. Portugal, it one the remarkable examples of 18th-cenitrry Partugusse ongineering, The main course the aqueduct covers km, but the whole network of camals extends through nearly....
for the of Coneers the redern nuticeale of Ooveles The erslect spuedd sates on beet alle wins ooher prockets.
Hlistry (18))
'1.5_ 'to_page Bec__pae__deeeeeee docomigtion cample' : 1.
Background [ei]]
"section on indeed [0, 1,2.3, 4,
Whire souse sapple euse the indernn exited lst stttaanc Thes lague edtury boateenng thei catloeend bas to trusish polible cal rou iffuc senvate: crty velice sping mest
'secLiso Li:le [Aguss Livros Aquadunt, History'
'Baskground"; Conntructionl
'Mae d'Auua]]
Cammun waly taiis, tesare albodee souse Losd cim squ eo
'section on Text'<TTh Agues Aqueducl (Porluguee::
'Weter was in scarce supply even for Lisbon's earlest.... "onnttuuction started 1731....
Aquaduto das Aguss Livres
in 1371, Flancicco de toandd propaaed uong the lictus pueee kuply Labort waar is Poruuaass Deng
Seturiiic Some time vidt peopise sul to the more foreaver uq squeeccs, time
'After dalvering its first water 1748, the aqueduct fed whole d'Agua (Mother
new....
Water) reservoir...
aquedart francing ayeer 170k
appainet
ogen
inck
'eection containe boblee or 11st' [0, 0, 0.0.00 Ol
mettee o movemmena sector the epuettcc over the Andtear volay.
Li zectioon suntarl ae [0. 0, 1, 1.
Construction (ei))
"sost.ion image, url"
[iblee:di vikimediis omdwriiedd wtt.e//dd//ooooooeeee WAguun
Merer Meko Portes thew Beiween
1750
uloo des, ulb_ddee__CC366111 NC33kgguus Livee 5,281
ty by Canlabio Vinee
hllps:/lugpoad
Castide
ula das Wot__dss_CCCC33 &cCr% Linee, 2ircl () O.....
cancingg
bought many
$10 Lmags _____attttr__eee pcmppcc 111, 1. 1.
(74) www.ghhhe prjjet
up spuntct carnd Pnter
Figure 6: Example samples from WikiWeb2M. Here illustrate two Wikipedia articles: Succulent Plant and Aguas Livres Aqueduct. The rendered webpage is illustrated on the left. On the right, a sample subset of fields are shown; these are stored in TensorFlow Sequence Example format.

---

Webpage
Image
Target Text
Predicted Text
50
Longpré -le-Sec
The main road in Longpré- le-Sec.
The church in Longpré-le- Sec.
Link to Image
Mural by Nandalal Bose.
A painting of Rabindranath the Santiniketan Ashram.
Santiniketan
Link to Image
1971 WOR-TV LD,, slide. This 'dotted 9' logo was used from 1970 to 1987.
WOR-TV logo, used from the early 1970s until the early 1980s.
WWOR-TV
WOR-TV
Link to Image
Table 17: Contextual image captioning examples. We include three random samples from the image captioning test set and compare the target image caption and predicted model output text.

---

Webpage
Type
Text
Joan Carling is an indigenous Filipino human rights activist and environmentalis! who has defended the rights of native and marginalized peoples for over two decades. She has served Secretary General of the Asia Indigenous Peoples Pact and has chaired the Cordillera People's Alliance in the Philippines. Carling has also contributed the United Nations Framework Convention on Climate Change and REDD+ activities and has served member of the United Nations Permanent Forum on Indigenous Issues. In September 2018 she received the Champions of the Earth Lifetime Achievement Award from the United Nations Environment Programme in recognition of her work as an environmentalist and defender of human rights.
Target
Joan Carling
Joan Carling is a Filipino environmentalist and human rights activist. [ROUGE- L 13.11]
Text
Joan Carling is Filipino indigenous human rights activist and member of the Igorot people. She is the former Secretary General of the Cordillera Peoples Alliance, a national Indigenous Peoples' organization. She was the Co-convener of the Indigenous Peoples Major Group on the Sustainable Development Goals. In 2018, she was designated a terrorist by the Philippine authorities for an alleged connection with the Communist Party of the Philippines and the New People's Army. In September 2018, 8, she received the Champions of the Earth Lifetime Achievement Award from the United Nations Environment Programme. [ROUGE-L
Multi
Alice Bemis Taylor was a philanthropist and was inducted into the Colorado Women's Hall of Fame in 2010. For her significant contributions to Colorado College, Colorado Springs Fine Arts Center and the Colorado Springs Day Nursery and other organizations, she was named "Lady Bountiful" by the press. Alice Bemis Taylor was an American philanthropist and philanthropist.
Target
Alice Bemis
Text
Taylor
BOOUGE-L21.085
Alice Cogswell Bemis Taylor was an American philanthropist and philanthropist. She was the founder of the Colorado Springs Day Nursery, the Colorado Springs Fine Arts Center, the Child Guidance Clinic and the Colorado Springs Commu- nity Chest. She was inducted into the Colorado Women's 's Hall of Fame in 2010.
Multi
BROUGE-L44.336,
The Tranco-Germall Parliamentary Assembly is a joint body of the German Bundestag and the French National Assembly formed to enable cooperation between both houses.
Target
The Franco-German Parliamentary Assembly is inter-parliamentary organisa- tion between the French and German parliaments. It was established in March 2019 following the Aachen Treaty, which was signed on 22 January 2019 by
Franco-
German
Text
Parliamentary
Assembly
Angela Merkel and Emmanuel Macron. [ROUGE-L 28.
The Franco- Franco-German German Parliamentary Assembly a joint parliamentary assembly of the French National Assembly and the German Bundestag. [ROUGE-L 59.09)
Multi
Table 18: Page description qualitative examples. We include random samples from the page description test set that had the largest metrie improvement with images included, and compare the target page description and predicted model output text.

---

Webpage
Scction
Type
Text
When Frederick II became King of Prussia in 1740, Quantz finally accepted a position as flute teacher, flute maker and composer.
Target
Johann
Court of Frederick
Quantz was a friend of Frederick the Great, who was a close friend of his father. [ROUGE-L 10.81]
Text
Joachim Quantz
Quantz friend of Frederick the Great, who great lover of the flute. [ROUGE-L. 16.22]
Multi
In 1833, arrangements were made to build the church on a property. donated by Edward Hanrick, on the corner of Lawrence Street and Adams Avenue.
Target
St. Peter Catholic Church
St. Peter Catholic Church was founded in 1834, but had no resident pastor until 1850. IROUGE-L 5.000
1800s
Text
St. Peter Catholic Church is the third oldest Catholic church in Montgomery, Alabama. [ROUGE-L 10.53] In 1920s, the Shanghai office was opened.
Multi
1900s
Target
P&T Group
-1950s:
Text
In the early 1900s, the firm expanded its (ROUGES 25.00)
Growth and Expansion
Multi
In 1905, the Hong Kong office was opened. [ROUGE-L 66.67]
Table 19: Section summarization qualitative examples. We include random samples from the section summarization test set which had the largest metric improvement and compare larget section summary and predicted model output text.
Feature Inputs
Page Desc.
Section Summ.
Image Caption.
Ref
Text Title Struet Caption Image
BLEURT
BLEURT
BLEURT
CLIPScore
CLIPScore
0.51
0.43
0.36
0.6850
0.7146
0.51
0.44
0.36
0.6845
0.7153
0.51
0.45
0.37
0.6865
0.7166
0.51
0.45
0.37
0.6851
0.7154
0.51
0.45
0.37
0.6878
0.7177
0.51
0.45
0.41
0.7340
0.7575
0.51
0.45
0.41
0.7329
0.7576
Table 20: Feature ablations with WikiWeb2M. We ablate over the section body text, title, structure, strutonre,captions, and images. We report BLEURT, CLIPScore, and RefCLIPScore metrics for the results in Table 5.
Input Section Type
Section Source
Metric
Task
Target Description Context
BLEURT CLIPScore RefCLIPScore
0.43
Section
WikiWeb2M
0.44
Summarization
0.45
WIT
0.40
0.7287
0.7527
Image Captioning
WIT
0.40
0.7307
0.7537
WIT
0.40
0.7325
0.7558
WikiWeb2M
0.41
0.7329
0.7576
Table 21: Section input ablations. We try using only the target section, both the target section and page description and/or context section(s), and vary the sections come from the smaller WIT or our WikiWeb2M superset. We report BLEURT, CLIPScore, and RefCLIPSS IPSoree metries for the results in Table 7.

---

Metric
Task
Model
ViT Data
BLEURT CLIPScore RefCLIPScore
im21k
0.51
Base
Page
JFT
0.51
Description
im21k
0.52
Large
JFT
0.52
im21k
0.45
Base
Section
JFT
0.45
Summarization
im21k
0.46
Large
JFT
0.46
im21k
0.41
0.7329
0.7576
Base
Image
JFT
0.40
0.7291
0.7534
Captioning
im21k
0.41
0.7374
0.7611
Large
JFT
0.41
0.7263
0.7527
Table 22: Pretrained model checkpoint ablations. We report BLEURT, CIIPPScree, and RefCLIPScore metrics for the results Table
Content Section Filter Threshold
Metric
Train
Test
BLEU-4
ROUGE-1
ROUGE-2
ROUGE-L
CIDEr
2
2
13.79
45,43
27.72
38.34
81.16
3
2
12.38
44.35
27.05
37.63
75.35
4
2
12.79
44.10
26.08
36.91
69.52
2
3
13.42
44.31
26.07
36.64
69.69
3
12.32
43.52
25.81
36.28
67.56
4
12.77
43.56
25.13
35.82
64.04
2
4
12.98
43.11
24.45
34.90
59.69
3
4
12.01
42,41
24.28
34.63
58.03
4
4
12.54
42.65
23.92
34.41
57.19
Table 23: Page description performance across different filtering thresholds. We change the threshold for how many rich content sections page must have to he included in our page description generation task dataset.
Sample Modalities
Metric
Task
Train
Test
BLEU-4
ROUGE-1
ROUGE-2
ROUGE-L
CIDEr
Multimodal
Combined
12.27
44.75
27.71
38.16
80.07
Page
Combined
Combined
13.79
45.43
27.72
38.34
81.16
Description
Multimodal
Multimodal
12.30
44.67
27.58
38.03
79.00
Combined
Multimodal
13.77
45.30
27.55
38.16
79.77
Multimodal
Combined
9.83
34.96
15.18
29.64
70.82
Section Summarization
Combined
Combined
9.93
34.86
15.16
29.53
70.86
Multimodal
Multimodal
9,74
34.89
15.10
29.56
70.22
Combined
Multimodal
9,84
34.78
15.06
29.43
70.17
Table 24: Comparison of only retaining multimodal page samples versus also allowing for text only pages (combined).

---

Prefix Inputs
Metric
Page
Page
Section
Section 1st
All Section
Images
BLEU-4
ROUGE-L
CIDEr
URL
Title
Title
Sentence
Content
13.79
38.34
81.16
13.00
38.33
81.02
12.59
38.47
81.68
Table 25: Prefix Global prefix ablations for page description generation. The input column "all section content" refers when all section indices, titles, body text, and captions are concatenated in order and then tokens contribute to the prefix up to the first 512 tokens.
Feature Inputs
Metric
Text Title
Struct
Caption
Image
BLEU-4
ROUGE-L
CIDEr
9.83
33.00
133.70
<<T
9.84
33.40
135.30
v
10.15
33.38
135.10
10.03
33.69
137.07
3.18
14.55
17.43
11.27
36.90
153.44
11.64
37.39
156.27
11.74
37.46
156.34
11.84
37.69
158.19
Table 26: Additional feature ablations with WikiWeh2M contextual image captioning. We ablate over the section body text, title, structure, captions, and images. We report BLEU-4, ROUGE-L and CIDEr metrics. We include rows already reported in the main text for ease of side by side comparison across all feature ablations.
Number of Input Images
Metric
Task
BLEU-4
ROUGE-L
CIDEr
13.44
38,52
81.52
Page
13.55
38.46
82.00
Description
14.00
38.50
81.49
10.12
29.43
69.89
Section
10.10
29.35
70.29
Summarization
9.67
29.37
70.29
11.84
37.69
158.19
Image Captioning
11.92
37.41
157.27
11.84
37.45
157.20
Table 27: Ablations varying the number of input images per task.
Contextual Image Captioning
Text Input
Metric
Image Input
Split
Target Section
Context Attribution Section
Desc.
B
R
C
Desc.
Nguyen et al.
Unknown
ResNet
23.83
48.80
276.60
11.84
37.69
158.19 19
Ours
WikiWeb2M
ViT
25.20
50.01
242.50
Table 28: Experimental results when reproducing the task set up of Nguyen et al. (2022). We do not use the same dataset splits since they released in prior work. Here set up the same sample inputs prior work. unlike our result in the main text which does not input the attribution description.

---

INSTRUCTIONS ANNOTATORS
We are collecting annotations for a task called "Wiki2Story." Wiki2Story has so far been defined as a conversion process from Wikipedia webpages Wikipedia "stories." The goal tum Wikipedia webpage into Instagram-like story which contains one story page per Wikipedia section. Each story page includes a section summary and paired image; see the below examples. We provide two examples of what an Introduction and History story page may look like for these sections of the Apple, Wikipedia article.
Introduction
History Apple History
Malus sleversil recognized as major progenitor species cullivated apple, and morphologically
Apple/ Introduction
An apple is an edible fruit produced by an apple (Malus domestica).
Apple trees are cultivated worldwide and the widely grown species the genus Malus. Introduction Apple/ Introduction
History Apple History
The apple thought to domesticated 4000-10000 years ago in Tian Shan Mountains.
We want to obtain annotations of these section summaries and have select the most appropriate image to pair with it. particular, the section summary should be a highlight of the section content. The highlight should: be self contained, condense the section's factual information into sentence of ideally fewer than 30 words, and retain enough detail for the reader learn something from the story page. The highlight supposed be an educational glimpse of the full section's content, remaining fully true to the original text.
We provide examples below both strong and weak summaries our use case. Note that we expect the summary to contain only factually correct information from what is provided in the original section text. We provide weak summary examples to demonstrate ways the summary style can be incorrect.
Example 1: The Proverb section of the Apple Wikipedia article.
SECTION TEXT
The proverb, "An apple day keeps the doctor away", addressing the supposed health benefits of the fruit, has been traced to 19th-century Wales, where the original phrase was "Eat an apple on going bed, and you'll keep the doctor from earning his bread". In the 19th century early 20th, the phrase evolved to "an apple day, doctor to pay" and "an apple day sends the doctor away"; the phrasing now commonly used was first recorded in 1922. Despite the proverb, 2015 study found evidence that eating an apple daily prevents visits a physician.
STRONG SUMMARIES

---

The proverb "An apple a day keeps doctor away" has been traced back 19th-century Wales, but has yet to be proven scientifically.
"An apple a day keeps the doctor away" originated in Wales in the 19th century with the phrasing "Eat an apple on going bed, and you'll keep the doctor from eamiing his bread."
The proverb "An apple a day keeps the doctor away" has had multiple phrasings over the years, first being traced 19th-century Wales.
WEAK SUMMARIES These are poor summaries because they use words like "this" or "It", have a dialogue-like style, or overly abstract the factual information from the Wikipedia section.
* This section talks about the phrase "An apple day keeps the doctor away" and all of the ways has been said.
* It talks about how apples don't actually prevent doctor visits.
* The apple proverb has existed for centuries.
Example 2: The Breeds section of the Dog Wikipedia article.
SECTION TEXT
Dogs are the most variable mammal on earth with around 450 globally recognized dog breeds. the Victorian era, directed human selection developed the modern dog breeds, which resulted in a vast range of phenotypes. Most breeds were derived from small numbers of founders within the last 200 years, and since then dogs have undergone rapid phenotypic change and were formed into today's modern breeds due artificial selection imposed by humans. The skull, body, and limb proportions vary significantly between breeds, with dogs displaying more phenotypic diversity than can be found within the entire order of carnivores. These breeds possess distinct traits related to morphology, which include body size, skull shape, tail phenotype, fur type and color. Their behavioral traits include guarding, herding, and hunting, retrieving, and scent detection. Their personality traits include hypersocial behavior, boldness, and aggression, which demonstrates the functional and behavioral diversity of dogs. As result, present day dogs are the most abundant carnivore species and are dispersed around the world. The most striking example of this dispersal that of the numerous modem breeds of European lineage during the Victorian era.
STRONG SUMMARIES
Around 450 dog breeds have been globally recognized, making dogs the most variable mammal with significant differences in behavioral traits and physical characteristics.
The breeding dogs during the Victorian Era resulted around 450 globally recognized dog breeds from small number of founders.
Dogs are the most variable mammal on Earth, having significant variance in skull, body, and limb proportions, also differing personality traits like sociability and boldness.
Present day dogs are the most abundant camivore with around 450 recognized dog breeds.
With around 450 globally recognized breeds high variance both physical and behavioral characteristics, dogs display more phenotypic diversity than all other carnivores combined.

---

WEAK SUMMARIES These poor summaries because they either too brief, reduce and abstract the factual information too much, or use words like "this" and 'it."
* There are many types of dogs on Earth.
* This section discusses the difference behavioral, physical, and personality traits dog breeds can have.
* It describes how dog breeding was done to result 450 breeds and mentions that dogs are the most variable mammal.
When selecting images, we want you to choose image you see fit to go with section content and summary text. It should be topically relevant and the image you feel is most visually appealing.
VI PLUGIN EXAMPLE (UPDATED)
Read the below section from Wikipedia page. The first sentence highlighted yellow. Does the first sentence provide strong and concise (fewer than 30 words) summary of the contents of the entire section?
If you need additional context on this section's s text and or topic to answer this question, can click here to see the description of the Wikipedia page. Otherwise, continue with the annotation task.
Section Title: Etymology Wikipedia Page Title: Apple
The word apple, formerly spelled appel in Old English, derived from the Proto-Germanic root "ap(a)laz, which could also mean fruit in general. This is ultimately derived from Proto-indo-European "ab(e))-, but the precise original meaning and the relationship between both words(clarification needed] is uncertain.
As late as the 17th century, the word also functioned as generic term for all fruit other than berries but including nuts such as 14th century Middle English word appel of paradis, meaning a banana. This use is analogous to the French language use of pomme.
Yes, it well summarizes the whole section
No, it does not well summarize the whole section
"HERE" WAS CLICKED ON FOR ADDITIONAL CONTEXT ABOVE, SHOW THE FOLLOWING PAGE DESCRIPTION TEXT:
An apple edible fruit produced by apple tree (Malus domestica). Apple trees cultivated worldwide and are the most widely grown species in the genus Malus. The tree originated Central Asia, where its wild ancestor, Malus sieversii, is still found today. Apples

---

have been grown for thousands of years Asia and Europe and were brought to North America European colonists. Apples have religious and mythological significance in many cultures, including Norse, Greek, and European Christian tradition.
IF THE ANSWER TO THE ABOVE QUESTION WAS NO, SHOW THE FOLLOWING SUMMARIZATION TASK:
Please write single sentence summarizing the Wikipedia section. Keep the summary factually accurate to the original text and summarize the content concisely; try to use 15-30 words at most. The goal is to provide an educational, interesting snippet for story page of this section.
Section Title: Etymology
Wikipedia Page Title: Apple


Reviews:
[{"review_id": "zeTKXLVeEZ", "reviewer": "Reviewer_X6bk", "summary": "", "questions": "What are the contributions of structural data work in different tasks? \n\n\nAs the first sentence can be a reasonable summary, how do you know if the model will directly borrow the first sentence to make the task trivial?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We’d like to thank Reviewer X6bk for their time spent reviewing our paper and thoughtful questions. We appreciate that they believed the paper is “well written with detailed analysis” and acknowledge WikiWeb2M to be a “large dedicated dataset of multimodal webpages.”\n\n>[Missing References]\n\nThank you for identifying this paper. *A multimodal framework for analyzing websites as cultural expressions* provides relevant context for why studying webpages from a multimodal perspective is important; we will update our draft to include the missing reference in related work.\n\nTo address your questions:\n\n>[Q1] *What are the contributions of the structural metadata made available by WikiWeb2M?*\n\nThe structural metadata made newly available with WikiWeb2M informs the section structure for each webpage; this tells us where each chunk of text and each image is located. This structural information is what allows us to define the Prefix Global attention over our webpage data. *I.e.*, without it, we would not be able to identify which images and section contents are the most relevant for each task. By knowing the section a body of text and set of images come from, we can utilize them as global tokens in our new attention mechanism.\n\nSecondly, we do find that simply including special structure tokens interleaved in the input sequence improves performance for page description generation and contextual image captioning (see Table 5 results, row 4 versus row 5). In future work, we agree that we could use the structural metadata in other ways. For example, it could be interesting to define a local attention whose attention neighborhood is adaptively defined by the tokens which belong to the same section or some other structural metadata available.\n\n>[Q2] *How do we ensure that the model does not borrow the first sentence, making the section summarization task trivial?*\n\nFor a given task, we **remove** the text that serves as the target summary or caption from our inputs to the model; this ensures there is no model “cheating.” *E.g.*, for section summarization, since we utilize the first sentence of a section as its pseudo target summary, we remove the first sentence from our inputs to the model and note this on L194 of the paper. We will revise the writing to make this more clear and explicit.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It is well written with detailed analysis.\nIt provides a large dedicated dataset  of MultimodalWebpage with a trained model on this dataset. ", "reasons_to_reject": "The tasks they proposed are either multi-modal summarization or captioning and don't have much novelty.\n\nThe contribution of structural metadata signaling is not clear. ", "questions_to_authors": "What are the contributions of structural data work in different tasks? \n\n\nAs the first sentence can be a reasonable summary, how do you know if the model will directly borrow the first sentence to make the task trivial?", "paper_topic_and_main_contributions": "The submission introduces three new generative tasks: page description generation, section summarization, and contextual image caption to verify multimodel webpage understanding. A dataset of 2M pages named WikiWeb2M also be introduced. The authors also design a novel attention mechanism Prefix GLobal to select the most relevant image and text content. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Pauwels, Luc. \"A multimodal framework for analyzing websites as cultural expressions.\" Journal of Computer-Mediated Communication 17.3 (2012): 247-265."}, "extra_text": {}}, {"review_id": "ixVVHzIEhA", "reviewer": "Reviewer_pHok", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer pHok for the thorough review of our paper and for identifying the benefits of both our data and new attention Prefix Global. We agree that a “key feature of WikiWeb2M is that it retains the complete web content, instead of information filtered for a particular use case” and that Prefix Global can help “address the long context problem” present with webpage data.\n\nWe address your concern below:\n>[R1] *It would be interesting to see the performance of LLMs with vision input (e.g., LLaVA) on the proposed dataset.*\n\nOur experiments utilize T5 [1] and ViT [2], with three variants of T5 defined by different encoder attention mechanisms (full attention [1], Transient Global [3], and our Prefix Global). T5 was in part chosen to compare our Prefix Global attention to prior work Transient Global [3], as they also used T5. We agree that evaluating our suite of webpage tasks with LLMs pretrained with visual input such as LLaVA would be interesting and leave this to future work.\n\n[1] Raffel et al. *Exploring the limits of transfer learning with a unified text-to-text transformer.* JMLR 2020.\n\n[2] Dosovitskiy et al. *An image is worth 16x16 words: Transformers for image recognition at scale.* ICLR 2021.\n\n[3] Guo et al. *LongT5: Efficient text-to-text transformer for long sequences.* NAACL 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. WikiWeb2M is a useful resource for multimodal webpage understanding. The inclusion of both image-caption pairs and long text articles in one place makes it a comprehensive dataset for various generative tasks.\n2. The proposed Prefix Global method is well-motivated. It helps lower computational complexity and retain performance. \n3. The experimental results demonstrate the utility of WikiWeb2M annotations. Comprehensive analyses were also conducted. ", "reasons_to_reject": "1. It would be interesting to see the performance of LLMs with vision input (e.g., llava) on the proposed dataset. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces the Wikipedia Webpage suite (WikiWeb2M) dataset, which consists of 2 million webpages, to study multimodal webpage understanding. The key feature of WikiWeb2M is that it retains the complete web content, instead of information filtered for a particular use case. WikiWeb2M enables a systematic study of several generative modeling tasks with text as the output: page description generation, section summarization, and contextual image captioning. To address the long context problem, the authors also propose a novel method called Prefix Global, which selects the most relevant image and text content as global tokens to attend to the rest of the webpage for context. The experimental results demonstrate that the new annotations from WikiWeb2M improve task performance compared to prior data. Comprehensive analyses of sequence length, input features, and model size were also conducted. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qkwc9rewRI", "reviewer": "Reviewer_dKPZ", "summary": "", "questions": "(1) Is there any noise present in the dataset parsing process? If so, including those noises and discussing their types will be beneficial.\n\n(2) As mentioned in line 236, the method utilizes the earlier content in a body of text. Have the authors explored other parts, such as the last content in a body of text (which is sometimes the conclusion)?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer dKPZ for the detailed review and clear questions. Thank you for identifying that Prefix Global can “enhance performance and efficiency.” We also agree that “the dataset and new attention mechanism will significantly benefit the development of multimodal webpage understanding.”\n\nWe now address your comments and questions:\n\n> [R1] *Additional evaluation metrics.*\n\nWe will additionally report BLEURT, CLIPScore, and RefCLIPScore with their references accordingly. As BLEURT uses a trained regression model to compute its scores, it can be time consuming to use for evaluation. Below we report BLEURT and CLIPScore on a random subset of 10k samples for all image captioning experiments in the main paper; additional BLEURT scores for other tasks will be added in an updated version of the draft, too. The results follow the same trends.\n\n**Table 5**\n| Feature Inputs                              | BLEURT            | CLIPScore | RefCLIPScore |\n| ----                                                | ----                    | ----              | ----                   |\n| Text                                              | 0.36                   | 0.6850        | 0.7146            |\n| Text, Title                                     | 0.36                   | 0.6845        | 0.7153             |\n| Text, Title, Struct                          | 0.37                   | 0.6865        | 0.7166             |\n| Text, Title, Caption                       | 0.37                   | 0.6851        | 0.7154             |\n| Text, Title, Struct, Caption            | 0.37                   | 0.6878        | 0.7177             |\n| Text, Title, Struct, Image              | **0.41**              | **0.7340**  | 0.7575             |\n| Tex, Title, Struct, Caption, Image | **0.41**              | 0.7329       | **0.7576**       |\n\n**Table 6**\n| Model | ViT Data| BLEURT | CLIPScore | RefCLIPScore | \n| ----      | ----         | ----                    | ----              | ----                  | \n| Base  | im21k     | **0.41**            | 0.7329        |  0.7576           |\n| Base  | JFT        | 0.4                    | 0.7291        | 0.7534            |\n| Large | im21k     | **0.41**            | **0.7374**  | **0.7611**       |\n| Large | JFT        | **0.41**            | 0.7263        | 0.7527             |\n\n**Table 7**\n| Input Section Type     | Section Source | BLEURT | CLIPScore | RefCLIPScore | \n|              ----                 |           ----          |        ----             |      ------       | ----                   |\n| Target                         | WIT                  | 0.4                    |     0.7287    | 0.7527             |\n| Target, Desc               | WIT                  | 0.4                    |    0.7307     | 0.7537             |\n| Target, Desc, Context | WIT                  | 0.4                   |    0.7325      | 0.7558            |\n| Target, Desc, Context | WikiWeb2M     | **0.41**            | **0.7329**   | **0.7576**       |\n\n> [Q1] *Is there any noise present in the dataset parsing process?*\n\nThe text and image content in WikiWeb2M is parsed from the webpage HTML. To the best of our knowledge there should not be any noise present in this part of the data parsing pipeline. For task specific processing, we believe noise may only be introduced in the processing of the first sentences of each section (to be used for either the section summarization task or as global inputs for page description generation).\n\nSpecifically, the code used to parse the first sentence may prematurely split a sentence from a period that does not signal the end of the sentence. We did manually inspect samples early on and found this to be rare (*e.g.*, 96/100 random samples were split correctly). Additionally, the sentences which were split prematurely could still be valid standalone sentences. Below we share four examples split prematurely:\n\n| Full First Sentence (By Manual Inspection) | Our Data Processor’s First Sentence |\n| ---------                                                          | -------                                                   |\n| In 2007, Saade was a founding member of What's Up!, a Swedish boy band which included Robin Stjernberg, Ludwig \"hejLudde\" Keijser and Johan Yngvesson. | In 2007, Saade was a founding member of What's Up! |\n| Unicamp offers over one thousand extension programs to the community, with different levels of minimum requirements (high school degree, undergraduate degree, etc.) and across all areas of study, focusing mainly on specialization courses and community outreach. | Unicamp offers over one thousand extension programs to the community, with different levels of minimum requirements (high school degree, undergraduate degree, etc.) |\n| The proposed biosynthesis of ascofuranone was reported by Kita et al., as well as by Abe et al. | The proposed biosynthesis of ascofuranone was reported by Kita et al. |\n| 1988 The choir sang the German premiere of Joseph Jongen's Mass for choir, brass ensemble and organ, Op. 130, which was not yet in print then, both in the Stiftskirche of Aschaffenburg and in St. Bonifatius. | 1988 The choir sang the German premiere of Joseph Jongen's Mass for choir, brass ensemble and organ, Op. |\n\nWe will add pseudo code for our sentence-splitter into the Appendix in case it is of use to anyone as well as a random set of examples from the dataset illustrating the first sentence produced by our code. Additionally, we do include the entire full text of each section in our annotations, which will allow other researchers to reprocess the raw text as they see fit.\n\n>[Q2] *Have the authors explored other parts of the text for prefix tokens, such as the last content in a body of text?*\n\nThis is an interesting point and not something we explicitly explored during our experiments. Our choice of utilizing earlier content in a body of text originates from the Lead Bias phenomenon, in which earlier content is typically more important. This phenomenon has historically been used to select more important content [1, 2, 3]. Not all webpages may reliably have a final text section which is conclusion-style, which also motivates using earlier page content.\n\n[1] Kedzie et al. *Content selection in deep learning models of summarization.* EMNLP 2018.\n\n[2] Xing et al. *Demoting the lead bias in news summarization via alternating adversarial learning.* ACL 2021.\n\n[3] Zhu et al. *Leveraging lead bias for zero-shot abstractive news summarization.* SIGIR 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The paper is very well-written and easy to follow. The organization of the manuscript effectively follows their contributions.\n\n(2) This paper's contributions, including both the dataset and the new attention mechanism, will significantly benefit the development of multimodal webpage understanding.\n\n(3) The experiments and analyses are conducted comprehensively, encompassing ablation studies and efficiency analysis.\n\n(4) Figure 4 demonstrates that the proposed method does not encounter performance degradation as the input sequence length increases, which is a promising solution for handling long texts.\n", "reasons_to_reject": "(1) Other evaluation metrics, especially those [1,2] that reflect human preference and those designed for multimodal evaluation [3,4], should be reported (see missing references section).\n", "questions_to_authors": "(1) Is there any noise present in the dataset parsing process? If so, including those noises and discussing their types will be beneficial.\n\n(2) As mentioned in line 236, the method utilizes the earlier content in a body of text. Have the authors explored other parts, such as the last content in a body of text (which is sometimes the conclusion)?\n", "paper_topic_and_main_contributions": "This paper introduces a large dataset, WikiWeb2M, for studying multimodal webpage understanding. Additionally, the authors propose a novel attention mechanism, Prefix Global, to enhance performance and efficiency. Extensive experiments and analyses demonstrate the superiority of both the dataset and the proposed method.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Bertscore: Evaluating text generation with bert, ICLR 2018.\n\n[2] BLEURT: Learning Robust Metrics for Text Generation, ACL 2020.\n\n[3] CLIPScore: A Reference-free Evaluation Metric for Image Captioning, EMNLP 2021.\n\n[4] EMScore: Evaluating Video Captioning via Coarse-Grained and Fine-Grained Embedding Matching, CVPR 2022."}}]

Author rebuttal:

Meta-review: The paper illustrates a new dataset, WikiWeb2M, designed to study multimodal webpage understanding. This dataset comprises 2 million webpages and retains the entirety of web content, rather than information tailored for specific use cases. The dataset facilitates the exploration of several generative modeling tasks that produce text outputs, including page description generation, section summarization, and contextual image captioning. To address challenges arising from long context, a novel attention mechanism named Prefix Global is proposed. This mechanism selects the most pertinent image and text content as global tokens, which then attend to the rest of the webpage for context. Experimental results indicate that annotations from WikiWeb2M enhance task performance compared to previous data. The studies also encompass comprehensive analyses concerning sequence length, input features, and model size, underscoring the dataset's significance and the efficacy of the Prefix Global method.

The paper's well-structured presentation, introduction of a comprehensive dataset, innovative attention mechanism, and robust experimental results make it a worthy contribution to the field. In particular: a) The paper introduces WikiWeb2M, a dedicated dataset for multimodal webpage understanding. This dataset is unique in its comprehensiveness, combining both image-caption pairs and extensive text articles, making it a valuable resource for a variety of generative tasks. b) The experiments conducted using the WikiWeb2M annotations show promising results. The paper also includes thorough analyses, such as ablation studies and efficiency analysis. Notably, the proposed method exhibits consistent performance, even as input sequence length increases, indicating its potential in handling long texts.
Decision: Accept-Main