WordScape: a Pipeline to extract multilingual, visually rich Documents with Layout Annotations from Web Crawl Data
Authors: Maurice Weber
Carlo Siebenschuh
Rory Marshall Butler
Anton Alexandrov
Valdemar Ragnar Thanner
Georgios Tsolakis
Haris Jabbar
Ian Foster
Bo Li
Rick Stevens
Ce Zhang
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Web Data
Common Crawl
Document Processing
Layout Detection

Abstract: We introduce WordScape, a novel pipeline for the creation of cross-disciplinary, multilingual corpora comprising millions of pages with annotations for document layout detection. Relating visual and textual items on document pages has gained further significance with the advent of multimodal models. Various approaches proved effective for visual question answering or layout segmentation. However, the interplay of text, tables, and visuals remains challenging for a variety of document understanding tasks. In particular, many models fail to generalize well to diverse domains and new languages due to insufficient availability of training data. WordScape addresses these limitations. Our automatic annotation pipeline parses the Open XML structure of Word documents obtained from the web, jointly providing layout-annotated document images and their textual representations. In turn, WordScape offers unique properties as it (1) leverages the ubiquity of the Word file format on the internet, (2) is readily accessible through the Common Crawl web corpus, (3) is adaptive to domain-specific documents, and (4) offers culturally and linguistically diverse document pages with natural semantic structure and high-quality text. Together with the pipeline, we will additionally release 9.5M urls to word documents which can be processed using WordScape to create a dataset of over 40M pages. Finally, we investigate the quality of text and layout annotations extracted by WordScape, assess the impact on document understanding benchmarks, and demonstrate that manual labeling costs can be substantially reduced.

WordScape: a Pipeline to extract multilingual, visually rich Documents with Layout Annotations from Web Crawl Data
Maurice Weber* ETH Zurich
Carlo Siebenschuh"
Rory M. Butler* University of Chicago
Anton Alexandrov
University of Chicago
ETH Zurich, INSAIT, Sofia University
Valdemar R. Thanner ETH Zurich
Georgios Tsolakis ETH Zurich
Haris Jabbar TU darmstadt
Ian Foster
Argonne National Laboratory, University of Chicago
Bo Li UIUC
Rick Stevens
Ce Zhang ETH Zurich
Argonne National Laboratory, University of Chicago
(maurice weber, ce zhang)@infffff..m.m .t..z ch; (siebenschuh, rorymb)@uchhhhbbbbooo f edu; faalexandrov, thannerv, (taolakis)@@tudde wt....cc
harisjabbarrgmail.. com; (stevens, feeesennsssstereeeeeee fanler)aanl..eeee. gov; 1bo0i wwwnoii....... edu
Abstract
We introduce WordScape, a novel pipeline for the creation of cross-disciplinary, multilingual corpora comprising millions of pages with annotations for document layout detection. Relating visual and textual items on document pages has gained further significance with the advent of multimodal models. Various approaches proved effective for visual question answering layout segmentation. However, the interplay of text, tables, and visuals remains challenging for variety document understanding tasks. In particular, many models fail to generalize well to diverse domains and new languages due to insufficient availability data. WordScape addresses these limitations. Our automatic annotation pipeline parses the Open XML structure of Word documents obtained from the web, jointly providing layout-annotated document images and their textual representations. In turn, WordScape offers unique properties as (1) leverages the ubiquity the Word file format the internet, (2) is readily accessible through the Common Crawl web corpus, (3) is adaptive to domain-specifi documents, and (4) offers culturally and linguistically diverse document pages with natural semantic structure and high-quality text. Together with the pipeline, we will additionally release 9.5M urls word documents which can be processed using WordScape to create dataset of over 40M pages. Finally, we investigate the quality of text and layout annotations extracted by WordScape, assess the impact document understanding benchmarks, and demonstrate that manual labeling costs can be substantially reduced.
training
'The first three authors contributed equally.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

wil
winl
Extraction URLs from
Common Crawl Snapshot
URL based deduplication
uris
urls
Document Download
uris
uris
deduplication
dock
deck
.dcc
88 Bounding Bos Armotation
Language iemniiictiion
Text Esuoction
docs jpp
Quality Filters
Overview over the WordScape pipeline for processing a single Common Crawl snapshot. First, extract and deduplicate all URLs from wat files that point Word documents. We then download the documents, apply malware filters, metadata extraction and deduplicate based on content. In the third step, convert downloaded documents to page images, extract text, run bounding box annotation algorithm, identify the dominant language and apply quality filters. The right side of the figure shows an example document with bounding box annotations.
Figure I:
1 Introduction
There is an abundance of digital, semi-structured data contained in visually rich documents such as PDFs or MS Word documents. However, while this information easily understood by humans, its semi-structured nature makes its analysis by automated data processing engines difficult. This difficulty stems, to a large extent, from the diversity how information in visually rich documents organized, across at least the three axes of culture, language, and industry. Therefore, effective use of such data often necessitates a costly and labour-intensive process of manual information extraction. Existing techniques in many automated document understanding tasks either based on conventional rule- based or machine learning (ML) approaches, relying on hand-crafted features, or more promising deep learning approaches which are trained large amounts of data. However, to date, both approaches often fail to generalize well due to a lack of compatible formats, or due to insufficient diversity in existing training datasets, especially for low-resource languages.
At the same time, we have witnessed tremendous progress the area of natural language processing (NLP) and computer vision applications, where pre-trained models [8] 25 31 24 27] have enabled researchers and practitioners to build useful applications by fine-tuning these models on specific downstream tasks. This progress has, to a large extent, been driven by leveraging large quantities of high-quality data extracted from the web.
In contrast to these techniques, the task of document understanding is an inherently multimodal problem, requiring models understand text, visual, and layout features and model their re- As consequence, next to advances in model design, considerable efforts have gone into creating datasets that combine these different modalities. These datasets can be grouped into two categories: on the one hand, there are human-labeled datasets like Docl DocLayNet 2 for document layout analysis, FUNSD [14] for form understanding or the RVL-CDIP document classification dataset []2] While this approach generates high-quality labels, it naturally limits the size to few hundred thousand samples due to the restrictive costs of numan-annotated labels. Alternatively, the automatic generation of ground truth labels has been leveraged to create datasets like PubLayNet (35), DocBank [19] and arXivdocs-weed weak (26]. While these datasets are larger in size, they typically sourced from the scientific domain and are thus mainly in the English language and lack the diversity needed to reflect the true distribution of documents prevalent in practice and across industries, cultures, and societies.
lations [3 22
Word documents are among the most widely used types of documents. While rendered Word documents appear semi-structured pdfs, the source code comprising Word documents consists of

---

highly structured XML files in the Open XML format, containing valuable information like reading order, the structure of tables and style information. Furthermore, Word documents are often used in more formal contexts and for professional writing such as academic papers, reports, business documents or official correspondence. This gives rise to the hypothesis that text found in such documents generally has higher quality than text on web pages like forums, social media updates or user-generated content. Here, we present WordScape, a pipeline that enables the automatic sourcing and annotation of diverse, multilingual, visually rich documents at scale, enabling researchers and practitioners to curate multimodal document understanding datasets, Similar to large-scale NLP dataset creation pipelines like CCNet [33,, we use the Common Crawl web corpus as our primary source of documents. We parse Common Crawl to collect urls pointing to MS Word documents embedded in websites, then parse the Open XML structure of these documents to extract text and identify the location and category of visual semantic entities like section headings, tables and figures on the rendered page images. To date, we extracted a total of 9.5M urls to Word files which we will publish and which can be processed using the WordScape pipeline to build a corpus of roughly 40M pages. In summary, we make the following contributions:
We present a novel pipeline to automatically extract and process millions of MS Word documents from the web, and open-source the codebase
We introduce a novel bounding box labeling algorithm based on the Open XML representation of MS Word documents.
We provide detailed analysis of the size, quality and distribution of datasets created using WordScape.
We validate one created dataset various layout analysis benchmarks and find that WordScape annotations can substantially reduce manual labeling efforts.
We will release 9.5M urls to word documents that we have collected from Common Crawl. These can be processed using WordScape to create a dataset of over 40M pages.
The remainder of this paper is organized as follows: In Section discuss related work. In Section we present our data creation pipeline and dataset metrics are presented in Sections and 5 We validate our dataset on downstream benchmarks in Section 6 and finally conclude in Section
2 Related Work
With the proliferation of pre-trained deep learning models, there has been a growing focus on high- quality, web-scale training datasets, both in the domains of natural language processing and computer vision. In the NLP domain, notable advances include the CCNet pipeline, C4 (25 34), OpenWebTex M.I. Pile v1 19I, S2ORC (20), Pythia (2], the RedPajama dataset (6), or the Refined Web dataset 21. Similar to these datasets and pipelines, this work aims to leverage data that is publicly available on the web to build large-scale, diverse, and multilingual data corpus. However, here focus on visual document understanding, an inherently multimodal domain, where next to text, layout and visual features of documents are also valuable source of data for downstream models.
Perhaps more similar to this work are multimodal web-scale datasets and pipelines like the Laion- 400M and Laion-5B datasets 30 29]. or the multimodal mmC4 (6 In contrast to these works, we focus on visually rich documents with layout annotations, rather than (image, caption) )- -pairs or text interleaved with images. There are multiple datasets in the visual document understanding domain such as the manually annotated DocLayNet (23) for document layout analysis, FUNSD [14 for form understanding or the RVL-CDIP document classification dataset [12.. Other notable datasets include the automatic, weakly labelled PubLayNet [35, DocBank 19). TableBank
(88 and arXivdocs-
weak (26] datasets. In a similar manner our approach, the LayoutReader dataset [32 leverages the
Open XML format of Word documents to construct a multimodal dataset of visually rich documents together with the text in reading order. However, LayoutReader does not include object detection labels for semantic entities and is released as a fixed dataset, rather than as a dataset creation pipeline. In Table T we show detailed comparison between WordScape and other datasets and pipelines.
httpe://commoncraa///
3

---

Table 1: Comparison with existing document layout datasets. (1) WordScape is released as a public pipeline together with 9.5M document urls; (2) 28 top-level categories detected via hierarchical topic modelling; (3) languages detected with fastText on single common crawl snapshot I6 15
Dataset
Pages
Classes
Annotation
Format
Document Types
Languages
Source
PubLayNet 35
360k
Automatic
Digital
Scientific articles
English
PubMed Central
DocBank
500K
13
Automatic
Digital
Scientilic Articles
English
arXiv
aexivooss-maak 26
127,472
23
Autematic
Digital
Scientific Articles
English
arXiv
Magazines, Technical Articles, Forms, Bank Statements, Advettisments
PRImA
305
10
Automatic
Scans
English
Financial Reports, Manuals
NecLayNer
80,863
11
Manual
Digital
Scientilii Articles, Laws Regulations,
Eaglish, Gemman French, Japanese
Patents, Government Tenders
Diginl, Scan, Photographs
Scientific articles, Terthooks, Books, Test papers. Magazines, Newspupers, Noces
Chinese People's daily, arXis, VKontakte
Doe S
9,080
74
Manual
Raglish, Chinese
WordScape (Ours)
9.5M wrl(())
30
Automatic
digital
28 Categorics (9)
136 languages
Common Crawl
3
Methodology
Our document processing pipeline builds on the Open XML structure of Word documents which contains valuable semantic information, and the hypothesis that the quality of text contained in such documents is generally higher than text found on HTML-based websites. As our primary source of data, the Common Crawl web corpus consisting of regular snapshots of the web with little dating back to 2013. On a high level, our pipeline consists of
overlap between different snapshots I three core steps: first parse a Common Crawl snapshot and extract all links that point Word files (i.e. URLs that end in ,docx or .doc). The second step is to send HTTP requests to these links and download the corresponding Word file. The final step consists of processing the document, resulting in a final multimodal dataset with page images, text contained on each page, and bounding box annotations for semantic entities like headings and tables on each page. An overview of the pipeline is shown in Figure T In this section, we present each step in more detail.
3.1 Parsing of Common Crawl
The first step in the WordScape pipeline is to extract urls that point to Word files from Common Crawl snapshots. Common Crawl provides data in raw (warc) format, UTF-8 encoded text (wet) and metadata (wat) files. Next to HTTP header data, each metadata file also contains a list of hyperlinks, from which we select all HTTP urls that end in .doc or _docx. Each wat file must be downloaded in its entirety to be correctly parsed. After the initial parsing of the wat files, the urls from each agent are merged, then deduplicated on a per-snapshot basis and finally deduplicated globally across all previously processed snapshots.
3.2 Document Download
In this step of the WordScape pipeline, we download the documents from the urls extracted from Common Crawl. This step outputs the downloaded Word source files, as well as metadata containing statistics on the quantity of successfully downloaded urls and failure or rejection reasons for each url. Failures relate mainly to benign HTTP errors. We reject document when response is successfully obtained but is not useful. A benign reason for rejection is either an unsuccessful HTTP response code (most commonly 403/404), an invalid URL, too many redirects/retries, response, an incorrect file format of the response, an incorrect content-type header, or internal hardware failure/cluster resource limitations. We furthermore reject potentially malicious documents by performing a check against OLE data structures (7) during download. As such we count any document that contains VBA code/macros, external relations, an OLE object pool, encryption, or flash embeddings Finally, we also reject excessively large files. The reason for this is that we aim to achieve a relatively even distribution of document pages, and to prevent mut-of-memory errors.
received HTTP
https::/ cettps://commocraau/ cttp///oooooccccccoooo gi github. io/ccc/c/// cc- wwwww- ctw..s/ttterrrste//pp///a/stttte 'While this is conservative malware filter, we emphasize that motivated adversaries can in theory still engineer malicious documents. Our malware filter is thus not security guarantee.

---

Upon a successful download, we save several metadata fields concerning the response and its analysis such as HTTP status, OLE information, as well as the file itself. In addition, the metadata includes a SHA-256 hash of the full response bytes: This allows to perform second global deduplication step against files which have identical content but are accessible under different urls, and to ensure that the document has not changed if it is downloaded second time. This serves as defense against potential dataset poisoning attacks, which have recently been shown to be practical in the context of web-scale datasets 4 Finally, the temporary metadata files recorded by the agents are merged, deduplicated via bytehash, and written to a database. We present metrics on this metadata in Section 4
3.3 Document Processing
The processing of Word documents consists of several steps, including language identification, bounding box annotation and text extraction. Here we describe each step in detail.
3.3.1 Bounding Box Annotation
we use a colorization scheme to extract bounding boxes of semantic entities from document page. In the first step, we parse the highly structured Open XML library and custom XML parsing code identify
Similar to the approach from 19
files of Word document using the python-doca the categories of different elements in the document.
We identify such elements by one of two methods: If the Word user has either used a built-in style (such as a heading formatter), or the element is natively tagged in the XML file (such as for tables), we use this information label the corresponding element. Clearly, this approach makes the assumption that using such a built-in functionality reflects the user's s intent, which is not always the case. Nevertheless, we expect this methodology to be accurate in the majority of cases. If, on the other hand, no built-in indicator can be found for an element, we fall back to heuristics, such as the distribution of used fonts in the document indicating headings, or successive numbered or bulleted paragraphs indicating succession of list items. This heuristics- approach is generally more noisy compared to the method based on built-in XML properties.
Once the category of a document element has been determined, we color it using the Open XML formats highlighting. formatting and text coloring features, by directly editing the XML. The colored document is then rendered via LibreOffice and each page is converted to an image. Colors on this image are then detected, providing bounding boxes for each different entity category. We provide more details on the annotation process in the Appendix.
3.3.2 Text Extraction
We extract text from a document on two levels of granularity. First, we extract the full document text from the Open XML structure using python- This document-level text is in reading order, due to the internal XML structure. Second, we extract the text from individual rendered pages using the PDFPlumber package. This allows us additionally extract word-level bounding boxes. It should be mentioned that the PDF-based extraction is less accurate due to the necessity to use heuristics when identifying and grouping characters into words. We discard any document that has less than total of 200 characters. On the page level, we keep pages without any text as they might contain figures or other relevant entities without text.
3.3.3 Language Identification
To identify the language of a document, we use the fastTeex language classifier [6 15 was trained on Tatoeba, Wikipedia and SETimes and can identify 176 languages using n--grams as features with the hierarchical softmax. We identify languages both on a document level, using the Open XML-based text, and on page level using the PDs-d text.
The classifier
httpe://www.lirreffffcc. org/
htrpss//github.cco/jjviiirt
5

---

10a
10
104
2023-14
2023-06
2021-43 unique urls
2020-40
2016-50
2013-48
wris
processed urls
documents
Figure 2: Number of urls and documents extracted from Common Crawl, for snapshots ranging from 2013 2023. Raw urls refers the initial urls extracted from each snapshot, prior any processing. The unique urls are globally deduplicated, starting from the most recent snapshot back to the oldest. Processed urls is the subset of urls which an HTTP request was sent, and "documents" refers to successfully downloaded documents.
3.3.4 Dataset Filters
provide several ways to filter subset of the core dataset created by WordScape, based on metadata collected during the annotation process. First, implement quality filter based the perplexity of the document text for models trained on Wikipedia, as well as a annotation reliability score to assess bounding box annotations. The latter metric captures the proportion of entities annotated using built-in or XML patterns vs. meuristic-basee annotations, as the former are generally more reliable. We present more details on the perplexity distribution in Section 5 and details the annotation quality score in the Appendix. allowing the creation of subsets with different requirements such the number of tables and other entities, or the language of the resulting dataset.
addition, we collect metadata for each document and each page,
4 Pipeline Statistics
In this section, we present statistics on running the WordScape pipeline. Specifically, we investigate the number of links pointing MS Word files in single common crawl snapshot, as well as the reasons that downloads failed, or were otherwise rejected/failed during the annotation process. We provide details the resources used to the WordScape pipeline in the Appendix.
Common Crawl Parsing To estimate the number of documents that can potentially be obtained using the WordScape pipeline, we parsed individual Common Crawl snapshots ranging from 2013 up to the March/April 2023 snapshot. We found that there are substantial duplicated Word file urls each snapshot: per-snapshot deduplication removes 60 80% of available urls. However, we found little overlap between snapshots, mirroring the fact that there is generally little overlap between the websites visited by Common Crawl. Furthermore, we noticed that the number of valid urls, ic. where document can be successfully downloaded, decreases substantially for older crawls: Out of all the urls visited for the 2013 snapshot, only 12.5% could be successfully downloaded, compared to 60.6% of urls from the most recent 2023 snapshot. This is to be expected older urls more likely to be inaccessible than newer ones. These observations are illustrated in figure 2
Document Download Out total of 5, 807, 634 requests urls from the November/Decemte snapshot, we found that 2, 441, 972 (42.1%) received a 200 return code and could thus be further processed. Out the successful responses, there were 364, 648 (14.9%) instances where the content- type header did not match a Word document and was thus rejected. Another 172, 772 (7.1%) documents were rejected because they did not pass our OLETools malware filter. This resulted in total of 904, 552 Word documents that could be successfully downloaded using the WordScape pipeline. We emphasize that we ran the requests at the beginning March 2023, i.e. roughly three months after the snapshot was published. It is likely that at later point in time, the number of responsive urls will be lower, leading to less documents. We provide further details on reasons for rejected downloads in the Appendix.

---

Dogio per Language
104
100
kn
Figure 3: Number of pages per language, produced by the WordScape pipeline run on 1.25.M Word documents extracted from the November/December 2022 Common Crawl snapshot.
Cundach Wark
Langagge
Dalaa
Bandaz
Dartagaas
Palih
Canch
Spankdu Cadlllio
Turkih
Mader (1433 Graek
130
Propeally 1932
12:0
190
100
1938
Peploctiy
Figure 4: Number of words in any given language subset, as a function of perplexity threshold. The figure shows the number of words with perplexity smaller equal the value the x-axis, for the seven languages with the highest (left) and lowest (right) number words. A word is defined as whitespace-dllimited sequence of characters with punctuation removed.
Document Processing further investigate how the WordScape pipeline performs, we ran the document processing step of the pipeline 1, 2,,, 383 of the successfully downloaded documents from the November/December 2022 snapshot. Out of these, for 248, 918 (~ 19.9%) of the documents, the annotation process was either rejected failed. The majority of the failures stem from the files being invalid Zip files, namely in 15.4% of all processed documents. Another 37.5k (~~ 3%) of the documents rejected because they contained less than 200 characters. We furthermore rejected all documents whose uncompressed file size was more than 20 times the compressed size as this indicates potentially malicious zip bomb. This process resulted in 002, 465 annotated documents, or 5, 481, 455 pages, including the document text and object detection bounding boxes.
Dataset Statistics
Language Distribution In the 5.5M annotated document pages, found total of 136 distinet languages, identified with fastText [16] 15) The page counts per language is highly skewed towards high-resource languages like Russian (2M pages) and English (1M pages), as opposed roughly 1k pages for Tajik and Urdu. Figure 3 shows the number of pages for the 50 most frequent languages.
Perplexity Scores We use the perplexity of a language model trained on a target domain to measure the quality of the text extracted using WordScape. We follow the approach used in [3 and use their 5-gram Kneser Ney models and SentencePiece tokenizers trained on Wikipedia. In this context, lower perplexity score indicates that the language is closer to the target domain and is thus expected shows the number of words with most certain perplexity value.
be of higher quality. We note that especially for Hungarian, Portuguese and Italian, the perplexity scores are relatively low, and large part of the corpus can be retained even when aggressively filtering out documents with moderately high perplexity. We provide further figures that illustrate the perplexity distributions for more languages in the Appendix.
Semantic Entity Distribution Semantic Entities like headings, tables and lists are the logical units that build up the structure of a document. Here we present statistics on the semantic entities that the WordScape pipeline annotates. This analysis is based on the 1.25M documents annotated from
7

---

60
26.084%
12.131%
18.6178
6.791%
4.023%
3.264%
3.559%
2.090%
0.604%
0.108%
0.3681
0.009%
0.003%
0.033%
0,0715
0.021%
838%4
30.8178
17.0067
3.083%
5 00824
5.701%
3.530%
6.402%
2.74%%
D.152
0.080%
0.0.3%
0.037%
0.003%
0.014%
22. 953%
92.870%
18.797%
7.139%
5.7945
5.989%
3.570%
2.510%
0.702%
0.154%
0.227%
0.051%
0.068%
0.26%
0.005%
0.025%
0835%
50.949%
15.798%
3.772%
2.838%
5.173%
2.727%
7.387%
0.69/%
0.037%
0.000%
0.676%
0.083%
0.05%
0.001%
0.01%%
14.113%
13.737%
4.490%
3.215%
2.014%
1.63%%
0.479%
0.639%
0.009%
5.015%
0.063%
0.000%
28.777%
11.660%
6.003%
1.142%
1.341%
0.067%
0.023%
0.01%%
0.000%
0.005%
uk
64.1%
33 33.565%
17997
4.239%
6.04.%
2.408%
0,400%
1.00%%
0.661%
0.031%
0.00015
0133%
0.009%
0.000%
0.000%
Votal
53,97094
30-349%
15. 8.
7,982%
6.039%
2.885%
2.538%
2.152%
9.540%
0.229M
0.062%
0. 0.044%
0.037%
0.0297
0.018%
0.013%
Euro, for0_
tabre
Sgure
Scetar
bupdec
quate
equatien
bibltography danotation
Figure 5: Proportion of layout element categories for the seven most frequent languages.
Governmnnt
Ceseral Helionee
Diter
Somer Gaider Tomplacs
Diher
Scenc
Spera
People Sackey
1-1
File
Use
Figure 6: Hierarchy of topics detected in the WordScape.
the November/December 2022 snapshot, containing a total of 173M entity bounding boxes. The most frequently appearing category are table cells (86M individual cells), making up roughly 50% of the entities. Table columns and rows also appear frequently, comprising another 20% of all entity bounding boxes. The next most frequent category list items, of which found 15M (8.9%). Further frequent categories are plain text (14.8M) and headings (7.6M). We found that, generally, the entity distribution is highly imbalanced; however, when excluding the individual table elements, the distribution flattens significantly.
In Figure 3 we show the semantic entity distribution, grouped by languages and where have excluded table elements and merged the different heading levels into single category. We see that the imbalanced nature of the categories persists across all languages considered. However, there are differences between languages in regard to the extent to which the classes are imbalanced (e.g. Russian documents are more imbalanced than French documents). To whether the pairwise occurrence of layout elements is correlated, we compute Spearman's rank correlation coefficient for pairs of layout elements in Figure 10 in the Appendix. We found that the elements are generally weakly correlated, except for (text, heading), (list, heading), (list, text), (form_field, text) and (footer, header). Further details on the semantic entities in the Appendix.
Topic Modeling Since Common Crawl snapshots cover websites across multiple, unfiltered do- mains, the Word documents extracted via WordScape can potentially themselves also cover a wide range of topics. To get better understanding of the topic distribution in WordScape, hicrarchical topic modelling classifier available in the Google Cloud NLP API over a 25k sample of WordScape documents in 11 languages support by the API (ru, en, it, ja, es, nl, zh, pt, ko, fr, de). This allows for a fine-grained analysis giving both high-level overview of the topics in the dataset, and also exposes more low-level details of aab-classifications. The top categories "/Law & Governnenn/GoveemeeettOo Templates" (9.0%), "/Jobs & Edducatom/EEuccatiooPPinnnn & Secondary Schooling (K-12)" (5.6%), "/People & Society/Religion & Belief" (3.5%), and "/Jobs & Edducation/EducaaiionCColll
ran the
found (15.3%), "/Reference/Generaa Reference//orms Guides &
& Uni- versities" (3. 33.4%). The full hierarchy is shown in Figure 6 We furthermore split the analysis across languages that where we found significant differences in topic distributions. While the most frequent top-level category in both Russian and Portuguese "Law & Government". accounting for 37% of documents, this category occurs relatively infrequently in other languages (< 13%). We provide a more fine- fine-graiin -rree-rrandd overview over the language specific topic distributions in the Appendix.
https://cloou.. google. col.lee.ccomaauuaaa... language

---

Training Object Detection Models on WordScape
To further assess the quality of the bounding box annotations extracted by WordScape, we conduct experiments on three different datasets. We measure the utility of the annotations by first training base model on WordScape annotations, then finetuning the model on a target benchmark. We attempt to show that, by leveraging the automatically annotated labels produced by WordScape, we can reduce the (manual) labeling cost on the target domain while still maintaining the original performance.
Text Detection on FUNSD We first consider the word-level text detection task on the FUNSD
Table 2: Text detection F1 @ loU 0.5 for Faster R- CNN FUNSD. N₂ are the WordScape samples; N the FUNSD samples.
dataset [4] This dataset is subset of the RVL-
CDIP 2 dataset and comprises 199 manually annotated, scanned forms. We trained Faster R-CNN [28] network on to 100k pages, anno- tated with word-level bounding boxes. In the second step, we finetuned the resulting model on 25 149 samples of the FUNSD dataset. In Table 2 we report the Fl score with IoU thresh- old 0.5. We can see that using only 10k WordScape samples and 25 finetuning samples substantially surpasses the text detection accuracy of the model trained on the full FUNSD dataset. By using WordScape annotations we can thus decrease the labeling cost 6-fold.
N, 25
Ny 50
N, 100
Ny 149
N₁
0.621
0.690
0.723
0.772
0.840
0.840
0.823
0.861
N. 50A
0.868
0.870
0.857
0.869
X, = 1006
0.872
0.869
0.850
0.882
Table Detection on ICDAR 2019 cT- DaR Here we present results on the ICDAR 2019 cTTDRR table detection task [1O]. We use the modern tables subset, the domain of which is closer to the WordScape domain, compared to the archival subset. It includes 600 training images and 240 test images. We compare mAP @ IoU [0.50:0.95] for the Ultralytics YOLOv8m model on the test set with and without any training on WordScape table documents. We resize images to 640 x 640 resolution and train 4 models with different training set sizes for 200 epochs using SGD, 0.01 learning rate, batch size of 16, 0.937 momentum and 5e 4 weight decay. We then finetune each model on 4 different subset sizes of cTDaR using AdamW with 5e 4 learning rate for 200 epochs, or until no improvement is observed for more than 30 epochs. We see that pre-training on WordScape improves results particularly in the low-resource regime.
Table 3: Table detection mAp @ loU [0.50:0.95] for YOLOv&m on ICDAR 2019 cTDaR. Np are the WordScape samples, N the cTDaR samples.
Ny = 75
Ny = 150
Ny = 300
Ny 600
0.869 0.008
0,888 1 0.011
0.949 0.006
0.974 1. 0.003
1.25k
0,906
0.012
0912 = 0.011
0.951 : 0.005
0,972 1 0.003
2.5k
0,914 0.009
0.929
0.008
0.960 + 0.004
0.974
+ 0.003
0.007
0.924
0.956
0.005
0.974
N₁ 10k
0,919 0.006
0,931 = 0.010
0.961 = 0.005
0.975 + 0.003
Layout Analysis on DocLayNet DocLayNet 2 is one of the largest numan-annotated document layout segmentation datasets, containing over 80k pages from variety of docu- ment sources. We train a YOLOv5 ob- ject detection model on 200k images obtained via WordScape, and then fine tune the model subsets of the Do- cLaynet training split, varying the fine tuning dataset sizes from 1A to the full 69k. The results are shown in Table 4 where that pre- training on WordScape leads to consistent performance improvements compared to (1) using random weights for intialization, and (2) pretraining with the same number of samples on the PubLayNet [35] dataset. This is particularly pronounced when less human labelled data available.
Table Document Layout Analysis mAp loU [0.50:0.95] for YOLOv5 on DocLayNet with different pretraining datasets. N2 is the number of finetuning samples.
N, = 1k
N₂ 5k
Ny 20k
Ny = 69k
Random Initialization
0.299
0.553
0.727
0.753
PubLayNet (200k)
0.467
0.659
0.720
0.745
WordScupe (200k)
0.508
0.679
0.734
0.755
Handcrafted Scientific Dataset WordScape's versatility arises from enabling access multi- lingual document pages with rich category structure. However, assessing its quality requires an

---

600
0.40
<00 060
0.25
0.35
0.30
0M9150:855
0.20
0.20
0.35
0.55
0.00
252
763
1080
1250
1.000
175a
2000
0.00
252
160
753
1926
175a
1968
2000
Figure 7: Layout analsysis mAP IoU [0.50:0.95] on the scientific paper dataset with YOLOv8 (left) and DETR (right) for varying WordSeape sample sizes. The figures show mean values for smaller (2.5k, 5k, 7.5k, red) and larger (15k, 20k and 25k, blue) pretraining sizes. The confidence bands arise as estimates of the quartiles 90.25 and q0.75.
equally refined dataset for downstream tasks. The growing interest in layout detection precursor to multimodal document understanding makes scientific literature particularly viable candidate.
We compiled dataset of diverse scientific content of N, 2, 500 pages from eight scientific domains (biology, chemistry, physics, mathematics, engineering. computer science, economics, and medicine) that span 120 subdomains from abstract algebra zoology. More than 41,000 instances were annotated by humans. total, there are 31 categories that can be grouped into metainformation (e.g. title), text body (c.g. paragraph), code (e.g. pseudocode), mathematical content (c.g. equation), and visual assets (c.g .table). Categorics relating to text are further split by indicating the presence of symbols. In addition, categories figure or table. Finetuning model on an information-denss annotation-rich scientific corpus is a formidable challenge for variety of reasons. It is particularly daunting for model pre-trained Wordscape, however, due to the (1) multi-disciplinary. information-dent scientific content, (2) the extensive, hierarchical class labels, (3) the high-resolution images stored PNG rather than JPG, (4) the human-made annotations and (5) the distributional shift from multi-lingual word documents to English-only PDFs.
present that relate captions the respective
DETR 3 and the current iteration YOLO represent state-of-the of-the-rrtt choices terms of accuracy and latency, respectively. In our experiment, both models are pre-trained pages stemming from WordScape with sample sizes ranging from 1000 to 100K. Subsequently, the models are finetuned on handerafted scientific dataset for up 2, 000 pages. The empirical results shown in figure and indicate that pre- traininingg on WordScape significantly reduces the need for downstream data.
7 Discussion
In this paper, we present a pipeline to create curated datasets consisting of high-quality, multilingual and diverse, visually rich documents with layout annotations. The pipeline is scalable to millions of pages and contains high-quality text, both for high- and low-resource languages. WordScape is the first pipeline that enables the creation of training datasets for large-scale multimodal document understanding models that fuse text, visual and layout features. As the main limitation of the pipeline, we identify the reliability of the bounding box annotations for certain semantic entities like headings, as they rely some extent on the assumption that formatting correlates reasonably well with user intent. In the future, we wish to explore more characteristics of the resulting dataset such as the amount of toxic offensive content, and train large-scale document understanding models that make full use of both text and image modalities in multiple languages. We are also excited to see how this dataset can further enhance existing web-scale multimodal and NLP datasets.
10

---

Acknowledgments and Disclosure of Funding
This work is partially supported by the National Science Foundation under grant No. 1910100, No. 2046726, No. 2229876, and Alfred P. Sloan Fellowship. CZ and the DS3Lab gratefully acknowledge the support from the Swiss State Secretariat for Education, Research and Innovation (SERI) under contract number MBB2.00036 (for European Research Council (ERC) Starting Grant TRIDENT 101042665), the Swiss National Science Foundation (Project Number 200021 184628, and 197485). Innosuisse/SNF BRIDGE Discovery (Project Number 40B2-0 187132), European Union Horizon 2020 Research and Innovation Programme (DAPHNE, 957407), Botnar Research Centre for Child Health, Swiss Data Science Center, Alibaba, Cisco, eBay, Google Focused Research Awards, Kuaishou Inc., Oracle Labs, Zurich Insurance, and the Department Computer Science at ETH Zurich. IF and RS acknowledge support from the U.S. Department of Energy under Contract DE-AC02-06CHI
References
[1] Apostolos Antonacopoulos, David Bridson, Christos Papadopoulos, and Stefan Pletschacher. realistic dataset for performance evaluation of document layout analysis. In 2009 10th International Conference on Document Analysis and Recognition (ICDAR), pages 296-300, 2009.
[2] Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallaban, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: suite for analyzing large language models across training and scaling, 2023.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pages 213-229 Springer, 2020.
[4] Nicholas Carlini, Matthew Jagielski, Christopher A Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. Poisoning web-scale training datasets is practical. arXiv preprint arXiv: 10149, 2023.
[5] Hiuyi Cheng, Peirong Zhang, Sihang Wu, Jiaxin Zhang, Qiyuan Zhu, Zecheng Xie, Jing Li, Kai Ding, and Lianwen Jin. M6doc: A large-scale multi-format, multi-type, multi-layout, multi-language, multi-annotation category dataset for modern document layout analysis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 15147, June 2023.
[6] Together Computer. Redpajama: An open source recipe reproduce llama training dataset, 2023.
[7] Microsoft Corporation. Object linking and embedding (ole) data structures.
[8] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[9] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprinl vrrXi:2201.000
2020.
[10] Liangeai Gao, Yilun Huang, Hervé Déjean, Jean-Luc Meunier, Qinqin Yan, Yu Fang, Florian Kleber, and Eva Lang. Icdar 2019 competition on table detection and recognition (ctdar). In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 15 1110-115. IEEE, 2019.
11

---

[11] Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus.
http://SkylionoO07 http:///iiooooo.gg.uu...o....... github. wtt.hubb..//ppeeeeeeeettCCrrrrpp 2019.
[12] Adam Harley, Alex Ufkes, and Konstantinos Derpanis. Evaluation of deep convolutional nets for document image classification and retrieval. In 2015 13th International Conference on Document Analysis and Recognition (ICDAR), pages 991-995. IEEE, 2015.
[13] Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layoutlmv3: Pre-training for document ai with unified text and image masking. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4083-4091, 2022.
[14] Guillaume Jaume, Hazim Kemal Ekenel, and Yean-Philipps Thiran. Funsd: A dataset for form understanding in noisy scanned documents. In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), volume 2, pages 1 1-6. IEEE, 2019.
[15) Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. Fasttext.zip: Compressing text classification models. arXiv preprint arXiv: 2016.
[16] Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguisties: Volume 2, Short Papers, pages 427-431, Valencia, Spain, April 2017. Association for Computational Linguistics.
[17] Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, and Furu Wei. Dit: Self-supervised pre-training for document image transformer. In Proceedings of the 30th ACM International Conference on Multimedia, pages 3530-3539, 2022.
[18] Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou, and Zhoujun Li. TableBank: Table benchmark for image-based table detection and recognition. In Proceedings of the Twelfth I. Language Resources and Evaluation Conference, pages 1918-1925, Marseille, France, May 2020. European Language Resources Association.
[19] Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, and Ming Zhou. DocBank: A benchmark dataset for document layout analysis. In Proceedings of the 28th International Conference on Computational Linguistics, pages 949-960, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics.
[20] Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969-4983 Online, July 2020. Association for Computational Linguistics.
[21] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The RefinedWeb dataset for Falcon I.LM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.00) 16, 2023.
[22] Qiming Peng, Yinxu Pan, Wenjin Wang, Bin Luo, Zhenyu Zhang, Zhengjie Huang, Yuhui Cao, Weichong Yin, Yongfeng Chen, Yin Zhang, Shikun Feng, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. ERNiE-layout: Layout knowledge enhanced pre- pre-training for visually-rich document understanding. In Findings of the Association for Computational Linguisties: EMNLP 2022, pages 7754-776,, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[23] Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed S Nassar, and Peter Staar. Doclaynet: A large human- annotated dataset for document- layout segmentation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3743-3751, 2022.
[24] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages PMLR, 2021.
12

---

[25] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-555 2020.
[26]
Johannes Rausch, Octavio Martinez, Fabian Bissig, Ce Zhang, and Stefan Feuerriegel. Doc- parser: Hierarchical document structure parsing from renderings. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 4328- -43884438,10 2021.
[27] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779-788, 2016.
[28] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-enn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[29]
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmar- czyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[30] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.0211 2021.
[31] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023.
[32] Zilong Wang, Yiheng Xu, Lei Cui, Jingho Shang, and Furu Wei. LayoutReader: Pre-training of text and layout for reading order detection. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 4735-4744, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[33] Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4003-4012, Marseille, France, May 2020. European Language Resources Association.
[34] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 483 Online, June 2021. Association for Computational Linguistics.
[35] Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. Publaynet: largest dataset ever for document layout analysis. In 2019 International Conference Document Analysis and Recognition (ICDAR), pages 1015-1022 IEEE, 2019.
[36] Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. arXiv preprint wwXiv:2304.06933 2023.
13

---

A Appendix
A.1 Details on Semantic Entity Annotation
One central part of the WordScape pipeline is the segmentation of page images into different semantic entities, As discussed in the main part of the paper, we segment the pages as follows:
1. Identify the classes different elements the documents by parsing the Open XML structure of Word documents.
2. Edit the XML tags such that the elements appear in a specific color the rendered page images. We first eliminate any highlighting, and then change the color of the font as well as the background, ensuring that the change in color does not affect the spatial composition of elements on the pages.
3. Match the colors on rendered pages using OpenCV II each category.
to get bounding box annotations for
In this way, we annotate the following set of semantic entities:
Title, Heading Level 1, Heading Level 2, Heading Level 3, Heading Level 4, Heading Level 5, Heading Level 6, Heading Level 7, Heading Level 8, Heading Level 9, Plain Text, List Item, Header, Footer, Table Header, Table Header Cell, Table, Table Cell, Table of Contents, Bibliography, Quote, Equation, Figure, Table Caption, Footnote, Annotation, Form Field, Form Tag, Table Row, Table Column.
As discussed previously, we have several ways to identify the category of an element. Here we discuss each technique in more detail.
Built-in Styles Word has a number of built-in formatting styles which users can apply to achieve well structured and formatted document. This includes styles for titles, headings with different levels, plain text, list items, footnotes and others. We take the usage of given builtin style as signal to identify the entity category of the element in the Open XMI. files and colorize the element accordingly.
Open XML tag Some elements cannot be readily identified via builtin styles, but rather using specific Open XML tags, accessible via the python- library. our implementation we use this methodology to identify the following categories: header, footer, text box, table, table cell, built-in table boxes as plain text. In the context of XML tag identification, we refer to table of content and forms as "built-in" opposed heuristically matched toc and forms. The colorization of figures is implemented by replacing the image files in the Word zip files with an image of the same shape and format (e.g., ipg or png encoded), but filled with the color corresponding to the figure entity.
content, built-in forms, figures. Note that we relabel text
Heuristics In the case that we cannot identify an element using built-in styles or XML indicators. we fall back to heuristics. These fall broadly under three categories:
Firstly, we compare the elements styling the styling of built-in elements. The following user action serves as an example: user creates a heading using the built-in heading feature and then styles that built-in element, and later copies and pastes the built-in heading to different location in the document, then edits its text content. The first element would be detected as a built-in heading, but the second element would not. The second copy-pat copy-pasted element possesses no built-in style name indicator: however, it possesses identical applied styling (font size, boldness, underlines etc.) to the known built-in element, and should therefore receive the same classification. We perform a second pass on each document after built-in elements have been found in order to classify any non-builtin elements with identical applied styling.
Second, we use content-aware heuristics specific to individual element types. For example, a paragraph in which every line break is immediately followed by number or special bullet character
14

---

is classified as a list, and text segments above certain length consisting only of underscores are classified form fields.
The last and most rudimentary heuristic deals only with distinguishing plain text elements from title and heading elements; choose this approach our last fallback because heading elements are crucial in outlining document structure. We rank elements according to font size, boldness and underlining, and use this information to create a hierarchy among text elements which matches Words built-in heading feature. For example, if document consists of elements with font sizes 20, 16 and 12, this approach would classify elements of size 20 as heading 1, size 16 as heading 2, and size 12 as plain text. As this is the last heuristic fallback, and therefore the least reliable, we employ various checks and restrictions on these classifications. For example, We configure a maximum length for headings classified this way, only rank font sizes as indicating a heading if they are larger than the most commonly appearing font size, and only classify document titles their styling is unique and of the largest font size. Finally, if an element does not match any heuristic, we simply label it as plain text.
Table Rows and Columns We find table rows and columns by post-processe the bounding boxes for table cells. Specifically, we divide table into the grid corresponding to the finest granularity; i.e., the smallest cell height for rows, and the smallest cell width for columns. After ordering the cells from top to bottom, row vertical position y is then defined as the sequence of all cells that vertically cover the cell with smallest height and starting at position y, ordered from left to right. Columns are determined analogously. Note that in this way we account for merged cells, i.e., the same (merged) cell can appear in multiple rows / columns. This is similar to the internal representation of tables in the Open XML standard.
A.2 Quality Filters
In WordScape, calculate several quality indicators based on which subsets of the output can be filtered.
Preliminary Filters As a preliminary step, during processing, we discard any document that has less than 200 characters in it. In addition, discard documents with more than 150 pages whose absolute (compressed) file size is larger than 10MB in order to maintain document diversity. We also discard documents whose uncompressed size is more than 20 times its compressed size, documents that contain excessively large images (> 22.4M pixels).
Text based characteristies For each document, we collect the number of characters, the num- ber of words the number of alphabetical characters, the number of numerical characters, the alphanumerical proportion, and the ratio of of alphabetical to numerical characters. In addition, we provide utilities to compute the perplexity of Wikipedia trained language models used in the CCNet pipeline [33]. Finally, each document is classified according to its dominant language using the FastText classifier [16 15). Here we also include the classification confidence in order to maintain the possibility to filter out low-confidence documents.
Bounding box annotations The WordScape bounding box annotations stem from either built-in and Open XML related sources, or from heuristics. We found that the former source is generally more reliable as the user has less degrees of freedom (e.g., tables) and choosing particular builtin style is a conscious action which we argue is strong signal to their intention. Heuristics on the other hand are based on relative font sizes and special characters and thus provide much weaker signal. To capture these different sources of bounding box annotation, we compute an annotation reliability metric, which as a weighted average over the proportion of the number of characters of reliably (i.e., builtin or XML tag based) annotated entities against the number of characters for heuristic-based annotations. Formally, we have the following score
N
b b
7/8 =
T1
(I)
Σ
Ci
+ h3
We define word white-space delimited sequence of characters with punctuation removed.
15

---

140000
120000
100000
# Documents
80000
60000
40000
20000
0.2
0.6
1.0
Annotation Quality Score
Figure 8: Distribution of the bounding box annotation quality score.
where the number entity categories, C is the number of entity with category by the number of built- built-in/reliably annotated characters, and h2 is the number of cturistic-annotatee characters. Since Tables and figures may not contain any characters but should still be counted as reliable, we set = 1 in those cases. We show the distribution of the quality scores in figure [ where we that the distribution is skewed towards documents with higher scores, and a spike at documents with score close 1.0. For documents with
close to exactly 1, observe that average they have around 50% fewer text entities, total words and list entities and significantly fewer heading annotations, which natural since those the entities that often rely on heuristics rather than built-in styles. Overall, we believe that the annotation quality score reflects the annotation confidence relatively well. However, it is important note that the highest reliability scores (close 1) could potentially imply low document diversity, e.g. documents that mainly contain tables and figures as these are always counted with 1.0.
A.3 Perplexity Distributions
We provide perplexity distribution plots for the 15 most common languages in figure 9 We observe fairly distinct distributions, with some languages showing more pointed curves (e.g. Ukrainian, Hungarian) and some languages with flat distributions (e.g. Czech, Turkish).
A.4 Download Failure Statistics
Here we provide further details on reasons that the http requests either failed, or successfully downloaded documents were rejected by WordScape. From table we can see that across snapshots, the most common reason is unsuccessful http code (e.g. 404), stemming from dead invalid links found in Common Crawl. This pattern gets amplified as we progress towards older snapshots.
Table 5: Frequency of the most common includes invalid URLs or URLs without a response, exceeded file sizes and miscellaneous errors. "HTTPCode" refers to unsuccessful HTTP Code (such as 404), "ContentType" invalid content- type header, "Retry Redireetd too many retries or redirects, "maldoc" a failed OLE check.
encountered during the download stage. "Other"
Other
HTTPCode
Contentlype
RetryRedirect
Maldoc
Total Rejections
Checked URLs
2023-14
3.847%
49.1511
11.915%
18. 18.121%
16.966%
1,701,770
4.142.849
2023-06
5.911%
44.832%
12.181%
18.875%
18.200%
1,616,608
3,830,526
2021-43
11.914%
39.727%
9.723%
28.329%
10.308%
2,294,023
3,400,950
2020-40
0.301%
57.873%
12.300%
18.714%
10.812%
1.718,184
2,761.522
2016-50
0.230%
52.471%
16.524%
23.447%
7.328%
912,971
1,209,775 775
2013-48
0.765%
60.94196 5
12.980%
20.844%%
4.469%
528,848
598,437
16

---

Perplexity Distributions
Russian
English
Ukrainian
Polish
Spanish: Castilian
0.004
0.002
0.010
0.003
0.002
0.005
0.01
0.001
Proportion In language subset.
0.000
0,000
000
0.00
0.000
2000
2000
Dutch: Flemish
French
Italian
Portuguese
B Bulgarian
0.0015
0.003
0.004
0.0010
0.002
0.0005
0.001
0.005
0.002
0.0000
0.000
0.00000
0.000
2000
20000
2000
2000
2000
Hungarian
Czech
Turkish
German
Greek, Modern (1453-), C 0.0015
0.015
0.0015
0.0015
0.0010
0.010
0.0011
0.0010
0.005
0.0005
0.0005
0.0000
0.0000
2000
20:00
2000
2000
2000
Perplexity
Figure 9: Perplexity Distributions for documents in the Top-15 languages from the Novem- ber/December 2022 Common Crawl snapshot.
A.5 Semantic Entity Distributions
Here we provide more details on the semantic entity distributions. The total number of entities for each of the 30 categories annotated by WordScape based 1.25M documents is presented in table Roughly 50% of all entities are table cells, resulting in a highly imbalanced class distribution. Excluding categories that correspond a more even distribution with the most dominant entity categories being list items and plain text. Table 7 presents the language specific entity counts, where we have omitted table structure elements and merged the different heading levels into one single category. Figure [0] shows Spearman's 's rank correlation coefficient for pairs of semantic entity counts a 5% significance level. Figure indicates how (un)balanced the semantic entity labels are for each language. Specifically, the figure shows the proportion of entities in the top-k semantic entity categories among all entities in the language subset. It can be seen that the Dutch, Russian, Polish and Ukrainian subsets appear to more unbalanced, compared Spanish or English.
elements of a table (i.e. Cells, Rows, Columns), find
A.6 Language Specific Topic Modelling
As highlighted in the main part of the paper, there exist considerable differences in the distribution of topics across languages. In Figure 12, it can be seen how the document type diversity varies acress languages. While the top-5 categories make up 53.7% of documents, they account for 80% of Korean and Portuguese documents. Figure 13 provides further evidence for this observation and shows the entire set of top-level categories for each language.
A.7 Computational Resources
Here we report detailed breakdown over the computational resources required to process one common crawl snapshot with WordScape. Running the first step of the pipeline, namely parsing of Common Crawl, in single node setup with 64 CPU cores and 512GB RAM, takes 49 hours to complete, or 3,087 CPU hours. We emphasize that this running time is heavily dependent on egress speed in the Common Crawl S3 bucket and might vary over time, depending on demand. Running the second step of the pipeline, with 64 cores and 256GB RAM takes 22.5 hours, or 1,440 CPU hours. Finally, the last step of the pipeline is the most CPU intense step and was run cluster
17

---

title
0.04
text
0.57 0.01
list
1.45-0.040.0
header
0.07
0.07-0.00..00
0.05
footer
0.15
toc
0.06
0.06 0.05
0.03 0.04
bibliography
0.00
0.01
quote
0.05 0.01
0.06
0.03 0.01 0.01 0.02 0.01
equation
figure
0.15 0.05
0.08
0.06
0.11 0.14 1.1400 0.04 0.01 0.02 0.03
footnote
annotation
0.04 0.01
0.04
0.04
0.02
0.02 0.03 0.02 0.00 0.01 0.02 0.02
form tag
0.02
form field
0.14 00.01 0.26 0.13 0.05 0,06 0.02 -0.01
0.02 0.03 0.01 0.12
table
0.03 0333-.000 0.000 0.02 0.10 0.11 0.04
0.02 0.05 0.02 0.02 0.04 0.07 equation
0.01
heading title
text
list
header
footer
toc bibliography
quote
figure footnote
annotation form, tag form field
Figure 10: Spearman's rank correlation coefficient for the occurrence of different layout elements. Pairs were the coefficient is not statistically significant at the 5% level are left blank.
ml
40.9%
63.0%
78.8%
86.
25:25
90.0%
33.15%
98.05
58.2%
98.2%
59.7%
40.8%
69.3%
81.2%
88.9%
95.9%
37.2%
98.4%
99.6%
98.0%
99.0%
pl 37.8%
99
8%
fr 32.9%
55.2%
74.0%
01.11
87.1%
92.9%
96.3%
98.8%
99.0%
99.7%
e0
32.1%
59,00
77.6%%
84. 456
89.0%
92.9%
96.2%
98.8%
96.4%
59.6%
30.8%
56.3%
73.4%
79, 8%
85.5%
90.5%
95.6%
58. 99.9%
99.7%
99.8%
WordScape
Top-1
Top-2
Top-3
Top-4
Top-5
Top-6
Top-7
Top-8
Top-9
Top-10
Figure 11: Layout diversity for the top-7 languages. Each column represents the proportion of top-k entities among all semantic entities in the language specific subset. This indicates how (un)balanced the semantic entity labels are for each language. The Dutch, Russian, Polish and Ukrainian subsets appear to more unbalanced, compared Spanish or English.
with 24 nodes and 24 CPU cores and 96GB RAM each, for about 22 hours, resulting in 12,672 CPU hours. In total, processing one snapshot of Common Crawl thus requires roughly 17k CPU hours.
A.8 Intended Use
This dataset creation pipeline, and the URLs to Word files extracted from Common Crawl snapshots intended to be used to generate training data for deep learning based document understanding and language models. The URLs are intended to be processed by the code made publicly available alongside this paper [T3] We emphasize that while the URLs static and hosted by the authors, the http responses may change in the course of time. For this reason, we provide SHA-256 hashes of the document contents downloaded during June 1st 12th 2023, that allow verify whether the content has changed in any way. Finally, we emphasize that the URLs provided can in some cases point to documents which are protected under copyright law, otherwise contain sensitive information, It is the responsibility of the user of the URLs to comply with these restrictions.
18

---

Table 6: Total number of entities after processing 1.25M documents from the November//Dceemer 2022 Common Crawl snapshot.
Entity Category
Entities
Frequency in Corpus
Table Cell
86,995,200 260
0.502776
Table Row
20,671,174
0.119466
List Item
15,544,374
0.089836
Table Column
15,276,984
0.088291
Plain Text
14,757,470
0.085289
Form Field
3,880,406
0.022426
Heading Level
3,161,658
0.018272
Table
2,935,625
0.016966
Heading Level 2
1,514,728
0.008754
Figure
1,295,668
0.007488
Table Header Cell
1,273,716
0.007361
Footer
1,145,431
0.006620
Heading Level 3
1,086,629
0.006280
Header
1,046,474
0.006048
Heading Level 4
629,615
0.003639
Heading Level
402,500
0.002326
Heading Level 6
270,680
0.001564
Title
267,039
0.001 543
Heading Level 9
265,021
0.001532
Table Header
170,712
0.000987
Heading Level
166,740
0.000964
Heading Level 8
119,500
0,000691
Table of Contents
38,479
0.000222
Form Tag
30,110
0.000174
Quote
21,240
0.000123
Equation
18,181
0,000105
Table Caption
16,289
0.000094
Footnote
14,233
0,000082
Bibliography
7735
0.000045
Annotation
6127
0.000035
Total
173,029,804
1.000000
Top-1
15.4%
20.0%
18.4%
13.7%
24.5%
25.3%
33.3%
17.4%
37.0%
36.8%
34.8%
Top-2
28.5%
35.4%
36.5%
26.9%
47.1%
38.1%
G1.1%
33.8%
59.2%
51.7%
51.5%
Top-3
38.3%
46.2%
50.0%
38.1%
58.5%
49.8%
72.2% 2%
45.1%
72.1%
61.8%
59.1%
Top-4
47.3%
55.0%
66.5%
Top-5
74.35%
de
ja
ko
zh
Figure 12: Topic diversity for language based subsets of WordScape data. Each row represents the proportion of documents that are contained in the top-k high level categories obtained via Google CLoud NLP API topic modelling. The French subset particularly diverse with the top-5 categories making up 53. 7% of documents, compared to over 80% for Korean and Portuguese documents.
19

---

Table 7: Semantic Entity Counts for each language. Based on 1.25M documents from the Common
Crawl November/December 2022 Snapshot.
lane
bendies
Iusdder
Scoke
10C
IrNio- prrpñ0
equatio
Spxre
Scrn
Decms bold
tikic
46,500
4.230810
6.451.755
211 2011
158,082
7.485
033
3,204
4,515
1$3,254
2,2M
9220
553
1,201455
1. 1,H0-250
0905 118
07:975
3,559.855
2.794
265.00 203
596
16.373
7,003
7.161
0.707
179.0
3,416
2.167
17,085
205.912
480
6635993
15:714
302
1.54
65.009
05.504
1.337
694
134
105,03
637
1.27%
450/001
147.255
7798 4777
348.786
13:335
555 231-225
434.999
104,904
59.734
2:200
192
554
100.193
131
243
1,029
181362
1AL109
2,300
1983
100.254
5.915
996.571
155.993
31:051
54,900
161
56,310
138
553 653.55
877 A00
166
1159
353.70
178
44.33
11.171
740
714
736
09
10
486
81 161
<5 500
148.314
2.993
21,043
00
170
756 150
110 987
16,058
185 A00
01
10
15.213
735
13315
16
860
11:17
100 064
OA 181
130
6.51
315,17
10213
16/91
76
170
IE
423
72.165
01.525
2304
2555
235371
120055
131.51
42302
15.263
10.200
202
216
27751
199
450
31
18.341
535
90
125
16
40
25-713
21,315
16,48
23000
SOLAZH
400 105
23105
1295
3200
13:45
155000
11354
0,311
1.706
145
41
4,201
121
16
40
17-761
21884
31
30.00
$6.374
1
1.50
077000
00-00
63136
100
24 M/
2.00
0.061
13,99
19
112
11
6222
196
21
78
10
566
155
11
06. 00.201
23.33
17
ASLAS1
2255
67353
14310
13.44
339
50
150
L345
15,655
2.85
156
23,854
2053-
95
12 22
9-711
41
172
122299
11552
681
5500
07355
5,500
1.5
211
41
LX10
94
455
425
4,82
116
96
30
14
245
50
11367
11240
1L557
5251
1181
76 05
22
6551
114
13 111
20 10
-2000
2000
91
10
1301
15
1264
1965
140
15
1966
4.00
15
11:00
38
1200
552UT
1989
564
14:51
1200
2,410
25
4,517
268
6
3621
2001
10
125
VII
4
220
500
150
6:550
365
2554
KI
25002
5055
135
8201
12:17
524
853
558
2
14
641
4600
21211
2,00
145
5155
639
1,02
3
1999
3001
181
1.00
EIIM
41.2
218
0
MM
21.89
4813
2006
2013
106
DOSTA
133
4370
5911
1.281
770
15
735
106
5220
6351
1ON
D19
10494
1.687
183
2008
2135
867
11
614
607
961
10163
455
114
1078
35
16
34
0
7212
20
MEDS
1300
1154
8863
1214
11
2579
2
18
104
21
196
540
8THH
2.00
115
12
1003
220
1224
121
196
19
263
164
157
4003
38
4006
1310
1,08
250
134
79
0
186
4812
4300
681
16
1007
144
134
437
0
36
158
509
11
4206
1016
24
1325
1A77
49
75
166
100
317
209
63
19
30.
198
315
122
136
LIST
128
TOO
1303
230
410
107
45
46
435
41
41
997
116
10
175
363
24
91
76
65
188
274
830
141
176
276
KY
45
331
114
152
15
18
30
KED
4ET
83
106
33
16
89
37
302
75
300
3U5
202
72
40
24
THE
13
10
THE
14
182
WE
nds
TH
IN
13
20

---

German
English
Spanish
Fewbir Swiler tirs
Torple & Socier
135
18%
Rosioes & Antoriia
Lise Gevernennt 13%
Retroeza
15%
35%
An teer
Redormec
Peypl
Resena telontia
tdhs
Peeston 5%
French
Italian
Japanese
Resoos & Inbunid
Reforeca
Poople Saide
11%
Redermoe
& Vavomeeee
Jon & Lencssn 25%
14%
Iole BiL
a
Engple Beout It%
Poogle Dounty Nts
Sporn
Korean
Dutch
Portuguese
19%
Law & CoNormnne 37%
33%
People 27%
Jdl Edaemien
23%
Selincc
Besene Iuboodal
7%
Haakh AP%
Sports
Acfornc 13%
Russian
Chinese
33%
17%
50%
Reteenox 18%
Haah 5%
Raference 79%
Figure Distribution of top-level content topics discovered via the hierarchical topic classifier from Google Cloud NLP api. The distribution is calculated based on a 25k sample of WordScape data.
21


Reviews:
[{"review_id": "tX5bZoA2h7", "reviewer": "Reviewer_nyYp", "summary": "", "questions": "", "limitations": "Malware is detected. But unsafe contents and copyright issues don’t seem to be addressed.", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- A large dataset for visual document understanding from the web.\n- The process for dataset construction is described in great detail, which is helpful for other researchers to reproduce the process.\n- The result shows the benefits of pretraning models using this dataset, which can reduce the labeling cost on the target domain. \n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "dnWeP6agns", "reviewer": "Reviewer_qD1P", "summary": "", "questions": "", "limitations": "The author discussed the limitations of the pipeline including the expirations of the links as well as the potential inaccuracies of bounding box labeling and text extraction. ", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "In the era of large (language) models, constructing high-quality datasets at a web scale is increasingly important. While most of the previous work focuses on the generating text-only data, this paper develops a pipeline to produce multimodal layout-rich document data. While there are relevant work before, this paper has the following the strength: \n1. The pipeline is carefully designed for processing and extracting MS word documents at web scale (of diverse domains and languages). Similar previous work either focuses on a single (or a few) domains (e.g., scientific documents in PubLayNet or DocBank) and the documents are mostly in the same language. And thus the generated dataset can be a good artifact for pre-training document image understanding models.\n2. The pipeline is documented in detail and the processing code will be released (it is still not accessible at the time of review). Most of previous work only releases the compiled dataset and the details of processings are often missing. The detailed documentation of the data processing as well as the released code can be helpful for the document analysis research community to reproduce and develop new document processing pipelines at scale. \n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "VEkyYtFBkQ", "reviewer": "Reviewer_fwpL", "summary": "", "questions": "", "limitations": "1. Lack of evidence for document type diversity: The major advantage of the proposed method compared to existing approaches is the ability to obtain more diverse data in terms of document types and languages. While Figure 3 presents statistics on language diversity, the paper fails to provide evidence of document type diversity, which is a critical aspect to support the paper's contributions. \n\n2. Insufficient validation of data quality: Annotating categories based on heuristics can lead to significant errors, but authors do not provide a clear analysis of the error rate or its severity. For example, the authors could perform random sampling of the data and compare the results with human annotations to quantify the discrepancies.\n\n3. Pretraining did not lead to the expected performance improvement and even resulted in performance degradation (Tables 1, 2; Figure 5): Considering existing works on document pretraining, large-scale pretraining should typically lead to a stable performance boost in downstream tasks (even when using all downstream task training data). Is this lack of improvement due to poor data quality as mentioned in point 2 above? Would pretraining with other large-scale datasets (e.g., PubLayNet, DocBank) in the same experiment also encounter similar issues? These aspects lack necessary exploration and discussion.\n\n4. The authors mentioned that they would label the 'footer' category, but in the visualization of the first example in Figure 1, the footer is not labeled.\n\n5. In Figure 5, what are the specific number of the pretrained data corresponding to the blue and red lines? Why are they represented as a range? This is confusing, please clarify.\n", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "(1) This paper propose a multi-language, cross-disciplinary, and diverse web-scale document image dataset, including layout detection and text content annotations. This is of great help in advancing general document intelligence.\n\n(2) Current automated annotation for large-scale document image datasets is limited in style. This work serves as a supplement to existing data.\n\n(3) The  proposed pipeline can extracts documents in over 100+ languages.\n\n(4). 9.5 million Word doc URLs are released, which can be used to create dataset with 40+ million pages.\n\n\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "kBbHmH9Cc2", "reviewer": "Reviewer_t2Lc", "summary": "", "questions": "", "limitations": "The discussion presents limitations, e.g., regarding the annotation quality. It would indeed be very helpful to understand the quality of the annotation component better.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The dataset generated by the pipeline is intended to contribute a more diverse set of documents along with useful annotations, expanding the semantic scope and content of existing annotated datasets. This may provide a corpus for (pre)training DL models with content of higher quality and coverage (beyond scientific documents). These claims are not supported by the analysis and experiments though.\n- The approach (hence pipeline) behind extracting and annotating the documents are well motivated. And the experiments illustrate the utility of the document corpus (besides limitations as explained below).\n- The paper is well written; clear and concise, especially the process of running the pipeline and annotating resulting documents are clearly described.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "S1pGo6c9zm", "reviewer": "Reviewer_knqo", "summary": "", "questions": "", "limitations": "One limitation discussed in the paper is that bounding box annotation can sometimes produce incorrect results. The authors attribute this to the fact that formatting does not always correlate well with user intent. Possible strategies or plans should be provided.", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "* This paper introduces WordScape, the first pipeline for creating a large-scale, multimodal and multilingual corpus for visual document understanding.\n* A novel bounding box labeling algorithm based on the Open XML representation is proposed, ensuring high-quality data generation without manual annotation.\n* Experimental results validate the efficacy of WordScape by showcasing how pretraining on its datasets significantly alleviates the burden of manual annotation in downstream tasks.\n* The authors have open-sourced the code and data, supporting further research in visual document understanding and multimodal models.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Reviewers agree this is an okay paper, although have several concerns and suggestions. Many of these are addressed in the responses & revision.

Summary And Contributions:
Presents a pipeline "WordScrape" that parses Open XML in Word documents from Common Crawl. This pipeline generates a multilingual, multimodal corpus for document layout detection. The corpus is described and experiments suggest that it reduces the need for manually annotated documents.

Strengths:
- Offers a large, diverse, multimodal dataset.
- The pipeline to create the dataset is the first of its kind.
- The pipeline is well-described.
- Proposes a novel bounding box labeling algorithm.
- Useful for pretraining models, alleviating some of the need for manual annotation in downstream tasks.
- The authors are open-sourcing the code and data.

Opportunities For Improvement and Limitations:
- Only the pipeline is released, not the dataset itself. 
- The dataset is composed of links, not the actual documents, yet handling the fact that links expire is not adequately addressed.
- Authors could better explain the specific contribution of WordScrape with respect to data quality, diversity, and coverage.
- The claim that this approach in particular reduces the need for manual annotation is not fully supported: the experiments only compare pretraining on datasets generated from WordScape with no pretraining at all. (Authors respond that the revision now includes more information on this.)
- More analysis analysis of the visual layouts of the created dataset. 
- Experiments can be improved (more specific suggestions provided by reviewers).
- The quality of the annotations can be better documented.

Correctness:
Addressed in author response/revision.

Clarity:
The paper is well written.

Relation To Prior Work:
Prior work is clearly discussed.

Documentation:
What's provided is done well. Most issues are addressed in response and revision. Remaining issue seems to be that authors don't specify plans for maintenance, and could be more detailed in suggestions for responsible use of the dataset.
Decision: Accept (Poster)