SWEb: A Large Web Dataset for the Scandinavian Languages
Authors: Tobias Norlund
Tim Isbister
Amaru Cuba Gyllensten
Paul Gabriel dos Santos
Danila Petrelli
Ariel Ekgren
Magnus Sahlgren
Venue: ICLR 2025 Poster (2025)
Keywords: dataset
pre-training
swedish
danish
norwegian
icelandic
Primary area: datasets and benchmarks
TL;DR: We present the hitherto largest pretraining dataset for the Scandinavian languages: the Scandinavian WEb (SWEb), comprising over one trillion tokens.

Abstract: This paper presents the hitherto largest pretraining dataset for the Scandinavian languages: the Scandinavian WEb (SWEb), comprising over one trillion tokens. The paper details the collection and processing pipeline, and introduces a novel model-based text extractor that significantly reduces complexity in comparison with rule-based approaches. We also introduce a new cloze-style benchmark for evaluating language models in Swedish, and use this test to compare models trained on the SWEb data to models trained on FineWeb, with competitive results. All data, models and code are shared openly.

Published as a conference paper at ICLR 2025
SWEB: A LARGE WEB DATASET FOR THE SCANDINAVIAN LANGUAGES
Tobias Norlund, Tim Isbister, Danila Petrelli, Ariel Ekgren,
Amaru Cuba Gyllensten, Paul Dos Santos, Magnus Sahlgren
AI Sweden
ABSTRACT
This paper presents the hitherto largest pretraining dataset for the Scandinavian languages: the Scandinavian WEb (SWEb), comprising over one trillion tokens. The paper details the collection and processing pipeline, and introduces a novel model-based text extractor that significantly reduces complexity in comparison with rule-based approaches. We also introduce new cloze-style benchmark for evaluating language models in Swedish, and use this test to compare mod- els trained the SWEb data models trained Fibe with competitive results. All data, models and code are shared openly.
INTRODUCTION
Large language models have made significant strides in recent years due to their general capabilities in language-processing tasks. This progress has been largely driven by the development of extensive and high-quality pretraining datasets sourced from open web data (Wenzek et al.] 2020; Brown et al., 2020 Abaddi al. 2022)] Penedo al., 2023 2024). However, the majority of research aimed at improving pretraining data focuses on high-resource languages such as English. Our goal is to create a large-scals scale and high-performing open pretraining dataset specifically for the Scandinavian languages: Swedish, Danish, Norwegian, and Icelandic.
Existing large-scale datasets for these languages primarily include mC4 (Xue et al. 2021), OSCAR (Abadji al.] 20022), and HPLT Datasets 1.2 (de Gibert et al. 2024). The Scandinavian portion of mC4 comprises approximately 100B tokens, TOB tokens for OSCAR 23.01, and 35B tokens for HPLT, which are all relatively small numbers considering that state-of-the-att large language models today are trained on trillions of high-quality tokens.
In this paper make the following contributions:
We release the largest date pretraining dataset for the Scandinavian languages: Scandinavian WEb (SWEb). SWEb is the result of running our proposed pipeline on 98 Common Crawl snapshots. SWEb contains 1.01 trillion tokens the Scandinavian lan- guages, approximately an order of magnitude more than other available open alternatives.
We introduce new cloze-style benchmark for evaluating language models in Swedish, HP-MEK, a subset of the Swedish Scholastic Aptitude Test (Högskoleprovet) used for university admissions in Sweden. Using HP-MEK, we show our data performs on-par with data from the recently proposed Web (Penedo al. 2024) pipeline.
We propose new comprehensive pipeline for curating pretraining data for large language models, built around a masel-based text extractor that significantly reduces complexity and is easily adaptable through rapid data annotatior Most notably, we demonstrate that pipeline returns about +60% more high quality tokens than Fib on the same input data.
Corresponding author: tobias
Data available here: https:/ //t::///uuge //wggggiigg htt.:/acc...cctttttteeeteeeeeeee s/SWbb
Code and extractor model is available here:

---

Published as a conference paper at ICLR 2025
2 BACKGROUND AND RELATED WORK
Early efforts to extract massive amounts of text from the open internet for LLM training start from WebText (Radford et al. 2019), developed for training GPT-2. In this case, outbound links from Reddit with a certain number of upvotes were used as the content selection criterion. Text was extracted using Dragnet (Peters et al. 2018) and Newspape and filtered with several heuristics, resulting in dataset of 40GB after deduplication. Soon after, CCNet (Wenzek et al. 2020) and C4 (Roberts et al., 2019) were proposed, both based on open web data from Common Crawl. C4 was initially developed exclusively for English but was later followed by a multilingual version, mC4 Xue et al. (02T). CCNet, on the other hand, was multilingual from the outset.
Both CCNet and C4 are based on the WET archives from Common Crawl, where all HTML, format- ting has been stripped, leaving only the text. However, this text still contains significant amount of noise in the form of menu and ad text, headers, footers, and sidebars, which are irrelevant to the page's ' primary content. A successful method for extracting primary content from WET archives is to deduplicate the documents at the line level. C4 globally deduplicates all lines, while CCNet deduplicates over subset of documents from the same Common Crawl dump. Line-by-ling dedu- plication is the primary extraction method in CCNet, whereas C4 additionally employs a range of English-specific cleaning heuristics.
Following extraction comes a language detection and filtering step. Whilst more computationally expensive, performing language detection post extraction been shown to achieve better detection accuracy than filtering pre extraction (especially for low-resource languages) (Wenzek et al.) 2020). Quality filtering differs slightly between the two, with C4 filtering using several heuristics, a bad words filter, and URL deduplication. In contrast, CCNet employs model- filter, using per- plexity as a quality measure with a KenLM model trained on Wikipedia.
CCNet has since been utilized in subsequent works such as RedPajama (vl and v2) (Together Com- puter 2023) and Dolma (Soldaini et af.) 2024). RedPajama- v2 runs CCNet on an expanded number of Common Crawl snapshots and filters for five high-resouree languages (none of which are Scandinavian, however). They also extend CCNet's quality filtering by pre-computing a larger set of popular quality signals but leave the thresholding and filtering to the user.
Recently, several works have moved away from Common Crawl's WET archives in favor of pro- cessing the raw HTML of webpages found in the WARC archives. Utilizing mor sophisticated text extraction turns out to be critical for the improving quality of the resulting data (Penedo et al.. 2024). In MassiveWehh (Rae al., (02T), the tree structure of HTML is utilized to more easily group and identify the primary content of pages. Some formatting is also retained, with the argument that this "diversity in formatting style translates effectively the generative capabilities of the Gopher models."
A similar approach is developed in NeuScraper (Xu et al. 2024), where model is trained on an element level - decide whether it should be extracted or not. Both RefinedWeb and FineWeb use the open-source framework Trafilatura (Barbaresi] (021) to extract text from HTML. Trafilatura is based on rules and heuristics on the DOM tree to ıdentify primary content and has been shown to be the best non-commercial extractor for certain domains (Lopuhin) 2019). However, quality issues are still prevalent, and in RefinedWeb (Penedo et al.) 2023) further (line-based) filters are added in an attempt to address these.
MassiveWeb introduce what they call "repetition filters" to remove documents with repetitive text, that is found beneficial with their extractor. These are also sucessfully reused in both RefinedWeb and later Fine Through systematic analysis, FineWet further adds small set of quality filters, that is shown through ablation experiments to yet increase quality. For a state of the art pipeline like Fine Web, the filtering can add up to 30 or more quantities and rules that might be difficult to oversee and adapt to new languages.
2

---

Published as a conference paper at ICLR 2025
Stagu Conkert
Stage Contant
Dommron (WWPC))
Centrnr (WET)
CENer
Conwe ITTI
Varsdon
Peraelptran Nakktown oallen)
Tot tomra barte
Dasiity (iteey
Deveppiction
Pt reporomst
The SWEb pipeline. We Common Crawl's preprocessed WET archives for content selection, and WARC for extraction. At the center stage sits our model based Markdown extractor, that is the primary workhorse produce our dataset.
Figure 1:
THE SWEB PIPELINE
As evident by the previous section, much focus has been placed on the development of heuristics and filters to enhance the quality the resulting data.
To move away from the extensive number of manual thresholds and complex extraction rules, propose a more data-driven alternative. By learning a model for extraction, this complexity can be significantly reduced.
We begin by describing our pipeline that, like existing approaches, consists of the overarching steps of content selection, extraction, quality filtering, and deduplication (Figure
3.1 STAGE 1: CONTENT SELECTION
Our pipeline begins with content selection, which aims to identify and select source documents from Common Crawl that are likely to be in one of the Scandinavian languages. Since the Scandinavian languages make up very small portion of the entire Common Crawl, we want to implement this step early to filter out all non-relevant content.
We CCNet to identify Scandinavian documents within the entire Common Crawl dataset. CCNet processes the WET archives, and after line-by-line deduplication, language detection is performed using fastText (Joulin al. (0166).. Documents with detected language above 0.2 for any of the four languages are selected for the next stage.
3.2 STAGE 2: CONTENT EXTRACTION AND FORMATTING
In Stage 2, we start from the documents indentified in Stage I but discard their content and instead Common Crawl's index download their original HTML from the WARC archives. This means we use CCNet and the WET documents solely for content selection, but not for extraction. In the WET archives, all formatting and structure, such header information, tables, text styles, bullet lists, and images, have been removed. We believe it is useful for language models to also model such structural information, in addition to plain text. Therefore, we aim extract also this information from the webpages, and retain it in Markdown format.
We propose a new method for extracting primary content from the webpages, consisting of two steps: 1) Convert HTML. to Markdown, 2) Extract primary content from the resulting Markdown through line-by-line filtering with trained model.
3.2.1 CONVERT HTML TO MARKDOWN
Since we want to preserve basic textual formatting, we choose to convert from HTML to Markdown with its very lightweight markup, thus does not add many extra tokens. We convert all incoming HTML documents to Markdown using Pandoc, stripping links and images. See Listing for an example.
No extraction has yet taken place, so these documents are still full of noise from menus, advertise- ments, and other extraneous content. We address this in the next step.
3

---

Published as a conference paper at ICLR 2025
Listing 1: A webpage converted markdown (translated, originally in Swedish), including title, top menu, headings and primary content. The document is truncated for brevity.
By LiCe, My Thoughta My Training
Blog
Blog
Running Times Over the Years
My EducaLion
Personsl Training
#2 Hednesday, Decsaber 14, 2012
10
The
13 ...but tonoszoN is rest day.
14
right in the middle of Louch veek periodize my craining, things, alsernating betwean heavy, madiun, and light weaks.1 and zunning was exactly ciset Lhing I thought about. when wake up Lhis nocning, But after sap Logetheer, aleep?!!
feel my
16
Possed by
19
Running Life az
20
3.2.2 MODEL-BASED CONTENT EXTRACTION
We observe that individual lines in the Markdown documents often correspond to specific elements such headers, paragraphs, navigation links. This makes lines an appropriate level for extraction. Therefore, we develop a custom annotation tool (details in Appendix [) to annotate which lines in these Markdown documents should be extracted and which should not. We ask annotators mark what is considered the "main content" on the current webpage, and make some principled decisions for quality and consistency:
1. We do not extract navigation text such menus buttons.
2. A significant portion of the webpages product pages. We decide to extract these only if there is a product description consisting of at least three complete sentences.
3. We extract tables if they are well-formatted and their content is tightly coupled to the main content.
4. On blogs article pages that include comments, extract such comments in addition to the main content.
5. We do not extract information from sidebars unless it clearly constitutes main content.
While not explicitly excluded as per our guidelines, advertisement text isn't considered to main content and is thus implicitly excluded. The full annotation guidelines can be found in Appendix C In total, we annotate 1 1,380 webpages, using 100 of these for validation and the remainder as training data for our extraction model.
Line Extraction Model Our dataset consists of Markdown documents with corresponding binary line annotations, see Figure IT We aim to train a model to predict this label for each line. For this purpose, choose use a transformer encoder, where each newline is replaced with special token [SEP]. We then feed the entire document through the encoder, with each [SEP] token representing the preced- ing line. This way, each line classification is contextualized by (theoretically) the full docu- ment context.
BCE
TRANSFORMER
Testetr
Figure 2: Illustration of our proposed line clas- sification model. Each newline is replaced by a special 
 token, and the corresponding embed- dings are used for classification
ho. Encoder((ro...)
4

---

Published as a conference paper at ICLR 2025
Content length
alphanumeric chars
Ratio headings per non heading
Unigram entropy
Num documenes
10*
107
10+
104
107
10000 20000
10000 5000500000000
0.4 0.6
0.00
0.05
0.10
0.33
0.20
Num chars
Ratio
Ratio
Entropy
Figure 4: Filtering distributions on two Common Crawl dumps, and exclude regions marked in red. We exclude documents whose content length is shorter than 100 chars (invisible in the chart).
Through linear projection of the output hidden state of each [SEP] token, we obtain logits for predicting the binary label of the current line. Let j denote the token index corresponding to each [SEP] token in the document. then get the predicted probability for the line as:
Pj = o: b)
where is the sigmoid function. The model is trained using binary cross-entropy loss between each Py and an annotated line label. See Figure 2 for an illustration. We apply a fixed threshold to p, to determine whether include exclude the line.
The Markdown documents can be very long, so the Longformer (Beltagy et al. 2020) architecture. Specifically, we use pre-trained model that supports up 16k tokens and has been trained for representation learning using masked language modeling" The Longformer is a linear complexity transformer. thanks to its local self-attention, where each token only at-
extraction precision/recall
1.0
Threshold 0.995 Thresheld 0.49
Threshold 6,050
0.8
Threshold 0,001
Precision
0.6
tends to fixed number of nearby tokens. We use a local attention window size of 256 tokens and no global attention, as this turned out to only impair generalization.
We fine-tune the entire model set 1,280 documents, and the results on the validation can be seen in Figure 3 We use the Adam optimizer with a constant learn- ing rate of 1e-5. The results show that despite our small-scale training data, we achieve an Fl score of 87%. Some more details and error analysis is provided in Appendix D
training
0.0
0.2
Recall
Figure 3: Precision/recall of our final line extrac- tion model. We pick a threshold of 0.05 at infer- ence, e.g. when applying the model for extraction.
Finally, we normalize the text using Fix Text For You (Speer 2019
3.3 STAGE 3: QUALITY FILTERING AND CLEANING
The third stage aims to filter for quality, reduce duplicate content and remove personally identifiable information (PII).
Quality Filtering A significant advantage of our model-based extraction is that it also implicitly performs much of the quality filtering. The extractor effectively learns to exclude content that is not of sufficient quality, such as spam and advertisements. This allows us to use only minimal set
https: //hggii ngfacee co/ ssverr ns wtt.://mmmeeeeeeeeeeeeeeeeeeeeem wor.tttt-ooooooo-oooooooooo 16384

---

Published as a conference paper at ICLR 2025
of simple filters remove edge cases where the extractor fails. Through qualitative analysis, developed four filters to exclude such edge cases:
1. Content length: We exclude cleaned documents that are shorter than 100 characters.
2. Ratio of alphanumeric characters: We exclude cleaned documents whose ratio al- phanumeric characters is lower than 0.4. These documents primarily consist of data tables and are not relevant without additional context.
3. Headings per non-heading word:
We note that in some documents, only headings are extracted with little or no accompanying text. We compute the ratio of the number of headings to the total number of words from non-heading lines. If the ratio is greater than 0.05, we exclude the document.
4. Unigram entropy: Also used in Together Computer (2023), this measures the diversity of the content and is computed using 2 -z/loooal log(: (olaa)) where the sum taken over counts of unique words in the normalised content. By manual inspection, we found threshold value of 3.0 to be reasonable, and exclude all documents below it.
In Figure 4 show the distributions of these four quantities, and in Appendix provide examples of documents that are filtered out.
De-duplication We used MinHashLSH (Leskovec et al., 2020) for document level near duplicate removal. The MinHash signatures were computed using unicode code point-level shingles of size 1 14 bands, and 8 hashes per band (a total of 112 Hashes). Deduplication was done per band in an iterative fashion: For each band in order, we grouped documents by their hashes within that band, and kept only one document per group. Following Wib (Penedo et al. (024), we only performed deduplication within snapshots, and not between them, as this was shown to increase downstream performance.
PII Replacement As final processing step, we make best effort at removing personally identi- fiable information from our data. To this end, we use regular expressions to replace email addresses and publicly facing IP-adresses with one of a few samples. This follows what has been done in previous works (Penedo et al. 2024 Soldaini et al. 2024).
4 EXPERIMENTS
How good is the data produced by our pipeline? To assess this question we conduct experiments against the recently proposed Web pipeline (Penedo et al.) 2024). We do this be performing a data ablation experiment. Here, train two language models data produced by 1) our pipeline and 2) the Fine Web pipeline respectively. We then evaluate the language models as a proxy for evaluating the datasets and, in turn, the pipelines.
FineWeb uses trafilatura (Barbaresi] 202T) as HTML extractor and relies on quality filter sets from both C4 and Gopher, as well as some novel additions. A notable difference is the fact that trafilatura (in the setting used by Fine produces plain text content, while SWEb for- mat as Markdown. As mentioned in Section 3.2 we primarily retain formatting via Mark- down as feature, but note that this may also affect the learning behavior of the model. In this work however, we do not perform spe- cific ablations to single out this particular fac- tor. Please see Appendix F where we show side-by-side comparisons of trafilatura vs our extractor outputs.
How will Sweden be able to itself in the international competition strengthen its position a leading knowledge A first step is look at the that govern the allocation of state research funds.
activate knowledge mark needs
C assert criteria
institutions
Proper aboes are on the way out, wbile sneskers are spreading, The following
longer causes any sensation: blazer, trousers, and white sneakers.
A manner propriety
ensemble
attine
Figure 5: Two examples from the HP-MEK task.
Translated to English (originally in Swedish).
Wee lowercased the and removed non-alphabetic characters before creating shingles.

---

Published as a conference paper at ICLR 2025
SWED
FineWee
Exp. Dataset
#Docs
#Tokens
Tokens/doc
SWEb
32.3M
25.2B 15.8B
779.7
Web
19.2M
820.3
Table I: Stats of experimental datasets SWEb and Fine
Figure 6: Venn diagram of documents in ex- perimental SWEb and Web datasets
4.1 BENCHMARK: HP-MEK
We investigated different benchmarks to evaluate the language models An appropriate bench- mark should give good "early signals" of performance, in small model and data scales. For the Scandinavian benchmarks, the Scandeval suite (Nielsen 2023) is commonly used. However, found neither of its subtasks to be appropriate for this study, as the models didn't reach good enough performace.
Instead, we chose to develop an alternative benchmark based on the Swedish Scholastic Aptitude Test (Högskoleprovet), that denote HP-MEK We download and extract the MEK (sentence completion) section all available historical tests, and end up with a total of 460 examples. HP- MEK is a cloze style test, with masked portions of provided passage. For each passage. four alternatives of the masked portions are available, see Figure 5 We evaluate a model by inserting each of the four alternatives into the passage, and pick the alternative with the highest joint log likelihood.
In our experiments, we see early and consistently increased performance as we train on successively more data, which speaks for it being a suitable indicator for performance at larger scales.
4.2 EXPERIMENTAL SETUP
We extract, filter and deduplicate the 2024-10 and 2024-18 Common Crawl snapshots using our pipeline to form an experimental dataset (SWEb). We also the Fine pipeline on the same input documents (selected from Stage 1) to form competing dataset (FineWeb). Table compares the two and Figure@/shore
Venn diagram of their document (url) sets.
We note that the SWEb pipeline extracts significantly more documents (+62%) and tokens (+60%) than Fibessb's pipeline. Most of FineWeb's documents are contained SWEb, while relatively few are uniquely selected by FineWeb. Interestingly, FineWeb extracts slightly more tokens per document on average, despite SWEb containing additional Markdown formating tokens.
We split the two datasets in 90/10 train/test splits and tokenize using the GPT-SW3 tokenizer (Ekgren al. 2024). Then, we train small language models on each training set respectively (Msw for SWEb and Mrw for FibeWeb), and use the Llama architecture with 1.82B parameters (including embeddings) with a 2048 sequence length, a global batch size 2 million tokens and a cosine decay learning rate schedule. Each model is trained for 10,811 steps, which corresponds to one full epoch for SWEb, and 1.6 epochs for FineWeb. checkpoint every 250 steps to evaluate progression throughout training.
4.3 RESULTS
In Figure 7 we show perplexity plots where each model is evaluated the each of the two test sets. We can first note that Msw achieves lower perplexity on its own data than Mrw. i.e. SWEb seems "easier" to fit despite it being trained more unique tokens. This could for example be due to the markdown formating, where markup tokens might casier to predict. Secondly, Msw performs relatively better FineWeh than MAw SWEb. We speculate this could also be due to the markdown, where Mpw gets more confused not having seen Markdown during training.
Available ottpss///uuggiggf.....//aaaaaa/s Sweden Mooie--ooeelll..l 1oddelsHHHMMKK

---

Published as a conference paper at ICLR 2025
SWEb
FineWeb train SWED test
70%
TineWeb
FineWebb train FineWeb test
60%
Perplexity
Accuracy
20r
50%
40%
30%
10
20%
08
5B
100
150
200
58
108
Training tokens
Training
Figure 7: Perplexity cross-evaluation. The two models are evaluated both SWEb and FineWet test sets.
Figure 8: Learning curves. Performance of our two ablation models on HP-MEK throughout training.
nument
15M
10M
0M
pestee
Patr
Resce
203922
Figure 9: SWEb distribution over the Common Crawl snapshots.
Next, evaluate Msw and Mrw HP-MEK, and plot learning curves in Figure We can that Msw closely matches Mrw throughout the training, suggesting the two datasets are on-par with each other with regards to this task. This suggests that able to match the trafilatura extractor with just 1,380 annotated extraction samples, and at the same time reduce the complex filtering to only four simple quantities.
THE SWEB DATASET
Language distribution
We run our pipeline on 98 Common Crawl dumps, starting from 2013-20 until 2024-26, to produce the Scandinavian Web (SWEb)
500M
dataset. SWEb comprises a total of 1.01 tril- lion tokens distributed over 1.2 billion docu- ments, resulting 3.6TB of raw (UTF-8) text.
400M
Documents
300M
This makes SWEb the largest open Scandina- vian dataset to date, an order of magnitude larger than the (to our knowledge) previously largest mC4 dataset. In Figure []] we show the document distribution across the Common Crawl dumps. As we can see, the amount of Scandinavian content has been steady since around 2017, averaging about 50M documents per dump.
200M
100M
Figure 10: Language distribution the SWEb dataset
To investigate the language distributon of SWEb, we use the fastText language indentification classifier by Joulin et al. (2016a b). Among the four Scandinavian languages, Swedish is the dominating one with 48% of documents classified as Swedish, 26% Danish and 20% as Norwegian, see Figure 10 Only 2.3% classified as Ice-
Using the GPT-SW3 (Ekgren
2024 tokenizer
8

---

Published as a conference paper at ICLR 2025
landic. A small portion of documents are classified non-scandinavian after content extraction, of which a majority is classified as English.
We release the SWEb dataset, the pipeline code, well trained extractor model open source license, and hope this will further research and development of high performant Scandinavian LLMs. We also provide datasheet detailing the dataset further in Appendix
DISCUSSION AND FUTURE WORK
Comparing to rule-based extractors such as trafilatura, our model based extractor offers greater flex- ibility the desired extraction output is demonstrated instead of encoded as heuristics. Our work also highlights the data efficiency with which this can be done, i.e just 1,380 annoated examples in our case. However, this also comes with cost. Running our model extractor for each document in- creases the compute required substantially over rule-based alternatives, which adds to these already compute-intensine
workloads. In extracting SWEb, we consumed 20k AMD MI250X GPU-hours which is a significant amount, but comparing to the budgets required for training the downstream LLMs it is still negligable.
While training LLMs on larger datasets have shown to yield higher performance, hypothesis is that there is only subset of high quality documents that are behind the performance boosts. For example, in FineWeb-Edu. further filtering web data towards "educational content" is shown to significantly boosts performance in reasoning- and knowledge-intensin on topic and content based filtering promising avenue for further refinement of SWEb towards particular LLM capabilities. This could potentially even be built into the extractor for more fine- grained control instead of binary post-hoc filter.
benchmarks. We see work
7 CONCLUSION
major bottleneck for pre-training LLMSs for smaller languages is the lack of large and high-quality open datasets. In this paper, we have presented the thus far largest open dataset for pre-training LLMs for the Scandinavian languages (Swedish, Danish, Norwegian and Icelandic). The dataset, which we call SWEb, comprises 1 trillion high-quality tokens in said four languages, and is openly shared in order to promote the development of LLMs for the Scandinavian languages. In creating SWEb, we have also developed a pipeline with a novel model-based text extractor that offers greater flexibility over the extraction process versus rule-based alternatives. We share both code and mod- els for the novel text extractor openly. This paper has introduced a new benchmark for Swedish, which we use to compare models trained using data with models trained using FineWeb and demonstrate that our data leads to models with performance on par with models trained from data using the state-of-the-ann pipeline Web.
ETHICAL CONSIDERATIONS
Handling Sensitive Content The SWEb dataset was created using publicly available Common Crawl data. During processing, efforts were made filter out low-quality and irrelevant material, such as advertisements, spam, and repetitive text. While our approach focuses on simplified and novel pipeline for text extraction, we do not implement specific filtering mechanisms for harmful or sensitive content. We acknowledge that pre-training data can influence the behavior of LLMs, potentially amplifying biases or generating harmful outputs. We encourage researchers and practi- tioners utilizing our pipeline and dataset to critically assess their data sources and apply appropriate filtering techniques based on their ethical and application-specffi requirements. Future work should consider integrating robust content moderation strategies to mitigate risks associated with unfiltered pre-training data. Refer to the datashect in the appendix for more details on the dataset's 's curation process.
Privacy and Data Protection To address privacy concerns, some personally identifiable informa- tion (PII) such as email addresses and publicly visible IP addresses were removed using regex-basee based filters. While these methods are widely adopted in the field, we acknowledge their limitations and the
9

---

Published as a conference paper at ICLR 2025
potential for residual PII. SWEb also adheres Common Crawl's policies, which respect robots. and nofollow directives to avoid restricted data. For additional information on how PII and privacy were handled, refer to the datasheet in the appendix.
Bias and Representation The SWEb dataset derived from Common Crawl, which reflects the inherent biases of web- base data. These biases may arise from factors such as the overrepresen- tation of content from certain languages, regions, or demographics with greater internet access and technological literacy. Additionally, the dataset may include content from domains or sources that reflect specific viewpoints, and, as with any web-based resource, there is a risk of including disin- formation other misleading content. While the dataset aims to provide valuable resource for Scandinavian languages, we encourage users to remain mindful of these potential biases and con- sider mitigation strategies during training and deployment. Further details on potential biases and dataset composition can be found in the datasheet the appendix.
Intended Use and Misuse Prevention The primary goal of SWEb is to support the research and development Scandinavian language models. This dataset should not be used train models that generate harmful content, misinformation, or other unethical outputs. Users are encouraged to implement safeguards and adhere ethical AI development principles. Refer to the datasheet the appendix for guidance on intended use and potential misuse.
Takedown Policy We provide mechanism for stakeholders to request the removal of specific content through our takedown policy. For more information on this policy, see the datasheet in the appendix.
ACKNOWLEDGMENTS
We gratefully acknowledge the Swedish Innovation Agency (Vinnova) for supporting this work. This work has also been supported by the Parallel computing center (PDC) at the Royal Institute of Technology in Stockholm, Sweden. We would like to express our deepest gratitude for providing the compute required for our experiments and building SWEb. We thank our annotators for their kind effort in helping out building our text extractor model.
REFERENCES
Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. Towards a Cleaner Document-Oriented Multilingual Crawled Corpus. arXiv e-prints, uary 2022.
Stti::2001006443 Jan-
Adrien Barbaresi. Trafilatura: A web scraping library and command-lin tool for text discovery and extraction. In Heng Ji, Jong C. Park, and Rui Xia (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing: System Demonstrations, pp. 122 131, Online, August 2021. Association for Computational Linguistics. doi: 10.18663/w1/20211ac-deeee act-demo..5 URL https:
//aclanthology wtw.aattthoooooooooo222222111111 20t.//2211..clcc. -ele- 15
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv:2004.05150, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., waapseerrfieeeeeeeee files//aper/20200
2020. URL https: httppss//proceeenngg..tttt n 96741 011e:45557c5brrr3447771111111113
10

---

Published as a conference paper at ICLR 2025
Ona de Gibert, Graeme Nail, Nikolay Arefyev, Marta Bañón, Jelmer van der Linde, Shaoxiong Ji, Jaume Zaragoza-Berna Bernabeu, Mikko Aulamo, Gema Ramírez- Rumírez-Sáncher Sánchez. Andrey Kutuzov, Sampo Pyysalo, Stephan Oepen, and Jörg Tiedemann. A new massive multilingual dataset for high- performance language technologies. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 1116-1128, Torino, Italia, May 2024, ELRA and ICCL. URL org. 2024. Lzec-main. 100.
Ariel Ekgren, Amaru Cuba Gyllensten, Felix Stollenwerk, Joey Öhman, Tim Isbister, Evangelia Gogoulou, Fredrik Carlsson, Judit Casademont, and Magnus Sahlgren. GPT-SW3: An autore- gressive language model for the Scandinavian languages. In Nicoletta Calzolari, Min- Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 11886-7900 Torino, Italia, May 2024. ELRA and ICCL. URL
org/ 2024 irec main. 695
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III au2, and Kate Crawford. Datasheets for datasets, 2021. httpst arxiv.
org/abs/ 1803. 09010
Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthij Douze, Hérve Jégou, and Tomas Mikolov. Fasttext.zip: Compressing text classification models. arXiv preprint 2016a.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759 2016b.
Jure Leskovec, Anand Rajaraman, and Jeffrey David Ullman. Mining of massive data sets. Cam- bridge university press, 2020.
Konstantin Lopuhin. Evaluating quality of article body extraction for commercial ser- vices and open-source libraries, 2019.
article extrration wwwpettiiioneebemmrrr...... benchmark
Dan Nielsen. ScandEvall: A benchmark for Scandinavian natural language processing. In Tanel Alumse and Mark Fishel (eds.). Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pp. 185-201, Tórshavn, Faroe Islands, May 2023. University of Tartu Library.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien I Launay. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.0/ / 16, 2023. URL httttt: http:///aawiiv.orga/////3000 01116
Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, al. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2446. 17557, 2024.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume l (Long Pupers), pp. 2227-2237, New Orleans, Louisiana, June 2018. Association for Computational Lin- guistics. doi: 10.18653/v 1/N18- 1202. URL
org/N18 1202
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
11

---

Published as a conference paper at ICLR 2025
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbi, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lor- raine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Ange- liki Lazaridou. Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cy- prien de Masson Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew J. Johnson, Blake A. Hechtman, Laura Weidinger, lason Gabriel, William Isaac, Edward Lockhart, Simon Osindero. Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language mod- els: Methods, analysis & insights from training gopher. CoRR, abs/2112.1144 2021. URL
attps://arxix> org/ htgga//////// aws///111 11446
Adam Roberts, Colin Raffel, Katherine Lee, Michael Matena, Noam Shazeer, Peter Liu, Sharan Narang, Wei Li, and Yanqi Zhou. Exploring the limits of transfer learning with unified text-to- text transformer. Google, Tech. Rep., 2019.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Lucy, Xinxi Lyu, Nathan Lambert, Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer Noah A. Smith, Hannaneh Hajishirzi. Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. arXiv preprint, 2024. URL https: Btt.:ttttte//eev...o///////////
Robyn Speer. ftfy. Zenodo, 2019. URL http::///....ctg/0002222//2/2//. wtw.:////.......11..... Version 5.5.
A Together Computer. Redpajama: An open source recipe to reproduce Ilama training dataset, April 2023.
jama Data
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Édouard Grave. Cenet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of The 12th Language Resources and Evaluation Conference, pp- 4003-4012, 2020.
Zhipeng Xu, Zhenghao Liu, Yukun Yan, Zhiyuan Liu, Chenyan Xiong, and Ge Yu. Cleaner pre- training corpus curation with neural web scraping. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 202/ Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 483-498, Online, June 2021. Linnutttiooalllinuusitiiess htw.tthhhhollg..../22111..111111 org/2021 naacc naacl main.41
Association for Computational Linguistics. doi: 10.18653/v 1/2022 www.l-mainnnne.. URL 7/aclanthology.c
12

---

Published as a conference paper at ICLR 2025
A SWEB DATASHEET
We provide a datasheet inspired by Geeru et al. (001)) )
Motivation
Purpose of the dataset
want to encourage open research and development of LLMs in the Swedish, Danish, Norwegian and Icelandic languages. We build and release SWEb to promote this objective and address the linguistic challenges specific to underrepresented Scandinavian languages, improving ac- cess language technology these regions.
Curated by
Al Sweden
Funded by
AI Sweden
Composition
Data Fields
Each data instance contains:
1. The source URL
2. The original Common Crawl WARC file path
3. The WARC date
4. The extracted text content, in markdown format
5. The detected language (using fastText classifier)
Data Splits
We split SWEb based Common Crawl dump, to allow for download based on time of crawl. We also include a default split containing the entire dataset.
Errors and noise
As outlined in this paper, we propose a novel model based approach to extract text from websites. However, the model not perfect and non-relevant content as well as noise are sometimes also erroneously extracted. We try to filter such examples in our third pipeline stage, but despite our best effort such examples may sometimes slip through.
Offensive and toxic content
As we don't 't attempt to filter based on content or topic in this work, SWEb might contain content that can be percieved offensive, threatening or otherwise toxic. When considering using this dataset, is important to be aware of this and that further processing might be necessary depending use case.
Dataset Curation
Curation rationale
We use Common Crawl as it is the (to our knowledge) largest and most diverse open corpus available in the Scan- dinavian languages.
Source data
The Common Crawl source data consist of large amounts of wehpages crawled from the open web. Common Crawl's crawlers has always respected nofollow and robots txt policies.
Time frames of collected data
We use all Common Crawl scraped dating back to week 50 of 2013 and up to week 26 of 2024.
Data processing steps
See Section 3
13

---

Published as a conference paper at ICLR 2025
Annotations
Among the data fields, only the detected language can be considered "annotated" by us.
Personal & sensitive information
We anonymize email addresses and public IP addresses us- ing regex patterns.
Considerations for using the data
Social impact of dataset
With SWEb, goal is make L.M training more cessible to the machine learning community by: (a) mak- ing the dataset creation process more transparent, by sharing our entire processing setup including the codebase used (b) helping alleviate the costs of dataset curation, both in time and in compute, for model creators by publicly releasing our dataset with the community.
Bias and Representation
While the Common Crawl data gathers diverse text sources, biases present in the original content may still exist. Users should critically assess how these biases may affect model training and outcomes, especially sensitive applications. It is recommended implement bias-mitigation techniques during training and model development.
Model Misuse
When training models with this dataset, it is crucial to vent harmful uses of the resulting models, such as generat- ing misleading dangerous content (c.g., disinformation, hate speech). Always consider the societal impact de- ploying models trained this data, and take precautions to implement appropriate safeguards.
Distribution
Distribution platform
The dataset will be distributed on the Huggingface Hub
License
The data is released under the CCO Creative Commons Li- cense. We make the following clarifications:
1. We not warrant or guarantee any rights to the underlying data contained within this dataset. Users are solely responsible for validating and se- curing the appropriate rights and licenses for their specific intended uses.
2. This license applies only to the structure and com- pilation of the dataset provided by us. We do not claim any database rights or ownership over the un- derlying data itself. Users must ensure compliance with any legal obligations, including those related to third-party content, copyrighted material, or per- sonal information (PII) that may be contained in the underlying data.
3. With the release of this dataset, our goal is to pro- mote and advance open research and the develop- ment of Scandinavian language models, showcase research outcomes as well as enable research vali- dation. Open datasets are essential to fostering in- novation and expanding knowledge in AI. We dis- claim any responsibility for other uses, including commercial applications. Users are responsible for ensuring the legality of their usage, especially in cases involving copyrighted material.
14

---

Published as a conference paper at ICLR 2025
Notice and take-down policy
Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please:
1. Clearly identify yourself, with detailed contact data such as address, telephone number email ad- dress at which you can be contacted.
2. Clearly identify the copyrighted work claimed to be infringed.
3. Clearly identify the material that is claimed to be infringing and information reasonably sufficient to allow us to locate the material.
4. You can reach at datasets ai.se
We will comply to legitimate requests by removing the af- fected sources from the next release of the corpus.
15

---

Published as a conference paper at ICLR 2025
B MARKDOWN ANNOTATION DETAILS
Text Annotation Tool
raggul Dillatiio
likkig
Maa te
ioplicaddi
teppippo/oppeeii ongloliteeeto
Previous
18/600
Next
Ignored
Arncestoos
PSYELOPEDIN
Telefon
maple woliom eieyeette hrerte tnooe Locheegi
cie ned in
paerral sox serien alggenes son anaire qjl an cosse for laacc
lerges
aegas
ravigeriig Hegan
Futege::
Talefie
Sperabide
relg mocis
Telafce ugottistng upatanee Haraal,, naketion gir sandar
Leciting
Blier
tecingrates
McdioWlli spar to accoed Claaniaacc
rietace touttver
tcalets.
Or
Aategert:
Soord
Figure 11: Our web based annotation tool. On the right side the original web page is displayed and on the left the corresponding markdown. Annotation is performed by selecting individual lines (marked green) that constitute the main content of the page.
We develop a web based tool that we use to annotate markdown documents, see Figure 11 The tool used annotate data for training and evaluating our text extractor model (Section 3.2.2
The annotation was performed by the authors as well as additional lab colleagues, in total a group of 12 people. We started by jointly annotating gold standard test of 100 examples (web pages). This was useful to align and develop our annotation guidelines.
Next, we annotated a first set of 400 training examples and trained first extractor model. This model served as a first baseline. We then iteratively annotated additional training data in batches of 300-500 examples, re-trained and re-evaluated after each iteration.
Judging what is "main content" in web pages is not always obvious however. When the evaluation didn't 't improve after new batch of annotations, we developed a method for discovering "confusing" training examples in the new batch that we could jointly discuss and align on. For each example in the new training batch, we compute the loss MM (I. y) L(M, (x), y), where the average over all BCE losses in the example and M,, is the model trained on all batches including iteration n.
By comparing this loss to the corresponding loss under the previous model Mn-1, we get a measure of how "surprising" this example is:
5 = 1 -1 MM (x,
Using this quantity, we could casily identify outliers and correct inconsistent annotations. By per- forming this post-annotation fix-up, we were able to improve performance on our set, for each annotated batch of data.
16

---

Published as a conference paper at ICLR 2025
C CONTENT EXTRACTION ANNOTATION GUIDELINES
The following description was provided to our annotators
In the provided annotation tool, please select individual lines by clicking and dragging across the lines you want select.
Please look at the rendered web page on the right. We want to extract the "main textual content" of the current page.
No navigation (menus, links, button texts etc) should be selected, except well formatted tables of content that link within the same page
Include headers of main content
If duplicate header, select the one closest to the main content
Include well formatted tables
Don't t include content of sidebars that is unrelated to the main content
It is OK leave the whole document unselectede if there is no relevant content
If there are many very similar-looking pages, they can be marked Ignored if they have al- ready been annotated. Bad pages without any good content should not be ignored however.
Include comment sections if there are any, but exclude navigation associated with those, e.g. Svara Rapportera inlägg similar.
Keep comment headings
If text is broken with e.g. "...", don't 't include
Select top heading exists
Keep at most consecutive newlines
Remove empty formatting lines (e.g **), except for dividers
Pages that primarily "data" (e.g. tables, numbers) without much text should be unse- lected. There should be at least three consecutive sentences of text. This puts a somewhat high bar for product pages
No HTML should be selected
17

---

Published as a conference paper at ICLR 2025
D CONTENT EXTRACTOR ERROR ANALYSIS
To analyze the errors made by our context extractor, we use the line-level annotations of the 100 validation documents. For each line in each document, the line is predicted be extracted or ignored. Therefore, for each document, we get number of FP and FN line classifications. In Figure we plot the distribution over number of FP and FN lines.
12
60
40
Docs
20
10
10
Num false positive lines
Num false negative lines
Figure 12: Distribution number of False Positive/Negative lines, in the test set documents
We can see that the vast majority of documents have less than 10 wrongly classified lines. The document with the highest number of false negative lines is shown in Figure 13 In this case, the table of contents, some headings as well as the reference list were wrongly not extracted.
18

---

Published as a conference paper at ICLR 2025
portas nod vntcesibars eive sorree ofter mnzoecaniiong vi) Asstendo 141 Rucinssa. Alls 22 vebure oioo, Lands con tin averaks crrrrrr Guater Adxl,, con raurore log
Wratt Blrt Gtantel stlawdoy www.hhagddsoooooeoe...... wwtp wtg. farina
taget
felgg flat sab startes felge glait tarrann flgesiage dn offire thii hillse plut beril adavis tepaskiffer
1:84
1312 tadiee
Flacce 12 silemtvens flvor pastraa for at axrape pascepaer 45 goed keaftig vieticas lies trie or kirte rpsole socra 1as as Piyets oar for andal, diso
pllctere toe or koein bavee gler Tjere lsear 1il replemantt (01221)) weatrate allw ls apxi apsiives Lerrace kb. 13 forttig flyst kos abograng surciige
Hllttt aletiagte Edtreteend vikanetis wwwpki/iiiooooooeeeeeeeoooomm wtwpa//taatatttttttrraaooo coveulesrr aygll Topusanttophrro..ccooo 19y!
Sesion aue
fenulnande Hizemooee rokaeedia wwwpssiiccainnmmeoo.oo. Degglent Jt.iiins tmp.b./p//lbb.. Aaslenu Nu.J
Netuger
Itrre Festrnp cefendt sig aobord "lyet besstri secuerrrr passegere:: cvandrr tem, disee ay diraere fire sver suer, fransiandd
Flyets
tilkepttxt digarede hande wan fffer, fiejet bake vrdie Batavia sakarty) daroo faretaget sery and Tipnngge canas bastra Spancet, madt. tan Lng alder oadiumet Comar, ((ES1) Flyers styrtand
rifrce
bodds
side wiet pu battt cenera 101P511 thati Arsif rejett and adtaenn Siber Sheee var cocile
iid atllablncreee lastras zhap pastagorer Kipe 19e pascagater or reie se bord. Elactt aszeagenirss der 1112 abbedd Kastria us der raanet ceertzige dhsneptiirr
famce meceillertree Corce He amnn demes went curebter 1ilmor 2A cen Jeta bower (11 Foyeca kentee panigere sode presaie kentpened, Ove menke anoraries gar zake
lig fntsprefere lofereenee
Hlaartaf wilfffrropeess snecca ang www.s/itoeeeaaaaaoooooooooooaaaa comnane 2013 rertren Bastaf abolPhss Late, anstrobstiea, (pgrit pe-iraroo austaf Aap3Pas cake westroluttisa ies)
(rree toors Lnnglas rricteedia www.oopddaiccmmmseoooo Heturta aal /r Kraccnsere Jagi iilpp fracemocre 200)
traes
Herrss aoyordo file splaae sitzar dta any wwwppooccaaaaaaccccccoooooo www.deaaaaccccceeeeeeeeeooao mwes www.eeeeeeeeee serrss depaerdr filer praittape-der totpe-eerra- Meyursderffer pegt
eent Sesmeseffe
Bediger
Ancrieebe conkee db
ret". Ankovenet oropteaten E. Rertet
sttre, salf: canh ceas, rad, (1990) 11900-12ien bot thin stroabladerr, ansodett:: Aftorsladet net 1odd ar statess mebverr. 196, Isle 61-900-0----- 900---0--4 (lette beaks)
stace More Hear to Threae Boe Eramute lngales Tames 22, Jesar 190,, 2.
Aeteeeee teptmins hars
feducer sasic inlo (wnritanniitttienn leedil 2915
arsttal-uutifleree gobrratte D. Mitiaanst 24, cesterier los Rapack badiiio horkerr Inskertandkkeelleeee hecnst saj bers
feeesaa Peleee Killee Breek Trem after Teikng Sooniepm, truse herre Vietim the
Jemaor 1947 S8M6
Figure 13: The extraction for true positives and blue lines are false negatives. True negative lines are not shown for illustrative purposes.
(ilyylykkkiyyyyykkkn.... i 1994 Green lines
19

---

Published as a conference paper at ICLR 2025
E
FILTERED EXAMPLES
We show examples of extracted documents that are are filtered out by each of our four quality filters.
E.1
CONTENT LENGTH < 100 CHARS
Sttps: /wwws buckerudnyst.oo no/ http://www.......oooaoaaoa/aaaea
30-ore-1 197--1879---11
Tato txa rill, Litt ekjoldete nyst,
NOX3, mv.
1t htpp::/wssowwweeeeee ovedan 2eo0nae/2001///// wwtp/////////////0 htth///////////oao////// fic.--rte--yp--eeeeee.. not torget /erbed/
Ohenana Clok in.L bygge nyLi mut
Lorgel
Achtung Die Kurve
E.2
RATIO OF ALPHANUMERIC CHARACTERS < 0.4
Sktpps //wwa www.wa..nneeeeee..... 2622 sandeback
CL98IC
Juniorsllevensian HO18
14
16
2+26*
sttpe httpes//nn.oo http://w....iiiieeeeeeeee..o /tateriii/ttaaaaaaeeee ltakerrr under winter-Dr etter Riving
1895 ** Arve 1900 xx A. 1904 x Arox 1906 xx Ares 1908 xx Arze 1912 x Ausr (=111161-> x A.sr 1920 x
1924 Arve 1928 vx Arwe 1932 vx Arve 1936 Arvx >- -=(1940)== xx leve - (1944)-- leve 1948
Arex 1952 Aras
Arax 1968 Aras 1972 kras R..s
Rexe 1992 Aoor 1996
2000
2004 2008 Ae
2012 ** Arwe 2016** Rosr 2020 <AVinter DL.A
Deltakerer etter SANASjONZA.
1924 Arre 1923 Arse 1932 Arex 1936 xx Rrvs (1940)- xx Ares en (1944) vx Ârse 1948 x Aexo 1992 Rear 1956 Rrva 1960 Arvs Aras 1968 Rran 1932 Beik 1976 Brvs Rras 2988 <x 2014 ex Arve 2018 Arae 2022
Arox 1998 Arva Arzs 2005 Arx. 2010
Deliakares vaRwinggN
1t //hi stori bo k.val www..a////vee........o...oo.oo.e sw/val//v 100
wwww/derkon...oooooo html
Gamia staden, Stortorget
2533
24,88 86
380
33.05
659 [ 20,45
312
11:199
102
4,43
68
52,
61
806
47,48
727
13,88
Summa
24,89
20,45
312
21,99
4,45
68
52,
68
806
47,48
727
13,99
212
betp://sww.va..
E.3 HEADINGS PER NON-HEADING WORD > 0.05
attps: //ww//////ee.. eahlgreacks. co/tor-dig http://wwww.....///////////////a
Glaskropp
Gl askropp cytologisk diagnos
Synonymer
20

---

Published as a conference paper at ICLR 2025
Spacialprovez, eytologisk diagnoscik
# ecortaspingessviionnng
Fronmateriel
12
13 128 Bōz el. notav
14
aed gul kuck, koniak boklen, simzil mindre nängdee macecial) Buck tolutande Lock, sterilt
16
Proviaguing
19
Enligt inrenittarand: kliniks regict Frovet skall Eherast etter sk013ringen traneporteras til1 Cytologen.
20
Ofixerade vatakor aia EBramnäla och lanas direk, til1 lab, persoral I0re ki 14.30.
21
Transport
23
24 Transport aka ske engdende till laboratorist for klinisk patolooi och dāz 1lnnas direkt till previals mmineen.
sttps://folk. sunsberdland. www..//aaaa.aa///occc..ooo///aaa
Qullet vázt. olive wanat 2 arw eJa, vennen, på lerdaq 18. nov, tyller du dzw Me gratulezer aå masse)! gla"
bestefarcidre aideferoldre.
(9-12 &r)
10
Silder. Fredheim
http:///tart.ercsoo...uu http:///aaaeaoeeee.oorrre/e//eee
inieeeeinnn
(2022-01-01 till 2622-03-13)
hivd/kategori
## Cytel/nivv
Vrkeshögska (reehigskoleeeamee eexamen
etusiedr
11 :
12 *
Ontattning
14 sp
15
Kenpeteasndi
denna studieenzet står jandende konpetenser
19 rckus:
20 \-
nemveppropramering
Kompetens databashantering lagring
22 data
23 \- Kenpeteasen att skaps dynamiska applikationer
24
iMrandcresultat
26
27 Efter avlagd studieerhet:
28 5- behärskar progsomering ned
(Kunekap)
30 \- Du ser skilinaden mellan statisks, interaktiva
31 och synaniska vebbsidor (Kunskap)
Kilenten
(Kanskap)
34 \- Du kan hypgt dynemicka vebaspger (Ftrdighet!
35 Du kan lagra data sakert databas
36 (FSrSighet)
problematik 1ssninger kring 38 lagra kinaliq information on en anvindare
39 [EDrnSilnincsstte)
uppfattas alika aütt Boes data och koppling till obkeherhet
42 prestanda (Pagssiinn.
43 Du uppfattas din nekt och diLL
E.4 UNIGRAM ENTROPY < 3.0
httpes//hestaae kett logen, se Foont ent, hooree//ii wnw.reeee//oo.oooooeo www.ooooo..l...omm id
21

---

Published as a conference paper at ICLR 2025
Catinkaox
Arabiskt Pullblod
Catinnka är ett Föddex 1976 Sverigee
Beat Arabiski Fullblod
Färge
12
Stocemil.j
13
skchorn-ck/
Tel akoborn ek
Ida skohosn ol jad ek Tean sntländaka Westroth, Willweckad Coempeosent Loa. Bn Cin detal] L111 hallenk
16a ekohorn ex mängd
10
13
14
blged andländska Lirkaad Forppresset detalf ballenn!
17
430
# leikgenansxei0
#Bfundurs Beidkjört Arneyl2013 2012-00 2--7- 12P10: 09100:0012, ja1r 20171
** Veieludagur rucnirn upp
10
11
Dwelarflokeer
14
15 Rétundur: Reidbjort Arneyl 2001-000220111111:11000::::000 joní 20191
17
18
Leikjandnsxei0
21
Retundors Heidbjort Arneyl 201 7-06-21713+ 08:0000000022 joní 20191
22

---

Published as a conference paper at ICLR 2025
F COMPARING OUR MODEL EXTRACTOR VS TRAFILATURA
We compare our model based extractor vs trafilatura (in the settings used by FineWeb).
https: http///wwwwppkkkkkkkko/ wtt..o//roosktt...t.oao http///ttttt////oooooo wttttt/bbboo////eo sokke//oounenns-------- htt.:///soooootttt.oo///oeees--s
Trafilatura
Model extractor (ours)
Inabundat
Ekail on klosterlasae andre aksemplez
2003
Nozisk, Boknal
Quarg Johannesen:
<Danne boken dreier seg on eksil og dannelse. Lesning ev Danto neg italiensk regol: wksil.
5. <Denne beken ireier seg on ckell dinaelse.
bannelse oppaiáe alliid ug Dac vesle som fina dannelse Norge, dannes ut fra evnen til distance 1 et Livanedvendig indre allez yluw wksil, bannelse
Lesning Dante may itallensk reqel: bannelse oppatár alliid og bare cksil. Det vesio son fins av danneise 1 Norge, dannea ut fea avnnn Lil diatanse livanedvendig indre aller Dannelse er det notsaile we bur lane. 811k faste nin se selvomsrg meg atadig max ina it av nozek kelter leg
turisma. sLsk faste min vomaogg neg stadig ner inn og ul av norsk kultuz (og underholdning) til 1eg ble uhalbrodaliq qudles or partil vag,
oksil.
livelang skaemplariske flykiningez as focradeze Ira Klostarlasse til Asbjarr Bunda,> 6. [fra Georg Cohennesens farord)
undecholdning
jeg ble ubelbowdellg
gudias og partilas en vas, awn livalang inceresse foz eksomplariska flykininger focradezw Sra Klosteslesse til Asbjarn Sunda.s (Sra Georg Cehannesena forard)
KlikksBant
lagez betikkez
9. Lalllgger Swndex nornalt imnen 1 vickedager
10 Byt: 1 alle väze buzikker
11
Klikk og bent
13
Trafllatura
Model extractor (ours)
Hur don än
Varl don än Lros
2. Päzgezna Re blokaze än igās
2
ciden är För möre
le
Rodelics
jay will inte bäre deras rist ner
Ellusionez av tröst son drar mlg ner ss ska vi halla hand jävla världen äe i brend
5. hetgs: /ttpppmaapc...c///////////aaaeap wew.ees.seeeeee se/vp- con--contttt/ttt /uplodda
Bur de lin
/20 420/4/10//44, Ved-don -e---d-------e-e ior-
6.
hot don an sincer ose sá ska vi striks högst
7. x*Hor dom dn*x
9 eet nej är alleid est nej
8.
10 Vark vi an gär aws vi nex
9. Fazgeene äs blukare an: igar
aldrig
igen, aldrig me:
ticen
ast konne jeg will inte höre derws rost mer
12 bele idd fär vi sage t:ll
13 jusen ing slar base still
12. I'lasionss ave LeBlat, asn deas niy aes
14 Bur in
sá ska vi hálla hand
jävla världen är brand
sargar
hálla hand
16 nur don an sinker oss sd 5ka vi striks höget
Heive jevia viciden or a. brand
17 ett nej är alleid est neg
16 O hur don än sanker oes ed sike vi skrika högst att är alleid ebs nej
18 En vacker stråle son föravann
mörk: 20 Innnn det blev kalls
det
Vert vi gån ser vs ner 20 aldrig ner igen, aldrig ner hels tiden såe stiga sill Ijusen runtonkring sthr
21 0ch on det var dira skrik son inte hördes
22 eller
din doster
kärnan, täroes iväg dada
23 skulle
still
24 Bur de Sn sarger oss sá ska vi hálla hand
25 Baiva 18v1a variden ar i brand
24 For de an eargar oas ed eka vi hália hand brand
26 hor don säncer ss ska vi skriks högst
25 Iva javla världen
27 nod ar alltid est neg
hur don in sänker oes sd eka vi ekeika högst
ett ned är alltid etz nej
vacker strile tärevann 28
innan blev nockt innan det blev kallt
32
dch det dine skesk inte hördes ciler var dotter con tordes ivig
hur skaile det kinnas, her skullc d5ss
36
hália hand
Heiva javia vicideen brand
hur dom in sänker des eá ske vi. skrika högst
40 ett ncd dr alitid etz nej
Albunapde
23

---

Published as a conference paper at ICLR 2025
Trafilatura
Model extractor (ours)
parinforras
Bi Vaage Norvik
Tea Vigstranda Norvika Ridabygda
Rauna vannet av bilferqen: Vaage-
3 ##
100 MCX pr passas
Narvik xon gilk der fro 1930 sil 1945.
4
Vei Lil Andalatas ble Lil
del bygget uncer andre
Tux Tnssmas jon
ferdig ev otkupasjonsmstte ve dendenniigg og veien Apret runct LidspunkLes Cus freden 1945.
7.
Tur Fra Vägat t:l Norvika Ridshygda Rauna i kjallvnmet billergen Vaage- gilk
tergen bygget
bathygger
bnten går burene ned og det Fostall, historie cunds delle san. hendel bene rundt den sragiske ulykken oksober
9. Vei Lil Andalames ble til stor del byogel ferdig
Agnis under andre
verdenskrig
Propogancamtnister Quieling regjeringen Gulbrand lunde n/frue onkon ved fergekalen på
sidepunktet for creden 1965, Dence rergen ble bygget samree båthyggge son bater turene ned del blir corcalz historie hendel runde den tragiske alytken aktaber 1942 hvor Kaltar Propagardemniitte Caisling regjeringen fande tergekaien på Vaage.
Turprisen oppgits pr passasjer
på
dette
max antalll Te kontakt alzernat og evl alleggier.
Eventuelt servering onbord!
fiskesugpe servert n/hyackt kres, dessert (Tilelerte
tonat/ct /ohi//hhli
bondepiker) og kokekatte, Re, 3a0,-
bankjpakke fre Braud ländverksbaterii Vestnes: Pásmurt bagetz
Turpriser sen oppgitt pazsaxjer basert max antall. kontakt for altornativer og evt allergior.
kanelbolle alt, sclekimmsollle, Kr. 130.-
10 3. Pasmurt begets red kylling & kerr!
11 kaneibolle alt, sciekinasbolle.
*vEventucit servering ambord)!*s
kyylling og Kr, 175.-
12 Hinessivena og aaren drikks fãs kiept soporat bord.
15. Bix tomat/chill basert krenet riskesuppe servers n/nybakt brad, deseert (I:ls.erte bonsepikerl kokekatte. Kr. 350.-
13 Timer
Mako, Passaejerer: 12
tansipakke tra Broud Ehnvertezskeri Vestnes: 2%. Pásmurt bagett med ost skinke
16
17 Vestnes
sancizolic
solskiansbolls. Kr. 110.-
bagett nen sylling kanelbolle ait. solskiansboli.. Rr. 120.-
20 4%. Pastaszlat mod kylling karri. Er. 175.-
21
Mineralvann og anaen drikke kjapt separat on bord.
23
Tiner
Pascantorer: Avgang:
27
Cucsprik:Eaczlen Norsk
24


Reviews:
[{"review_id": "Oe3Iqi0KZe", "reviewer": "Reviewer_picV", "summary": "The paper introduces SWEb, the largest pretraining dataset for Scandinavian languages, containing over one trillion tokens across Swedish, Danish, Norwegian, and Icelandic. SWEb aims to address the scarcity of large-scale, high-quality datasets specifically tailored for these languages. To create this dataset, the authors develop a model-based text extraction pipeline that enhances efficiency and reduces complexity compared to rule-based methods. Key contributions include:\n\n- SWEb Dataset: An extensive web dataset of Scandinavian languages with over one trillion tokens, surpassing existing resources by an order of magnitude.\n- Model-Based Extraction Pipeline: A novel, data-driven text extraction model that effectively filters high-quality content, yielding about 60% more usable tokens than previous approaches like FineWeb.\n- Swedish Benchmark (HP-MEK): A new cloze-style benchmark for Swedish, derived from the Swedish Scholastic Aptitude Test, to evaluate language models trained on SWEb and demonstrate competitive performance against models trained on FineWeb.\n\nThe authors openly release the SWEb dataset, extraction pipeline, and the HP-MEK benchmark to support further research and development in Scandinavian language modeling​", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality\n\nThe SWEb dataset is original in its approach to handling Scandinavian languages. The authors create a model-based extraction process that moves away from rule-heavy, manual extraction methods, simplifying the pipeline. They also introduce HP-MEK, a benchmark specific to Swedish, which adds value by providing a relevant evaluation tool for Scandinavian models.\n\nQuality\n\nThe quality of the work is evident in the detailed steps of the SWEb pipeline. The authors carefully build a process to select and clean high-quality data, resulting in 60% more usable tokens than previous approaches like FineWeb. They validate the dataset with clear metrics, comparing models trained on SWEb and FineWeb to show the effectiveness of their extraction model.\n\nClarity\n\nThe paper is organized well, making each pipeline stage easy to understand. Diagrams and examples help clarify complex steps like content extraction and filtering. The authors document their choices for filtering and quality control, making the approach easier to follow and replicate.\n\nSignificance\n\nSWEb is significant because it makes a high-quality, large-scale dataset available for Scandinavian languages, which traditionally have fewer resources. This dataset and the HP-MEK benchmark can help researchers build better models for these languages, making SWEb a useful resource for Scandinavian language research.", "weaknesses": "1. Limited Applicability Beyond Scandinavian Languages\nWeakness: The SWEb pipeline is tailored specifically for Scandinavian languages, potentially limiting its scalability or adaptability to non-Scandinavian or low-resource languages. This narrow focus may reduce the general utility of SWEb’s approach in multilingual or global settings where language resources are scarcer.\n\nRecommendation: It would be beneficial to discuss adapting the pipeline to other language families or the performance challenges in non-Scandinavian languages. Providing generalization strategies, such as multilingual training or enhanced language detection techniques, could expand SWEb’s relevance. Including preliminary results or small-scale tests on other language groups would further strengthen the paper’s broader applicability.\n\n2. Reliance on Manual Annotation in Model Training\nWeakness: Although SWEb’s model-based extractor is innovative, it relies on manually annotated data (1,380 samples) for training the extraction model, which may be resource-intensive and impractical for other languages or domains. Annotating thousands of samples for every new language could become a bottleneck, especially in low-resource contexts.\n\nRecommendation: Introducing semi-supervised or weakly supervised learning approaches to reduce the dependency on manually annotated data could improve scalability. Alternatively, SWEb could consider leveraging existing rule-based systems or transfer learning from similar languages to jumpstart the model training in new language settings, potentially improving data efficiency.\n\n3. Evaluation Restricted to Swedish Benchmark\nWeakness: The evaluation uses only a Swedish benchmark (HP-MEK) to compare SWEb against FineWeb. While effective for a Swedish-specific evaluation, this choice does not cover other Scandinavian languages in the dataset (e.g., Danish, Norwegian, Icelandic), making it difficult to generalize the effectiveness of SWEb’s extraction pipeline across the entire language set.\n\nRecommendation: Expanding the evaluation to include benchmarks for other Scandinavian languages or adapting HP-MEK for Danish, Norwegian, and Icelandic could enhance the assessment’s robustness. Providing a language-specific performance analysis would offer insights into whether the extraction model’s quality varies across languages and help optimize future language-specific models.\n\n4. Lack of Qualitative Analysis of Extracted Content\nWeakness: The quantitative metrics (e.g., token count, perplexity, accuracy) demonstrate SWEb’s improvements but lack a qualitative assessment of the extracted text's relevance or coherence. Without this, it’s challenging to understand how well the extraction model preserves the content’s intended meaning or cultural context, especially when removing ads and navigation elements.\n\nRecommendation: Including a qualitative evaluation of content extracted by SWEb compared to FineWeb, such as reader surveys or manual inspection of content fidelity, would offer a deeper understanding of its cultural and contextual accuracy. This analysis would help validate the extractor’s ability to maintain high content relevance, especially for Scandinavian-specific terms, phrases, or topics that might be lost during processing.\n\n5. Limited Error Analysis in Content Extraction\nWeakness: The paper does not provide an in-depth error analysis for the types of errors encountered during extraction, such as failures to remove advertisements, incorrect content classification, or issues in handling specific webpage structures. This gap makes it difficult to assess the limitations of the extraction model and how it could be improved.\n\nRecommendation: An error analysis that categorizes extraction mistakes (e.g., missed ads, incorrectly retained menu items, or misclassified headers) would clarify the model’s boundaries and suggest refinements. Detailing any challenges in handling regional dialects or slang in Scandinavian languages would identify areas for improvement in future iterations of SWEb.\n\n6. Computational Expense of the Extraction Process\nWeakness: The computational requirements for SWEb’s extraction model, which consumed 20,000 GPU hours on AMD MI250X GPUs, may be prohibitive for many research labs or developers working with limited resources. This factor could limit the pipeline’s accessibility and adoption.\n\nRecommendation: Considering optimizations in the extraction model, such as fine-tuning on smaller, more frequent batches or experimenting with lighter transformer architectures, could reduce computational demands. Additionally, presenting a cost-benefit analysis comparing SWEb’s compute usage to the downstream performance gains would offer a more balanced view of the pipeline’s scalability and efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uQM9ZcCjLd", "reviewer": "Reviewer_SsWA", "summary": "The paper describes the creation of a currently largest Scandinavian data set for training language models.\nA new method for collecting texts from the web is proposed, based on an encoder model, and compared with a rule-based method.", "questions": "182: What were are the annotators exactly instructed to do? How did they mark the main content? \n\n195 isn't => is not\n\n200-202: what are the \"binary line annotations\" exactly? \nCan they be seen in the example in Listing 1?\n\n300-301: What is \"trafilatura\"? Explanation citation? \n\nWhy is markdown better than plain text? \n\n308: alternative benchmarks => alternative to what?\n\n309: to evaluate performance on => performance of what? Of the proposed text extraction method? \n\n\n315: didn't => did not\n\n351: which model? \n\n376: on the two test sets -- which ones? the one from 90/10 split and the other is HP-MEK?\n\n\nFigure 1: what is \"en\"? It is not discussed in the text.\n\n\n\n426: as the desired extraction output is demonstrated instead of encoded as heuristic => the meaning of the sentence is unclear; what does \"demonstrated\" means? What does \"encoded as heuristic\" means?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The data set will be a valuable tool for researching language models and understanding their properties for Scandinavian languages, which are all under-resourced and under-investigated.", "weaknesses": "Some important points about the evaluation are not fully clear.\n\n\"Benchmark HP-MEK\" is unclear: what was the motivation to created this test set and what was exactly evaluated exactly on it?  \nThe new text extraction method or language models trained on the extracted texts? \nFor language models (section 4.2) it is said that there is 90/10 training/test splitting.\nTherefore it is not clear what was involved in evaluations (text extractor or language models or both) and how (on which test set/s).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rh9hvjWSlN", "reviewer": "Reviewer_dCGi", "summary": "This work introduces a new pretraining dataset for Scandinavian languages, as well as a cloze-style evaluation dataset for Swedish language models.", "questions": "The pretraining dataset supports Scandinavian languages, but the benchmark targets Swedish. With the Swedish benchmark as a starting point, how much work would it be to generate similar benchmarks for other Scandinavian languages?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Usefulness and relevance: Improving the quality and assessing that this quality has indeed improved for LLMs targeting languages other than English is a clearly relevant topic that will have uses even outside of academia.\n\n- Open-source: Authors promise the release of the dataset, benchmark, and utils used to generate/evaluate them.\n\n- Technical details and examples: The paper features many detials about the implementation. Even if it was not open-source, I feel fairly confident that this work could be majorly reproduced.", "weaknesses": "While it's a sound contribution, I'm not sure ICLR is the right venue for this work. It lacks algorithmic or theoretical novelty, and it's rather a (very good) application of well-known NLP principles to process a new dataset for specific languages.\n\nFor example, the heuristics mentioned in the paper are very similar to \"old\" related work, e.g. [1]\n\nhttps://arxiv.org/abs/1912.07076", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bAAxYLIgzO", "reviewer": "Reviewer_4Vny", "summary": "The paper presents a new pretraining corpus for Scandinavian languages:Swedish, Danish, Norwegian, and Icelandic. To create this data, they propose a new pipeline that uses a model-based text extractor trained with a small amount of human-labelled data on what is considered the main content in a markdown version of an HTML webpage. They contrast their proposed approach against the FineWeb pipeline, which extracts plain text directly from HTML pages. They further introduce a new cloze style test with a dataset, HP-MEK consisting of 460 examples based on the Swedish Scholastic Aptitude Test (Högskoleprovet),  to benchmark pre-trained performance and show that on a small dataset+model setting, their extractor can match the performance of FineWeb filter, despite being simpler in complexity and using an extractor that is only trained with 1380 human-labeled examples.\n\nOnce validated, they use this pipeline to create SWEb, which includes 1.01 trillion tokens.", "questions": "See weakenesses.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors present a novel model-based extractor, trained with 1380 human-labelled examples, to identify the main context from documents (HTML converted to markdown).\n2. They introduce a new task and benchmark to validate their pipeline against a baseline, FineWeb and show that their pipeline, while being simpler, results in close performance on this task.\n3. They present a new pretraining corpus of 1.01 trillion tokens for Scandinavian languages, which will be a valuable resource to the community.", "weaknesses": "1. It is unclear why the authors chose to retain the markdown tags for pretraining (section 4). They also do not explain much about why HP-MEK is a reasonable benchmark for the pretraining setting.\n2. No direct analysis is presented on the efficacy of the model-based extractor, given that it is one of the main component of the paper apart from reporting an F1 score. The downstream application in section 4 is useful but it doesn't say much about what this extractor does or is capable of filtering.\n3. The dataset pipeline only includes filters like content length, # of alphanumeric characters, and unigram entropy for quality filtering. However, it would have been useful if there was any direct consideration of what is considered high-quality data in the context of pretraining and if additional checks were made in place about safety and fairness of representation in the dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)