SWEb: A Large Web Dataset for the Scandinavian Languages

## Abstract

Abstract This paper presents the hitherto largest pretraining dataset for the Scandinavian languages: the Scandinavian WEb (SWEb), comprising over one trillion tokens. The paper details the collection and processing pipeline, and introduces a novel model-based text extractor that significantly reduces complexity in comparison with rule-based approaches. We also introduce a new cloze-style benchmark for evaluating language models in Swedish, and use this test to compare models trained on the SWEb data to models trained on FineWeb, with competitive results. All data, models and code are shared openly.

## 1 Introduction

Large language models have made significant strides in recent years due to their general capabilities in language-processing tasks.
This progress has been largely driven by the development of extensive and high-quality pretraining datasets sourced from open web data (Wenzek et al., ; Brown et al., ; Abadji et al., ; Penedo et al., ; ).
However, the majority of research aimed at improving pretraining data focuses on high-resource languages such as English.
Our goal is to create a large-scale and high-performing open pretraining dataset specifically for the Scandinavian (north-germanic) languages: Swedish, Danish, Norwegian, and Icelandic.

Existing large-scale datasets for these languages primarily include mC4 (Xue et al., ), OSCAR (Abadji et al., ), and HPLT Datasets 1.2 (de Gibert et al., ).
The Scandinavian portion of mC4 comprises approximately 100B tokens, 10B tokens for OSCAR 23.01, and 35B tokens for HPLT, which are all relatively small numbers considering that state-of-the-art large language models today are trained on trillions of high-quality tokens.

In this paper we make the following contributions:

- •
We release(^1^11Data available here: [https://huggingface.co/datasets/AI-Sweden-Models/SWEb](https://huggingface.co/datasets/AI-Sweden-Models/SWEb)) the largest to date pretraining dataset for the Scandinavian languages: Scandinavian WEb (SWEb). SWEb is the result of running our proposed pipeline on 98 Common Crawl snapshots. SWEb contains 1.01 trillion tokens in the Scandinavian languages, approximately an order of magnitude more than other available open alternatives.
- •
We introduce a new cloze-style benchmark for evaluating language models in Swedish, HP-MEK, a subset of the Swedish Scholastic Aptitude Test (Högskoleprovet) used for university admissions in Sweden. Using HP-MEK, we show our data performs on-par with data from the recently proposed FineWeb (Penedo et al., ) pipeline.
- •
We propose a new comprehensive pipeline for curating pretraining data for large language models, built around a model-based text extractor that significantly reduces complexity and is easily adaptable through rapid data annotation(^2^22Code and extractor model is available here: [https://github.com/aidotse/SWEb](https://github.com/aidotse/SWEb)). Most notably, we demonstrate that our pipeline returns about +60% more high quality tokens than FineWeb on the same input data.

## 2 Background and Related Work

Early efforts to extract massive amounts of text from the open internet for LLM training start from WebText (Radford et al., ), developed for training GPT-2.
In this case, outbound links from Reddit with a certain number of upvotes were used as the content selection criterion.
Text was extracted using Dragnet (Peters et al., ) and Newspaper(^3^33[https://github.com/codelucas/newspaper](https://github.com/codelucas/newspaper)) and filtered with several heuristics, resulting in a dataset of 40GB after deduplication.
Soon after, CCNet (Wenzek et al., ) and C4 (Roberts et al., ) were proposed, both based on open web data from Common Crawl.
C4 was initially developed exclusively for English but was later followed by a multilingual version, mC4 (Xue et al., ).
CCNet, on the other hand, was multilingual from the outset.

Both CCNet and C4 are based on the WET archives from Common Crawl, where all HTML formatting has been stripped, leaving only the text.
However, this text still contains a significant amount of noise in the form of menu and ad text, headers, footers, and sidebars, which are irrelevant to the page’s primary content.
A successful method for extracting primary content from WET archives is to deduplicate the documents at the line level.
C4 globally deduplicates all lines, while CCNet deduplicates over a subset of documents from the same Common Crawl dump.
Line-by-line deduplication is the primary extraction method in CCNet, whereas C4 additionally employs a range of English-specific cleaning heuristics.

Following extraction comes a language detection and filtering step.
Whilst more computationally expensive, performing language detection post extraction been shown to achieve better detection accuracy than filtering pre extraction (especially for low-resource languages) (Wenzek et al., ).
Quality filtering differs slightly between the two, with C4 filtering using several heuristics, a bad words filter, and URL deduplication.
In contrast, CCNet employs a model-based filter, using perplexity as a quality measure with a KenLM model trained on Wikipedia.

CCNet has since been utilized in subsequent works such as RedPajama (v1 and v2) (Together Computer, ) and Dolma (Soldaini et al., ).
RedPajama-Data v2 runs CCNet on an expanded number of Common Crawl snapshots and filters for five high-resource languages (none of which are Scandinavian, however).
They also extend CCNet’s quality filtering by pre-computing a larger set of popular quality signals but leave the thresholding and filtering to the user.

Recently, several works have moved away from Common Crawl’s WET archives in favor of processing the raw HTML of webpages found in the WARC archives.
Utilizing mor sophisticated text extraction turns out to be critical for the improving quality of the resulting data (Penedo et al., ).
In MassiveWeb (Rae et al., ), the tree structure of HTML is utilized to more easily group and identify the primary content of pages.
Some formatting is also retained, with the argument that this “diversity in formatting style translates effectively to the generative capabilities of the Gopher models.”

A similar approach is developed in NeuScraper (Xu et al., ), where a model is trained to – on an element level – decide whether it should be extracted or not.
Both RefinedWeb and FineWeb use the open-source framework Trafilatura (Barbaresi, ) to extract text from HTML.
Trafilatura is based on rules and heuristics on the DOM tree to identify primary content and has been shown to be the best non-commercial extractor for certain domains (Lopuhin, ).
However, quality issues are still prevalent, and in RefinedWeb (Penedo et al., ) further (line-based) filters are added in an attempt to address these.

MassiveWeb introduce what they call “repetition filters” to remove documents with repetitive text, that is found beneficial with their extractor.
These are also sucessfully reused in both RefinedWeb and later FineWeb.
Through a systematic analysis, FineWeb further adds a small set of quality filters, that is shown through ablation experiments to yet increase quality.
For a state of the art pipeline like FineWeb, the filtering can add up to 30 or more quantities and rules that might be difficult to oversee and adapt to new languages.

## 3 The SWEb Pipeline

As evident by the previous section, much focus has been placed on the development of heuristics and filters to enhance the quality of the resulting data.

To move away from the extensive number of manual thresholds and complex extraction rules, we propose a more data-driven alternative.
By learning a model for extraction, this complexity can be significantly reduced.

We begin by describing our pipeline that, like existing approaches, consists of the overarching steps of content selection, extraction, quality filtering, and deduplication (Figure 1).

Figure: Figure 1: The SWEb pipeline. We use Common Crawl’s preprocessed WET archives for content selection, and WARC for extraction. At the center stage sits our model based Markdown extractor, that is the primary workhorse to produce our dataset.
Refer to caption: x1.png

### 3.1 Stage 1 : Content Selection

Our pipeline begins with content selection, which aims to identify and select source documents from Common Crawl that are likely to be in one of the Scandinavian languages.
Since the Scandinavian languages make up a very small portion of the entire Common Crawl, we want to implement this step early to filter out all non-relevant content.

We use CCNet to identify Scandinavian documents within the entire Common Crawl dataset.
CCNet processes the WET archives, and after line-by-line deduplication, language detection is performed using fastText (Joulin et al., ).
Documents with a detected language score above 0.2 for any of the four languages are selected for the next stage.

### 3.2 Stage 2 : Content Extraction and Formatting

In Stage 2, we start from the documents indentified in Stage 1 but discard their content and instead use Common Crawl’s index to download their original HTML from the WARC archives.
This means we use CCNet and the WET documents solely for content selection, and not for extraction.
In the WET archives, all formatting and structure, such as header information, tables, text styles, bullet lists, and images, have been removed.
Like MassiveWeb (Rae et al., ), we believe that such structural and layout information can be valuable for language models to learn from, and therefore, we aim to extract this information from the webpages.

We propose a new method for extracting primary content from the webpages, consisting of two steps: 1) Convert HTML to Markdown, 2) Extract primary content from the resulting Markdown through line-by-line filtering with a trained model.

#### 3.2.1 Convert HTML to Markdown

Since we want to preserve basic textual formatting, we choose to convert from HTML to Markdown with its very lightweight markup, thus does not add many extra tokens.
We convert all incoming HTML documents to Markdown using Pandoc, stripping links and images.
See Listing 1 for an example.

Figure: Listing 1: A webpage converted to markdown (translated, originally in Swedish), including title, top menu, headings and primary content. The document is truncated for brevity.

No extraction has yet taken place, so these documents are still full of noise from menus, advertisements, and other extraneous content.
We address this in the next step.

#### 3.2.2 Model-Based Content Extraction

We observe that individual lines in the Markdown documents often correspond to specific elements such as headers, paragraphs, or navigation links.
This makes lines an appropriate level for extraction.
Therefore, we develop a custom annotation tool (details in Appendix B) to annotate which lines in these Markdown documents should be extracted and which should not.
We ask annotators to mark what is considered the “main content” on the current webpage, and make some principled decisions for quality and consistency:

- 1.
We do not extract navigation text such as menus or buttons.
- 2.
A significant portion of the webpages are product pages. We decide to extract these only if there is a product description consisting of at least three complete sentences.
- 3.
We extract tables if they are well-formatted and their content is tightly coupled to the main content.
- 4.
On blogs or article pages that include user comments, we extract such comments in addition to the main content.
- 5.
We do not extract information from sidebars unless it clearly constitutes main content.

While not explicitly excluded as per our guidelines, advertisement text isn’t considered to be main content and is thus implicitly excluded.
The full annotation guidelines can be found in Appendix C.
In total, we annotate 1,380 webpages, using 100 of these for validation and the remainder as training data for our extraction model.

##### Line Extraction Model

Figure: Figure 2: Illustration of our proposed line classification model. Each newline is replaced by a special 
 token, and the corresponding embeddings are used for classification
Refer to caption: x2.png

Our dataset consists of Markdown documents with corresponding binary line annotations, and we aim to train a model to predict this label for each line.
For this purpose, we choose to use a transformer encoder, where each newline is replaced with a special token [SEP].
We then feed the entire document through the encoder, with each [SEP] token representing the preceding line.
This way, each line classification is contextualized by (theoretically) the full document context.

$$ $h_{0:n}=\text{Encoder}(x_{0:n})$ (1) $$

Through a linear projection of the output hidden state of each [SEP] token, we obtain logits for predicting the binary label of the current line.
Let $j$ denote the token index corresponding to each [SEP] token in the document.
We then get the predicted probability for the line as:

$$ $p_{j}=\sigma(Wh_{j}+b)$ (2) $$

where $\sigma$ is the sigmoid function.
The model is trained using binary cross-entropy loss between each $p_{j}$ and an annotated line label.
See Figure 2 for an illustration.
We apply a fixed threshold to $p_{j}$ to determine whether to include or exclude the line.

Figure: Figure 3: Precision/recall of our final line extraction model. We pick a threshold of 0.05 at inference, e.g. when applying the model for extraction.
Refer to caption: x3.png

The Markdown documents can be very long, so we use the Longformer (Beltagy et al., ) architecture.
Specifically, we use a pre-trained model that supports up to 16k tokens and has been trained for representation learning using masked language modeling(^4^44[https://huggingface.co/severinsimmler/xlm-roberta-longformer-base-16384](https://huggingface.co/severinsimmler/xlm-roberta-longformer-base-16384)).
The Longformer is a linear complexity transformer, thanks to its local self-attention, where each token only attends to a fixed number of nearby tokens.
We use a local attention window size of 256 tokens and no global attention, as this turned out to only impair generalization.

We fine-tune the entire model on our training set of 1,280 documents, and the results on the validation set can be seen in Figure 3.
We use the Adam optimizer with a constant learning rate of 1e-5.
The results show that despite our small-scale training data, we achieve an F1 score of 87%.

Finally, we normalize the text using Fix Text For You (Speer, ).

### 3.3 Stage 3 : Quality Filtering and Cleaning

The third stage aims to filter for quality, reduce duplicate content and remove personally identifiable information (PII).

##### Quality Filtering

Figure: Figure 4: Filtering distributions on two Common Crawl dumps, and exclude regions marked in red. We exclude documents whose content length is shorter than 100 chars (invisible in the chart).
Refer to caption: x4.png

A significant advantage of our model-based extraction is that it also implicitly performs much of the quality filtering.
The extractor effectively learns to exclude content that is not of sufficient quality, such as spam and advertisements.
This allows us to use only a minimal set of simple filters to remove edge cases where the extractor fails.
Through qualitative analysis, we developed four filters to exclude such edge cases:

- 1.
Content length: We exclude cleaned documents that are shorter than 100 characters.
- 2.
Ratio of alphanumeric characters: We exclude cleaned documents whose ratio of alphanumeric characters is lower than 0.4. These documents primarily consist of data tables and are not relevant without additional context.
- 3.
Headings per non-heading word: We note that in some documents, only headings are extracted with little or no accompanying text. We compute the ratio of the number of headings to the total number of words from non-heading lines. If the ratio is greater than 0.05, we exclude the document.
- 4.
Unigram entropy: Also used in Together Computer (), this measures the diversity of the content and is computed using $\sum-x/total*\log(x/total)$ where the sum is taken over counts of unique words in the normalised content. By manual inspection, we found a threshold value of 3.0 to be reasonable, and exclude all documents below it.

In Figure 4, we show the distributions of these four quantities, and in Appendix D, we provide examples of documents that are filtered out.

##### De-duplication

We used MinHashLSH (Leskovec et al., ) for document level near duplicate removal.
The MinHash signatures were computed using unicode code point-level shingles of size 16(^5^55We lowercased the text and removed non-alphabetic characters before creating shingles.), 14 bands, and 8 hashes per band (a total of 112 Hashes).
Deduplication was done per band in an iterative fashion: For each band in order, we grouped documents by their hashes within that band, and kept only one document per group.
Following FineWeb (Penedo et al., ), we only performed deduplication within snapshots, and not between them, as this was shown to increase downstream performance.

##### PII Replacement

As a final processing step, we make a best effort at removing personally identifiable information from our data.
To this end, we use regular expressions to replace email addresses and publicly facing IP-adresses with one of a few samples.
This follows what has been done in previous works (Penedo et al., ; Soldaini et al., ).

## 4 Experiments

How good is the data produced by our pipeline?
To assess this question we conduct experiments against the recently proposed FineWeb pipeline (Penedo et al., ).
FineWeb uses trafilatura as HTML extractor and relies on quality filter sets from both C4 and Gopher, as well as some novel additions.
A notable difference is the fact that trafilatura (in the setting used by FineWeb) produces plain text content, while SWEb format as markdown.
Please see Appendix E where we show side-by-side comparisons of trafilatura vs our extractor outputs.

### 4.1 Benchmark: HP-MEK

Figure: Figure 5: Two examples from the HP-MEK task. Translated to English (originally in Swedish).
Refer to caption: x5.png

We investigated different alternative benchmarks to evaluate performance on.
An appropriate benchmark should give good “early signals” of performance, in small model and data scales.
For the Scandinavian benchmarks, the Scandeval suite (Nielsen, ) is commonly used.
However, we found neither of its subtasks to be appropriate for this study, as the models didn’t reach good enough performace.

Instead, we chose to develop an alternative benchmark based on the Swedish Scholastic Aptitude Test (Högskoleprovet), that we denote HP-MEK(^6^66Available at [https://huggingface.co/datasets/AI-Sweden-Models/HP-MEK](https://huggingface.co/datasets/AI-Sweden-Models/HP-MEK)).
We download and extract the MEK (sentence completion) section of all available historical tests, and end up with a total of 460 examples.
HP-MEK is a cloze style test, with masked portions of a provided passage.
For each passage, four alternatives of the masked portions are available, see Figure 5.
We evaluate a model by inserting each of the four alternatives into the passage, and pick the alternative with the highest joint log likelihood.

In our experiments, we see early and consistently increased performance as we train on successively more data, which speaks for it being a suitable indicator for performance at larger scales.

### 4.2 Experimental setup

We extract, filter and deduplicate the 2024-10 and 2024-18 Common Crawl snapshots using our pipeline to form an experimental dataset (SWEb).
We also run the FineWeb pipeline on the same input documents (selected from Stage 1) to form a competing dataset (FineWeb).
Table 1 compares the two and Figure 6 shows a Venn diagram of their document (url) sets.

Figure: Table 1: Stats of experimental datasets SWEb and FineWeb
Refer to caption: x6.png

We note that the SWEb pipeline extracts significantly more documents (+62%) and tokens (+60%) than FineWeb’s pipeline.
Most of FineWeb’s documents are contained in SWEb, while relatively few are uniquely selected by FineWeb.
Interestingly, FineWeb extracts slightly more tokens per document on average, despite SWEb containing additional markdown formating tokens.

We split the two datasets in 90/10 train/test splits and tokenize using the GPT-SW3 tokenizer (Ekgren et al., ).
Then, we train small language models on each training set respectively ($M_{\text{SW}}$ for SWEb and $M_{\text{FW}}$ for FineWeb), and use the Llama architecture with 1.82B parameters (including embeddings) with a 2048 sequence length, a global batch size of  2 million tokens and a cosine decay learning rate schedule.
Each model is trained for 10,811 steps, which corresponds to one full epoch for SWEb, and 1.6 epochs for FineWeb.
We checkpoint every 250 steps to evaluate progression throughout training.

### 4.3 Results

Figure: Figure 7: Perplexity cross-evaluation. The two models are evaluated on both SWEb and FineWeb test sets.
Refer to caption: x7.png

In Figure 8, we show perplexity plots where each model is evaluated on the each of the two test sets.
We can first note that $M_{\text{SW}}$ achieves lower perplexity on its own data than $M_{\text{FW}}$, i.e. SWEb seems “easier” to fit despite it being trained on more unique tokens.
This could for example be due to the markdown formating, where markup tokens might be easier to predict.
Secondly, $M_{\text{SW}}$ performs relatively better on FineWeb than $M_{\text{FW}}$ on SWEb.
We speculate this could also be due to the markdown, where $M_{\text{FW}}$ gets more confused not having seen markdown during training.

Next, we evaluate $M_{\text{SW}}$ and $M_{\text{FW}}$ on HP-MEK, and plot learning curves in Figure 8.
We can see that $M_{\text{SW}}$ closely matches $M_{\text{FW}}$ throughout the training, suggesting the two datasets are on-par with each other with regards to this task.
This suggests that we are able to match the trafilatura extractor with just 1,380 annotated extraction samples, and at the same time reduce the complex filtering to only four simple quantities.

## 5 The SWEb dataset

Figure: Figure 9: SWEb distribution over the Common Crawl snapshots.
Refer to caption: x9.png

Figure: Figure 10: Language distribution over the SWEb dataset
Refer to caption: x10.png

We run our pipeline on 98 Common Crawl dumps, starting from 2013-20 until 2024-26, to produce the Scandinavian Web (SWEb) dataset.
SWEb comprises a total of 1.01 trillion tokens(^7^77Using the GPT-SW3 (Ekgren et al., ) tokenizer), distributed over 1.2 billion documents, resulting in 3.6TB of raw (UTF-8) text.

This makes SWEb the largest open Scandinavian dataset to date, an order of magnitude larger than the (to our knowledge) previously largest mC4 dataset.
In Figure 9, we show the document distribution across the Common Crawl dumps.
As we can see, the amount of Scandinavian content has been steady since around 2017, averaging about 50M documents per dump.

To investigate the language distributon of SWEb, we use the fastText language indentification classifier by Joulin et al. (; ).
Among the four Scandinavian languages, Swedish is the dominating one with 48% of documents classified as Swedish, 26% as Danish and 20% as Norwegian, see Figure 10.
Only 2.3% are classified as Icelandic.

We release the SWEb dataset, the pipeline code, as well as our trained extractor model open source license, and hope this will further research and development of high performant Scandinavian LLMs.
We also provide a datasheet detailing the dataset further in Appendix A.

## 6 Discussion and Future work

Comparing to rule-based extractors such as trafilatura, our model based extractor offers greater flexibility as the desired extraction output is demonstrated instead of encoded as heuristics.
Our work also highlights the data efficiency with which this can be done, i.e just 1,380 annoated examples in our case.
However, this also comes with a cost.
Running our model extractor for each document increases the compute required substantially over rule-based alternatives, which adds to these already compute-intensive workloads.
In extracting SWEb, we consumed 20k AMD MI250X GPU-hours which is a significant amount, but comparing to the budgets required for training the downstream LLMs it is still negligable.

While training LLMs on larger datasets have shown to yield higher performance, a hypothesis is that there is only a subset of high quality documents that are behind the performance boosts.
For example, in FineWeb-Edu, further filtering web data towards “educational content” is shown to significantly boosts performance in reasoning- and knowledge-intensive benchmarks.
We see work on topic and content based filtering as a promising avenue for further refinement of SWEb towards particular LLM capabilities.
This could potentially even be built into the extractor for more fine-grained control instead of as a binary post-hoc filter.

## 7 Conclusion

A major bottleneck for pre-training LLMs for smaller languages is the lack of large and high-quality open datasets. In this paper, we have presented the thus far largest open dataset for pre-training LLMs for the
Scandinavian languages (Swedish, Danish, Norwegian and Icelandic).
The dataset, which we call SWEb, comprises 1 trillion high-quality tokens in said four languages, and is openly shared in order to promote the development of LLMs for the Scandinavian languages.
In creating SWEb, we have also developed a pipeline with a novel model-based text extractor that offers greater flexibility over the extraction process versus rule-based alternatives. We share both code and models for the novel text extractor openly.
This paper has introduced a new benchmark for Swedish, which we use to compare models trained using our data with models trained using FineWeb, and we demonstrate that our data leads to models with performance on par with models trained from data using the state-of-the-art pipeline FineWeb.

#### Acknowledgments

We gratefully acknowledge the Swedish Innovation Agency (Vinnova) for supporting this work.
This work has also been supported by the Parallel computing center (PDC) at the Royal Institute of Technology in Stockholm, Sweden. We would like to express our deepest gratitude for providing the compute required for our experiments and building SWEb.
We thank our annotators for their kind effort in helping out building our text extractor model.

## Appendix A SWEb Datasheet

We provide a datasheet inspired by Gebru et al. ():

| Motivation |  |
| --- | --- |
| Purpose of the dataset | We want to encourage open research and development of LLMs in the Swedish, Danish, Norwegian and Icelandic languages. We build and release SWEb to promote this objective and to address the linguistic challenges specific to underrepresented Scandinavian languages, improving access to language technology in these regions. |
| Curated by | AI Sweden |
| Funded by | AI Sweden |
| Composition |  |
| Data Fields | Each data instance contains:<br>1.<br>The source URL<br>2.<br>The original Common Crawl WARC file path<br>3.<br>The WARC date<br>4.<br>The extracted text content, in markdown format<br>5.<br>The detected language (using fastText classifier) |
| Data Splits | We split SWEb based on Common Crawl dump, to allow for download based on time of crawl. We also include a default split containing the entire dataset. |
| Errors and noise | As outlined in this paper, we propose a novel model based approach to extract text from websites. However, the model is not perfect and non-relevant content as well as noise are sometimes also erroneously extracted. We try to filter such examples in our third pipeline stage, but despite our best effort such examples may sometimes slip through. |
| Offensive and toxic content | As we don’t attempt to filter based on content or topic in this work, SWEb might contain content that can be percieved as offensive, threatening or otherwise toxic. When considering using this dataset, it is important to be aware of this and that further processing might be necessary depending on use case. |
| Dataset Curation |  |
| Curation rationale | We use Common Crawl as it is the (to our knowledge) largest and most diverse open corpus available in the Scandinavian languages. |
| Source data | The Common Crawl source data consist of large amounts of webpages crawled from the open web. Common Crawl’s crawlers has always respected nofollow and robots.txt policies. |
| Time frames of collected data | We use all Common Crawl scraped dating back to week 50 of 2013 and up to week 26 of 2024. |
| Data processing steps | See Section 3. |
| Annotations | Among the data fields, only the detected language can be considered “annotated” by us. |
| Personal & sensitive information | We anonymize email addresses and public IP addresses using regex patterns. |
| Considerations for using the data |  |
| Social impact of dataset | With SWEb, our goal is to make LLM training more accessible to the machine learning community by:<br>(a) making the dataset creation process more transparent, by sharing our entire processing setup including the codebase used<br>(b) helping alleviate the costs of dataset curation, both in time and in compute, for model creators by publicly releasing our dataset with the community. |
| Bias and Representation | While the Common Crawl data gathers diverse text sources, biases present in the original content may still exist. Users should critically assess how these biases may affect model training and outcomes, especially in sensitive applications. It is recommended to implement bias-mitigation techniques during training and model development. |
| Model Misuse | When training models with this dataset, it is crucial to prevent harmful uses of the resulting models, such as generating misleading or dangerous content (e.g., disinformation, hate speech). Always consider the societal impact of deploying models trained on this data, and take precautions to implement appropriate safeguards. |
| Distribution |  |
| Distribution platform | The dataset will be distributed on the Huggingface Hub |
| License | The data is released under the CC0 Creative Commons License.<br>We make the following clarifications:<br>1.<br>We do not warrant or guarantee any rights to the underlying data contained within this dataset. Users are solely responsible for validating and securing the appropriate rights and licenses for their specific intended uses.<br>2.<br>This license applies only to the structure and compilation of the dataset as provided by us. We do not claim any database rights or ownership over the underlying data itself. Users must ensure compliance with any legal obligations, including those related to third-party content, copyrighted material, or personal information (PII) that may be contained in the underlying data.<br>3.<br>With the release of this dataset, our goal is to promote and advance open research and the development of Scandinavian language models, showcase research outcomes as well as enable research validation. Open datasets are essential to fostering innovation and expanding knowledge in AI. We disclaim any responsibility for other uses, including commercial applications. Users are responsible for ensuring the legality of their usage, especially in cases involving copyrighted material. |
| Notice and take-down policy | Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please:<br>1.<br>Clearly identify yourself, with detailed contact data such as an address, telephone number or email address at which you can be contacted.<br>2.<br>Clearly identify the copyrighted work claimed to be infringed.<br>3.<br>Clearly identify the material that is claimed to be infringing and information reasonably sufficient to allow us to locate the material.<br>4.<br>You can reach us at datasets@ai.se<br>We will comply to legitimate requests by removing the affected sources from the next release of the corpus. |

## Appendix B Markdown annotation details

Figure: Figure 11: Our web based annotation tool. On the right side the original web page is displayed and on the left the corresponding markdown. Annotation is performed by selecting individual lines (marked green) that constitute the main content of the page.
Refer to caption: extracted/5904980/Figures/annotation_tool.png

We develop a web based tool that we use to annotate markdown documents, see Figure 11.
The tool is used to annotate data for training and evaluating our text extractor model (Section 3.2.2).

The annotation was performed by the authors as well as additional lab colleagues, in total a group of 12 people.
We started by jointly annotating a gold standard test set of 100 examples (web pages).
This was useful to align and develop our annotation guidelines.

Next, we annotated a first set of 400 training examples and trained a first extractor model.
This model served as a first baseline.
We then iteratively annotated additional training data in batches of 300-500 examples, re-trained and re-evaluated after each iteration.

Judging what is “main content” in web pages is not always obvious however.
When the evaluation didn’t improve after a new batch of annotations, we developed a method for discovering “confusing” training examples in the new batch that we could jointly discuss and align on.
For each example $x$ in the new training batch, we compute the loss $l^{M_{n}}(x,y)=\mathcal{L}(M_{n}(x),y)$, where $\mathcal{L}$ is the average over all BCE losses in the example and $M_{n}$ is the model trained on all batches including iteration $n$.

By comparing this loss to the corresponding loss under the *previous* model $M_{n-1}$, we get a measure of how “surprising” this example is:

$$ $\delta=l^{M_{n-1}}(x,y)-l^{M_{n}}(x,y)$ (3) $$

Using this quantity, we could easily identify outliers and correct inconsistent annotations.
By performing this post-annotation fix-up, we were able to improve performance on our test set, for each annotated batch of data.

## Appendix C Content Extraction Annotation Guidelines

The following description was provided to our annotators
In the provided annotation tool, please select individual lines by clicking and dragging across the lines you want to select.

- •
Please look at the rendered web page on the right. We want to extract the “main textual content” of the current page.
- •
No navigation (menus, links, button texts etc) should be selected, except well formatted tables of content that link within the same page
- •
Include headers of main content
- •
If duplicate header, select the one closest to the main content
- •
Include well formatted tables
- •
Don’t include content of sidebars that is unrelated to the main content
- •
It is OK to leave the whole document unselectede if there is no relevant content
- •
If there are many very similar-looking pages, they can be marked Ignored if they have already been annotated. Bad pages without any good content should not be ignored however.
- •
Include comment sections if there are any, but exclude navigation associated with those, e.g. Svara / Rapportera inlägg or similar.
- •
Keep comment headings
- •
If text is broken with e.g. “…”, don’t include
- •
Select top heading if it exists
- •
Keep at most 2 consecutive newlines
- •
Remove empty formatting lines (e.g **), except for dividers (———)
- •
Pages that are primarily “data” (e.g. tables, numbers) without much text should be unselected. There should be at least three consecutive sentences of text. This puts a somewhat high bar for product pages
- •
No HTML should be selected

## Appendix D Filtered examples

We show examples of extracted documents that are *filtered out* by each of our four quality filters.

### D.1 Content length < 100 chars

Figure: [https://www.buskerudmynt.no/produkt/norske-mynter-etter-1874/norske-argangsmynter/50-ore/olav-v-1974-1991/50-ore-1977-kv.-0](https://www.buskerudmynt.no/produkt/norske-mynter-etter-1874/norske-argangsmynter/50-ore/olav-v-1974-1991/50-ore-1977-kv.-0)

Figure: [https://www.ovedanielsson.se/2021/08/30/ohrmans-fick-inte-bygga-nytt-mot-torget/embed/](https://www.ovedanielsson.se/2021/08/30/ohrmans-fick-inte-bygga-nytt-mot-torget/embed/)

Figure: [https://jesper.nu/spel/achtung-die-kurve](https://jesper.nu/spel/achtung-die-kurve)

### D.2 Ratio of alphanumeric characters < 0.4

Figure: [https://www.innebandystats.se/statistik/219645/kevin-sandeback](https://www.innebandystats.se/statistik/219645/kevin-sandeback)

Figure: [https://nn.wikipedia.org/wiki/Kategori:Deltakarar_under_vinter-OL_1984_etter_øving](https://nn.wikipedia.org/wiki/Kategori:Deltakarar_under_vinter-OL_1984_etter_%C3%B8ving)

Figure: [https://historik.val.se/val/val2010/alkon/K/valdistrikt/12/80/0102/alderkon.html](https://historik.val.se/val/val2010/alkon/K/valdistrikt/12/80/0102/alderkon.html)

### D.3 Headings per non-heading word > 0.05

Figure: [https://www.sahlgrenska.se/for-dig-som-ar/vardgivare/laboratoriemedicin/analyslistan/specialprover-cytologisk-diagnostik/16648.html/](https://www.sahlgrenska.se/for-dig-som-ar/vardgivare/laboratoriemedicin/analyslistan/specialprover-cytologisk-diagnostik/16648.html/)

Figure: [https://folk.sunnhordland.no/publications/211951](https://folk.sunnhordland.no/publications/211951)

Figure: [https://start.arcada.fi/sv/kurser/303000/2021-2022/IA-2-004/0](https://start.arcada.fi/sv/kurser/303000/2021-2022/IA-2-004/0)

### D.4 Unigram entropy < 3.0

Figure: [https://hastkatalogen.se/content/horse/info.php?id=31999](https://hastkatalogen.se/content/horse/info.php?id=31999)

Figure: [https://www.nilssonsilammhult.se/hallmobler/ida-skohorn-ek/](https://www.nilssonsilammhult.se/hallmobler/ida-skohorn-ek/)

Figure: [https://kaldarsel.is/author/heidbjort-arney/](https://kaldarsel.is/author/heidbjort-arney/)

## Appendix E Comparing our model extractor vs trafilatura

We compare our model based extractor vs trafilatura (in the settings used by FineWeb).

Figure: [https://www.ark.no/produkt/boker/dokumentar-og-faktaboker/eksil-9788202253912](https://www.ark.no/produkt/boker/dokumentar-og-faktaboker/eksil-9788202253912)

Figure: [https://gipomusic.se/track/vad-dom-an-tror/](https://gipomusic.se/track/vad-dom-an-tror/)

Figure: [https://fjordsaga.com/no/turer/2-i-kjolvannet-av-bilfergen-vaage-norvik](https://fjordsaga.com/no/turer/2-i-kjolvannet-av-bilfergen-vaage-norvik)

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: N/A
soundness: 3
strengths: Originality

The SWEb dataset is original in its approach to handling Scandinavian languages. The authors create a model-based extraction process that moves away from rule-heavy, manual extraction methods, simplifying the pipeline. They also introduce HP-MEK, a benchmark specific to Swedish, which adds value by providing a relevant evaluation tool for Scandinavian models.

Quality

The quality of the work is evident in the detailed steps of the SWEb pipeline. The authors carefully build a process to select and clean high-quality data, resulting in 60% more usable tokens than previous approaches like FineWeb. They validate the dataset with clear metrics, comparing models trained on SWEb and FineWeb to show the effectiveness of their extraction model.

Clarity

The paper is organized well, making each pipeline stage easy to understand. Diagrams and examples help clarify complex steps like content extraction and filtering. The authors document their choices for filtering and quality control, making the approach easier to follow and replicate.

Significance

SWEb is significant because it makes a high-quality, large-scale dataset available for Scandinavian languages, which traditionally have fewer resources. This dataset and the HP-MEK benchmark can help researchers build better models for these languages, making SWEb a useful resource for Scandinavian language research.
summary: The paper introduces SWEb, the largest pretraining dataset for Scandinavian languages, containing over one trillion tokens across Swedish, Danish, Norwegian, and Icelandic. SWEb aims to address the scarcity of large-scale, high-quality datasets specifically tailored for these languages. To create this dataset, the authors develop a model-based text extraction pipeline that enhances efficiency and reduces complexity compared to rule-based methods. Key contributions include:

- SWEb Dataset: An extensive web dataset of Scandinavian languages with over one trillion tokens, surpassing existing resources by an order of magnitude.
- Model-Based Extraction Pipeline: A novel, data-driven text extraction model that effectively filters high-quality content, yielding about 60% more usable tokens than previous approaches like FineWeb.
- Swedish Benchmark (HP-MEK): A new cloze-style benchmark for Swedish, derived from the Swedish Scholastic Aptitude Test, to evaluate language models trained on SWEb and demonstrate competitive performance against models trained on FineWeb.

The authors openly release the SWEb dataset, extraction pipeline, and the HP-MEK benchmark to support further research and development in Scandinavian language modeling​
weaknesses: 1. Limited Applicability Beyond Scandinavian Languages
Weakness: The SWEb pipeline is tailored specifically for Scandinavian languages, potentially limiting its scalability or adaptability to non-Scandinavian or low-resource languages. This narrow focus may reduce the general utility of SWEb’s approach in multilingual or global settings where language resources are scarcer.

Recommendation: It would be beneficial to discuss adapting the pipeline to other language families or the performance challenges in non-Scandinavian languages. Providing generalization strategies, such as multilingual training or enhanced language detection techniques, could expand SWEb’s relevance. Including preliminary results or small-scale tests on other language groups would further strengthen the paper’s broader applicability.

2. Reliance on Manual Annotation in Model Training
Weakness: Although SWEb’s model-based extractor is innovative, it relies on manually annotated data (1,380 samples) for training the extraction model, which may be resource-intensive and impractical for other languages or domains. Annotating thousands of samples for every new language could become a bottleneck, especially in low-resource contexts.

Recommendation: Introducing semi-supervised or weakly supervised learning approaches to reduce the dependency on manually annotated data could improve scalability. Alternatively, SWEb could consider leveraging existing rule-based systems or transfer learning from similar languages to jumpstart the model training in new language settings, potentially improving data efficiency.

3. Evaluation Restricted to Swedish Benchmark
Weakness: The evaluation uses only a Swedish benchmark (HP-MEK) to compare SWEb against FineWeb. While effective for a Swedish-specific evaluation, this choice does not cover other Scandinavian languages in the dataset (e.g., Danish, Norwegian, Icelandic), making it difficult to generalize the effectiveness of SWEb’s extraction pipeline across the entire language set.

Recommendation: Expanding the evaluation to include benchmarks for other Scandinavian languages or adapting HP-MEK for Danish, Norwegian, and Icelandic could enhance the assessment’s robustness. Providing a language-specific performance analysis would offer insights into whether the extraction model’s quality varies across languages and help optimize future language-specific models.

4. Lack of Qualitative Analysis of Extracted Content
Weakness: The quantitative metrics (e.g., token count, perplexity, accuracy) demonstrate SWEb’s improvements but lack a qualitative assessment of the extracted text's relevance or coherence. Without this, it’s challenging to understand how well the extraction model preserves the content’s intended meaning or cultural context, especially when removing ads and navigation elements.

Recommendation: Including a qualitative evaluation of content extracted by SWEb compared to FineWeb, such as reader surveys or manual inspection of content fidelity, would offer a deeper understanding of its cultural and contextual accuracy. This analysis would help validate the extractor’s ability to maintain high content relevance, especially for Scandinavian-specific terms, phrases, or topics that might be lost during processing.

5. Limited Error Analysis in Content Extraction
Weakness: The paper does not provide an in-depth error analysis for the types of errors encountered during extraction, such as failures to remove advertisements, incorrect content classification, or issues in handling specific webpage structures. This gap makes it difficult to assess the limitations of the extraction model and how it could be improved.

Recommendation: An error analysis that categorizes extraction mistakes (e.g., missed ads, incorrectly retained menu items, or misclassified headers) would clarify the model’s boundaries and suggest refinements. Detailing any challenges in handling regional dialects or slang in Scandinavian languages would identify areas for improvement in future iterations of SWEb.

6. Computational Expense of the Extraction Process
Weakness: The computational requirements for SWEb’s extraction model, which consumed 20,000 GPU hours on AMD MI250X GPUs, may be prohibitive for many research labs or developers working with limited resources. This factor could limit the pipeline’s accessibility and adoption.

Recommendation: Considering optimizations in the extraction model, such as fine-tuning on smaller, more frequent batches or experimenting with lighter transformer architectures, could reduce computational demands. Additionally, presenting a cost-benefit analysis comparing SWEb’s compute usage to the downstream performance gains would offer a more balanced view of the pipeline’s scalability and efficiency.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 182: What were are the annotators exactly instructed to do? How did they mark the main content? 

195 isn't => is not

200-202: what are the "binary line annotations" exactly? 
Can they be seen in the example in Listing 1?

300-301: What is "trafilatura"? Explanation citation? 

Why is markdown better than plain text? 

308: alternative benchmarks => alternative to what?

309: to evaluate performance on => performance of what? Of the proposed text extraction method? 


315: didn't => did not

351: which model? 

376: on the two test sets -- which ones? the one from 90/10 split and the other is HP-MEK?


Figure 1: what is "en"? It is not discussed in the text.



426: as the desired extraction output is demonstrated instead of encoded as heuristic => the meaning of the sentence is unclear; what does "demonstrated" means? What does "encoded as heuristic" means?
soundness: 2
strengths: The data set will be a valuable tool for researching language models and understanding their properties for Scandinavian languages, which are all under-resourced and under-investigated.
summary: The paper describes the creation of a currently largest Scandinavian data set for training language models.
A new method for collecting texts from the web is proposed, based on an encoder model, and compared with a rule-based method.
weaknesses: Some important points about the evaluation are not fully clear.

"Benchmark HP-MEK" is unclear: what was the motivation to created this test set and what was exactly evaluated exactly on it?  
The new text extraction method or language models trained on the extracted texts? 
For language models (section 4.2) it is said that there is 90/10 training/test splitting.
Therefore it is not clear what was involved in evaluations (text extractor or language models or both) and how (on which test set/s).

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: The pretraining dataset supports Scandinavian languages, but the benchmark targets Swedish. With the Swedish benchmark as a starting point, how much work would it be to generate similar benchmarks for other Scandinavian languages?
soundness: 4
strengths: - Usefulness and relevance: Improving the quality and assessing that this quality has indeed improved for LLMs targeting languages other than English is a clearly relevant topic that will have uses even outside of academia.

- Open-source: Authors promise the release of the dataset, benchmark, and utils used to generate/evaluate them.

- Technical details and examples: The paper features many detials about the implementation. Even if it was not open-source, I feel fairly confident that this work could be majorly reproduced.
summary: This work introduces a new pretraining dataset for Scandinavian languages, as well as a cloze-style evaluation dataset for Swedish language models.
weaknesses: While it's a sound contribution, I'm not sure ICLR is the right venue for this work. It lacks algorithmic or theoretical novelty, and it's rather a (very good) application of well-known NLP principles to process a new dataset for specific languages.

For example, the heuristics mentioned in the paper are very similar to "old" related work, e.g. [1]

https://arxiv.org/abs/1912.07076

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: See weakenesses.
soundness: 3
strengths: 1. The authors present a novel model-based extractor, trained with 1380 human-labelled examples, to identify the main context from documents (HTML converted to markdown).
2. They introduce a new task and benchmark to validate their pipeline against a baseline, FineWeb and show that their pipeline, while being simpler, results in close performance on this task.
3. They present a new pretraining corpus of 1.01 trillion tokens for Scandinavian languages, which will be a valuable resource to the community.
summary: The paper presents a new pretraining corpus for Scandinavian languages:Swedish, Danish, Norwegian, and Icelandic. To create this data, they propose a new pipeline that uses a model-based text extractor trained with a small amount of human-labelled data on what is considered the main content in a markdown version of an HTML webpage. They contrast their proposed approach against the FineWeb pipeline, which extracts plain text directly from HTML pages. They further introduce a new cloze style test with a dataset, HP-MEK consisting of 460 examples based on the Swedish Scholastic Aptitude Test (Högskoleprovet),  to benchmark pre-trained performance and show that on a small dataset+model setting, their extractor can match the performance of FineWeb filter, despite being simpler in complexity and using an extractor that is only trained with 1380 human-labeled examples.

Once validated, they use this pipeline to create SWEb, which includes 1.01 trillion tokens.
weaknesses: 1. It is unclear why the authors chose to retain the markdown tags for pretraining (section 4). They also do not explain much about why HP-MEK is a reasonable benchmark for the pretraining setting.
2. No direct analysis is presented on the efficacy of the model-based extractor, given that it is one of the main component of the paper apart from reporting an F1 score. The downstream application in section 4 is useful but it doesn't say much about what this extractor does or is capable of filtering.
3. The dataset pipeline only includes filters like content length, # of alphanumeric characters, and unigram entropy for quality filtering. However, it would have been useful if there was any direct consideration of what is considered high-quality data in the context of pretraining and if additional checks were made in place about safety and fairness of representation in the dataset.

Meta-review: This paper introduces SWEb, a large-scale pretraining dataset consisting of over one trillion tokens for Scandinavian languages, presented with a detailed data collection and processing pipeline including a novel model-based text extractor. Reviewers recognized SWEb's significant contribution to the field, as it provides a high-quality resource for the traditionally under-resourced languages. The availability of this dataset, along with the new Swedish benchmark can allow the development of improved language models and significantly advance Scandinavian language research. Reviewers also appreciate the detailed methodology and open release of the resources. 

One of the main concerns pointed out by reviewers is evaluation restricted to the Swedish benchmark. The authors mention that their study on other benchmarks for other Scandinavian languages (e.g. Scandeval) didn't provide a good signal at the small scale resulting in opting for the Swedish benchmark which is the majority language. A few of the other limitations of the work are the lack of qualitative and error analysis of the model-based extractor and its extracted content, the lack of ablation studies regarding markdown tags during pretraining, and the absence of an ethics section. The authors address these concerns in the rebuttal and mention adding them in the final version of the paper. 

Given the substantial value of the dataset and the authors' efforts to address the reviewer’s concerns, I am recommending this paper to be accepted, with a suggestion to add the ethics section along with feedback mentioned by the reviewers in the final version of the paper.

Decision: Accept (Poster)