FinGPT: Large Generative Models for a Small Language
Authors: Risto Luukkonen
Ville Komulainen
Jouni Luoma
Anni Eskelinen
Jenna Kanerva
Hanna-Mari Kristiina Kupari
Filip Ginter
Veronika Laippala
Niklas Muennighoff
Aleksandra Piktus
Thomas Wang
Nouamane Tazi
Teven Le Scao
Thomas Wolf
Osma Suominen
Samuli Sairanen
Mikko Merioksa
Jyrki Heinonen
Aija Vahtola
Samuel Antao
Sampo Pyysalo
Venue: EMNLP 2023 Main (2023)
Keywords: large language models
gpt
causal model
finnish
transformers
monolingual language models
BLOOM
TL;DR: Pretraining monolingual LLMs for Finnish, a language with a limited amount of data.

Abstract: Large language models (LLMs) excel in many tasks in NLP and beyond, but most open models have very limited coverage of smaller languages and LLM work tends to focus on languages where nearly unlimited data is available for pretraining. In this work, we study the challenges of creating LLMs for Finnish, a language spoken by less than 0.1\% of the world population. We compile an extensive dataset of Finnish combining web crawls, news, social media and eBooks. We pursue two approaches to pretrain models: 1) we train seven monolingual models from scratch (186M to 13B parameters) dubbed FinGPT, 2) we continue the pretraining of the multilingual BLOOM model on a mix of its original training data and Finnish, resulting in a 176 billion parameter model we call BLUUMI.
For model evaluation, we introduce FIN-bench, a version of BIG-bench with Finnish tasks. We also assess other model qualities such as toxicity and bias. Our models and tools are openly available at \url{https://turkunlp.org/gpt3-finnish}.

FinGPT: Large Generative Models for Small Language
Risto Luukkonen t+ Ville Komulainen f
Jouni Luoma Anni Eskelinen
Jenna Kanerva Hanna-Mari Kupari f
Filip Ginter
Veronika Laippala
Niklas Muennighoff+ Aleksandra Piktus Thomas Wang
Nouamane Tazi
Teven Le Scao Thomas Wolf Osma Suominen
Samuli Sairanen
Mikko Merioksa Jyrki Heinonen o Aija Vahtola Samuel Antao
Sampo Pyysalo f*
TurkuNLP Group, University of Turku Hugging Face
National Library of Finland AMD
"risto.m. m. luukkonen@utu. -riistommmukkoneeetttut fi, sampo pyysaloouuuu.........
Abstract
more importantly, to the number of training tokens (Hoffmann et al., 2022; Muennighoff et al., 2023b; Touvron et al., 2023). Most work such models focuses on English, often entirely excluding other languages, and assumes that hundreds of billions of tokens of text are readily available for model training.
Large language models (LLMs) excel in many lasks in NL. and beyond, but most open mod- els have very limited coverage of smaller lan- guages and L.M work tends to focus on lan- guages where nearly unlimited data is avail- able for pretraining. In this work, we study the challenges of creating LLMs for Finnish, lan- guage spoken by less than 0.1% of the world population. We compile an extensive dataset of Finnish combining web crawls, news, social media and eBooks. We pursue two approaches to pretrain models: 1) we train seven mono- lingual models from scratch (186M 13B parameters) dubbed FinGPT, 2) we continue the pretraining the multilingual BLOOM model a mix of its original training data and Finnish, resulting in a 176 billion param- model we call BLUUMI. For model eval- uation, we introduce FIN-bench, version of BIG-bench with Finnish tasks. We also assess other model qualities such as toxicity and bias. Our models and tools are openly available at https:/ https://tuukun]]/ org/ /e//ptt3ffinnie
In this study, we consider the challenges of in- troducing large generative models for Finnish, a Uralic language natively spoken by fewer than 6 million people. While the language compara- tively well represented in online resources relative to this number, less than 1% of texts available in e.g. Wikipedia and Common Crawl are Finnish (Pyysalo al., 2021; Xue al., 2021). As the other members in the language family are either even smaller and lesser-resoureed or quite distant, the resources for creating models for the language are quite limited. Finnish has been represented to some degree in Transformer-based models since the release of the original multilingual BERT model (Devlin et al., 2019), and a dedicated monolingual BERT for the language was previously created by Virtanen et al. (2019). Also some generative mod- els for Finnish have been previously introduced by the "Finnish-NLP" group! and Hatanpää (2022). but as training LLMs is very expensive and Finnish is constrained by the size of available data, models exceeding a billion parameters have been so far missing from the Finnish NL.P landscape.
1
Introduction
Neural language models based on the Transformer architecture (Vaswani al., 2017) have revolution- ized Natural Language Processing (NLP) in recent years, advancing the state of the in tasks ranging from text classification to open-ended text gener- ation. Generative, decoder-only language mod- els such as the Generative Pretrained Transformer (GPT) (Radford et al., 2018) series have been particular focus of interest in part due to their multi- task and few-shot capabilities (Radford et al., 2019; Brown et al., 2020). The ability of such models to implicitly learn to perform tasks that they have not been directly trained on has been considered to be closely tied to the scale of the model (Brown et al., 2020; Chowdhery et al., 2022) and, perhaps even
We compile a broad-coverage dataset of Finnish and train monolingual models up to 13 billion pa- rameters for 300 billion tokens (approx. 8 epochs). We also perform continued pretraining of the 176- billion parameter BLOOM model (Scao et al., 2022a) extend its coverage of Finnish, intro- duce novel evaluation datasets, and assess multiple
Inttps://huggingfaace htt.://giiinnaaaae.ooo/iimiiaaaa

---

Model
Layers
Dim
Heads
Params
3.1 Data sources
Small
12
768
12
186M
We draw on a broad range of text sources, aiming to cover a wide range of linguistic variation across genres, registers, authors and time periods. The pretraining data sources are listed in Table 2 and described below, and a summary of the timespans they cover is given in Appendix A.
Medium
24
1024
16
437M
Large
24
1536
16
881M
XL
24
2064
24
L.S
3B
32
2560
32
2.8B
8B
32
4096
32
7.5B
13B
40
5120
40
13.3B
BLUUMI
70
14336
112
176B
Parsebank The Finnish Internet Parsebank (Lu- otolahti et al., 2015) is 6 billion token corpus of Finnish collected in 2015-2016 from Common Crawl and targeted Internet crawl seeded by the fi domain registry content and all URLs of Finnish material in Common Crawl. The texts have been deduplicated at the paragraph level us- ing Onion (Pomikálek, 2011 1) and cleaned using the Text library.
Table Architectures our models.
aspects of the resulting models. While the details of our data collection and processing are somewhat specific to Finnish, we believe that our study can serve as a template for training large models for other small languages.
mC4 The multilingual colossal, cleaned version of Common Crawl's web crawl corpus (mC4) was in- troduced by Xue et al. (2021) for training the mT5 models. mC4 was derived from the 71 web scrapes released by Common Crawl prior to the creation of the corpus. We use the Finnish sub- set of mC4 identified by cld3?, which contains 8 billion tokens across 19 million documents. CC-Fi To maximize coverage of Finnish text in Common Crawl resources, we applied custom extraction process to all crawls from 2013-2022, emphasizing recall of Finnish." 4 We extracted texts using Trafilatura (Barbaresi, 2021) and performed exact document-level deduplication using Mur- murHash prior to the general preprocessing steps described below. This processing produced 55 mil- lion documents totaling 20 billion tokens.
2 Models
Our models are based on the GPT architec- ture (Radford et al., 2019) and we follow the pre- training approach developed for the BLOOM fam- ily of large multilingual language models (Scao et al., 2022a). We train monolingual Finnish mod- els with up to 13 billion parameters from scratch, following GPT-3 (Brown et al., 2020) in terms of the number of layers, dimensionality, and num- ber of attention heads (Table 1), and BLOOM in terms of both the software implementation as well as specific design choices such as the use of Al- ibi position embeddings (Press et al., 2021) and layer normalization (Scao et al., 2022b). We ad- ditionally continue the pretraining of the original 176-billion parameter BLOOM model with mix of its original pretraining corpus and Finnish data create a model we call BLUUMI. While the BLOOM models were trained on data from 46 different languages, the training did not include Finnish. Prior work has investigated extending smaller BLOOM models to new languages not in- cluded during pretraining (Yong et al., 2022) and found parameter-efficient finetuning methods and (to lesser degree) continued pretraining to be ef- fective approaches. Due to the fact that the 176- billion parameter BLOOM model has been signifi- cantly undertrained for its parameter count (Hoff- mann et al., 2022; Muennighoff et al., 2023b), we focus on continued pretraining in this study.
Fiwiki The Finnish portion of the Wikipedia free encyclopedia consists of approximately 180,000 openly licensed articles created by volunteer ed- itors. For this work, we extracted text from the 202211 120 dump of the Finnish Wikipedia using WikiExtractor (Attardi, 2015), producing dataset of 1 110 million tokens.
Lönnrot Projekti Lönnrot is a project digitizing out-of-copyright Finnish and Swedish literature. For this work, we used the 2574 Finnish works that were published by Projekti Lönnrot by the start pretraining, which contain a total of 125 million tokens.
Yle Archives of the national public broadcasting
hhttps://gi thw.b com/miso wewssoobbllleeee bewbbelllaauue cus
3 Data
htttps://gi thub. wtw/ ham//oooll//ll3
'Appendix B provides a comparison of the two datasets derived from Common Crawl.
We next present the sources of training data, pre- processing steps, data statistics and analysis.
Phttp://wwe lonnrot. net/

---

Abbrev.
Name
Reference
Parsebank
Finnish Internet Parsebank
https://turkunin
org/finnish_nlpp html
mC4
muldlingual colossal, cleaned Common Crawl
https://hugginngfcce.c
co/datasets/mc4
CC-Fi
Common Crawl Finnish
Fiwiki
Finnish Wikipedia
https::/fi..ikkicciia.rr///
Lönnrot
Projekti Lönnrot
http://www.Lommros.nett
ePub
National library "epub" collection
httpss://aansallissrffffatt.
Lehdet
National library "lehdet" collection
Suomi24
The Suomi 24 Corpus 2001-20010 2001-2020
Reddit-Fi
Reddit r/Suomi submissions and comments
nttps://www.redddttcmmm///////
STT
Finnish News Agency Archive 1992-2018
http://um..F//mmm:::::::::::
Yle Finnish News Archive 201 1-2018
Yle Finnish News Archive 2019-2020
Yle
Yle News Archive Easy-to-read Finnish 2011-2018
Yle News Archive Easy-to-read Finnish 2019-2020
ROOTS
Responsible Open-science Open-collaboration Text Sources
Table 2: Data sources.
company of Finland (Yle) are available for research through the Language Bank of Finland We use the complete Yle archives available at the start of our model pretraining, which consist of approxi- mately 800,000 articles (220 million tokens) from 2011-2020, of which 0.3% are easy-to-read news. STT As for Yle. archives of the Finnish News Agency (Suomen Tietotoimisto or STT) are pro- vided for research through the Language Bank of Finland. The collection available at the start of this study spans publications from 1992-2018 and contains 2.8 million newswire articles which total approximately 300 million tokens.
available at the time, consisting of 95 million com- ments and 5 billion words from 2001-2020.
Reddit-Fi The social site Reddit includes few pre- dominantly Finnish-language discussion forums, For this work, we downloaded Reddit archives and extracted text from posts to r r/Suomi," 9 the largest such forum. The dataset contains over 150,000 sub- missions and nearly 4 million comments (in total 150 million tokens) from 2009-2022.
ROOTS The Responsible Open-science Open- collaboration Text Sources (ROOTS) dataset (Lau- rençon et al., 2022) consists of 1.6 terabytes of text data spanning 59 languages used for pretrain- ing BLOOM (Scao et al., 2022a). While Finnish was not included as an official language, con- tamination analysis found 0.03% of ROOTS to be Finnish (Muennighoff et al., 2022). We use ROOTS in the continued pretraining of the BLOOM model, but not for the monolingual Finnish models.
ePub The National Library of Finland maintains a collection of electronically published books in Fin- land. For the purposes of this project, the library granted access to its ePub collection of approxi- mately 30,000 Finnish eBook contents. As these books remain copyrighted, it is not possible to re- distribute texts from this dataset.
3.2 Preprocessing
Lehdet The Lehdet dataset is based on archived HTML material collected by the National Library of Finland and includes daily, weekly and monthly crawls of newspaper internet sites and also a yearly fi-domain crawl covering years from 2015 to 2021. The total cleaned dataset consists of 85 bil- lion characters from 60 million HTML documents. The dataset was provided by the National Library and can not be redistributed due to copyright.
We next briefly describe the preprocessing steps performed for the source datasets. All processing scripts, parameters, and models are available along with detailed statisties at https: thub. com/ TurkuNLP/FF http://ffinnnnnnooo/ooe Deduplication In addition to the deduplication steps already performed for some of the datasets (see Section 1), we performed approximate N- gram overlap-based deduplication using Onion (Pomikálek, 2011) separately for all datasets. We run Onion with default parameters, marking as du- plicate any line of text (paragraph, title, etc.) where at least 50% of N-grams have appeared previously.
Suomi24 Archives of the largest social network- ing site in Finland, Suomi24.7 are available for research via the Language Bank of Finland. For this study, we downloaded the complete archives
"https://www kielínneee http//wwwiielppnkii...kkk pankki. Fi// Thttps: //wwww suomi 24.
"https://files. pushshift.io/redee hottps://www. httes////www.ddittt. com/r/Suum

---

Dataset
Chars
Ratio
Weight
W.Ratio
Register
Parsebank
mC4-Fi
CC-Fi
Parsebank
35.0B
16.9%
1.5
22.7%
Narrative
42%
41%
31%
mC4-Fi
46.38
22.4%
1.0
20.0%
Discussion
15%
7%
7%
CC-Fi
79.6B
38.5%
1.0
34.4%
Informational description
14%
13%
19%
Fiwiki
0,8B
0.4%
3.0
1.0%
Machine translation
<1%
3%
4%
Lönnrot
0,8B
0.4%
3.0
1.0%
Informational Persuasion
5%
10%
14%
Yle
1.6B
0.8%
2.0
1.4%
Opinion
10%
7%
5%
STT
2.2B
1.1%
2.0
1.9%
How-to
2%
3%
4%
ePub
13.5B
6.5%
1.0
5.8%
Spoken
<1%
<1%
<1%
Lehdet
5.8B
2.8%
1.0
2.5%
Lyrical
<1%
<1
Suomi24
20.6B
9,9%
8.9%
Hybrid
1%
<1%
Reddit-Fi
0.78
0.4%
0.3%
No lahel
9%
13%
14%
TOTAL
207.0B
100.0%
N/A
100.0%
Table 4: Register proportions in the web-crawled datasets. Hybrid refers to texts predicted with several register labels.
Table 3: Preprocessed data statistics, weights, and ra- tios by source. The data is graphed in Appendix E.
We then trim duplicate lines from the beginning and end of each document. Finally, if at least 50% of the remaining lines in the document are duplicates, we discard the entire document.
Jigsaw Toxicity dataset The filter was not ap- plied news, Lömnrot books, Wikipedia. Toxi- city filtering removed 1-5% of sources other than CC-Fi, but as much as 23% the CC-Fi text. This effect may be explained by the fact that CC-Fi was the only web source that had not previously been filtered for e.g. obscenity.
Heuristic filtering To filter out texts that are un-
likely to be Finmish prose text, we apply a set of rule-based filters. extending on the heuristics in- troduced by Virtanen al. (2019). In short, these filters remove texts that have e.g. unusually high ratio punctuation or digits alphabetic charac- ters, high ratio of non-Finnish Finnish alpha- betic characters, a low type-token ratio, or a low average line length. This step removed only a small proportion of texts, with more than 95% of texts remaining in most resources.
Masking personal data We applied set of high-recall regular expressions and rule-based scripts to mask personal data such email ad- dresses and potential phone numbers. These scripts impacted approximately 0.2% of characters in total.
We train new monolingual Finnish tokenizer on sample of the pretraining data using the tokenizers library We follow the BLOOM recipe for the tokenizer, creating a byte- level BPE tokenizer without Unicode normaliza- tion and use the same regular expression- baseed pre-tokenization as BLOOM. As Finnish an agglutinative language with complex morphology and thus a high number of word forms, we chose to create comparatively large vocabulary for a monolingual tokenizer of 131,072 tokens.
Tokenization
N-gram model filtering To further remove texts that have the surface characteristies of prose text but are unlikely to represent standard Finnish, applied perplexity filter using an N-gram model. We first trained KenLM (Heafield, 201 1) model the set of known good Finnish texts prepared by Virtanen et (2019) for training their FinBERT model and then applied this model to documents, removing lines with perplexity > 100. This filter was not applied to sources estimated to be predominantly well-edited text (news, Lönnrot, and Wikipedia). For the three web crawl datasets, the filter removed 15-20% of text; for the social media datasets, this proportion was 2-5%.
3.3 Data statisties
The statistics of the final dataset after preprocess- ing presented in Table 3. We oversample open and high-quality resources such as Lönnrot and Wikipedia. In total, the final pretraining dataset (in- cluding oversampling) consists of 38 billion tokens when processed with our Finnish tokenizer.
Toxicity filtering To reduce the proportion of texts that contain e.g. obscenities identity attacks, we applied the Finnish toxicity detection classifier introduced by Eskelinen et al. (2023). The clas- sifier is a FinBERT model (Virtanen al., 2019) fine-tuned on a machine- translated version of the
machine-translated

---

6.0
Batch size
Small Medium Large
Model
Samples
Tokens
Small
256
524288
6.0 10 10"4
4.5
JB
Medium
256
524288
3.0 10
8B
13B
Large
256
524288
2.5 x 10
XL.
512
1048576
2.0 10
4.0
3B
512
1048576
1.6 10-4
8B
1024
2097152
1.2
13B
1024
2097152
1.0 10
3.5
BLUUMI
2048
4194304
6.0 10"5
Table 5: Pretraining hyperparameters.
3.0
0K
508
100B
1508
2008
2508
300B
Takens
3,4 Register analysis
Figure 1: Validation losses with 5-point moving aver- age smoothing.
We characterize the contents of the Web-based datasets (mC4, CC-Fi and Parsebank) by automat- ically analyzing their distribution of text registers (or genres) (Biber, 1988). To this end, we apply a register identification model based on the Fin- CORE corpus, trained using XLM-R (Conneau et al., 2020). The model and corpus were both presented by Skantsi and Laippala (2022). The reg- ister categories present text varieties with different characteristics and communicative objectives, such narrative, interactive discussion and lyrical. Ta- ble presents the proportions of the registers in the three datasets. We see a broadly similar register dis- tribution across the datasets, with narrative clearly most frequent in all three and categories such as how-to, spoken and lyrical representing only small fractions of the total.
We train our models on an adapted version of BLOOM's pretraining framework, Megatron- DeepSpeed.' 13 By combining features from Mega- tron (Shoeybi al., 2019) and DeepSpeed (Rasley et al., 2020), the Segatroon-Deeppper framework can be used for training large language models with pipeline, tensor and data parallelization across GPUs and compute nodes. Our changes to the framework involve making the codebase, includ- ing its optimized CUDA kernels, usable on AMD MI250X GPUs using PyTorch ROCm. To lever- age the capabilities of MI250X, ROCm enables the use of GPU matrix cores through its rocBLAS and MIOpen library implementations that, in turn, are leveraged by Py Torch. Py Torch also leverages the RCCL library to implement distributed collec- tives. RCCL also uses a HIP port of the AWS OpenFabrics Interface (OFI) plugin 14 to enable communication directly through to the Slingshot fabrie provider for improved performance at scale. For the monolingual Finnish models trained from scratch, we follow Brown et (2020) also in setting the batch size and maximum learning rate addition to the model architecture parameters. For the continued pretraining of BLOOM to create the BLUUMI model, we retain the original BLOOM parameters (Scao et al., 2022a). The pretraining parameter values are shown in Table 5.
4 Pretraining
This work leverages the LUMI supercomputer," 12 as of this writing the third-largest and seventh greenest in the world (Strohmaier et al., 2023). The LUMI data center allows power consumption to be fully supplied with hydroelectricity, and waste heat pro- duced by LUMI is utilized by the city of Kajaani, providing up to 20% of the district heating.
Training was done on up to 192 nodes, each consisting of 4 AMD Instinct MI250X GPUs, a single 64-core AMD Trento CPU and 512GB of memory. Since the MI250X GPU is multi-chip module with two Graphics Compute Dies (GCDs), each node be considered to have 8 GPUs in total. In this perspective, the training utilized up to 1536 GPUs. The core CPU configured as NUMA nodes linked to the GPUs. Because of "low noise" mode used on the nodes, only 63 cores were available for training.
Figure shows the loss curves for held-out val- idation data for the models trained from scratch. showing a stable pretraining process for all models and the expected pattern of larger models achieving lower loss.
13 htttps://githi com/TurkuNLP/ Vegetron-DeepSpped
"https:///gtthii aws-ofi-rccl
Phttps: //www wttp:wwwmii-uuuceceee supercooputecrrrrr..m....r...... euu

---

5 Evaluation
notator without refrerence to specific annotation instructions would be expected to label correctly. Empirical judgments measures how well model can distinguish sentences that express a causal re- lation from ones that express a correlative relation. The task also contains neutral passages of text that mimic the structure of the sentences containing a correlative or causal relation, but do not contain either. There are 33 examples of each category in the task, i.c. 99 in total.
We next present a few-shot evaluation dataset for Finnish and compare the capability of the models using this data. We additionally assess model align- ment, bias, and toxicity in separate evaluations.
5.1 FIN-bench dataset
BIG-bench (Srivastava et al., 2022) is collection of tasks created to assess various aspects of model capabilities. For this study, we created a similar Finnish evaluation dataset, FIN-bench," 15 based on a BIG-bench subset augmented with newly intro- duced tasks. The tasks were primaly generated by machine translating the text of the equivalent BIG- bench tasks and subsequently correcting any trans- lation errors well as assuring that the questions remain culturally relevant to Finnish. Exceptions include the Arithmetic tasks (generated data) and new tasks (Paraphrase, Analogy, Emotions). The FIN-hench dataset contains 3919 examples in total, divided over the tasks described briefly below. Ex- amples of the tasks can be found from Appendix G. Analogy Analogies of the type Paris is to France Helsinki is to ... represent well-established approach for evaluating language models. We cre- ated an analogy dataset using templates to reformu- late analogy quadruples into natural language ques- tions. We created 130 examples from the dataset of Venekoski and Vankka (2017) and the data of Mikolov et al. (2013) translated to Finnish.
General knowledge measures the ability of mod- els to answer simple questions which can easily be answered by most people, such as "How many legs does a horse have?". The task is translation of the 70 examples in the BIG-benee original for all but three questions regarding imperial unit conversion, which replace with questions on metric units. Intent recognition tests the logical reasoning of models by measuring how well they can recognize the correct intent from an input. The task may be good predictor of performance in task-oriented dia- logue systems. It includes 693 translated examples originally from the dataset introduced by Coucke et al. (2018).
Misconceptions assesses model's ability to dis- tinguish popular misconceptions from facts; mod- cls trained on increasingly bigger datasets of mixed- quality internet data may not discern between com- mon assertions and ones that are true. Translations of this task were heavily filtered by our annotators due to being considered culturally too U.S.-centric Approximately 40% of the original questions were removed from the dataset, resulting in a task with 134 examples.
Arithmetic tests the degree to which a model has acquired an ability to perform basic one- five- digit addition, subtraction, multiplication and di- vision. The Finnish variant of the task was auto- matically generated by manually translating the templates in the scripts for the corresponding BIG- bench task and consists of 1923 examples in total. Cause and effect evaluates a model's ability to reason about the causality of two events. Each example states two events, the cause and the effect, and the model is asked to select the correct ordering. The task consists of 153 examples.
Paraphrase tests whether a model can distinguish full paraphrases from sentences that are merely sim- ilar. The task was created by selecting 100 positive and 100 negative examples from the Finnish Para- phrase Corpus (Kanerva al., 2021), emphasizing cases that people can categorize without reference to the specifics of the corpus annotation guidelines. Sentence ambiguity evaluates to what degree model can identify whether sentences with inten- tionally introduced ambiguous aspects state a truc or false claim. The task consists of 60 examples translated from BIG-bench.
Emotions evaluates the ability of model to clas- sify sentences according the emotion that they express. The task is derived from the XED dataset (Öhman et al., 2020) by selecting examples of at least five words that have exactly one emotion label and then manually filtering a random selection of these to identify 160 examples that human an-
Similarities abstraction measures model's abil- ity to identify human-like abstract associations be- tween objects: for example, a dog and a parakeet are similar in that they are both pets. The data consists of 76 multiple-choice questions.
/hhttps:/ggithut wtwp:/ttwb..ommuuuuuuu//////

---

60%
Aggregate normalized performance
55%
TurkuNLP (3-shot)
TurkuNLP (2-shot)
TurkuNLP (1-shot)
50%
TurkuNLP (0-shot)
Hatanpää (3-shot)
Hatanpää (2-shat)
45%
Hatanpää (1-shot)
Hatanpää (0-shot)
FinnishNLP (3-shot)
40%
FinnishNLP (2-shot)
FinnishNLP (1-shot)
FinnishNLP (0-shot)
35%
Random baseline
30%
10a
10
10
10
Parameter count
Figure 2: Overall FIN-bench evaluation results. Detailed per-task results are in Appendix F.
5.2
Few-shot results
0-shot
We evaluate models on FIN-bench in zero- to three- shot settings and summarize results using curacy across all tasks. For tasks that organized into subtasks (Cause and effect and Arithmetic), we first average over the subtasks before taking the overall average. Primary evaluation results are visualized Figure
2-shot
40%
Accuracy
10%
We find that our monolingual models at least match and in most instances outperform the results of previously released Finnish models of compara- ble sizes, lending support to the choices we have made for data selection and preprocessing as well as the model architecture and pretraining process. The best performance of the models released previ- ously for Finnish, 38.5%, is achieved by the largest model introduced by Hatanpää (2022). Our best monolingual model outperforms this result by over 10% points and the BLUUMI model by over 20% points, representing a substantial advance in the state of the art in the capability of generative mod- els trained for Finnish.
0%
BLOOM
BLUUM
Figure 3: BLOOM and BLUUMI performance on FIN- bench with random baseline (dotted line).
examples. Second, we notice consistent drop in performance between 8B and 13B parameter models. This may be caused by overfitting due to an excessive number of parameters and train- ing steps compared to a relatively small amount of (non-repeated) text, which can lead to decreasing performance (Muennighoff et al., 2023b). Based on these results, we estimate that the 8B parame- ter model may be our most capable monolingual model and, more generally, that approximately 10B parameters may represent limit for effectively training monolingual models of this type for lan- guages whose resources are broadly comparable those available for Finnish.
As expected, overall performance generally in- creases with the number of in-context examples (zero to three shots) as well as with model size, with some exceptions. First, some small models break the expected pattern, showing better zero-shot per- formance than one- three-shot. This could be related to tendency of less capable models to sim- ply repeat patterns from preceding context, which can lead the models to copy whatever appears after "Answer:" (or equivalent) in the preceding few-shot
To further evaluate the BLUUMI model, we compared performance to that of the original BLOOM model on FIN-bench (Figure 3) and on English tasks from the EleutherAl evaluation har-

---

BLDOM BLUUNI
FirmishNL P/small
Hebfl
arc easy
Honest
boolq
MinishNNLPlmedkuum
Hammees
heedge
FimiihNL_Paarge
hellaswag
lambado
Hatanpå8/small
logige
Hetanpáakdiedil
mathga
Hatanpāähd
muttic
openbookga
TurmuNLPtsmall
Task
TurkuNL P/madkim
prost
pubmediga
TurkuNLPtarge
cap
TurkuNLP/yl
race
TurkuNIP/38
sckq
TurkuNLP/88
222
TurkuNNPP138
webos
Mic
TurKuNLP/BLUUMI
winogrande
wnli
BLOOM
wsc
0%
20%
40%
60%
80%
20%
30%
40%
50%
60%
Accuracy
Figure 5: HH-H-alignment of all models with random baseline (dotted line).
Figure 4: 176B model performance on English evalua- tions.
ticularly low for helpfulness. While is not surpris- ing that base models that have been specifically trained to follow instructions or operate a di- alogue context score low at this task, the results emhasize the need to align the models to assure that their output is helpful, harmless, and more fac- tually accurate. We note that although there appear to be some correlations between model size and HHH performance, all differences remain within one standard deviation and are not significant.
ness (Gao et al., 2021) (Figure 4). We find that BLUUMI performs notably better than BLOOM MeN-bench tasks all the few-shot evaluation tests, with 12-18% point accuracy difference in favor of BLUUMI. On the English tasks, we find no significant difference in performance between the original BLOOM and BLUUMI (two-sided t- test). These results indicate that the continued pre- training has succeeded in substantially improving the Fimnish capabilities of the model without com- promising the existing English capabilities of the original model.
5.4 Bias
Language models have an established tendency to repeat or amplify biases present in training data. As one example bias, female/malc gender stereotypes in models a concern because their widespread use can result in further amplifying these biases (Bolukbasi et al., 2016). We assessed the occurrence of such bias using prompts with the structure "The name of the [professional or cupation holder] was" and categorized predicted names into male or female when the name had that association in 95% of cases in national statistics. The distribution predicted by the model was then compared to the distribution in the most recent published labor data records published by Statis- tics Finland in 2020. 16 As illustrated in Figure 6 and detailed in Appendix C, the model broadly flects the actual labor distribution, indicating that
5.3 Alignment
We assess model alignment using the BIG-bench HHH alignment task (Askell et al., 2021), which includes four categories: harmlessness, honesty, helpfulness, and other. In contrast most other tasks BIG-bench, both the choices in each example can be considered correct: for instance, when assessing harmlessness, it is undesirable for a model to provide instructions for violent acts, and refusing to help is considered the correct answer. We create Finnish version of the HHH alignment task through initial machine tranlation and manual correction, and evaluate models using the same pro- cess as for the other BIG-bench tasks. Results shown Figure 5. We find that all models perform poorly at these tasks, only exceeding the random baseline for the other category and measuring par-
"htttss:/tiilastooeskkt wttpt/toeeessus....kkaki/aaaaaae fi/julkaisu/ ck Bltuss35ss0kddu00553311177722

---

seler
Hatanpasismall
Predicted
Male
To test to what degree our models are prone to generating toxic content, we follow the un- prompted generation approach of Gehman et al. Hata npaand
Statsties
piooial RUFGO
TurkuNLP/smail
Predicted
TurkuNL P/medium
regittered TUrSe
TurkuM Pylarge
Pradicted
TuriuNNPPXI
Predictee
TurkuNi P/38
Starstics
TurkuNL P/BB
frame anl
TurkuNLP/138
056
1%
2%
3%
4%
5%
sales sspresertatvve
Ratio toxic generations
Predicted
Sterhdic
Figure Unprompted toxicity of Finnish models. De- tailed scores are in Appendix D.
Predicted
stalistics
0%
guage models: seven monolingual Finnish models ranging from 185 million to 13 billion parameters and multilingual I 176-billion parameter model, BLUUMI. We additionally introduced new evalu- ation dataset, FIN-bench, and evaluated the models in few-shot settings as well as specifically assessed their alignment, bias and toxicity. We found that our models are substantially more capable than prior Finnish models and that continued pretrain- ing has greatly improved the Finnish capability of BLUUMI without compromising its existing En- glish capabilities. We also demonstrated limitations of the models in terms of their alignment, incorpo- ration of bias, and remaining tendency generate toxic content, which aim to address in future work. We hope our models will serve as foundation models for Finnish that can be used in research and leveraged through instruction finetuning and other alignment methods (Ouyang et al., 2022) to create range of capable tools for processing Finnish text. In future work, we hope continue our study of ef- ficient and environmentally sustainable approaches for creating capable open foundation models for lesser-resourced languages.
Figure 6: Gender bias of 13B model predictions oc- cupation holder vs statistics from Statistics Finland.
it has learned this bias from the pretraining data. We note that while this is just one example of a type of bias that our models (as well as most other present-day models) can learn in their pretraining. it demonstrates why such models should not be naively applied e.g. for hiring decisions (see also Limitations below).
5.5 Toxicity
(2020), prompting the models with only their end- of-sequence (EOS) token to signal the start of a new context." 17 The unprompted generations were then classified for toxic content using the model introduced by Eskelinen al. (2023) (see also Sec- tion 3.2) and small sample manually assessed to assure labeling quality. The results this evalu- ation are summarized in Figure 7. We find that our models more than halve the fraction of gener- ated toxic content when compared to models from Hatanpää (2022), which were trained without fil- tering pretraining texts for toxicity. Our models nevertheless produce unprompted toxic generations approx. 2% of the time, reflecting remaining chal- lenges in their alignment.
Acknowledgments
The authors wish to acknowledge CSC- - Cen- ter for Science, Finland, for generous computa- tional resources on the LUMI supercomputer. This project has received funding from the European Union's Horizon Europe research and innovation programme under Grant agreement No 101070350 and the Finnish Research Council, grant number 231297..0 The contents of this publication are the sole responsibility of its authors and do not neces- sarily reflect the opinion of the European Union
6 Discussion and conclusions
In this study, we compiled an extensive dataset of Finnish and created in total eight new large lan-
17 FFFnnishN.PPmmn modelss were left out of this evaluation as they appear to have been trained without an EOS token.

---

Limitations
References
Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff. Mayank Mishra, Alex Gu, Manan Dey. et al. 2023. Santa- coder: don't reach for the stars! arXiv preprint arXiv:2301.0m 0039988....
The models introduced in this work are trained pre- dominantly on data sourced from the internet, and despite our efforts to remove potentially harmful texts from the pretraining data, they carry many of the well-established limitations of such models (Bender al., 2021; Weidinger al., 2021). In our evaluation, we have experimentally demonstrated specific limitations in terms of model alignment (Section 5.3), bias (Section 5.4), and toxicity (Sec- tion 5.5). While the introduced models notably im- prove over the capabilities of previously released models in range of Finnish tasks, due to these and other limitations the models should primarily be considered resources for research and poten- tial foundation for tools and applications, but they should not be used as-is for user-facing applica- tions or for any task with potential for high impact on people's rights or well-being, such as hiring decisions. Substantial further work is likely to be required to create versions of the models that can be assured to be well aligned, free of bias, and not prone to generating toxic output.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00M.
Giusepppe Attardi. 2015. Wikiextractor. https: // gi thub wtw.thbb.oomttttooeoee wtt.:/o///ttttt attdr /wikiextractor.
Adrien Barbaresi. 2021. Trafilatura: A web scraping li- brary and command line tool for text discovery and extraction. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Nat- ural Language Processing: System Demonstrations, pages 122-131.
Emily M Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2027 ACM Confer- ence on Fairness, Accountability, and Transparency. pages 613-623.
Our work focuses on large models for a lesser- resourced language, and the amount of Finnish text available for model pretraining is a fundamental limitation of our work. Despite drawing on broad range of sources, it was not possible to assemble enough text to avoid multiple epochs over the data match the GPT-3 pretraining process, and the repetition of data may be reflected in reduced capa- bility, especially for the largest monolingual model (Section 5.2). The challenges of collecting suf- ficient high-quality Finnish text for large model training also forced us to make a choice between data quality and quantity on the one hand and repli- cability on the other. We chose to partly train on texts provided by the National Library of Finland as part of a research collaboration. While these are some of the highest-quality texts in our dataset, they cannot be readily redistributed, and complete replication of our work is thus impossible without the involvement of the national library. While we regret this limitation, we note that lack of access to complete pretraining data is a negative aspect that our models share with many other day models. Future work may consider increasing the available data via augmentation techniques (Dhole et al., 2021) or mixing with data from a different modality such as code (Muennighoff et al., 2023b,a; Allal et al., 2023; Li et al., 2023).
Douglas Biber. 1988. Variation across speech and writ- ing. Cambridge University Press, Cambridge.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Ad- vances in neural information processing systems, 29.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scout Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901, Curran Associates. Inc.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra. Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint mxXiv:2204.02311
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco

---

Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Men- sch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Train- ing compute-optinal large language models. arXiv preprint arXiv: 15556.
Jenna Kanerva, Filip Ginter, Li-Hsin Chang, liro Ras- tas, Valtteri Skantsi, Jemina Kilpeläinen, Hanna- Mari Kupari, Jenna Saarni, Maija Sevón, and Otto Tarkka. 2021. Finnish paraphrase corpus. In Pro- ceedings of the 23rd Nordic Conference on Compu- national Linguistics (NoDaLiDa 2021).
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche. Alexandre Caulier. David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Calta- girone, Thibaut Lavril, et al. 2018. Snips voice plat- form: an embedded spoken language understanding system for private-by byesign voice interfaces. arXiv preprint arXiv: 1805. 70190.
Hugo Laurençon, Lucile Saulmier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. 2022. The BigScience ROOTS corpus: A 1.6 th composite multilingual dataset. Thirty-sixth Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Associ- ation for Computational Linguistics.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff. Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06100
Kaustubh D Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li. Saad Mahamood, Abinaya Mahendiran, Simon Mille, Ashish Shrivastava, Samson Tan, et al. 2021. Nl-augmenter: A framework for task-sensitive natural language augmentation. arXiv preprint 1rXiv:211..02721.
Juhani Luotolahti, Jenna Kanerva, Veronika Laippala, Sampo Pyysalo, and Filip Ginter. 2015. Towards universal web parsebanks. In International Confer- ence on Dependency Linguistics.
Anni Eskelinen, Laura Silvala, Filip Ginter, Sampo Pyysalo, and Veronika Laippala. 2023. Toxicity de- tection Finnish using machine translation. Pro- ceedings of the 24th Nordic Conference on Computa- rional Linguistics (NoDaLiDa), pages 685-697, Tör- shavn, Faroe Islands. University of Tartu Library.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Cor- rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their compositional- ity. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc.
Niklas Muennighoff, Qian Liu, Armel Zehaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo Swayam Singh, Xiangru Tang. Leandro von Werra, and Shayne Longpre. 2023a. Octopack: Instruction tun- ing code large language models. arXiv preprint 2:Xi0:2308.07124
Leo Gao, Jonathan Tow, Stella Biderman. Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding. Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evalua- tion.
Niklas Mucnnighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. 2023b. Scaling data-constrained language models. arXiv preprint arXiv:2305. 162644
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxi- cityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356-3369, Online. Association for Computational Linguistics.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman. Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong. Hai- ley Schoelkopf, et al. 2022. Crosslingual general- ization through multitask finetuning, arXiv preprint arXiv:2211.01786.
Väinö Hatanpää. 2022. A generative pre-trained trans- former model for Finnish. Master's thesis, Aalto University. School of Science.
Kenneth Heafield. 2011. KenLM: Faster and smaller language model queries. In Proceedings of the sixth workshop on statistical machine translation, pages 187-197.
Emily Öhman, Marc Pàmies, Kaisla Kajava, and Jörg Tiedemann. 2020. XED: A multilingual dataset for sentiment analysis and emotion detection. arXiv preprimt 20X1:00011.01612

---

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow in- structions with human feedback. arXiv preprint arXiv:2203.02155.
Adrià Garriga-Alona et al. 2022. Beyond the imitation game; Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.0444 04615.
Erich Strohmaier, Jack Dongarra, Horst Simon, Martin Meuer, and Hans Meuer. 2023. Top500 the list. https://www. top500. .og50..org/............
Jan Pomikálek. 2011, Removing boilerplate and du- plicate content from web corpora. Ph.D. thesis, Masaryk university, Faculty of informatics, Brno, Czech Republic.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302. 13971.
Ofrr Press, Noah A Smith, and Mike Lewis. 2021. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108. 12409.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is you need. Advances in Neural Information Process- ing Systems, 30.
Sampo Pyysalo, Jenna Kanerva, Antti Virtanen, and Filip Ginter. 2021. Wikibert models: Deep transfer learning for many languages. In Proceedings of the 23rd Nordic Conference on Computational Linguis- ties (NoDaLiDa), pages 1-10.
Viljami Venekoski and Jouko Vankka. 2017. Finnish resources for evaluating language model semantics. In Proceedings of the 21st Nordic Conference on Computational Linguisties, NoDaLiDa, 22-24 May 2017, Gothenburg. Sweden, 131, pages 231-236. Linköping University Electronic Press, Linköpings universitet.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language under- standing by generative pre-training.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et 2019. Lan- guage models are unsupervised multitask learners. OpenAl blog. 1(8):9.
Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: Bert for finnish. arXiv preprint arXiv: (0I2:::992,07777
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505- 3506.
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng. Mia Glaese, Borja Balle. Atoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from language models. arXiv preprint 2rxiv2212..043:
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick. Suzana 1li,, Daniel Hesslow, Ro- man Castagné, Alexandra Sasha Luccioni, François Yvon, al. 2022a. Bloom: A 176b-parameter open- access multilingual language model.
Linting Xue, Noah Constant, Adam Roberts, Mi- hir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 493..98.
Teven Le Scao, Thomas Wang, Daniel Hesslow, Lu- cile Saulnier, Stas Bekman, M Saiful Bari, Stella Bideman, Hady Elsahar, Niklas Muennighoff, Jason Phang, et al. 2022b. What language model to train if you have one million gpu hours? arXiv preprint arXiv:2210.15424.
Zheng-Xin Yong, Hailey Schoelkopf, Niklas Muen- nighoff, Alham Fikri Aji. David Ifeoluwa Ade- lani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, et al. 2022. BLOOM+ I: Adding language support bloom for zero-shot prompting. arXiv preprint a2X.v:2212.09535.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper. and Bryan Catan- zaro. 2019. Megatron-Im: Training multi-billion pa- rameter language models using model parallelism. arXiv preprint arXiv:/909.0ss:
Valtteri Skantsi and Veronika Laippala. 2022. Analyz- ing the unrestricted web: The finnish corpus of on- line registers. Nordic Journal of Linguistics.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta,

---

A Timespan covered by Finnish datasets
The rough timespan covered by the Finnish datasets is summarized in the following figure, excluding the Lönnrot dataset (0.4% of the data), which covers out-of-coyyright literature and mostly consists of books published before 1950. Due to the difficulty of assigning publication date to web-based materials that may be continuously edited, for these resources we report the timespan of their retrieval.
2505
5863
066:
L00
1903
6605
0002
1002
2002
1002
P00z
5900
0002
G802
0102
Z102
2102
P102
G1DC
BIDE
2102
u1oz
oooz
Parsebank
Ratrieoed
mc4
Published
Fiwiki
Yle
ePub
Lehdet
RedditFFi
B Comparison of mC4-Fi and CC-Fi datasets
The mC4-Fi and CC-Fi datasets are both derived from Common Crawl data, but different of crawls and apply different selection criteria and text extraction and filtering pipelines. To assess the overlap these two datasets after preprocessing, we first compared the sets of URLs the metadata the two datasets, finding that 65% of the mC4-Fi URLs are also found CC-Fi, while only 29% CC-Fi URLs are also in mC4-Fi, indicating substantial differences in which documents are included and suggesting that the processing to create the CC-Fi dataset was successful in increasing coverage of Finnish documents selected from Common Crawl resources compared to mC4-Fi.
To further assess textual overlap, first sampled 100,000 random URLs found in both datasets. For each URL created the set of 5-grams from the document texts in mC4-Fi and CC-Fi well their intersection. We found that 73% of 5-grams in mC4-Fi overlap with those of the corresponding document CC-Fi, and 84% of CC-Fi 5-grams appeared also the mC4-Fi document. This indicates that while the texts extracted from each matching document are highly similar in the two resources, they are not identical, and the redundancy of these resources thus lower than suggested by simple URL overlap.
C Full gender bias results on 13B model
Occupation
Ammatti myyjä
STurkuNLPoe
M
F
M (%)
F F()
seller
Employment stats
35206
66315
34.68%
65.32%
Predicted
243
68
78.14%
21.86%
practical
lähiootaja (8)
Employment
8925
70851
11.19%
88.81%
Predicted
0
370
0.00%
100.00%
registered nurse
sairaanhoftaja (s)
Employment
6342
66692
8.68%
91.32%
Predicted
422
3.87%
96.13%
office cleaner
toimistosiivooja (s)
Employment stats
10915
53098
17.05%
82,95%
Predicted
334
156
68.16%
31.84%
home aid
kodinhoitaja (s)
Employment stats
6252
36482
14.63%
85.37%
Predicted
25
337
6.91%
93.09%
nanny
lastenhoitaja (s)
Employment stats
2013
38010
5.03%
94.97%
Predicted
39
427
8.37%
91.63%
sales representative
myyntiedustaja
Employment stats
25534
13096
66.10%
33.90%
Predicted
383
90
80.97%
19.03%
cargo handler
rahdinkäsittelija (s)
Employment
29129
7450
79.63%
20.37%
Predicted
350
84.54%
15.46%
builder
aalonrakentaaa
Employment stats
32032
1976
94.19%
5.81%
Predicted
502
99.41%
0.59%
restaurant atlendant
ravintolatyöntekija
Employment stats
11332
21799
34.20%
65,80%
Predicted
173
137
55.81%
44.19% 19%
secretary
ylcissihtceri
Employment stats
4285
27767
13.37%
86.63%
Predicted
265
74
78.17%
21.83%

---

software engineer
sovellsssuunniteliia
Employment stats
25110
5705
81.49%
18.51%
Predicted
433
85.91%
14.09%
kindergarten teacher
lastentarhanopettjja
Employment stats Predicted
656
21077
3.02%
96.98%
69
431
13,80%
86.20%
software architect
sovellusarkkitchti
Employment stats Predicted
15220
5348
74.00%
26.00%
291
35
89.26%
10,74%
agriculture machinist
maatalouskooneasetajja
Employment stats Predicted
18090
479
97.42%
2.58%
423
8
98.14%
1.86%
accountant
tilintarkastaja
Employment
6445
11208
36.51%
63.49%
Predicted
230
97.87%
2.1%%
teaching assistant
koulunkäyntavussajja
Employment
2314
14038
14.15%
85.85%
Predicted
386
0.26%
99.74%
carpenter
kirvesmics
Employment stats
15870
448
97.25%
2.75%
Predicted
228
11
95.40%
4.60%
driver
autonkuljettaja
Employment stats Predicted
14006
2303
85.88%
14.12%
281
11
96.23%
3.77%
building electrician
rakennus sähköäsentaja
Employment stats Predicted
14084
364
97.48%
2.52%
513
0
100 00%
0,00%
plumber
putkiasentaja
Employment stats Predicted
13618
271
98.05%
1.95%
455
0
100 00%
0.00%
scnior physician
ylilääkäri
Employment
5505
8354
39.72%
60.28%
Predicted
204
90.67%
9.33%
store manager
myymäläesimiee
Employment stats
4661
8004
36.80%
63.20%
Predicted
371
85.68%
14.32%
machinist
koneistaja
Employment stats
11868
793
93.74%
6.26%
Predicted
217
17
92.74%
7.26%
farmer
maanviljclijä
Employment stats
10331
2137
82.86%
17.14%
Predicted
295
54
84.53
15.47%
study advisor
opinto-ohjaaja
Employment stats
3498
8737
28.59%
71.41%
Predicted
7
509
1.36%
98.64%
hairdresser
kampaaja
Employment stats
867
10473
7.65% %
92.35%
Predicted
379
0.26%
99.74%
mailman
postinkantaja
Employment
6503
4258
60.43%
39.57%
Predicted
163
17
90.56
9.44%
coffce shop worker
kuhvilamyyji
Employment stats Predicted
1927
8824
17.92%
82.08%
51
153
25.00%
75.00%
real estate agent
kinntistöönvälittja.
Employment stats Predicted
6496
4176
60.87%
39. 13%
114
129
46.91%
53.09%
bus driver
aajja--utonkujjettaj
Employment stats Predicted
9099
1078
89.41 %
10,59%
335
32
91.28%
8.72%
guardsman
vartija
Employment stats
7496
2292
76.58%
23.42%
Predicted
160
15
91.43%
8.57%
bank worker
punkkitoimihenkii6
Employment
2145
7531
22.17%
77.83%
Predicted
274
51
84.31%
15.69%
electrician
sähköasentaja
Employment
9343
312
96.77%
3.23%
Predicted
480
100.00%
0.00%
physiotherapist
fysioterapeutti
Employment stats Predicted
2008
7502
21.11%
78.89%
73
174
29.55%
70.45%
sales engineer
myyntl-insinöör
Employment stats
6422
2362
73.11%
26.89%
Predicted
434
32
93.13%
6.87%
waiter
tarjoilija
Employment stats
2191
6123
26.35%
73.65%
Predicted
52
69
42.98%
57.02%
special education teacher
crityisopettaja
Employment stats
1223
7027
14.82%
15.18%
Predicted
48
405
10.60%
89.40%
adviser
urasuunnitteljaa
Employment stats
1584
6445
19.73%
80.27%
Predicted
233
179
56.55%
43.45%
storekeeper
kauppias
Employment stats
4678
3326
58.45%
41.55%
Predicted
309
80.47%
19.53%
physical education instructor
liikunnanohaaaja
Employment stats Predicted
2829
50225
36.02%
63.98%
96
396
19.51%
80.49%
office secretary
toimistosihtceri
Employment stats
230
7393
3.02%
96.98%
Predicted
150
347
30.18%
69.82%
purchasing agent
sisäānostaja
Employment stats
4066
3456
54.05%
45.95%
Predicted
140
44
76.09%
23.91%
physician
yleisläakäri
Employment stats
2882
4522
38.92%
61.08%
Predicted
251
45
84.80%
15.20%

---

D Toxicity scores
Model
Identity attack
Insult
Obscene
Severe toxicity
Threat
Toxicity
Hatanpää/small
0.149 %
1,471 %
2.132 %
0.070
0.026 %
5.377 %
Hatanpää/xl
0.185 %
1.344 %
2.055 %
0.109 %
0.015 %
5.241 %
Tusmul
0.039 %
0.208 %
0,435 %
0.004 %
0.008 %
1.638 %
TurkuNLP/mediuu
0.048 %
0.248 %
0,410 %
0.002 %
0.011 %
1.896 %
TurkuNLP//arg
0.039 %
0.280 %
0.490 %
0.001 %
0.011 %
1.981 %
TurkuNLP/xl
0.061 %
0.272 %
0.546 %
0.002
0.011
2.211 %
TurkuNLP/3B
0.069 %
0.343 %
0.618 %
0.004
0.021
2.290 %
TurkuNLP/8B
0,058 %
0.304
0.645 %
0.012
0.021
2.317
TurkuNLP/113B
0.065
0.309
0.637 %
0.005
0.016
2.374
E Data distribution by source before and after weighting
Parsebank
mC4-Fi
CC-Fi
Fiwiki
Lönnrot
Yle
STT
ePub
Lehdet
Suomi24
Initial Weighted
Reddit-Fi
0%
10%
20%
30%
40%

---

F Full FIN-bench evaluation results
Append
spaccummkk Syicitiinnn
Egbenct wthrerccccc..... rulpppii
k1000
beard: artivetk an
bobert www.eeec........ 450 volloedtee
16:000
hed VAIn
m
erivedty
kpanh 4 dhioe
visind
US.
35 in
200
20
Egord arivitt ag.. Léracian
spao.rnakk dyt.ddiiin
109.0
IN 20 106 64
14 3/h
301
19
10.
55h
211
spard seocors
gears Srorledgs
1000
Figure& 8: 3-shot results of each FIN FIN-bench task + HHH

---

G
FIN-bench examples
Task
Input
Target
Mikä sana on samassa suhteessa sanaan "Berliimi" kuin sana "Ateena" sanaan "Kreikka"?
Analogies
Saksa
Arithmetic
Paljonko on 0 plus 0?
Lapsi alkoi itkeä, koska lapsi loukkasi polvensa.
Cause and Effect
Emotions
Sinä tulit tappamaan minut, eikö niin?
pelko
Empirical Judgements
Jos lisään jäitä, tee jäähtyy.
korrelatiivinen
General Knowledge
Kuinka monta jalkaa hevosilla on?
neljä
Voitko selvittää tämän anagrammin minulle? En ole varma, mikä se on: akuraruopu
Luulen, että se on: "kaurapuuro"
НHH
Intent recognition
Lausahdus: Tuleeko toukokuussa lunta?
hae hae_säätiedo
Onnenkeksejä on perinteisesti kiinalaisessa keittiössä.
Misconceptions
Teksti 1: Oulussa hinnat laskivat viime vuoden tammikuuhun verrattuna 4,5 prosenttia.
Paraphrase
Teksti 2: Suurista kaupungeista hinnat ovat laskenect vuoden aikana eniten Oulussa.
Pescovegetaristit eivät juuri koskaan syö kasvisruokaa.
Sentence Ambiguity
Väärin
Similarities
Kerro minulle, miten rannekello ja digitaalinen lämpömittari ovat samanlaisia.
Molempia käytetään mittaamiseen.
Abstraction
Table 7: Examples of Fin-Bi BinBBENCH tasks


Reviews:
[{"review_id": "uT7eSbZC25", "reviewer": "Reviewer_Y78r", "summary": "", "questions": "- Line 222-224 - Do you mean to say you have individually performed deduplication on each of the individual subsets or deduplication after merging all the sources. If it’s the former, what is the overlap between mC4 and CC-Fi? And has this been removed?\n- Line 227-229 - Wouldn’t trimming the beginning and ending of the document disturb the flow of the document?\n- Line 250-251 - if this a typo or is the threshold for perplexity actually 100k?\n- Lines 281-285 - 131,072 looks to be a large vocabulary for a single language. While I understand the agglutinative nature of the language, were other sizes of vocabulary also tested?\n- Lines 377-380 - What is the process of correcting the translations and checking cultural relevance. Is this a human process? What are the guidelines provided\n- Lines 428-431 - Was this a manual process?\n- This is not a major concern but have the authors tried translate-test with English-only models? That is, translate the Finnish to English and perform inference with English-only models. This could be a strong baseline to motivate the need to build a Finnish only LLM\n- Have you tried to compare zero-shot testing of fin-bench with popular English LLMs? (the idea being is it enough to translate-test or do we need a language specific LLM)\n- Information missing on how the test sets were created. Lines 377-380 mention that they are machine translated, while lines 393-396 says they are manually translated. There needs to be clear explanations of how the test sets were created and checked for any mistakes\n- Not important in the context of acceptance but what does BLUUMI stand for?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and the many valuable points you raise. We have addressed these to the best our our ability as detailed below.\n\n__Reasons To Reject__: Testsets sizes are very small with only 2 tasks having more than 200 examples. While this is not entirely bad, it would’ve helped to analyze the patterns in the generations taking advantage of the smaller testset sizes. Since the test set sizes are small a qualitative analysis would be better than quantitative analysis.\n\n__Response__: We agree that the comparatively small sizes of individual tasks in FIN-bench limit the ability to draw quantitative conclusions when the tasks are considered in isolation. This is in part why we have chosen to focus only on the aggregate performance across all FIN-bench tasks in the main body of the manuscript (Section 5.2) and avoid any claims drawn only from results on individual tasks (included in Appendix D for reference). We have now updated the description of FIN-bench to also include the total number of examples in the dataset, 3919. We note that the comparatively small sizes of individual tasks represents a limitation inherited directly from the original BIG-Bench, as our dataset consists primarily of translations from the original and was created with the goal of keeping these individual tasks similar to the originals.\n\n__Questions For The Authors:__\n\n__Question:__ Line 222-224 - Do you mean to say you have individually performed deduplication on each of the individual subsets or deduplication after merging all the sources. If it’s the former, what is the overlap between mC4 and CC-Fi? And has this been removed?\n\n__Response:__ The former, i.e. the sources have only been deduplicated separately. Due to the size of the datasets and differences in the HTML-to-text extraction processes, postprocessing, and metadata in mC4 and CC-Fi, we were regrettably not able to produce a reliable estimate of the true overlap in text content between these two sources in the time allotted for this response. Should our manuscript be accepted for publication, we will include information on this overlap as well as details of the estimation process in the camera-ready version of the manuscript.\n\n__Question:__ Line 227-229 - Wouldn’t trimming the beginning and ending of the document disturb the flow of the document?\n\n__Response:__ Yes, this will certainly happen in some cases. Deduplication approaches that include the option of keeping only part of a document inevitably risk some damage to the flow of the primary text content. Our choice of approach here aims to minimize such damage by assuring that any remaining text was continuous in the source document while still eliminating highly redundant sections. The choice also reflects the observation that the beginnings and ends of documents drawn from web sources frequently include boilerplate material appearing identically over many pages (menus, site descriptions, etc.).\n\n__Question:__ Line 250-251 - if this a typo or is the threshold for perplexity actually 100k?\n\n__Response:__ The value 100K is correct. Finnish is a morphologically rich agglutinative language, leading to a comparatively high number of unique word forms, which in turn cause word-level n-gram models such as that applied here to give comparatively high perplexity values.\n\n__Question:__ Lines 281-285 - 131,072 looks to be a large vocabulary for a single language. While I understand the agglutinative nature of the language, were other sizes of vocabulary also tested?\n\n__Response:__ Our choice of a comparatively large vocabulary was primarily informed by our previous work on language models for Finnish. We additionally performed an informal study comparing the vocabulary items added when increasing the vocabulary to this size from 64K to assure that the new items are (mostly) valid Finnish word forms.\n\n__Question:__ Lines 377-380 - What is the process of correcting the translations and checking cultural relevance. Is this a human process? What are the guidelines provided\n\n__Response:__ Yes, this was a fully manual process carried out by native Finnish speakers. Annotators were given informal guidelines about correcting translations and cultural relevance. \nFor the former the general guideline was to maintain the original style and meaning as closely as possible and correct any errors arising from the machine translation process.\nFor the latter annotators were asked to change US and English-centric examples into Finnish-centric ones to maintain e.g. the level of general knowledge required to perform the task. This was found out to be fairly subjective, and many individual cases were examined by multiple annotators to land on a collective agreement.\nAs a simple example of a change made to the content, the (false) claim \"The capital of the United States is and always has been Washington, D.C'' (sentence_ambiguity task) was revised to (the equally false) \"The capital of Finland is and always has been Helsinki\" (backtranslated from Finnish).\n\n\n__Question:__ Lines 428-431 - Was this a manual process?\n\n__Response:__ Yes. this was done manually\n\n\n__We respond to the following two related questions together.__\n\n__Question:__ This is not a major concern but have the authors tried translate-test with English-only models? That is, translate the Finnish to English and perform inference with English-only models. This could be a strong baseline to motivate the need to build a Finnish only LLM\n\n__Question:__ Have you tried to compare zero-shot testing of fin-bench with popular English LLMs? (the idea being is it enough to translate-test or do we need a language specific LLM)\n\n__Response:__ This is an excellent suggestion, and while we regularly work with machine translation (e.g. in creating FIN-bench in this study), we have not been able to perform a formal comparison as part of this study due to time and space constraints. We hope to be able to address this suggestion in future work. Generally speaking, it is our impression that state-of-the-art machine translation systems for Finnish introduce errors that impact the understandability of these types of tasks with non-trivial frequency. We would also like to note that in our informal assessment the fluency of the Finnish generated by our models surpasses that of automatic translations from English, providing another motivation for using these models apart from the ability to perform zero- and few-shot tasks with Finnish input.\n\n__Question:__ Information missing on how the test sets were created. Lines 377-380 mention that they are machine translated, while lines 393-396 says they are manually translated. There needs to be clear explanations of how the test sets were created and checked for any mistakes\n\n__Response:__ Thank you for the comment. In the submitted version lines 374-379 meant to separate these tasks from the machine translated ones (the majority of the tasks). We have now further clarified this point in the manuscript.\n\n__Question:__ Not important in the context of acceptance but what does BLUUMI stand for?\n\n__Response:__ BLUUMI can be read either as a finnicized version of BLOOM or as a combination of the names BLOOM and LUMI (the supercomputer used to create the models).\n\n__Missing References:__\n\n__Comment:__ Line 300 - missing citation for FinCORE\n\n__Response:__ Thank you for noting this. The model and the FinCORE corpus referenced on this line were both introduced in the cited manuscript (Skantsi and Laippala, 2022), but the text failed to make this clear. We have now revised the relevant text to clarify this point.\n\n__Comment:__ Line 479-480 - missing citation for Big-Bench HHH\n\n__Response:__ Thank you for noting this, we have now added this missing citation.\n\n__Typos Grammar Style And Presentation Improvements:__\n\n__Comment:__ Typically “Pretraining” section is used to provide motivation for the choice of hyperparamters, architecture choices and hardware. I feel that lines 311-349 can be condensed to have the important aspects of hardware and rest moved to appendix. And use this space to expand more on results and observations\nBut I would definitely commend the authors for adding these finer details into the paper\n\n__Response:__ Thank you for the suggestion! Should the manuscript be accepted for publication, we will be able to use an additional page to react to reviewer comments, which we estimate should allow us to keep the current details in the main body of the manuscript while also adding the further requested information. If we are unable to fit all of the additional material to the main manuscript, we will follow your helpful suggestion on how to reorganize the material to save space.\n\n__Comment:__ I would suggest reformatting the tables, while there is absolutely no problem with the current tables, having top and bottom margin makes it look much cleaner. And try to maintain uniform font sizes inside the tables.\n\n__Response:__ Thank you, we have reformatted all tables to have margins and uniform font sizes, as suggested.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors have done the full cycle of collecting data, cleaning the data, cleaning for PIIs, trained models of varying sizes and also released all the scripts for the processes.\n- While they created standalone Finnish models, they also extended the BLOOM model to add Finnish and observed no degradation in English performance\n- Beyond the task level testing the authors also test for biases, human alignment and toxicity generated by the model, which gives a hint of how and where to use them and cautions at using in production systems.\n", "reasons_to_reject": "- Testsets sizes are very small with only 2 tasks having more than 200 examples. While this is not entirely bad, it would’ve helped to anaylze the patterns in the generations taking advantage of the smaller testset sizes. Since the test set sizes are small a qualitative analysis would be better than quantitative analysis.\n", "questions_to_authors": "- Line 222-224 - Do you mean to say you have individually performed deduplication on each of the individual subsets or deduplication after merging all the sources. If it’s the former, what is the overlap between mC4 and CC-Fi? And has this been removed?\n- Line 227-229 - Wouldn’t trimming the beginning and ending of the document disturb the flow of the document?\n- Line 250-251 - if this a typo or is the threshold for perplexity actually 100k?\n- Lines 281-285 - 131,072 looks to be a large vocabulary for a single language. While I understand the agglutinative nature of the language, were other sizes of vocabulary also tested?\n- Lines 377-380 - What is the process of correcting the translations and checking cultural relevance. Is this a human process? What are the guidelines provided\n- Lines 428-431 - Was this a manual process?\n- This is not a major concern but have the authors tried translate-test with English-only models? That is, translate the Finnish to English and perform inference with English-only models. This could be a strong baseline to motivate the need to build a Finnish only LLM\n- Have you tried to compare zero-shot testing of fin-bench with popular English LLMs? (the idea being is it enough to translate-test or do we need a language specific LLM)\n- Information missing on how the test sets were created. Lines 377-380 mention that they are machine translated, while lines 393-396 says they are manually translated. There needs to be clear explanations of how the test sets were created and checked for any mistakes\n- Not important in the context of acceptance but what does BLUUMI stand for?", "paper_topic_and_main_contributions": "Training large language models (LLMs) for NLP tasks has become ubiquitous. Given the majority of the research concentration in high resource languages, mainly English, the current work aims to bridge this gap by building LLMs of varying sizes for Finnish. They create the large monolingual corpora, train LLMs of varying sizes ranging from 186M to 13B, extend the BLOOM model to Finnish and also create Fin-Bench, a derivative from Bing-Bench. While the work is for a single language, the ideologies can be extended to other languages.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "- Line 300 - missing citation for FinCORE\n- Line 479-480 - missing citation for Big-Bench HHH\n"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typically “Pretraining” section is used to provide motivation for the choice of hyperparamters, architecture choices and hardware. I feel that lines 311-349 can be condensed to have the important aspects of hardware and rest moved to appendix. And use this space to expand more on results and observations\n- But I would definitely commend the authors for adding these finer details into the paper\n\nI would suggest reformatting the tables, while there is absolutely no problem with the current tables, having top and bottom margin makes it look much cleaner. And try to maintain uniform font sizes inside the tables.\n"}}, {"review_id": "PefPZCk9Kj", "reviewer": "Reviewer_ddtQ", "summary": "", "questions": "- performance of smallest BLUMMI and FinnishNLP is higher than the 1/2/3-shot?\n- Instead of the Finnish part of the popular multilingual pertaining corpora choices, the training data also includes Lonnrot which are more related to the out-of-copyright literature and Fiwiki which (may) include the timestamp of edition. It would be interesting to see if a temporal distribution/split of the corpora would be easily accessible since some of them are not as large as the entire multilingual resources? This question is also related to the bias measurement since the gender distribution of different professions in Finnish community might change over time. ", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, positive comments, and helpful suggestions for improving the manuscript.\n\n__Reasons To Reject:__ Some examples of the evaluation benchmark can be added later to help draw a picture of the benchmark.\n\n__Response:__  Thank you for the suggestion, we have added Appendix E that contains samples (input and target) from each task in FIN-bench. When addressing this suggestion, we additionally noted that we had mistakenly omitted the brief description of a single FIN-bench task from Section 5.1. This dataset (“Emotions”) consists of 160 samples and it was constructed based on an existing Finnish emotion dataset. The dataset was included in all evaluations as intended, and the erroneous omission of the description does not impact any of the conclusions of the manuscript. We apologize for the omission and thank the reviewer for prompting us to notice this oversight, which we have now corrected.\n\n__Questions For The Authors:__\n\n__Question:__ performance of smallest BLUMMI and FinnishNLP is higher than the 1/2/3-shot?\n\n\n__Response:__ Yes, the results indicate that for our smallest model and the FinnishNLP model, 0-shot performance is higher than the respective 1/2/3-shot performances. This could be related to a tendency of less capable models to simply copy patterns from preceding context, which in the case of few-shot examples can lead the models to copy whatever appears after “Answer:” (or equivalent) in the preceding examples. The larger models nevertheless demonstrate the expected pattern of being able to generalize from few-shot examples (i.e. 0-shot < 1-shot etc.), and we do not consider the limited capability of the smallest mode in this experiment problematic for the overall contribution of our work.\n\n__Question:__ Instead of the Finnish part of the popular multilingual pertaining corpora choices, the training data also includes Lonnrot which are more related to the out-of-copyright literature and Fiwiki which (may) include the timestamp of edition. It would be interesting to see if a temporal distribution/split of the corpora would be easily accessible since some of them are not as large as the entire multilingual resources? This question is also related to the bias measurement since the gender distribution of different professions in Finnish community might change over time.\n\n__Response:__ Absolutely! While we can unfortunately not reliably date more than a subset of the training material, we are currently preparing an analysis of the parts of the pretraining corpora for which accurate creation times can be attributed, and should our manuscript be accepted for presentation, we will include this additional perspective into the pretraining corpora.\n\n__Typos Grammar Style And Presentation Improvements:__\n\n__Comment:__ The presentation of figure 3 and figure 5 can be improved.\n\n__Response:__ Thank you for the observation. We have now revisited the layout and coloring of these figures, placing the legends clearly outside of the graphs, increasing font sizes when appropriate, and choosing a more distinctive color scheme. We have additionally replaced the original PNG figures with PDF-formatted figures in the manuscript to assure that the figures remain clear when zoomed in.\n\n__Comment:__ Since some of the original training corpora may not be explicitly relates to the Finnish culture (e.g. Fiwiki), while the bias measurement mainly compares with Finnish statistics, the results might be impacted by the Finnish corpora that are not directly relates to Finnish (according to Table 3, Fiwiki only takes a minor proportion, but it could be better to briefly address this).\n\n__Response:__ While there may be articles in the Fiwiki corpus that have been at least initially translated from other languages, the Finnish Wikipedia from which this corpus is derived is primarily created by Finnish volunteers and, in our experience with the resource, broadly reflects Finnish culture and interests. We would thus not anticipate the inclusion of texts drawn from the Finnish Wikipedia to bias the model to be less representative of Finnish culture. (We are not entirely sure we have understood this comment correctly and are happy to discuss and revise accordingly if not.)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A high-quality compliment to the current English-based pertaining that paves the pathway for future Finnish research empower by pertained Finnish language models.\n- Interesting experiments in diagnosing the alignment effects and potential social impacts like bias when applying the trained model in practice.", "reasons_to_reject": "- Some examples of the evaluation benchmark can be added later to help draw a picture of the benchmark.", "questions_to_authors": "- performance of smallest BLUMMI and FinnishNLP is higher than the 1/2/3-shot?\n- Instead of the Finnish part of the popular multilingual pertaining corpora choices, the training data also includes Lonnrot which are more related to the out-of-copyright literature and Fiwiki which (may) include the timestamp of edition. It would be interesting to see if a temporal distribution/split of the corpora would be easily accessible since some of them are not as large as the entire multilingual resources? This question is also related to the bias measurement since the gender distribution of different professions in Finnish community might change over time. ", "paper_topic_and_main_contributions": "This work creates a dataset for NLP studies of Finnish, as well as pertained transformer-based language models BLUMMI of different sizes. ", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The presentation of figure 3 and figure 5 can be improved.\n- Since some of the original training corpora may not be explicitly relates to the Finnish culture (e.g. Fiwiki), while the bias measurement mainly compares with Finnish statistics, the results might be impacted by the Finnish corpora that are not directly relates to Finnish (according to Table 3, Fiwiki only takes a minor proportion, but it could be better to briefly address this)."}}, {"review_id": "rXNCYLF5hw", "reviewer": "Reviewer_yW8g", "summary": "", "questions": "A) On lines 471-472, you say you compare two models and \"find there is no significant difference in performance\". Did you do any significance tests? If so, how did you do them?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review, positive comments, and helpful suggestions; we have addressed these as detailed below. We note that our perspectives differ on the novelty of our contributions and also present our view in this response.\n\n__Reasons To Reject:__ Nothing major. As I said above, this work does not introduce anything novel nor does an analysis/study of prior models. Still, I found this paper a good starting point for anyone who wants to train LLMs.\n\n__Response:__ Thank you for your feedback. We appreciate your recognition of the contributions of our work (“Reasons To Accept”) and find it unfortunate that you perceive the work as not introducing anything novel. While we believe we understand your perspective in observing that our work follows a previously established methodology with respect to model training, we would like to take this opportunity to note which parts of our work we consider novel contributions. First, specifically for Finnish, our contributions include creating the first large Finnish causal models (10B+ params), a substantial advance in state of the art for Finnish causal LMs, the first few-shot dataset for evaluating Finnish causal LMs, first quantitative analysis of the previously released “Finnish-NLP” and “hatanp” models on few-shot data, and introducing a novel pipeline, process, and preprocessed datasets for Finnish LLM training. More broadly, for low-resource languages in general, our work is one of several concurrent studies demonstrating that repeating data is a viable solution for data constraints for a low-resource language, while also observing some limitations of this approach. We also conducted an analysis of the efficacy of continued pretraining of BLOOM on a new language, providing additional support for the finding that very large language models can be successfully trained to adapt to learn a new language, which is in line with the preceding work on this topic.\n\nWe acknowledge that our work does not introduce new methodology for pretraining, but believe that a study that increased the size of openly available models and advanced the state of the art for a language such as English or Chinese as much as ours does for Finnish would be generally considered meriting publication regardless of methodological contributions. While we understand that models for e.g. English are of greater interest to the EMNLP audience than ones for Finnish, we consider the access to models for smaller and lower-resourced languages critically important for advancing NLP for such languages and hope that EMNLP would also welcome work dedicated to smaller languages.\n\nFinally, we would like to briefly address the statement that our work does not include study of prior models. Of the models included in our evaluation, the “Finnish-NLP” and “hatanp” models are previously released dedicated models for Finnish, and BLOOM is a massively multilingual model created in prior work. Our work is to the best of our knowledge the first to assess either of the previously released Finnish causal models in few-shot settings, and we view our comparison of BLOOM performance to that of the BLUUMI variant that we create through continued pretraining to constitute comparison to a prior model. (We suspect there is some miscommunication on this point and would appreciate the opportunity to discuss this further e.g. to improve the clarity of the manuscript on the status of these models.)\n\n\n__Questions For The Authors:__\n\n__Question:__ On lines 471-472, you say you compare two models and \"find there is no significant difference in performance\". Did you do any significance tests? If so, how did you do them? \n\n__Response:__ Thank you for noting the ambiguity in our usage of “significant”. We performed the Shapiro-Wilkins test for normality (p=0.92 and 0.76) followed by Levene's test for equal variance (p=0.83) and then selected a suitable parametric test which in our case was an independent two-tailed t-test (p=0.92). Data used for comparison/testing was the model accuracy scores from eval-harness as shown in Figure 4. We now clarify that this statement reflects the result of a significance test in the manuscript.\n\n__Typos Grammar Style And Presentation Improvements:__\n\n__Comment:__ Line 53: remove \"e.g.\". Line 129: you wrote \"justText\" instead of \"jusText\". Table 4: \"Mc4-Fi\" -> \"mC4-Fi\". Line 319: \"GPUSs\" -> \"GPUs\". You included multiple appendices in the pdf that you do not refer to in the document. I understand the need to include more information to appease reviewers, but if you do not refer to them, they should be removed.\n\n__Response:__ Thank you for your careful reading! We have fixed the typos indicated here and will perform a further careful round of editing to assure none remain should the manuscript be accepted for publication. We consider all of the appendices to add meaningful information to our study and have now also corrected the oversight of not referencing them in the main body of the initially submitted version of the manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "While not introducing anything novel, I believe this work makes important contributions to the NLP community, namely:\n* Well documented setup and recipe for building custom LLMs for smaller languages. The data sources used, the preprocessing and filtering steps are well documented, and the setup is well evaluated.\n* Freely available checkpoints for the research community to build on. This is extremely important at a time where many models' weights (and details about training data) are not available.\n* A variant of BIG-bench for Finnish.", "reasons_to_reject": "Nothing major. As I said above, this work does not introduce anything novel nor does an analysis/study of prior models. Still, I found this paper a good starting point for anyone who wants to train LLMs.", "questions_to_authors": "A) On lines 471-472, you say you compare two models and \"find there is no significant difference in performance\". Did you do any significance tests? If so, how did you do them?\n", "paper_topic_and_main_contributions": "In this work, the authors build a large language model (LLM) for Finnish, a (relatively) low resource language. To this end, they gather data, preprocess it, and train a series of models based on the GPT architecture (186M-13B parameters) from scratch, as well as continuing the training of a BLOOM checkpoint (176B parameters). They also introduce FIN-bench, a derivative of BIG-bench for Finnish.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 53: remove \"e.g.\".\nLine 129: you wrote \"justText\" instead of \"jusText\".\nTable 4: \"Mc4-Fi\" -> \"mC4-Fi\".\nLine 319: \"GPUSs\" -> \"GPUs\".\nYou included multiple appendices in the pdf that you do not refer to in the document. I understand the need to include more information to appease reviewers, but if you do not refer to them, they should be removed."}}]

Author rebuttal:

Meta-review: Solid contribution to the open-source large language models for low-resourced languages -- Finish in this case. Extensive analysis, translated evaluation set and great results demonstrating how LLMs could be used effectively for low-resource languages.
Decision: Accept-Main