On the Impact of Cross-Domain Data on German Language Models
Authors: Amin Dada
Aokun Chen
Cheng Peng
Kaleb E Smith
Ahmad Idrissi-Yaghir
Constantin Marc Seibold
Jianning Li
Lars Heiliger
Christoph M. Friedrich
Daniel Truhn
Jan Egger
Jiang Bian
Jens Kleesiek
Yonghui Wu
Venue: EMNLP 2023 Findings (2023)
Keywords: large language models
cross-domain datasets
data diversity
data quality
benchmark
TL;DR: German language models trained on a diverse cross-domain dataset outperform models trained on high-quality data alone, leading to new state-of-the-art results.

Abstract: Traditionally, large language models have been either trained on general web crawls or domain-specific data. However, recent successes of generative large language models, have shed light on the benefits of cross-domain datasets. To examine the significance of prioritizing data diversity over quality, we present a German dataset comprising texts from five domains, along with another dataset aimed at containing high-quality data. Through training a series of models ranging between 122M and 750M parameters on both datasets, we conduct a comprehensive benchmark on multiple downstream tasks. Our findings demonstrate that the models trained on the cross-domain dataset outperform those trained on quality data alone, leading to improvements up to 4.45% over the previous state-of-the-art.

On the Impact of Cross-Domain Data on German Language Models
Amin Dada', , Aokun Chen 2:3, Cheng Peng"., 2,3, 3, Kaleb E Smith', Ahmad Idrissi- Yaghir Constantin Mare Seibold 1.7 Jianning Li", Lars Heiliger', Christoph M. Friedrich Daniel Truhn Jan Egger".9, Jiang Bian Jens Kleesiek Yonghui Wu
Institute for Medicine (IKIM), University Hospital Essen (AöR), Essen, Germany
Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FI,, USA Cancer Informatics and eHealth core, University of Florida Health Cancer Center, Gainesville, FL, USA NVIDIA, Santa Clara, CA, USA
Department of Computer Science, University of Applied Sciences and Arts Dortmund, Dortmund, Germany
Institute for Medical Informatics, Biometry and Epidemiology (IMIBE), University Hospital Essen (AöR), Essen, Germany Clinic for Nuclear Medicine, University Hospital Essen (AöR), Essen, Germany
"Department of Diagnostic and Interventional Radiology. University Hospital RWTH Aachen, Aachen, Germany
Cancer Research Center Cologne Essen (CCCE), West German Cancer Center Essen,
University Hospital Essen (AöR), Essen, Germany
German Cancer Consortium (DKTK. Partner site Essen), Heidelberg, Germany
Department of Physics, TU Dortmund, Dortmund, Germany
Abstract
2020), or encoder-decoder models (Raffel et al., 2020; Zeng et al., 2022). As these have shown impressive performance for natural language un- derstanding, many works built upon them for spe- cific tasks such as medical language understand- ing (Rasmy et al., 2021; Lee et al., 2020) hu- man interaction (Bai et al., 2022), or advance them through intelligent design choices. For instance, He et al. (2021) introduced the disentangled attention mechanism that encodes the token position and in- formation separately, enabling it to surpass human performance on the SuperGLUE benchmark (Wang et al., 2019). However, as the data requirements for LLMs can become difficult to acquire for smaller institutions, it poses the questions whether strictly increasing the data amount is the only solution if it is possible enable the training of LLMs through more elaborate data selection process.
Traditionally, large language models have been either trained on general web crawls or domain- specific data. However, recent successes of generative large language models, have shed light on the benefits of cross-domale datasets. To examine the significance of prioritizing data diversity over quality, we present German dataset comprising texts from five domains. along with another dataset aimed contain- ing high-quality data. Through training series of models ranging between 122M and 750M parameters both datasets, we conduct com- prehensive benchmark on multiple downstream tasks. Our findings demonstrate that the models trained the cross-domain dataset outperform those trained on quality data alone, lcading to improvements up to 4.45% over the previous state-of-the-art, The models available at: https://hugg// inggicegfee.... co/ wwiikkk-eeeeee essen
The Pile (Gao et al., 2020) has played a crucial role in emphasizing the advantages of augmenting web-crawl datasets with a wide array of domain- specific data. This realization has prompted the training of large language models (LLMs) on The Pile and similar datasets (Gao et al., 2020; Lau- rençon et al., 2022). Gunasckar et al. (2023) pre- sented 1.3B parameter model trained on dataset of 7B carefully curated tokens that is comparable to GPT3.5 on HumanEval indicating that data qual- ity apart from general scaling laws (Kaplan et al., 2020) play an important role as well.
1 Introduction
With established scaling laws (Kaplan et al., 2020), increasing the model size and datasets has become a consistent scheme in the rapidly evolving field of Large Language Models (L.MMs) (Kaplan et al., 2020; Touvron et al., 2023; Smith et al., 2022). This pattern exists for any recently proposed archi- tecture (Yang al., 2023), independent of decoder- only (Brown et al., 2020b; Chowdhery al., 2022; Touvron et al., 2023), encoder-only (He et al., 2021; Devlin et al., 2019; Liu et al., 2019; Clark et al.,

---

However, notable disparity arises when consid- cring the German language, as there is currently a lack of comparable variety-focused datasets. To address this gap, we propose various methods for curating a diverse German dataset, even for do- mains with limited resources, such as the medical field. Our methods are applicable to a wide range of languages, approximately 74% of this dataset is acquired through automatic translation and web- crawl filtering techniques. We demonstrate the advantages of such datasets over pure web-crawl- based datasets with a set of encoder-only models we call GeBERTa.
2 Related Work
2.1 German LMs
Subsequent the immense success achieved by English models, a surge of transformer models trained in various other languages has emerged (Martin et al., 2020; Chan et al., 2020). This devel- opment has been made possible through the avail- ability and utilization of web-based multilingual corpora. Most prominently OSCAR (Ortiz Suarez and Gabay, 2022) and CC100 (Wenzek et al., 2020). While Oscar only performs some heuristies to filter quality data, the CC100 pipeline makes delib- erate attempl to filter for higher-quality texts by using an n n-gram model to filter for texts that are similar to Wikipedia articles (Wenzek et al., 2020). We base our study on GC4 a dataset that was col- lected with the CC100 pipeline. In German, (Chan et al., 2020) released GBERT Base: GBERT Larges GELECTRA and GELECTRA which trained OSCAR, Wikipedia, the Open Par- allel Corpus (Tiedemann, 2012), and legal texts (Ostendorff al., 2020a). In their study, Chan et al. explored the impact of data quantity on model performance. Their dataset consisted of a combi- nation of legal documents and a small subset of medical documents from the Open Parallel Corpus, which introduced cross-domain aspect to their data. However, they did not explicitly measure the specific effects of incorporating cross-domain data. In addition, GouBERT (Scheible al., 2020) was released, German RoBERTa (Liu al., 2019) model that was solely trained on the German por- tion of the OSCAR dataset.
In 2020, Chan al. published set of German transformer (Vaswani al., 2017) models based on BERT and ELECTRA, achieving state-of-the- art results on two downstream datasets. Around the same time, GottBERT was released, a German RoBERTa model (Scheible et al., 2020). Since the two publications were only evaluated on two common downstream datasets and both dispensed with hyper-paramettr tuning, it is unclear how they compare. In this work, we train new German LMs with the DeBERTa (He et al., 2021) architecture and assemble a comprehensive benchmark of eight downstream datasets. By fine-tuning the hyper- parameters of our models and the previously re- leased ones, we showcase the enhanced model per- formance achieved through the DeBERTa architec- ture and our cross- cross-domain dataset. (Chan al., 2020) and (Scheible et al., 2020) were based on the web-based OSCAR dataset. Moreover, Chan et al., 2020 explored the impact of data quantity on model performance. The aim of this work is to quantify the effects of data quality and the inclusion of cross-domain data on encoder-only models.
2.2 Cross-Domain Pre-training
Our contributions can be summarized as follows:
Encoder-only transformers are usually initially trained on general web crawls, news articles, books, and subsequently fine-tuned for specific domains. A well-known example is BioBERT (Lee et al., 2020), domain-specific LM further pre-trained on PubMed texts based on BERT (Devlin et al., 2019), which is pre-trained on Wikipedia and BooksCor- pus. Alsentzer al., 2019, however, show that pre-training on medical data (Clinical BioBERT) is superior to pre-training on general data (Clini- cal BERT) for several downstream medical tasks. Alternatively, there are some works that only train on the target domain, for instance, (Gu et al., 2021)
We empirically show that pre-training lan- guage models on heterogenous datasets results in better downstream performance.
We assemble an extensive benchmark of eight tasks and thoroughly evaluate previously pub- lished models.
We present new German language models that achieve state-of-the-e results.
We release our models and the source code for compiling the pre-training dataset.
htaps:://geraa--nie coppoonnnl-groouu- gi tiub. io/projects/ ac4-corpus.Ad html html
(last access: 20-06-2023)

---

and (Yang et al., 2022). On the other hand, gen- erative LLMs are often trained on more diverse datasets that incorporate multiple domains. After the release of GPT-2 (Radford et al., 2019) and GPT-3 (Brown et al., 2020a), the research focus has shifted to diverse large datasets. The Pilc com- bines web data with forum posts, scientific publi- cations, and articles from a large range of domains and demonstrates that data diversity is important factor for the performance of LLMs. Although the composition of the training data for GPT-3.5 and GPT-4 remains unknown, several publications have demonstrated their capabilities in multiple domain- specific tasks, including medicine (Adams et al.. 2023; Nori et al., 2023), implying that the training data incorporates various domains.
outweigh any potential drawbacks associated with reduced data quality.
3.1 Quality-Focused Dataset
We combine GC4, German news articles, and other relevant resources to create our first dataset D Qquality While (Wenzek et al., 2020) had previ- ously applied deduplication based on document hashes, the creators of GC4 implemented ad- ditional deduplication step. They employed Mon- goDB deduplication to identify duplicates across snapshots (Wenzek et al., 2020). To construct our dataset, we relied on all pre-processed GC4 head snapshots, which showed the highest similarity to Wikipedia articles. Additionally, we incorporated the WMT 2022 German monolingual news crawl corpus (Kocmi et al., 2022) and the 80220301.d6 German Wikipedia dump?.
The use of cross-domain datasets in other lan- guages remains rare, mainly because of the effort involved in compiling such datasets. In addition, the availability of domain -omainn-pecific data is low in many languages. For example, a recent study found only less than 2 million publicly available medi- cal documents in German (Bressem et al., 2023). this study, we present different methods ere- ate corresponding German datasets based on web crawls or English domain-specifi. datasets. Al- though, due to the use of automatic translation and filtering methods, the quality is lower compared to the English datasets, we can show that our cross- domain dataset leads to better performance of German language models.
Given the potential overlap between these datasets, we applicd the deduplication algorithm introduced by (Lee et al., 2022). The deduplication process involved two stages: initially, we dedupli- cated GC4 and the news corpus separately, and sub- sequently, deduplicated the combined datasets. We configure the minimal matching substring length to 100 tokens. To ensure document integrity, any doc- ument containing duplicates was removed entirely. Interestingly, despite the previous deduplication efforts by (Wenzek et al., 2020), which removed 70% of the texts from each GC4 snapshot, and the subsequent MongoDB deduplication, which elimi- nated an additional 46% of the remaining data, we discovered that out of an initial 74.6 billion tokens in GC4, 35.3 billion were duplicates. Moreover, out of the 16 billion news tokens, 9.3 billion were identified as duplicates. Conversely, when consid- ering the concatenation of all datasets, only 0.72 billion token duplicates were found.
3 Data Collection
CC100 prioritizes quality in its construction. While this emphasis on quality might be beneficial, it also has certain limitations when it comes to the dataset. Specifically, domains that encompass in- formal noisy texts, such as social media posts medical documents, may be at a disadvantage due to the focus on quality. To address this concern and explore the trade-off between quality and vari- ety, we have developed two distinct datasets. The first dataset D Dquality is centered around ensuring high quality, featuring carefully curated and reli- able content. However, recognizing the importance of incorporating diverse texts, we have taken steps extend the quality-focused dataset to include broader range of material, thereby increasing its variety. By creating these two datasets with dif- ferent emphases, we aim to examine whether the advantages gained from a variety- focused approach
3.2 Variety-Fnhancen Dataset
In this section, aim to expand upon the quality- focused dataset we discussed earlier, with the pri- mary objective of enhancing its diversity. To achieve this, we introduce several additions to D Qquality- We partition the data we gather into four distinct domains: informal, medical, legal, and literature. The subsequent section will provide a detailed account of how we obtain these domain- specific datasets. By including data from various
tht ht/hhugggigaacccc acc. co/ wwt.satsaesss/aeaaaaa sestttastsswwii wwtp://iiiiiiiaaee (last access: 22-06-2023)

---

Quality Integration
Variety Integration
GC4
Filtered
W
D Dariety
Figure I: Overview of our data set collection process. The deduplicated quality-focussed data based GC4, news, and Wikipedia. We extend this data set with more variety-focussed data sources that form filtered multilingual data sets, translations, and curated data.
Category
Source Data
Trans.
Data
Size
#Docs
#Tokens
Wikipedia
No
9GB
2,665,357
1.9B
News
No
28GB
12,305,326
6.1B
GC4
No
90GB
31,669,772
19.4B
Reddit 2019-2023 (GER)
No
5.8GB
15,036,592
1.3B
Holiday Reviews
No
2GB
4,876,405
428M
OpenLegalData:
No
5,4GB
308,228
IB
German cases and laws
Charite doctoral theses abstracts
No
28MB
16,947
5M
Flexikon
No
106MB
74.136
23M
NTS of Animal Experiments
No
24MB
50,310
4M
German Guideline Program
No
13MB
4,348
3M
in Oncology
Springer Abstract
No
79MB
34,035
15M
CC medical texts (GER)
No
3.6GB
2,000,000
682M
Medicine Dissertations
No
1.4GB
14,496
295M
Pubmed abstracts
Yes
8.5GB
21,044,382
1.7B
MIMIC
Yes
2.6BB
24,221,834
695M
MMCC-Patennt--eaa
Yes
2.1GB
1,743,344
414M
German Fiction
No
1.1GB
3,219
243M
English books
Yes
7.1GB
11,038
1.6B
Total
167GB
116,079.769
35.8B
Table 1: Sizes of used data sources after pre- processing in the form of bytes, number of documents, and tokens. We integrate from top to boutom: formal, informal, legal, medical, and literature data.

---

domains, we strive to incorporate wider range of linguistic styles and content. We sacrifice quality by adding automatically translated texts, filtered web-crawl data, and potentially noisier data from social media with the intention of increasing the data variety. To ensure that our results are compa- rable in terms of data volume, we take the step of removing random subset of the GC4 dataset. We refer this dataset D variely.
the Stanza library (Qi et al., 2020), where they were parsed into individual sentences. These sentences were then grouped into chunks, each constrained to a maximum of 128 tokens. The number of tokens was counted using the tokenizer provided with the translation model. This constraint was considered since it was observed that larger chunks of text re- sulted in decrease in translation quality. For the translation phase, the Fairseq WMT" 19 English German model was used. The model was config- ured with a context length of 156 and employed a beam search with a single beam, a setting chosen to optimize translation speed.
Informal Drawing inspiration from the work of Blombach et al., 2020, we have identified Ger- man content within the Pushshift Reddit Dataset (Baumgartner et al., 2020). To accomplish this, we gathered all submissions and comments posted between 2019 and 2023 from the dataset. We then performed several pre-processing steps, including unescaping HTML content, removing URLs, and filtering out texts with fewer than 20 words. To identify German content accurately, we employed the fasttext language identification model (Joulin et al., 2017). We set threshold language identi- fication score of 0.9, excluding posts that scored below this value.
To cover the legal domain, the Open galData corpus consisting of more than 250,000 German cases and laws was used (Ostendorff et al., 2020b).
Legal
We employed the Corpus of German- Language Fiction conduct pre-training on Ger- man literature. This corpus comprises 2,735 German-language prose books from the Gutenberg- DE Edition, supplemented by an additional 484 books that were manually translated into German. Additionally, we applied the previously mentioned method to translate the BooksCorpus adding an- other 1 1,0088 books.
Literature
As result of this process, we obtained collec- tion of 15 million texts from Reddit. However, to augment our informal data further, we incorporated an additional 4.9 million texts from corpus of reviews sourced from a German holiday booking website, as published by Guhr et al., 2020.
The resulting datasets are summarized Table
Experiments and Results
Medical In contrast to English, which bene- fits from existing large medical text corpora like
4.1 Pre-training
MIMIC (Johnson et al., 2016), the availability of such resources in German limited due strict privacy regulations. However, we have managed to collect relatively small set of public medical datasets and websites. Using this data, we applied the 5-gram approach described in (Wenzek et al., 2020) to filter the latest release of the OSCAR cor- pus for medical texts. Out of the 207 million total documents in OSCAR, focused selecting the top 2 million documents where the 5-gram model had the lowest perplexity. Additionally, we have gathered 1 14,49 publicly available medicine disser- tations written in German.
Following (He et al., 2021), pre-train DeBERTabase- DeBERTaaarge and with the same configuration they used on both of our datasets and DcBERTaatarge only on Dvariety- In the fol- lowing, we will refer to the models as G and GeBERTaY We train with an accumulated batch size of 2048 for 1M steps on the dynamic masked-language modeling task with the AdamW optimizer (Loshchilov and Hutter, 2019). We uti- lize DeepSpeed ZeRO optimization and the De- BERTa Hugging Face implementation. We train a sentencepiece tokenizer (Kudo and Richardson, 2018) on a random sample of 100M sentences of each dataset. For the base and large models, we set the vocabulary size to 50k tokens, while the vocabulary of the xlarge model has 128k tokens.
To further enrich this dataset, extended dataset of six million PubMed abstracts, clini- cal notes from MIMIC III (Johnson al., 2016) and PMC-Patients-ReCCT translated into German. Prior to translation, the documents underwent a tokenization process using
(Zhao et al., 2023) was
http://huuggnngaeae..o co/ htt///acbooo///ea 19-en-de (last access: 21-06-2023)

---

Formal
Informal GE18
Medical GRAS
Literature DROC
Model
GE14
GQuAD
TS
GGP
JS
Avg
EM
87.10
72.19
55.07
51.27
72.34
78.17 17
62.90
77.18
88.03
73.65
GBERT base
#:.12
#1.39
=3.34
86.19
74.09
56.87
48.02
70.62
77.53
65.97
71.17
88.06
72.
GELECTRA base
+0..5
+1.80
=2.94
87.15
72.76
56.13
51.12
71.25
78.18
65.71
74.60
88.61
74.05
GottBERT base
+0.19
+:120
+0.80
87.80
78.01
62.19
51.81
74.70
78,08
66.04
80,13
87.67
75.53
GeBERTa base
+0.19
+0.7
+1.58
+0.16
-6.23
+0.32
88.06
78.54
62.06
53.16
74.83
78.13
68.37
81.85
89.14
76.51
GeBERTa base
+0022
+0.65
+1.39
=6.23
+0.32
Table 2: Fl-scores and standard deviation of the hase models.
Table 4 presents the hyper-parameters used in the pre-training of the different GeBERTa models.
Guhr et al., 2020 introduced a large sentiment clas- sification benchmark with positive, negative, and neutral labels. We combine sb10k (Cieliebak et al., 2017) and PoTTS (Sidarenka, 2016), resulting in a dataset of 14,980 tweets.
4.2 Downstream Tasks
We evaluate the existing German models and ours comprehensive benchmark. The bench- mark encompassed various task types, including question-anss answering, classification, and named en- tity recognition (NER). Additionally, we intro- duced new task focused on hate-speech detection, utilizing two existing datasets. When the datasets provided train, development, and test sets, we uti- lized them accordingly. In cases where such sets were not available, randomly split the data into 80% for training, 10% validation, and 10% test set. Specifically for the GEI8 dataset, we employed stratified splits based on the labels for the valida- tion set.
GermanQuAD (GQuAD) For English mod- els, the Stanford Question Answering Dataset (SQuAD) (Rajpurkar al., 2016) is popular downstream task. Each example consists of a para- graph from Wikipedia along with a question and the indices the answer the paragraph. the second version, questions without an answer were added, increasing the task difficulty. Möller et al., 2021 created German QuAD, which is based on the German Wikipedia articles corresponding to the English ones used in SQuAD.
GGPONC 2.0 (GGP) Based on clinical practice guidelines in oncology (Borchert et al., 2022) pub- lished NER dataset. Consisting of 1.8 million tokens from 10,000 documents, it is the largest available Germans medical NLP dataset. They pub- lished annotations with varying numbers of classes and span lengths. We focus on the most complex case with 8 fine-graine semantic classes and long entity spans.
GermEval 2014 (GE14) GermEval 2014 is NER dataset based on German Wikipedia and news articles. contains a total of 590,000 tokens from 31,000 sentences. It includes 12 entity types such location and person.
GermEval 2018 (GE18) This multiclass hate- speech classification dataset comprises 5,009 Ger- man tweets. It contains two coarse and fine-grained annotations. We focus on the latter more challeng- ing task. The dataset exhibits significant imbal- ance, with only 11.9% of the samples marked as insults and a mere 1.4% labeled as profanity.
GRASCCO (GRAS) is a collection of synthetic clinical case reports (Röhrig et al., 2022). Follow- ing the same annotation scheme in GGPONC 2.0 annotated version of GRASCCO ated (Bressem et al., 2023). Also, on this dataset, we evaluate the fine- fine-grained classes and long entity spans.
Tweet Sentiment (TS) We noticed that in GE18, the validation F1-score is good indicator of the test score. We believe this is due the rela- tively small number of examples for the underrepre- sented classes. We, therefore, created another hate speech detection task from two existing datasets.
DROC
The Corpus of Character References German Novels (DROC) comprises about 393,000 tokens from 90 German novels. 52,079 tokens were

---

annotated as character references differentiating between four entity types.
Our hyper-parameter search also improved the results that were previously reported on GE18 for base by 0.37% and for GELECTRA by 1.74%. Furthermore the performance of GouBERT base on GE14 by 0.31%. In some other cases, we were not able to reproduce the same per- formance as previously reported. We attribute this to the differences in evaluation. For instance, the sults reported for GouBERT are the best result from several runs, while we report the average across five runs. Table 5 presents the peramme search space used for fin-ttuning the models on the different downstream tasks.
GBERT base
Jsynee (JS) (Lohr et al., 2018) published dataset of medical textbook cases with topic la- bels. Due to copyright restrictions, they released source code that generates the dataset based on the book PDFs. We were able to acquire 7 out of 10 of these books resulting in 544 samples and six classes.
4.3 Fine-Tuning
Following (Liu et al., 2019), we perform limited parameter search for each model and task combi- nation. We use warmup step ratio of 6% and consider the batch sizes 16 and 32. Depending on the model size we adapt the learning rate range. The maximum number of epochs was set to 20. The final parameters were set according to the best validation performance of each sweep. Finally, we perform five test runs with early stopping and re- port the average test performance.
The high standard deviation observed across all models in GE18 validates our initial concerns about its instability. Conversely, in the case of TS, the standard deviation is noticeably lower across all models.
Discussion
Our extensive experiments provide clear evidence that the models that we have published have achieved a significant step forward in the process- ing of natural language in the German language (NLP). It is important note that certain improve- ments can undoubtedly be attributed the De- BERTa architecture. This can be seen in improved performance on tasks such as question answering and named entity recognition with larger entity spans. However, a direct comparison between models trained on web crawl data alone and those trained on cross-domain data shows a clear im- provement, which can be attributed to including diverse pre-training data. We support this claim with a significance test between GeBERTa Q and GeBERTa y which is shown in Table 6.
4.4 Results
We present our results for the base models in Ta- ble 2. While GeBERTa surpasses the existing models on almost all tasks, GeBERTa achieves the best results. Particularly clear differences com- pared to previous models can be seen on GQuAD (+1.55%) and GRAS (+ (2.1%). As expected, train- ing informal texts improves the results down- stream tasks from this domain, namely GE18, and TS. This is also true for the medical datasets and DROC. Interestingly, the performance also im- proves on GE14 and GQuAD. Both tasks are based on Wikipedia and can therefore clearly be assigned to the formal domain. Nevertheless, the model ben- efits from cross domain pre- pre-training here as well. Table 3 presents the performance of the larger models, Analogous
base
Furthermore, our results have shown the poten- tial of a cross-domain dataset, mainly built by au- tomatic translation and filtering of existing multi- lingual resources, to improve the quality of these models. The recently published Falcon model (AI- mazrouei et al., 2023) was trained using combi- nation of web-crawled data and curated English cross- data. Therefore, we strongly believe that generative models can also benefit from adopt- ing our data collection methods for assembling multilingual, cross-domain datasets.
the base mod- els GeBERTa large outperforms GeBERTay al- though the difference here smaller. For instance, on GQuAD it performs better than GeBERTay (+1.02%) and, interestingly, also on TS despite that it was not trained on informal data. This suggests that for larger models, the specific do- mains in the pre-training dataset not always relevant. However, GE18 is clearly worse than GeBERTa large In addition, GeBERTay achieves higher results on average across all tasks. Finally GeBERTa? starge scores the highest across all tasks but GGP and GRAS.
Q
An interesting finding from our research is that our base model achieved a comparable level of per- formance to the previous large state-of-the-ar mod- els. This finding suggests that the required com-

---

Formal
Informal GE18
Medical GGP
Literature DROC
Model
GE14
GQuAD
TS
GRAS
Avg
Fl
EM
88.48
81.51
63.41
54.37
73.60
79.17
69.28
76.32
90.29
76.63
GBERT large
10.23
20.84
=1.01
11.65
20.61
20.14
20.80
14.42
10.15
10.63
88.39
80.51
63.71
55.41
73.84
79.09
70.16
73.73
89.83
76.37
GELECTRA large
20.13
20.41
21.15
1154
20.88
10.09
20.92
12.35
20.27
10.60
GeBERTa Q
88.69
83.54
67.40
51.84
75.70
78.13
68.91
81.07
89.57
77.18
large
10.44
20.84
1:017
10.47
1:0660
GeBERTa V large
88.84
82.52
65.15
53.76
75.32
78.35
70.02
82.16
90.39
77.67
100.4
GeBERTa V xlarge
89.04
85.05
67.71 71
55.80
76.25
76.71
67.92
82.42
90.63
77.98
1:0 26
10.63
00.62
1..42
2.0700
10.08
11,00
14.70
10.21
1:0.0
Table 3: Fl-scores and standard deviation of the large models.
putational resources can significantly reduced by using a diverse dataset in conjunction with ef- ficient model architectures. As a result, these ad- vances become more accessible to a wider range of people and lead to savings in power consumption and reduction in the associated environmental impact. A recently published study French model (Antoun et al., 2023) further supports this notion, showing that using novel model architec- tures helps to make training more data efficient. We hypothesize that training on diverse datasets could further improve efficiency in similar way, given their training on a dataset derived from the CC100 pipeline.
putational resources required to train large-scale models can have sigmificant carbon footprint, con- tributing to environmental concerns.
Finally, while the majority of our dataset consists of filtered multilingual datasets and translations, this type of data is difficult to obtain in languages with limited resources. We believe that this is an issue that will need to be addressed in future work.
References
Lisa Adams, Daniel Truhn, Felix Busch, Avan Kader, Stefan Nichues, Marcus Makowski, and Keno K Bressem. 2023, Leveraging gpt-4 for post hoc trams- formation of free-text radiology reports into struc- tured reporting: multilingual feasibility study. Ra- diology, (07(4))2330755.
6 Conclusion
conclusion, this paper presents German lan- guage models that achieve state-of-the-ar results on various natural language processing tasks by pre-training on more diverse datasets. The experi- ments show that training language models on cross- domain datasets leads to improved performance compared to models trained on web crawl data alone. The results suggest that the computational resources required for training can be significantly reduced while still achieving high performance through the use of diverse datasets and efficient model architectures. Overall, the research provides evidence that incorporating diverse, multilingual data is crucial for building high-quality language models.
Ebtesam Almazrouei, Hamza Alobeidii, Abdulaziz AL- shamsi. Alessandro Cappelli. Ruxandra Cojocaru. Merouane Debbah, Etienne Goffinet, Daniel Hes- low, Julien Launay. Quentin Malartic, Badreddine Noune, Baptiste Pannier, Guilherme Penedo. 2023. Falcon-40B: an open large language model with state-of the performance.
Emily Alsentzer, John R Murphy, Willie Boag, Wei- Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. 2019, Publicly available clinical bert embeddings. arXiv preprint arXiv:1904.0332.
Wissam Antoun, Benoit Sagot, and Djamé Seddah. 2023. Data-efficient french language modeling with camemberta.
Bai, Tong Wang, Alessandro Sordoni, and Peng Shi. 2022. Better language model with hypernym class prediction. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1352-1362, Dublin, Ireland. Association for Computational Linguistics.
Limitations
Despite the remarkable capabilities of language models (LMs), it important to recognise their lim- itations. One major concern is the high energy con- sumption associated with training LMs. The com-
Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The

---

pushshift reddit dataset. In Proceedings of the inter- national AAAI conference on web and social media, volume 14, pages 830-839.
Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin. Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellsters Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. PaLM: Scaling Language Modeling with Pathways. CoRR, abs/2204.02311.
Andreas Blombach, Natalie Dykes, Philipp Heinrich, Besim Kabashi, and Thomas Proisl. 2020. corpus of German Reddit exchanges (GeRedE). In Proceed- ings of the Twelfth Language Resources and Evalua- tion Conference, pages 1510-6316, Marseille, France. European Language Resources Association.
Florian Borchert, Christina Lohr, Luise Modersohn, Jonas Witt, Thomas Langer, Markus Follmann, Matthias Gietzelt, Bert Amrich, Udo Hahn, and Matthieu-P. Schapranow. 2022. GGPONC 2.0 - the German clinical guideline corpus for oncology: Cu- ration workflow, ammotation policy, baseline NER taggers. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3650- 3660, Marseille, France. European Language Re- sources Association.
Mark Cieliebak, Jan Milan Deriu, Dominic Egger, and Fatih Uzdilli. 2017. A Twitter corpus and benchmark resources for German sentiment analysis. In Proceed- ings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 45- 51, Valencia. Spain. Association for Computational Linguistics.
Keno K Bressem, Jens-Michalis Papaioannou, Paul Grundmann, Florian Borchert, Lisa C Adams, Leon- hard Liu, Felix Busch, Lina Xu, Jan P Loyen, Ste- fan M Niehues, et al. 2023. Medbert, de: A compre- hensive german bert model for the medical domain. arXiv preprint 233.0:2303.08179
Kevin Clark, Minh-Thang Luong, Quoe Le, and Christopher D Manning. 2020. Electra: Pre- tre-training text encoders discriminators rather than generators. arXiv prepriat arXiv:2003. 10555.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan. Prafulla Dhariwal, Arvind Neelakantan. Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan. Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020a. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Jacob Devlin, Ming-Wor Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186 Association for Computational Linguistics.
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang, Ho- race He, Anish Thite. Noa Nabeshima, al. 2020. The pile: An 800gb dataset of diverse text for lan- guage modeling. arXiv preprint arXiv:2101.000 0000777.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020b. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama. Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-specific lan- guage model pretraining for biomedical natural lan- guage processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1233.
Branden Chan, Stefan Schweter, and Timo Möller. 2020. German's 's next language model. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6788-6796, Barcelona, Spain (On- line). International Committce on Computational Lin- guistics.
Oliver Guhr, Annh-Kathrin Schumann, Frank Bahrmann, and Hans Joachim Böhme. 2020. Training a broad-coverage German sentiment classi- fication model for dialog systems. In Proceedings of the Twelfth I Language Resources and Evaluation Conference, pages 1627-1632, Marseille, France. European Language Resources Association.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gauray Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tswyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes. Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben
Suriya Gunasekar, Yi Zhang, Iyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo

---

de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. 2023. Textbooks are all you need.
Katherine Lee. Daphne Ippolito. Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Buni Burch, and Nicholas Carlini. 2022. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 8424-8445, Dublin, Ireland. Association for Computational Linguistics.
Pengcheng Hc, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. DEBERTA: DECODING- ENHANCED BERT WITH DISENTANGLED AT- TENTION. In International Conference on Learning Representations (ICLR).
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019, Roberta: A robustly optimized BERT pretraining 1692.
Alistair F.W. Johnson, Tom J. Pollard, Lu Shen, Li wei H. Lehman, Mengling Feng. Mohammad Ghassemi. Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Reger G. Mark. 2016. MIMIC-III, freely ac- cessible critical care database. Scientific Data, 3(1).
approach. CoRR,
Christina Lohr, Sven Buechel, and Udo Hahn. 2018. Sharing copies of synthetic clinical corpora without physical distribution - a case study to get around IPRs and privacy constraints featuring the German JSYNCC corpus. In Proceedings of the Fleventh In- ternational Conference on Language Resources and Evaluation (LREC 2018), Miyazaki. Japan. European Language Resources Association (ELRA).
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Con- ference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Pa- pers, pages 427-431, Valencia, Spain. Association for Computational I Linguistics.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray. Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
003661.
Louis Martin, Benjamin Muller, Pedro Javier Or- tiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoit Sagot, 2020. CamemBERT: tasty French language model. In Proceedings of the 58th Anmual Meeting of the As- sociation for Computational Linguistics, pages 7203- 7219, Online. Association for Computational Lin- guistics.
Tom Koemi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grund- kiewicz, Barry Haddow, Rebecca Knowles, Philipp Kochn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel, and Maja Popović. 2022. Findings of the 2022 conference on machine translation (WMT22). In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 1-45, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computa- tional Linguistics.
Timo Möller, Julian Risch, and Malte Pietsch. 2021. GermanQuAD and GermanDPR: Improving non- English question answering and passage retrieval. In Proceedings of the 3rd Workshop on Machine Reading for Question Answering, pages 42-50, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Taku Kudo and John Richardson. 2018. SentencePicce: simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023. Capabili- ties of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303. 13375.
Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Ed- uardo González Ponferrada, Huu Nguyen, et al. 2022. The bigscience roots corpus: A 1.6 tb composite mul- tilingual dataset. Advances in Neural Information Processing Systems, 05:31809-31826
Pedro Ortiz Suarez and Simon Gabay. 2022. A data- driven approach to named entity recognition for early Modern French. In Proceedings of the 29th Inter- national Conference on Computational Linguistics. pages 3722-3730, Gyeongju. Republic of Korca. In- ternational Committee on Computational Linguistics.
Malte Ostendorff, Till Blume, and Saskia Ostendorff. 2020a. Towards an Open Platform for Legal Infor- mation. In Proceedings of the ACM/IEEE Joint Con- ference Digital Libraries in 2020, JCDL '20, page 385-388, New York, NY, USA. Association for Com- puting Machinery.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: pre-trained biomedical language representation model for biomedical text mining. Bioinformatiic, (0(4)):2344-22)

---

Malte Ostendorff. Till Blume, and Saskia Ostendorff. 2020b. Towards an open platform for legal informa- tion. In Proceedings of the ACM/EEFF Joint Confer- ence on Digital Libraries in 2020, JCDL '20, page 385-388, New York, NY, USA. Association for Com- puting Machinery.
Casper, Zhun Liu, Shrimai Prabhumoye. George Zerveas, Vijay Korthikanti, Elton Zhang. Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. 2022. Using deepspeed and megatron to train megatron-turing nlg 530b, large-scale generative language model.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics: System Demonstrations, pages 101-108, Online. As- sociation for Computational Linguistics.
Jörg Tiedemann. 2012. Parallel data, tools and inter- faces in OPUS. In Proceedings of the Eighth In- ternational Conference on Language Resources and Evaluation 12), pages 22 181-221,, Istanbul, Turkey, European Language Resources Association (ELRA). Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 188):
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:230..13971.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-1 5551.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu-
ral Language Processing, pages 2383-2392. Austin, Texas. Association for Computational Linguisties.
Alex Wang, Yada Pruksachatkun. Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Superglue: A stickier benchmark for ceneral-purpoose language understand- ing systems. In Advances in Neural Information Processing Systems 32: Annual Conference on Neu- ral Information Processing Systems 2019, NeurlPS 2019. December 8-14, 2019, Vancouver, BC, Canada, pages 3261-3275.
Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi 2021. Med-bert: pretrained contextual- ized embeddings on large-scale structured electronic health records for disease prediction. NPJ digital medicine, :(1):86.
R Röhrig et al. 2022. Grascco- the first publicly share- able, multiply-aleenated german clinical text cor- pus. In German Medical Data Sciences 2022-Future Medicine: More Precise, More Integrative, More Sus- tainable!: Proceedings of the Joint Conference of the 67th Anmual Meeting of the German Association of Medical Informatics, Biometry, and Epidemiol- ogy EV (gmds) and the 14th Annual Meeting of the IMMF-Technole Methods, and Infrastructure for Networked Medical, volume 296, page 66. IOS Press.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Con- neau, Vishrav Chaudhary, Francisco Guzmán, Ar- mand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the Twelfth Lan- guage Resources and Evaluation Conference, pages 4003-4012, Marseille, France. European Language Resources Association.
Raphael Scheible, Fabian Thomezyk, Patric Tippmann, Victor Jaravine, and Martin Bocker. 2020. Got- tbert: a pure german language model. CoRR, abs/2012.02110..
Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin, and Xia Hu. 2023. Harnessing the power of Ilms in prac- tice: A survey on chatgpt and beyond. arXiv preprint arXiv:230..13712
Uladzimir Sidarenka. 2016. PoTTS:
The Potsdam Twit- ter sentiment corpus. In Proceedings of the Tenth In- ternational Conference Language Resources and Evaluation 16), pages 1133-1 141, Portorož, Slovenia. European Language Resources Association (ELRA).
Xi Yang. Aokun Chen, Nima PourNejatian, Hoo Chang Shin, Kaleb E. Smith, Christopher Parisien, Colin Compas, Cheryl Martin, Anthony B. Costa, Mona G. Flores, Ying Zhang, Tanja Magoc. Christopher A. Harle, Gloria Lipori, Duane A. Mitchell, William R. Hogan, Elizabeth A. Shenkman, Jiang Bian, and Yonghui Wu, 2022. A large language model for elec- tronic health records. npj Digital Medicine, 5(1):194.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared

---

Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang. Hanyu Lai. Ming Ding. Zhuoyi Yang. Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm-I 130:: An open bilingual pre-trained model. arXiv preprint arXiv:2210.0241..
Zhengyun Zhao, Qiao Jin, Fangyuan Chen, Tuorui Peng, and Sheng Yu. 2023. Pme-patients: A large-scale dataset of patient summaries and relations for bench- marking retrieval-basee -based clinical decision support sys- tems.
A Appendix

---

Hyper-parameter
GeBERTa xlarge
GeBERTa large
GeBERTa base
Number of Layers
24
24
12
Hidden size
1536
1024
768
FNN inner hidden size
6144
4096
3072
Attention Heads
24
16
12
Attention Head size
64
64
64
Dropout
0.1
0.1
Warmup Steps
10k
10k
10k
Learning Rates
le-4
le-4
2e-4
Batch Size
2k
2k
2k
Weight Decay
0.01
0.01
0.01
Max Steps
IM
1M
1M
Learning Rate Decay
Linear
Linear
Linear
Adam
le-6
le-6
le-6
Adam 81
0.9
0.9
0.9
Adam 82
0.999
0.999
0,999
Gradient Clipping
1.0
1.0
1.0
Table 4: Hyper- papemeparmeetrss for pre-traa me-training GeBERTa
Hyper-parameter
xlarge
large
base
Warmup Ratio
0.06
0.06
0.06
Learning Rates
156-6-66-6,77-66.8--6
(7--6.8e-6.9c-6.1e-5)
(1e-5.2e-5.3e-544-55
Batch Size
[16,32)
(16,32)
[16,3)]
Weight Decay
0.01
0.01
0.01
Maximum Training Epochs
20
20
20
Learning Rate Decay
Linear
Linear
Linear
Adam
le-6
le-6
le-6
Adam 81
0.9
0.9
0.9
Adam B2
0.999
0.999
0.999
Gradient Clipping
1.0
1.0
1.0
Table 5: Hyper-parameter search space for different model sizes.
Dataset
GE14
GQuAD
GE18
TS
GGP
GRAS
JS
DROC
p-value
0.04
<0.01
0.01
0.42
0.30
0.04
0.27
<0.01
Table 6: t-test with the null hypothesis that the GeBERTaY V achieves lower results than 2 ? reveals statistical relevance (p-value < 0.05) for 5/8 datasets.
The test
base"


Reviews:
[{"review_id": "MKuf1z9quW", "reviewer": "Reviewer_oKki", "summary": "", "questions": "I noticed the batch size for training models is quite small (2048 sentences If I get it right). Is it expected?\n\n[update after author response] I thank authors for their detailed response. I think the statement of the contribution from the paper should be updated as in your response. Please update the work that way. Meanwhile I update the score of the work from 2 to 3. I think it has merits but there are also key weakness (novelty, results are not that strong).", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your time, constructive feedback, and valuable questions raised.  We appreciate your sentiment that the models released with this paper will be of great benefit to the community. In the following, we love to address unclear aspects of the paper and answer insightful questions. \n\n**Addressing lack of contributions**\n\nThe concept of cross-domain data is indeed not novel, and a positive effect of training on more diverse data is an intuitive hypothesis. However, we did not find conclusive evidence for this. For example, in the CamemBERT paper [1], OSCAR and CC100 were compared. The authors of CamemBERT reported no significant difference between models trained on OSCAR and CC100. While both are web-crawl-based, CC100 was filtered for quality content, potentially limiting its diversity. In contrast to this, the authors of “The Pile” [2] hypothesized that their improvement can be attributed to the higher variety in the pile compared to CC100. Most recently, in [3], the authors reported that carefully processed web data leads to better results than the pile. Supporting the claim that focusing on diversity outweighs focusing on quality.\n\nWhile evidence was shown for generative large language models we were not able to find any research on the effects of cross-domain training of encoder models. As this previous work led to no clear conclusion, we investigate the tradeoff between quality-focused data vs. variety-focused data for encoder-only models in a controlled setting where both datasets are equally large.\n\nSpecifically, we would like to add that there is limited research on the effects of automatically translated English data cross-domain. For example, we found statistically significant improvements in the literature dataset DROC (see below), although almost all books used for pretraining were translations from English ones.  We also reported a significant improvement in the medical task GRAS (see below) while about 90% of our medical pretraining data was acquired through automatic translation or automatic filtering of common crawl. We think that this contribution is interesting for the non-English community, especially for low-resource languages.\n\n**Reduction of required computational resources**\n\nWe agree that the performance improvement from the variety-focused data is not clearly visible for all downstream tasks. For clarification, we would like to note our claim of reduced computational resources also mentions the model architecture (i.e., DeBERTa) as a factor:  \n“The results suggest that the computational resources required for training can be significantly reduced while still achieving high performance through the use of diverse datasets and efficient model architectures.“\nWe support this claim with the observation that our base model with 122M parameters trained on V achieves an average F1 score of 76.51 while GBERT large scores 76.63, and GELECTRA large 76.37, both with 335M parameters, reducing the required model size almost by a factor of 3 while maintaining a similar performance. We attribute the fact that our base model performs similarly as large models to the variety-focused data, but also to the more efficient DeBERTa model architecture. Our base model trained on Q scored an average F1 score of 75.53, supporting the hypothesis that data variety is necessary to reach the performance of large models with base models and the reduction of computational resources that entails this.\n\n\n**Benchmark significance**\n\nDue to your interest in the significance of the results from the models trained on the Q and V datasets, we performed a t-test with the null hypothesis that the base model trained on V achieves lower results than the base model trained on Q.\n\n\n| Dataset | GE14   | GQuAD  | GE18   | TS     | GGP    | GRAS   | JS     | DROC   |\n|---------|--------|--------|--------|--------|--------|--------|--------|--------|\n|    p-value     | 0.04   | <0.01  | 0.01   | 0.42   | 0.30   | 0.04   | 0.27   | <0.01  |\n\n\nThe test reveals statistical relevance (p-value < 0.05) for 5/8 datasets. Since the standard deviation for GE18 is much higher, we increased the number of runs initiated with random seeds from 5 to 10. Due to the limited available time for this rebuttal, we only performed additional runs for GE18. \n\nWe would like to note that the statistically non-significant results are partly due to problems with the datasets. For example, the standard deviation in JS is very high, which can be explained by the very small size. In addition, GGP is the only dataset on which all models perform approximately equally well, which may indicate that this dataset might be saturated.\n\n**Correction of data quantity**\n\nWhile it is true that the size of the pretraining dataset plays a significant role in the quality of a model, GBERT and GELECTRA were trained on multiple datasets (see Table 1 in [4]) with a total size of 162GB. In comparison, our datasets consist of 167GB, which we only perceive as a minor difference in quantity and not as the primary factor for the performance difference.\n\n\n**Why this batch size**\n\nEach input consists of 512 tokens, so a batch size of 2048 means that each optimization step was done with 512 * 2048 tokens. We followed the original DeBERTa configuration [5].\n\nWe sincerely appreciate your valuable feedback, which has greatly contributed to the refinement of our work. We hope that we addressed your concerns and will be happy to answer any further questions. If you think that our additional experiments and clarifications improve the quality of this work, we will happily include them in the final version. \n\n\nReferences\n\n[1] Martin, L., Muller, B., Ortiz Suarez, P., Dupont, Y., Romary, L., Clergerie, ., Seddah, D., & Sagot, B. (2020). CamemBERT: a Tasty French Language Model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 7203–7219). Association for Computational Linguistics.\n\n[2] Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., ... & Leahy, C. (2020). The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027.\n\n[3] Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cappelli, A., Alobeidli, H., ... & Launay, J. (2023). The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116.\n\n[4] Chan, B., Schweter, S., & Moller, T. (2020). German′s Next Language Model. In Proceedings of the 28th International Conference on Computational Linguistics (pp. 6788–6796). International Committee on Computational Linguistics.\n\n[5] He, P., Liu, X., Gao, J., & Chen, W. (2020). Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Including cross-domain datasets seems to be useful in 7/8 cases and 6/8 cases. It is not obvious whether the gain is always significant (e.g. it seems very minor for GE14, GGP).\n* Models will be released together with the paper and I think the released models will be useful for the community.", "reasons_to_reject": "- The finding of having more cross-domain datasets for training pretrained models is not a novel idea  IMHO. It has been well known that the more diversed data we train a ML model, the better the model could be across different domains. With that to me the novelty of the paper is very limited.\n- The claim that the required computational resources can be significantly reduced by using a diverse dataset is very misleading IMO. First of all if we train the a pretrained model on two different datasets (Q and V), not all improvements are that significant from their benchmark (maybe the biggest gains are with GE18, 68.37 and I am not sure the level of improvement is that significant). Second, it is true that the paper achieves much stronger results than previous work, but I think it is simply because the models trained with much larger data (Please correct me here if I was wrong - GBERT, GELECTRA are trained on GC4, while your models are trained on GC4 + WMT 2022 German monolingual news crawl + Wikipedia). This shows that increasing the training data is still the most crucial aspect regarding the training data and not the domains diversity). So I agree computational resources can be reduced, but I am not sure the level of degradation is that significant as claimed in the paper.", "questions_to_authors": "I noticed the batch size for training models is quite small (2048 sentences If I get it right). Is it expected?\n\n[update after author response] I thank authors for their detailed response. I think the statement of the contribution from the paper should be updated as in your response. Please update the work that way. Meanwhile I update the score of the work from 2 to 3. I think it has merits but there are also key weakness (novelty, results are not that strong).", "paper_topic_and_main_contributions": "This paper presents German language models that achieve strong results on several NLP tasks including GE14, GQuAD, GE18, TS, JS. The key finding of the paper is to claim that training language models on cross-domain datasets leads to improved performance compared to models trained on web crawl data alone.  Specifically, a large high-quality curated dataset from GC4, WMT 2022 German monolingual news crawl and Wikipedia is used as a baseline. Another dataset with the same size is used for comparison and that one includes cross-domain datasets related to Informal, Medical, Legal, Literature. Experiments show that training pretrained models with different size improve quality accuracy in 7/8 cases for base model and 6/8 cases for large model. While this is expected, it is not clear all of the improvement is significant. Another claim that the required computational resources can be significantly reduced by using a diverse dataset is misleading IMO though, because there is a lack of datapoints to convince this.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "q5haLyK0mH", "reviewer": "Reviewer_Rbed", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review. We appreciate your insights and the time you took for the review. We're pleased to hear your agreement with the potential impact of our work in the future, as well as the release of a number of pretrained large language models with state-of-the-art performance evaluated using 8 benchmark datasets, leading to improvements up to 6.1% over previous best results. We believe that these aspects will indeed prove beneficial for the broader community.\n\nFurthermore, we would like to take this opportunity to emphasize our contribution regarding empirical evidence for the benefits of variety-focused pretraining data as opposed to quality-focused data, particularly concerning encoder-only models. We're excited about the potential implications of our findings for the field, and we hope this contribution will resonate beyond the scope of German NLP.\n\nWe are glad that you have recognized our experiments provide sufficient support regarding the advantages of variety-focused data over quality-focused data. We provide additional significance test results in our answer to reviewer 1.\n\nWe thank you again for your positive assessment of our work, and we are happily available for any further questions that may arise.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "By releasing large amounts of German monolingual data for various domains and pre-trained models, which are not available in previous studies, it will be very helpful for future related research.", "reasons_to_reject": "I don't see any reason to reject this paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper describes the German corpus collected for training large German language models and analyses the performance of those language models. In the case of German, research on learning language models using large monolingual corpus is outdated, and unlike English, domain specific data is not actively used for learning. In this study, the authors first collected quality-focused data using CC100 and domain variety-enhanced dataset (adding informal, medical, legal, literature domain data to quality focused data).\n\nIn this paper, the following three models were trained by following the configuration of DeBERTa. First, $GeBERTa_{large}^Q$ is the $DeBERTa_{large}$ model trained with quality-focused data. $GeBERTa_{large}^{V}$ is a model with the same configuration trained on variety-enhanced data. Finally, $GeBERTa_{xlarge}^{V}$ is a larger model trained with variety-enhanced data.\n\nThe above models were finetuned on eight downstream tasks and their performance was compared with the models presented in previous studies. The results showed that the models trained with variety-enhanced data performed relatively better.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qSzNAECLMQ", "reviewer": "Reviewer_BK21", "summary": "", "questions": "N/A", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We want to highlight that our results show that cross-domain training outperforms quality-focused training, with up to 6.1% improvement over the previous state-of-the-art. Along with this result, we introduce (1) a German dataset comprising texts from five domains and (2) a number of pretrained large language models with state-of-the-art performance evaluated on a benchmark of 8 datasets. Our findings suggest that using a large corpus translated from English fills the gap of access to large-scale domain-specific German corpora, which will benefit NLP in many non-dominant languages.\n\nWe appreciate your constructive feedback, and we are happy to address your concerns:\n\n**Dataset Composition**\n\nAs you noted, we combined multiple datasets and employed a wide range of preprocessing methods. However, we would like to clarify that we did not rely on existing preprocessing scripts. Instead, we merged existing methods into a new framework for non-English variety-focused data acquisition and introduced new approaches, such as the translation of large English corpora and the filtering of common crawl for medical texts.  Since the translation models and the web-crawl data sets are available for a wide range of languages, we believe this work will also impact other languages. This finding holds relevance, particularly for languages where domain-specific resources are scarce.\n\n**Automated Programs and Data Quality**\n\nYour point about the challenges of maintaining data quality when employing automated programs is valid and is the core idea that drove our research. We believe there is a trade-off between data quality and dataset variety. While curated sources might restrict noise, incorporating diverse data, even from low-quality sources like social media, brings the advantage of capturing real-world variations. To investigate this tradeoff, we constructed two equally large datasets. The first dataset is geared towards data quality, incorporating carefully curated texts from reliable sources such as Wikipedia, news articles, and CC100 [1], for which the authors state that it consists of quality-filtered web data. In contrast, the second dataset showcases data processed using automated methods, including translation. We acknowledge that this data has lower quality due to automated processing. However, we show that despite this loss in quality, the variety-focused datasets lead to models that perform better. Beyond the specific nuances of data quality and variety, our work also introduces several pretrained large language models with state-of-the-art performance evaluated using 8 benchmark datasets for the German community.\n\nWe appreciate your review and hope that we were able to address your concerns. We fully agree with your concern about the risk of low-quality data when applying automated processing scripts. Our results indicate that the benefits of data variety outweigh the drawbacks of lower data quality. We hope that our explanations provide a clearer perspective on the intentions and outcomes of our research. We are happy to respond to further questions and comments.\n\n\n\n[1] Wenzek, G., Lachaux, M.A., Conneau, A., Chaudhary, V., Guzman, F., Joulin, A., & Grave, E. (2020). CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data. In Proceedings of the Twelfth Language Resources and Evaluation Conference (pp. 4003–4012). European Language Resources Association.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The author build a German dataset comprising texts from five domains.", "reasons_to_reject": "It looks like the authors merged multiple datasets and then processed them using other people's preprocessors, while using some automated programs to aid in the construction, but it's difficult to control the quality of the data when constructing with programs. I think the contribution of this paper is very limited.", "questions_to_authors": "N/A", "paper_topic_and_main_contributions": "The author present a German dataset comprising texts from five domains, along with another dataset aimed at containing high-quality data.", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents new sets of German encoder-only language models using a cross-domain dataset. The model is based on DeBERTa architecture, and it achieves an impressive performance on almost all the NLP tasks evaluated on compared to previous models (GBERT, GELECTRA, GottBERT). 

The reviewers found the models development to be useful especially since it achieves impressive result and since the authors planned to release their code to the community. Also, their results highlight the use of a cross-domain/heterogeneous dataset for model pre-training. There is a small concern about limited contributions of the paper apart from the cross-domain corpus, and building a new set of models, but the issues have been addressed by the authors.
Decision: Accept-Findings