Building a Large Japanese Web Corpus for Large Language Models
Authors: Naoaki Okazaki
Kakeru Hattori
Hirai Shota
Hiroki Iida
Masanari Ohi
Kazuki Fujii
Taishi Nakamura
Mengsay Loem
Rio Yokota
Sakae Mizuki
Venue: COLM (2024)
Keywords: corpus, web, Common Crawl, Japanese, continual pre-training, Llama 2, Mistral, Mixtral
TL;DR: This study builds a large Japanese web corpus from the Common Crawl archive, and demonstrated its effectiveness by continual pre-training on Llama 2 7B, 13B, 70B, Mistral 7B v0.1, and Mixtral 8x7B.

Abstract: Open Japanese large language models (LLMs) have been trained on the Japanese portions of corpora such as CC-100, mC4, and OSCAR. However, these corpora were not created for the quality of Japanese texts. This study builds a large Japanese web corpus by extracting and refining text from the Common Crawl archive (21 snapshots of approximately 63.4 billion pages crawled between 2020 and 2023). This corpus consists of approximately 312.1 billion characters (approximately 173 million pages), which is the largest of all available training corpora for Japanese LLMs, surpassing CC-100 (approximately 25.8 billion characters), mC4 (approximately 239.7 billion characters) and OSCAR 23.01 (approximately 74 billion characters). To confirm the quality of the corpus, we performed continual pre-training on Llama 2 7B, 13B, 70B, Mistral 7B v0.1, and Mixtral 8x7B as base LLMs and gained consistent (6.6-8.1 points) improvements on Japanese benchmark datasets. We also demonstrate that the improvement on Llama 2 13B brought from the presented corpus was the largest among those from other existing corpora.

Published as a conference paper at COLM 2024
Building Large Japanese Web Corpus for Large Language Models
Naoaki Okazakiž Kakeru Hattori, Hirai Shota, Hiroki lida, Masanari Ohi,
Kazuki Fujii, Taishi Nakamura, Mengsay Loem; Rio Yokotaj Sakae Mizuki
Taihi
Department of Computer Science, School of Computing, Tokyo Institute of Technology
$ Global Scientific Information and Computing Center, Tokyo Institute of Technology 2-12-1 Ookayama, Meguro-ku, Tokyo, 152-8550 Japan
fokazakiēc, seakeru.hattor@@n/ee
chota.hirai@m.
hiroki. hiroki.iida@nlp. iida@nlp.c,
masanari ohi@nlp.c kazuki. fujii@rio. gsic, taishi. nakamura@r gsic, mengsay. loemênlp.c, rioyokota@rio. gsic, sakae. www.mmikiillp...mmm c}. titech.ac, ac. jp
Abstract
Open Japanese large language models (LLMs) have been trained on the Japanese portions of corpora such as CC-100, mC4, and OSCAR. However, these corpora were not created for the quality of Japanese texts. This study builds large Japanese web corpus by extracting and refining text from the Common Crawl archive (21 snapshots of approximately 63.4 billion pages crawled between 2020 and 2023). This corpus consists of approximately 312.1 billion characters (approximately 173 million pages), which is the largest of all available training corpora for Japanese LLMs, surpassing CC-100 (approximately 25.8 billion characters), mC4 (approximately 239.7 billion characters) and OSCAR 23.01 (approximately 74 billion characters). To confirm the quality of the corpus, we performed continual pre-training on Llama 2 7B, 13B, 70B, Mistral 7B v0.1, and Mixtral 8x7B Instruct as base LLMs and gained consistent (6.6-8.1 points) improvements on Japanese benchmark datasetss We also demonstrate that the improvement on Llama 2 13B brought from the presented corpus the largest among those from other existing corpora.
Introduction
ChatGPT, released by OpenAl in late 2022, established a milestone toward achieving general- purpose artificial intelligence. Research topics using large language models (LLMs) attract much attention including chain of thoughts (Wei al. 2022, Kojima al. 2022 Wang al.
2023a Trivedi et al. (2023), instruction tuning (Wang et al. 2023bb c), evaluation (Suzgun et al.
2023 Zheng et al. [2023), hallucination (Rawte et al. 2023) | et al. 2024), bias (Ladhak
eet al. | Feng et al. 2023), detecting generated text (Mitchell et al. 2023) watermark-
ing (Kirchenbauer et al. 2023), poisoning (Wan et al. 2023), detecting traininin data (Shi
et al. 2024), unlearning Yao et al.) 2023), and efficient interence (Dettmers et al.) 2023).
While there are various motivations, such as raising the level of research and development in natural language processing, elucidating the mechanisms of LLM intelligence, the security risks of relying on a handful of foreign companies, and achieving responsible artificial intelligence, several Japanese companies and universities have actively been developing open LLMs that achieve good performance on Japanese text. However, in many cases, the quality of the training data for building Japanese LLMs was not satisfactory because they were mostly developed overseas as a part of multilingual corpora.
Table lists representative corpora used for training LLMs. Some popular corpora such as
Pile-CC (Gao et al.) 2020), ROOTS [Laurençon et al. 2023) Dolma (Soldaini et al. 2023), and
RedPajama-Data-v/ cannot be used for training Japanese LLMs, including no Japanese text.
httpp:://www stg.gteerr.i//bboo//reeaaaaaaaa

---

Published as a conference paper at COLM 2024
Corpus
Size (en)
Size (ja)
Source
Text
Langdet
statmt. oorg nggams
976 BT
2012, 2013
WE1?
cld2
Buck et al. 2014)
Dedup: Remove lines with exact hash values (MurmurHash, 64 bit).
Clean: Remove email addresses and HTML fragments; normalize
punctuation and capitalization (Moses truecaser).
CC-100 (CCNett
532 BT
26 BL 2018
WET
fastText
WWeneek et al. 2020
Dedup: Remove paragraphs with exact hash values (SHA-1, 64 bit).
Clean: Use documents with high likelihood computed by the lan- guage model trained on Wikipedia.
C4
156 BT
April 2019
WET
langdetect
Raffel et al. 2020)
Dedup: Remove exact matches in three-sentence units.
Clean: Keep only lines ending in punctuation mark; remove docu-
ments with one or two sentences; remove lines with 4- words; etc.
mC4
2,733 BT
240 BL. 71 months
WET
cld3
Xue et al. 2021
Dedup: Same as C4
Clean: Same C4, but they use modified rule for keeping text
(because the punctuation filter of C4 is specific to English): the text must be at least 200 characters long and contain at least 3 lines. fastText
OSCAR 23,01
523 BW
74 BL
Nov Dec 2022
WET
Abadji et al 2022
Dedup: TLSH
Clean: Remove documents with high likelihood computed by the
language models trained on harmful content or with URLs registered in Blacklists UT1.
Pile-CC
cld2 Dedup: MinHash (Jaccard coefficient threshold is approximately 0.5).
2,312 BL
2013 2020
jusText
Gao et al. 2020
Clean: Use documents similar to those in OpenWebTer Web Text2ex2 (documents
with a Reddit score of or higher) based on a fast classifier. (original)
ROOTS
484 BB
(539 domains)
fastText
Laurençon et al. 2023
Dedup: Sim (6-gram)
Clean: Remove documents with too few words, with a high percent- age of repeated characters, with a high percentage of repeated words, with too many emojis, or with too few function words, etc.
Web
trafilatura fastText Dedup: MinHash (character 5-gram, approximate threshold 0.8)
600 BT
Until Jun 2023
[Penedo et al. 2023)
Clean: Blacklist URL filter by UT1; remove repetitions; per-document
filter; per-line modification; remove lines matching more than 50 tokens based on sulfix array.
Dolma
2,415 BT
May 2020 to Jun 2023
WET
fastText
(ooldini et al. 2023
Dedup: Remove documents with identical URLs; remove identical paragraphs within documents.
Clean: Paragraph filtering of MassiveWeb (Rae et al. 2021 punctua- tion rules of C4; etc.
ClueWeb 22
*16.7TT 3,301 BT Search engine
BlingFire (original)
Overwijk et al. 2022)
Dedup and Clean: Documents were sampled from a commercial
(Non-commercial use)
search engine based on the distribution of page visits.
RedPajama-Datta-22
T0.5
All (84 dumps)
WET
fastText
Dedup: Same as CCNet
Clean: Text-quality estimation rules used by C4, RefinedWeb, etc.
This study (raw)
3,684 BL 3,684 BL 2020-40 to 2023-23
trafilatura (original)
Dedup and Clean: None.
This study (clean)
312 BL 312 BL 2020-40 to 2023-23 trafilatura (original)
Dedup: MinHash (character 5-gram, Jaccard coefficient threshold is approximately 0.9).
Repetition detection of RefinedWeb and quality filter specially designed for Japanese text.
Table 1: Representative corpora that can be used to pre-train LLMs. BL, BW, BT, and TT stand for billion letters, billion words, billion tokens, and trillion tokens, respectively. Ex- tracted from the corresponding papers, numbers in English size are not directly comparable because of the difference in units. The number of Clue may include text in other languages. Numbers in Japanese size are in Unicode characters and are directly comparable. Text, Langdet, Dedup, and Clean explain methods for text extraction, language detection, deduplication, and cleaning. WET in text extraction indicates that the corpus uses the text extraction results distributed by Common Crawl in WET format.
2

---

Published as a conference paper at COLM 2024
Therefore, CC-100 (Wenzek et al. 2020), mC4 (Xue et al. 2021 and OSCAR 23.01 (Abadji itt al. 2022) are candidates for training Japanese LLMs, including a certain amount of Japanese text and released with permissive licenses. However, the quality of Japanese text in these corpora is unsatisfactory because they contain noise in the HTML- HTML-to--tex conversion (Common Crawl WET format) and because they have incorporated no special efforts to improve text quality for Japanese. Although we recognize the importance of efforts in building multilingual corpora, it is difficult to build useful and reliable Japanese corpus without the knowledge of Japanese language. Concurrently to our work, |Enomotoo et al. (2024) explored filtering methods for Japanese web corpora, but this study did not demonstrate the effectiveness of the filtering methods on generative LLMs.
Therefore, this paper explores method to construct a large-scale, high-quality Japanese web corpus that can be used for training Japanese LLMs. The presented method includes lightweight language detection that boosts the processing speed of text extraction in the target language. This strategy is applicable not only to Japanese but also to other languages with less text than English. In addition, we specially design a filtering method to find good-quality Japanese text. The corpus developed in this study is made from 21 snapshots of Common Crawl (from CC-MAIN-2020-11 to corpus after cleaning is 173,350,375 pages and 312,093 428,689 characters. To confirm the quality of the corpus, we perform continual pre-training on Llama 2 7B, 13B, 70B [Touvron] et al. 2023), Mistral 7B v0.1 (Jiang et al. 2023), and Mixtral 8x7B Instruct (Jiang et al. 2024 as base LLMs. Experimental results demonstrate that continual pre-training consistently improves the base model's performance by 6.6-8.1 points on Japanese benchmark datasets. We also demonstrate that the improvement on Llama 2 13B brought from the presented corpus was the largest among those from other existing corpora. The models trained on the presented corpus are available the web site and Hugging Face") The implementations of this study are available GitHut 15
2003-223) and the size of the
2 Related work
A number of large corpora were built from Common Crawl |archiess Common Crawl is non-profit organization that crawls websites and provides their archives. The crawled data was initially distributed in ARC forma S| but since the summer of 2013, it has been stored in Web ARChive (WARC) and Web Text (WET) formats. According to Statistics of Common Crawl Monthly Archives 10 the total amount of accessible archives (data crawled since 2013 till 2023) is 251,325,355 35,,155,,7 pages. Compact Language Detector 2 (cld2 II estimated that about 5 percent of these web pages are written in Japanese.
Table I summarizes the corpora derived from Common Crawl (except for ClueWeb22, where documents are sampled from a commercial search engine). All permissive corpora that contain Japanese text rely on the data in WET format (HTML pages were converted to text by Common Crawl). The advantage of processing data in WET format is the reduction of processing time and data transfer size. Wenzek et al. (2020) proposed pipeline to extract multilingual text from Common Crawl to train a multilingual BERT model (XLM-
2Continual pre-training on a base model starts with the weights (parameters) of the base model and performs another round of pre-training on new data (the Japanese corpus in this study). This is also called as continued pretraining.
hhttps:/swwaloow 11m. github.
httpp::/huuginggf ace. wtt.ceccc.eeeeeoeoooooeeeooeeeee tokyotech-
'The entire pipeline except for deduplication:
6 @Deduplicaion:: httpps://gitubb. http://wllllwwwo.ww 11m//ou/ubmmmmrr
/https:/ccommoncrawl.orr
https: http:://waattt....oorrrrrroeerer
10
3

---

Published as a conference paper at COLM 2024
R) (Conneau et al. 2020), and released its implementation (CCNet
12 and data (CC-100)
Xue al. (2021) constructed mC4, multilingual extension of C4 Raffel et al. 2020) to train m15, multilingual variant of T5. Abadji et al. (2022) released OSCAR 23.01 with adult
content filtering (based on an æ-gram language model) and duplicate removal (based on
locality sensitive hashing) added to the previous release.
However, the corpus construction procedure often introduces aggressive filtering rules because the WET data usually includes irrelevant text, e.g., noises HTML-to-text conver- sion, JavaScript codes, navigation menus, and footers. For example, C4 (Raffel et al. 2020) had to introduce filtering rules such "remove lines with the word JavaScript"" (to remove JavaScript code) and "remove pages with curly brackets ()" (because curly brackets are rarely used in a natural language but often used in a programming language). Therefore, we use WARC instead of WAT as the source format of Common Crawl archives.
We follow the design principle of Web (Penedo et al. (023): scale first (avoid labour intensive human curation process), strict deduplication (respect the value of deduplication for large language models), and neutral filtering (avoid undesirable biases introduced by machine-learning bbaed filtering). The construction procedure of Refined has much in common with MassiveWeb (Rae et al. 2021). Collecting web pages with their own web crawler, Rae et al. (2021) used Google's SateSearch filter to remove harmful content, and extracted text based on the HTML. DOM structure. They also removed documents that contained lot of repetitions, low-quality text, or duplicated information. In this study, extend their pipelines to extract high-quality Japanese documents efficiently.
We also considered using ClueWeb22 (Overwijk et al. 2022 Japanese corpus. Although Clue has nice and unique properties such as real distri- bution (web pages are extracted by the crawler of commercial search engine and follow the distribution of web search activities), large-scale quality content (content extraction pipeline based on the search engine's production-quality content understanding system), and rich information (annotations of content structure, in-links tiiiikk//u-llikke out-links of web pages, visual features of contents, etc), we could not use ClueWeb22 in this study because an LLM trained on ClueWeb22 cannot be released under an open license.
source for building large
3 Method
Figure
shows the corpus building pipeline. This pipeline is roughly divided into the following three stages: (1) Extracting Japanese text from WARC files in Common Crawl (Section ; (2) Selecting Japanese text carefully with quality filtering (Section 3.2 dedu-
plication (Section 3.3, and host filtering (Section 3.4 (3) Cleaning extracted text (Section
3.5
3.1 Extracting Japanese text
Common Crawl snapshots are stored as buckets in Amazon S3 and can be accessed via S3 or the web server 14 We extract HTML content from WARC data using the WARCIO library 15
(Step 1, Figure
We will defer to the explanation of Step until later. We then apply
Trafilatura Barbaresi]
2021) for extracting text from HTML markups (Step 3). Step detects the language of text based on linear binary classifier (see A.p for the detail) and extracts Japanese text only.
Because about 5% of the entire Common Crawl is written in Japanese, it is possible to reduce the processing time of Steps 3 and 4 by 95% if we can target Japanese web pages only. However, in order to improve the accuracy of language detection, it is desirable to remove HTML markup before applying the language detection. Therefore, we should apply text
hhtps://huggiigfffc.ccm https//aaw...c/atttttttte///eeee net
15
15
4

---

Published as a conference paper at COLM 2024
Step
Downloading WARC files
63,352,266,406 pages in Common Crawl (2020-200) 2023)
Step
Rapid Japanese detection
This makes the processing time times faster
Step 3
Text extraction
Extracting text from HTML (by using Trafilatura)
Step
Precise Japanese detection
2,686,080,913 pages (3,684,320,219,69) characters)
Step 5
Quality filtering
646,238,066 pages (1,202,868,044,6 characters)
Step
Deduplication (MinHash)
185,766,463 pages (353,191,872,8)) characters)
Step 7
Filtering by hostnames
173,350,375 pages (312,674,522,600 characters)
Step
Normalizing punctuations
Normalize Japanese punctuations into " "and"
Step 9
Removing footers
173,350,375 pages (312,093, 688, characters)
Figure 1:
The pipeline for building large Japanese web corpus.
extraction first (Step 3), followed by language detection (Step 4). To make matters worse, text extraction takes more processing time than language detection; we cannot reduce the processing time in this processing order.
This is the motivation behind Step 2: we select web pages that are highly likely to be in Japanese in a rapid manner without text extraction. A web page proceeds Steps 3 and only when one of the following two conditions is met: (1) the HTML declares the language as Japanese, for example, <html lang="ja">; (2) the precise Japanese detection (Section A.1 recognizes the content within <title> tag as Japanese.
To evaluate the validity of this rule, we compared the results of rapid and precise language detection. More specifically, assuming that the result of the precise language detection is
correct, measured the accuracy of rapid language detection on 10 WARC files (394,192 pages, 12,052,992,7000 bytes in total with gzip compression). The precision, recall, and F1 score were 0.888, 0.967, and 0.926, respectively. This result indicates that although rapid Japanese language detection may discard about 3.3% of Japanese web pages, the relatively high precision reduces the processing time for Steps 3 and 4 for non-Japanese web pages. In a benchmark experiment using CPU of Intel Xeon Gold 6130 processor (2.1 GHz), the processing time for Steps 1-4 was about 15 times faster than Steps 3, and 4 (without rapid language detection) because Step 2 removes number of web pages from the target.
This stage obtained 2,686,080,911 pages and 3,684,320,219,688 Japanese characters from 21 snapshots of Common Crawl (from CC-MAIN-20200-00 to CC-MAIN-2023-23),
3.2 Quality filtering
This process (1) removes web pages with many repetitions, (2) selects web pages that contain good-quality Japanese text, and (3) removes web pages that may contain harmful expres- sions. For processing (1), we adopted the rules of Rae al. (2021) to remove documents with many duplicated contents (see Appendix A.2).
As for processing (2), standard has been established for assessing the quality of Japanese text. Therefore, we designed several rules for (2) selecting Japanese text of good quality. This study removes text that satisfies either of the following conditions as irrelevant for LLMs (thresholds in parentheses).
VCC--MIIN CCCCAAINN22222222251110222221210
from 0 9 in CC-MAIN-2023-223
5

---

Published as a conference paper at COLM 2024
1. Number of characters (less than 400 characters)
2. Percentage of hiragana characters (less than 0.2)
3. Percentage of katakana characters (greater than 0.5)
4. Percentage of Japanese characters (hiragana, katakana, kanji, punctuations) (less than 0.5)
5. Average number of characters in sentence (less than 20 more than 90)
6. Number of characters in the longest sentence (more than 200)
7. Percentage of sentences ending with an ellipsis symbol (greater than 0.2)
For example, Rule ensures that text includes a certain amount of function words such as ga (subject case marker), wo (object case marker) zi (position case maker roughly correspond- ing to to in English). Rule 3 rejects a web page containing a lot of product or service names (e.g., found in SEO sites), which are often expressed in katakana characters. Rule 7 removes web page that looks like an RSS feed (a collection of snippets of other web pages). These rules were adjusted manually as the authors examined the text to be deleted and extracted.
Although Web used UT1 blocklis 18 for removing harmful content, its coverage for Japanese pages may be insufficient. In order to remove web pages that may contain harmful expressions (3), we manually created a list of "NG expressions, which may indicate harmful (e.g,, adult, violent, aggressive) contents. If the percentage of characters that match NG expressions is greater than 0.05, exclude the text from the corpus.
This quality filtering reduced the size of the corpus to 646,238,066 pages 4,200,88880444,66 characters). Examining the text before and after the quality filtering, we observe that web pages that may be unuseful for training LLMs (e.g., e-commerce sites) have disappeared.
3.3 Deduplication
Because Common Crawl visits and crawls the same website multiple times, the archive includes web pages that are identical or similar because minor modifications or repro- ductions. Lee et al. (2022) reported that deduplication, removing duplicated text from the corpus, not only reduces memorization of LLMSs but also improves pre-training efficiency.
1997) in similar manner Therefore, we performed deduplication using MinHash (Broder to Lee et al. (2022). When using MinHash for deduplication, is common to create r buckets, where each bucket is concatenation of b MinHash values, compare pairs of buckets of two documents, recognize the two documents as duplicate if any of bucket pairs are identical. In this study, we adopted a setting where 20, r 40 that a pair of documents with Jackard coefficien [9 of 0.9 can be approximately detected as a duplicate with a probability of 92.5%. When pair of documents was recognized as a duplicate, we kept the one crawled recently and removed the older one. the snapshot used in this experiment, the non- duplicate rate of web pages collected between March and June 2023 ranged from 77.8 to 87.9 percent, while the non-duplicate rate of web pages collected before February 2023 dropped to less than 40 percent, and the rate of web pages collected around 2020 was less than 20 percent. This deduplication process reduced the corpus size to 185,766,463 pages (353,191, 882, characters).
3.4 Filtering by hostnames
we found some irrelevant content in the corpus. Therefore, we created a block lis 20 consisting of hostnames that met the following criteria.
Even after the quality filtering in Section 3.2
/tttpps://di.... http:///isi.uuccciollll.eee wwt.ffckkissst..........r.rrasae
19 this study, features were constructed with 5-grams of characters.
We used the word blocklist as "a list for blocking," following the name "UT1 blocklist." We are reluctant to use the word "black list" because some hosts in this list may not be "black." For example, hospital website explaining woman's maternity prone be included in the list.

---

Published as a conference paper at COLM 2024
1. Included in the UT1 blocklist.
2. Percentage of pages containing the name of a dating site (greater than 0.001).
3. Percentage of pages containing NG expressions (greater than 0.005).
4. #wikipedia. org
5. *, 4.5ch.mm net
Although the UT1 blocklist includes some Japanese websites, we introduced Criteria 2 and 3 to improve the coverage. The authors manually adjusted the thresholds for the percentage 2T We applied Criterion because we planned to use Wikipedia text extracted from the dump. This filtering process resulted in corpus of 173,350,375 pages (312,674,522, characters).
3.5 Cleaning extracted text
Sections 3.1 to 3.4 process text at the document level, i.e., filter out irrelevant documents without altering the text within a document. In this study, to avoid an unexpected side effect, we minimally edit the text: punctuation normalization (replacing Western-style punctuations with Japanese-style ones) and footer trimming (removing footers that were left by Trafilatura). Refer to Appendix A.3 for the details. The process does not delete web pages, but the number of characters decreased from 312,674,522,606 to 312,093,428,689,
Experiments
4.1 Models and training data
In order to evaluate the usefulness of the presented corpus, conducted continual pre- training of popular LLMs that excel in English 22 We used Llama 2 7B, 13B, 70B, Mistral 7B v0.1, and Mixtral 8x7B Instruct v0.1 as base models and performed continual pre-training of these base models on the presented corpus to examine whether the corpus improves their Japanese knowledge and skill. The training data for continual pre-training was mixture of the presented corpus, Japanese Wikipedia, Ref,nedWeb, and arXiv component The
2ile Gao et al. 2020). Specifically, we prepared approximately 104.9 billion tokens of training data, assuming a sequence length of 4,096 tokens and a batch size of 1,024 with
25,000 steps for continuous pre-training. The ratio of Japanese to English tokens was set to 9:1, with 5% of the training data being the English text from Reb,nedWeb, 5% being The Pile's arXiv paper text (English), and the remaining 90% being Japanese text. The breakdown of the Japanese text was about 1.6 billion tokens of Japanese Wikipedia, and the presented Japanese corpus occupied the rest. We also used AlgebraicStack (Azerbayev et al., 2024) for Mistral's continual pre-training. For Mixtral, we used both AlgebraicStack and The We used different ratio (72:28) of Japanese to English tokens for continual pre- pre-ningning on Mixtral 8x7B Instruc 25
Vault (Manh al., 2023 24
We did not change the architecture of base LLMs; the embedding sizes of tokens, hidden layers, feed-forwart layers, the number of attention heads, and the number of layers were
We placed emphasis on recall at the expense of precision to reduce harmful behaviors of trained models. This design results including a lot of non-harmful websites in the block list.
22 may be straightforward to evaluate the corpus by training L.MSs from scratch, but we chose continual pre-training because the broader goal of this effort was to build high-performance L.LMs that excel in Japanese under the limited amount of computing budget. We can assess the quality the corpus by checking performance gains from the base models and also comparing the performance with other models.
We expected that the RefinedWeb and arXiv would help maintain the base LLMs' English knowl- edge and skill during continual pre-training.
2e completed the continual pre-training experiments on Llama well before we began with Mistral 7B. By then, our objective had expanded to enhance logical inference capabilities in Mistral and Mixtral, leading us to incorporate AlgebraicStack and The Vault.
We changed this ratio as an attempt to improve the performance in English.
7

---

Published as a conference paper at COLM 2024
unchanged from the base models in continual pre-training. Before continual pre-training,
we added Japanese vocabulary to Llama 2 and Mistral 7B tokenizers. The total number of vocabulary was 43,176 for LLMs based on Llama 2 and 42,800 for those based on Mistral 7B. We employed AdamW (Loshchilov & Hutter] 2019), with hyperparameters set to $ = 0.9,
82 = 0.95, = 1.0 x 10 10-a. We used re-warming and re-decaying the learning rate (lbrahim et al. 2024) with 1,000 warm-up steps. For Llama 2, the maximum learning rate was set at 1.0 10 with decay rate of 1 1/30, while for Mistral and Mixtral, it was 2.0 10 with decay rate of 1 /10. The batch size was 1,024. We used 0.1 for weight decay and 1.0 for gradient clipping. In addition, we used Flash Attention (Dao al. 2022) to improve computational and memory efficiency.
In order to examine the impact of a corpus on the performance of LLMs, we performed con- tinual pre-training on ClueWeb22 and llm-ip-corpus v1.0.1 as well as the presented corpus. We also compared our LLMs to others with similar numbers of parameters: CyberAgentL.M2 7B (CALM2 7B, a Japanese LLM trained from scratch); Aapanese-StableeM-BB Baaeee------a------a--- (JSLMB 7B, continual pre-training LLM Llama 7B); Youri 7B (a continual pre-training LLM on Llama 2 7B); Qwen 7B; Nekomata 7B (a continual pre-training LLM on Qwen 7B); Japanese Stable LM Base Gamma 7B (JSLMG 7B, continual pre-training LLM on Mistral 7B); Qwen 14B; KARAKURI LM 70B (Karakuri 70B, a continual pre-training LLM on Llama 70B); Staisee-Staaee---as----ee
(JSLBB 70B, continual pre-training LLM on LLama 2 70B); and Qwen 72B. Refer to Appendix A.4 for the complete list of URLs of the LLMs used in the experiments.
4.2 Evaluation Dataset
We used Ilm-jp-eva and Im-evaluation-harnns. as Japanese evaluation benchmarks. Ilm-
28
jp-eval is benchmark consisting of tasks for multiple choice (MC) question answering, open question answering (QA), reading comprehension (RC), and natural language inference
(NLI) tasks. JCommonsense(A (Kurihara et al. 2022) is employed for MC, JEMHopQA (Ishii
et al. 2023 and NIILC (Sekine) 2003) for QA, JSQuAD (Kurihara et al., 2022) for RC We decided to exclude the NLI dataset from the benchmark because the distribution of ground-truth NLT labels is highly imbalanced, which made the evaluation unstable 29 For eeovallation-hareess we used Japanese subsets of XL-Sum (Hasan et al., 2021) for the automatic summarization task and MGSM (Shi et al. 2023) for the arithmetic reasoning task, respectively. In addition, WMT 2020 (Barrault et al. 2020) was used for the lapanese-Engllee and English-Japanes. machine translation.
4.3 Results
Table rrpports the performance of LLMs on the Japanese benchmark datasets. The row "+ this study" shows the performance of the LLM after we applied continual pre-training to the base model on the presented corpus. The rows "+ this study" demonstrate that the presented corpus consistently improves the performance of the base models via continual pre-training by 6.6-8.1 points on average. The models built by this study outperform other models built from scratch or continual pre-training, establishing the state-of-the-ar performance in each model size (7B, 13B, 8x7B, and 70B). This fact shows the usefulness of the presented corpus in building LLMs that excel in Japanese.
We can compare different Japanese corpora in continual pre-training of Llama 2 13B. All corpora (ClueWeb22, lmmjjp-corpus v1.0.1, and presented corpus) improved the base model's performance by 5.0-7.0 points, which verifies the effectiveness of the continual pre-training, The presented corpus yielded the best improvement (7.0 points) of all corpora; in particular, it drastically enhances the performance of question answering (JCom, JEMHop, NIILC),
26 wtttp/////ttthaa..c///////////oo
27
28 httttsss/giituuu.mmm//aalllli//p http:///wwww./.ooooooor/r/r//r// commit #9b42d41
An LLM may obtain a high score only if the model happens to predict the majority label without understanding and solving the task.
8

---

Published as a conference paper at COLM 2024
Corpus
QA
QA
QA
RC
Sum
Ja-En
En-Ja
Math
Avg
JCom
JEMHop
NILLC
JSQuAD
XL-Sum
WMT20
WMT20
MGSM
CALM2 7B
21.98
50.47
50.66
77.99
2.33
14.99
23.45
6.00
30,98
JSLMB 7B
36.10
44.78
44.32
83.18
21.95
12.26
19.46
7.20
33.66
Youri
46.20
47.76
49.99
85.06
19.57
19.71
26.71
6.40
37.67
Llama 2
38.52
42.40
34.10
79.17
19.05
17.37
17.83
7.60
32.01
this study
48.08
50.78
59.68
85.73
18.30
15.11
25.10
12.40
39.40
Qwen 7B
77.12
42.34
23.76
85.94
13.71
18.01
16.89
21.60
37.42
Nekomata 7B
74.17
49.28
50.22
87.07
16.76
18.15
26.73
12.40
41.85
JSLMG 7B
73.64
46.43
55.68
89.10
22.93
15.61
23.90
16.80
43.01
Mistral 7B
73.01
42.45
27.22
85.63
20.06
17.33
14.05
17.60
37.17
+ this study
85.70
49.15
55.19
88.02
19.88
16.67
24.94
22.40
45.24
Llama 2 13B
68.19
44.55
41.74
85.51
21.33
19.81
21.36
13.20
39.46
+ ClueWeb22
76.76
49.29
56.02
89.55
20.15
23.32
29.08
11.60
44.47
+ 11m-jp
74.71
50.85
60.34
90.28
21.91
17.89
25.89
18.00
44.98
+ this study
78.37
50.63
64.06
90.07
21.68
17.71
27.37
21.60
46.44
Qwen 14B
88.29
42.43
32.20
89.80
18.51
22.24
22.23
38.80
44.31
Mixtral 8x7B
84.00
50.33
31.07
88.08
20.02
20.63
19.56
45.20
44.86
this study
92.58
58.43
56.87
91.48
25.89
20.74
27.05
43.60
52.08
Karakuri 70B
85.79
51.25
57.13
91.00
14.64
21.13
25.40
27.20
46.69
JSLMB 70B
91.15
49.25
60.42
91.92
25.73
23.35
27.65
41.60
51.38
Llama 2 70B
86.86
46.56
52.56
90.80
23.61
23.98
26.43
35.60
48.30
+ this study
93.48
62.90
69.60
91.76
22.66
22.98
30.43
48.40
55.28
Qwen 72B
92.94
55.66
45.18
91.59
21.79
23.56
25.61
63.20
52.44
Table 2: Benchmark evaluation on Japanese tasks. A horizontal line groups LLMs with the same number of parameters. A horizontal dash line groups LLMs from the same base model. A bold number indicates the maximum value in the LLMs with the same number of parameters.
reading comprehension (JSQuAD), and arithmetic reasoning (MGSM), Although did not incorporate a special effort to improve these tasks, the model acquired knowledge about Japan and the Japanese language from the presented corpus. In contrast, the presented corpus did not improve the summarization (XL-Sum) task. This trend is observed across other base models, except for Mixtral 8x7B Instruct, potentially indicating that adding Japanese vocabulary may have detrimental effect on this task. In addition, we found that the continual pre-training on the presented corpus improved the performance in English-Japanese translation but degraded that in the reversed direction (Japanese-Englle translation). We will explore mitigation of this phenomenon in the future, e.g., by changing the ratio of Japanese to English tokens in the training data and promoting language transfer using an Agglish-Japansse parallel corpus during continual pre-training.
Conclusion
In this paper, we built large Japanese web corpus by extracting and refining text from the Common Crawl archive (21 snapshots of approximately 63.4 billion pages crawled between 2020 and 2023). This corpus consists of approximately 312.1 billion characters (approximately 173 million pages), which is the largest of all available training corpora for Japanese LLMs. We confirmed the usefulness of the corpus by performing continual pre-training Llama 2 7B, 13B, 70B, Mistral 7B v0.1, and Mixtral 8x7B Instruct base LLMs. The experiments demonstrated consistent (6.6-8.1 points) improvements in Japanese benchmark datasets, and established the ofate-of-the-art performance in each model size (7B, 13B, 8x7B, and 70B). We also observed that the improvement on Llama 2 13B brought from the presented corpus the largest among those from other existing corpora.

---

Published as a conference paper at COLM 2024
Future directions include efforts towards the safety of LLMs, such as reducing harmful generations (e.g, discrimination, exclusion, toxicity, hallucination)). Currently, we only use the lists of NG expressions and hostnames, but it is desirable to establish more robust filtering methods to remove harmful text for pre-training Japanese LLMs. In addition, although our study focused the continual pre-training setting, we want to evaluate the presented corpus by training Japanese LLMs from scratch. Although we evaluated the LLMs in downstream tasks such as qutestio-ansseringg and summarization, is questionable whether this can measure the "general intelligence" of an LLM. At the same time, training an LLM on a pre-training corpus requires huge computations. Therefore, we want explore lightweight method for assessing the effectiveness of pre-training corpora without building LLMs.
Limitations
We could not present an ablation study of each step in Figure F
because of a large amount of computational resources required to train LLMs. Instead, we reported the performance of each step of the corpus construction procedure in Section 3
This study focuses on Japanese, and the construction of corpora for other languages is outside the scope. For English, it may be good strategy to make use of good quality corpora that have already been developed. Although some ideas in this paper could be helpful in building corpora for other languages, for example, reducing processing time with rapid language detection, different countries to share the task of building corpora for their own, as assessing text quality requires the knowledge of the language.
believe that it would be better for researchers from
Ethics statement
Article 30-4 of the Copyright Act in Japan permits use of copyrighted work without the permission of the copyright holder as long as the use is not person's purpose to personally enjoy or cause another person to enjoy the work. This is quote from the Japanese translation of Article 30-4 of Copyright Ac
It is permissible to exploit work, in any way and to the extent considered necessary, in any of the following cases, or in any other case in which it is not a person's purpose to personally enjoy or cause another person to enjoy the thoughts or sentiments expressed in that work; provided, however, that this does not apply the action would unreasonably prejudice the interests of the copyright owner in light of the nature or purpose of the work or the circumstances of its exploitation: (snip)
The nuance of enjoy probably needs clarification. Enjoying work roughly corresponds to consuming the idea of a work (for any purpose, including entertainment, learning, and communication, regardless of the monetary gains from the work). The intention of Article 30-4 is to define an exceptional use case where computer program consumes works directly for machine learning and information analysis, whereas human cannot do that.
This provides the justification for building the presented corpus and LLMs. To the best of our knowledge, we have taken every measure to keep the corpus as non-toxic and unbiased as possible, but we are unaware of any direct ethical consequences caused by the LLMs trained on the presented corpus.
8 Reproducibility statement
All models developed this study (continual pre-training on Llama 2 7B, 13B, 70B, Mistral 7B v0.1, and Mixtral 8x7B Instruct) have already been released on Hugging Face. The bench-
ch2sc3sb5at4
10

---

Published as a conference paper at COLM 2024
mark datasets used in this study are also publicly available. Therefore, it is straightforward to reproduce our experimental results reported in T
Acknowledgements
This paper is based on results obtained from project, JPNP18002, commissioned by the New Energy and Industrial Technology Development Organization (NEDO). In addition, the experiments of continual pre-training of LLMs was supported by the "Support Program for Building Large Language Models" of the Al Bridging Cloud Infrastructure (ABCI) developed and operated by the National Institute of Advanced Industrial Science and Technology (AIST). We used the datasets and findings released by the Japanese LLM Study Group (LLM-jp) in the evaluation experiments. We also received suggestions on tokenization from Tatsuya Hiraoka of MBZUAI.
References
Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. Towards cleaner document-oriented multilingual crawled corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 4344 4435,, June 2022. URL httttttts///ccnttttt org/2022.1rec-1 .463
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. In the Twelfth International Conference on Learning Representations, 2024. h/ppeeeeee/eeeeeeeeeeee..ee. wttpe/rrreieeeeeeerrmuummm nett/forum?dd=4WooRR90
Adrien Barbaresi. Trafilatura: web scraping library and command tool for text discovery and extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pp. 122-131, August 2021. doi: (0.18633/v1/0021 acl-demo.15. URLhhttss:/bcclaattolo.e org/ acl .
Loic Barrault, Magdalena Biesialska, Ondiej Bojar, Marta R. Costa-jusså, Christian Fed- ermann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu Lo, Nikola Ljubešić, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Santanu Pal, Matt Post, and Marcos Zampieri. Findings of the 2020 conference on machine translation (WMT20). In Pro- ceedings of the Fifth Conference on Machine Translation, pp. 1-55, November 2020. URL https:://allanthooogy.oo org/ wmt-
Andrei Z Broder. On the resemblance and containment of documents. In Proceedings of the Compression and Complexity of Sequences 1997, pp. 21, 1997. ISBN 0818681322. doi:
10.1109 /TT//EE.......66609..
Christian Buck, Kenneth Heafield, and Bas van Ooyen. N-gram counts and language models from the Common Crawl. In Proceedings of the Ninth International Conference on Language Resources and Evaluation, pp. 3579-3584, May 2014. URL http::/www..t. www.wwwwwww.c.oowoo./mm org/
proceedings/Irec222 htteecc22144dff//// /199 Paper.pdff
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Compulational Linguistics, pp. 8440-8451, July 2020. doi: 10.18653/v1 2005333/1122222acmmmia.. acl-main 747
URL https::/aclaathholgge org/2020
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAtten- tion: Fast and memory-efficient exact attention with 10-awareness. In Ad- vances in Neural Information Processing Systems, volume 35, pp. 16344-16359, 2022. URL https://ppoceeedings. www.eip.cc/ppppp.eeeeeee caappaprfflllee files/paper htts:/papprr/222/f1lee /2022/f1l/ 67755555222200000022000001166133
11

---

Published as a conference paper at COLM 2024
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. In Adoances in Neural Information Processing Systems, volume 36, pp. 10088-10115, 2023. URL https::/pproceeeddngg www.ips.cc/pppaaaepeeeeee files/ 2bbaaa22222a222220400 wtw.eeaaee..0000-aaaa--orrrrrrre
Rintaro Enomoto, Arseny Tolmachev, Takuro Nitssuma, Shuhei Kurita, and Daisuke Kawa- hara. Investigating web corpus filtering methods for language model development in Japanese. In Proceedings of the 2024 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies (Volume 4: Student Research Workshop), pp. 154-160, June 2024. doi: 11111665//1122224maaaaaa https://aclanthol..og org. 2024. naacl-srve
URL
Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading unfair NLP models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11737-11762, July 2023. doi: 11118653/v1/2023.ad
656. URL https://aclanthologgy www.ooggg///..... acl long. 656
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy, The Pile: An 800GB dataset of diverse text for language modeling. arXiv:2101.0000 2020.
Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong- Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. XL-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 4693-4703, August 2021. doi: 10.18653 1118663331/22211 findings-acl.413, URL
www.o/gg//2222/2 findings-acl. 413
Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats L. Richter, Quentin Anthony, Tim- othée Lesort, Eugene Belilovsky, and Irina Rish. Simple and scalable strategies to con- tinually pre-train large language models. Transactions on Machine Learning Research, 2024. URL /RLlitttss:/oopnreeieee wttp:/wwwwwewewww/eeeeeeeemee http:forrrumiiiiimmmeeecCC0O
Ai Ishii, Naoya Inoue, and Satoshi Sekine. Construction of a Japanese multi-hop QA dataset for QA systems capable of explaining the rationale. In the 29th Annual Meeting of Japanese Association for Natural Language Processing (NLP2023), pp. 2088-2099 March 2023. URL meet inieeetngn//2//pdll http:/200///ppf.dr//88--/1- dir/Q8-14.paa
https://www anlp http:pppppppppcccc
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B. arXiv:2310.06825, 2023.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts. arXiv:2401.0441 2024.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Gold- stein. A watermark for large language models. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp. 17061-17084, 23-29 Jul 2023. URL aagsssmmlr..
html
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Ad- vances in Neural Information Processing Systems, volume 35, pp. 22199-22213, URL https::/pproceedigssneeeee www.sps.eunispcceoeeeee cc/pppp http:/pppprffflls//eeaaaeeeeeese http:/ppppe//22/ffllee.. 1abb22221eccaaafffe1111999911111
2022.
12

---

Published as a conference paper at COLM 2024
Kentaro Kurihara, Daisuke Kawahara, and Tomohide Shibata. JGLUE: Japanese general language understanding evaluation. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 2957-2966, June 2022. URL https:://cclanthol.gg/ org/ 2022
1rec-1.317
Faisal Ladhak, Esin Durmus, Mirac Suzgun, Tianyi Zhang, Dan Jurafsky, Kathleen McKe- own, and Tatsunori Hashimoto. When do pre- traininiiig biases propagate to downstream tasks? a case study in text summarization. In Proceedings of the 17th Conference of the Euro- pean Chapter of the Association for Computatioonal Linguistics, pp. 3206 3219, May 2023. doi: 10.18653/v1 http//1122222c///// eacl-main.2333 URL httpttttt//cllaeaeee http://loy.a.tooooooo.ooooooooo. org org. 2acl:22223.ccllllc maci-main...a... 234
Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, et al. The BigScience ROOTS corpus: A 1.6TB composite multilingual dataset. arXiv:2303.09115, 2023.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8424 8445, May 2022. doi: (0.18633/v1/200 acl-long.577. URL https: pttts://aacaathol..og
wwwpo//g......./. 201-222222..clnnn 577
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https:///pennreiii..teefffooooe
Bkg6RiCqY7
Dung Nguyen Manh, Nam Le Hai, Anh T. V. Dau, Anh Minh Nguyen, Khanh Nghiem, Jin Guo, and Nghi D. Q. Bui. The vault: A comprehensive multilingual dataset for advancing code understanding and generation. In Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023), pp. 244,244, December 2023. doi: 10. 18653/ /tt//v1/2200 w1/2023.nlposs- 1.25. /Rtphttss://accanthoo
www.oogg..........o www.os-sl.111. 1.25
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. DetectGPT: Zero-shot machine-generated text detection using probability curvature. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of 23-29 Jul 2023. URL https
Proceedings of Machine Learning Research, pp.
//proceedings. mlr. pttppprss//////mmceeeeeeeee html
Arnold Overwijk, Chenyan Xiong, Xiao Liu, Cameron VandenBerg, and Jamie Callan. ClueWeb22: 10 billion web documents with visual and semantic information.
arXiv:2211.15845 2022.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The RefinedWeb dataset for Falcon LLM: Outperforming curated corpora with web data, and web data only. arXiv:2306.01116 2023.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' d'Atume,, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, lason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray
13

---

Published as a conference paper at COLM 2024
Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training Gopher. arXiv:2112.11446, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. 074- 074.htm html
http:tttp://mmml http:////mmm...rrppppprrr///////
Vipula Rawte, Swagata Chakraborty, Agnibh Pathak, Anubhav Sarkar, S.M Towhidul Islam Tonmoy, Aman Chadha, Amit Sheth, and Amitava Das. The troubling emergence of hallucination in large language models an extensive definition, quantification, and prescriptive remediations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2573, December 2023. doi: emnlp-main.155 URL/Intoss///claathholoo horgeoollooooooooo/ooooooo wtwp/222.emmlppppmrp. emnlp-mainn main. 155
Satoshi Sekine. Development of question answering system for encyclopedias, the 9th Anmual Meeting of Japanese Association for Natural Language Processing (NI. (Ni.P2003), pp. 637-640 March 2003. URL https//www..amppppppprreedddee/s http:///annnmmmmmttnn/e/////// idf adf_ddrr/C766.pp
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language models are multilingual baain-of-thought reasoners. In the Eleventh International Conference on Learning Representations, 2023. URL https://opente: //tp:////e///eww...///// /ttppe/rreeiee..eefformmnnddd FR3wGCk-IXp
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In the Twelfth International Conference on Learning Representations, 2024. URL
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Khyathi Chandu, Jennifer Dumas, Li Lucy, Xinxi Lyu, lan Magnusson, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Pete Walsh, Hannaneh Hajishirzi, Noah A. Smith, Luke Zettlemoyer, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: An open corpus of trillion tokens for language model pretraining research. Technical report, Allen Institute for AI, 2023. URL https://allennai gi thub. 1o/dolma/ docs/assets/dolmm http://oo...maaaaaateetvooo.... 1.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challeng- ing BIG-bench tasks and whether thoough can solve them. In Findings of the Asso- ciation for Computational Linguistics: ACL 2023, pp. 13003-13051 July 2023. doi: 10. 10.18653/ URL htttttsss//cclanttooooy// 002./ findings-aan ac1.824
ho0d2..innings-ace
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09220 2023.
14

---

Published as a conference paper at COLM 2024
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with mhain-of-thoughee reasoning for knowledge-inteessve multi-step questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10014-10037 July 2023. doi: (1/8663//112223.a/// URL httpss://aclaanholoooooo///////p
acl wonganl---oooooooooeoeooooo 557.
Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. Poisoning language models during instruction tuning. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 35413-35425, 23-29 html
mlr. Jul 2023. URL https::/prrceeddingggg http://eeennn........rrss//////a prt.sprr///22222a2233
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In the Eleventh International Conference on Learning Represen- fations, 2023a.
httpss///ppnnreview wtwp:erreeiie.nee///rrummmm/mm idd 1NIMMrW
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Tz Beltagy, and Hannaneh Hajishirzi. How far can camels go? Exploring the state of instruction tuning on open resources. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023b. URL https::/opeerreiie..eeeee wttpe//wwwwweeeeeooorrro/e/////e
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 13484-13508, July 2023c. doi: (0.18653/v1/200 acl-long.754, URL https: httpps///cclanthol..g
org/2023. acl- long. 754
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-though prompting elicits reasoning in large lan- guage models. In Advances in Neural Information Processing Systems, volume 35, pp. 24824-24837, 2022. URL / httpp:://proceedinss.eerre neur www.enwwwws...pppprrrrre.e 0ile/995609613524cfffff1... http/6660122222411../3///////0e0
http://pper..lllespppre///////// pdf
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Fran- cisco Guzmán, Armand Joulin, and Edouard Grave. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the Twelfth Language Re- sources and Evaluation Conference, pp. 4003-4012, May 2020. ISBN 979-10-95546-344. URL 1.49-
https://aalanthhooog..oo////////
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to- text transformer. In Proceedings of the 2021 Conference of the North American Chapter of
the Association for Computational Linguistics: Human Language Technologies, pp. 483-498, June 2021. doi:
naac maacl-main.44
Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. In Socially Responsible Language Modelling Research, 2023. URL httph:::p:tpppt htettpp://pperreviiw... http://perreviiieeeeffforrrrraii www..n/mffourmiid--mmm
wKe6jE065x
Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A. Smith. How language model hallucinations can snowball. In Proceedings of the 41st International Conference on Machine Learning, pp. 59670-59684, 2024. URL httpps://proceedingg mlr. pre.s/rres//35//
zhang24ay. html
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirly-seventh Conference on Neural Information Processing Systems Dalasets and Benchmarks Track, 2023.
URL wtwpaannwwwww.eeefffoooooottoooo
15

---

Published as a conference paper at COLM 2024
A Appendix
A.1 Precise language detection
Following the pipeline of Wef (Penedo et al. 2023), we built a language detector with linear discriminator using character n-grams teatures. We used multilingual Wikipedia texts as the training data for the linear discriminator, and the feature space was constructed from the training data with character unigrams, bi-grams, and trigrams that satisfy one of the following criteria.
1. Within the top 400,000 occurrences the training data for all languages.
2. Within the top 400,000 occurrences in the training data of Japanese.
3. Within the top 100,000 occurrences in Chinese training data.
4. Within the top 10,000 occurrences in the training data for each language.
If only Criterion (1) was used, there would be possibility that features related to Japanese characters would not obtained sufficiently. Therefore, Criterion (2) aimed to obtain features specific to Japanese. Criterion (3) made it easier to distinguish between Japanese and Chinese, which share Chinese characters. We aimed to stabilize the detection results for texts in languages with insufficient training data using Criterion (4). The total number of distinct features was 821,484.
For training the linear discriminator, we used the dump of Wikipedia CirrusSearch (as of June 12, 2023). In training the language detector, we reduced the amount of training data to 1/20th of the dump because the total amount was too large for training the discriminator. We sampled 100,000 instances in the remaining data for development and evaluation sets. We trained an L2-regularized L2-loss Support Vector Machine on the training data using LIBLINEAR 2.46. The regularization coefficient was set to 10 empirically by the search on the development set. We confirmed that the performance of this classifier was quite good: 0.996 F1 score on the test set and 0.989 F1 score whatlang-corper
A.2 Rules for removing repetitions
The removal of documents that contain many repetitions aims to filter out documents that do not seem to have useful information and to prevent the behavior LLMs from repeatedly generating the same words. Specifically, we follow the rules of Rae et al. 2022 and remove a document when the percentage of any of the following exceeds the threshold (shown in parentheses).
1. Number of lines duplicated in other lines total number of all lines (0.30)
2. Number of paragraphs that are duplicates of other paragraphs / total number of paragraphs (0.30)
3. Number of characters that appear in other lines total number of all characters (0.20)
4. Number of characters in paragraphs that appear in other paragraphs / total number of characters (0.20)
5. Number of occurrences of the most frequent 2-gram / total number of occurrences all 2-grams (0.20)
6. Number of occurrences of the most frequent 3-gram / total number of occurrences of all 3-grams (0.18)
7. Number of occurrences of the most frequent 4-gram / total number of occurrences all 4-grams (0.16)
32
ntu, http://wwww..iiuuu.u..o///o////a
https://githbb http://ggghhuucccc/aaaaaaaaaaaaa
16

---

Published as a conference paper at COLM 2024
8. Total number of occurrences of 5-grams appearing more than once Total number occurrences of all 5-grams (0.15)
9. Total number of occurrences of 6-grams appearing more than once Total number of occurrences of all 6-grams (0.14)
10. Total number of occurrences of 7-grams appearing more than once / Total number occurrences of all 7-grams (0.13)
11. Total number of occurrences of 8-grams appearing more than once Total number of occurrences of all 8-grams (0.12)
12. Total number of occurrences of 9-grams appearing more than once Total number of occurrences of all 9-grams (0.11)
13. Total number of occurrences of 10-grams appearing more than once / Total number of occurrences of all 10-grams (0.10)
A.3 Clearning the text
In order to normalize the punctuation in the corpus to ", and ". ", we replaced ", wwih and ". "wihh" ". respectively, based the following rule.
1. Check the number occurrences the symbols". ", "and" ", "appear in the document. However, preceding an alphanumeric character is excluded from the replacement with
If ", "appears more often than ", replace ", with
2. Check the number of occurrences the symbols". "and ". "appear in the document. If " appears more often than ". replace ". with preceding alphanumeric character is excluded from the replacement with ".
However, ".
This punctuation normalization process replaced "with ". in 290,318 documents (0.17%) and ", "wit ". " in 1,107,319 documents (0.64%).
Although Trafilatur used for text extraction removes the navigation and footer text of web pages, we sometimes see footer text that have not be removed by Trafilatura, Therefore, expressions such as "Trackback list for this article", "All rights reserved", and "Click" in the last three lines of text were removed from the text they occupy more than 30% of the text in character. This footer removal process removed footers at the end of 12,617,787 documents (7.3)) of the pages.
A.4
List of LLMs used in the experiments
Base models
Llama 7B: http:s//wngiiiggaccccccmmmmm///a
Llama 13B: http://wnggiiffaeeccoocrce/elllm
Llama 70B: http:://wheeiiifffccccocccimm//a
Mistral 7B v0.1: http:://wweiiiiffe..coossssaaass
Mixtral 8x7B Instruct v0.1:
Other models for comparison
Rinna Youri 7
Qwen 7B: TBAktpppr//uggiiigseooocc///////
17

---

Published as a conference paper at COLM 2024
Rinna Nekomata 7B: http:://wwggginaaae..er/m/mmm//n
Japanese Stable LM Base Gamma 7B:
Qwen 14B:
LM
LM 70B:
Qwen 72B: httpss/hugggingffeeee.//e///////
18


Reviews:
[{"review_id": "veYWuhLz5t", "reviewer": "Reviewer_QC9E", "summary": "This paper presents an effort to create a very large corpus of trustworthy Japanese text that could be used to train LLMs. They describe the method used for obtaining candidate data, cleaning, and deduplicating it. In order to show the quality of the resource, the authors train different LLMs on this data and show that it actually improved the metrics on a benchmark of Japanese NLP tests.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your positive review and useful suggestions for our paper.\n\n> R1: the quality of hostname filtering\n\nThis is a valid point. Unfortunately, there is no public data available for comprehensively detecting harmful Japanese web pages; we found that the coverage of the UT1 blocklist was insufficient for filtering out harmful websites in Japan. We could not use a commercial filtering service to recognize harmful URLs because we were unsure of whether the TOS would allow us to build an LLM that could be used for commercial purposes. This was the reason why we created our own list for hostname filtering with tight thresholds (0.001 and 0.005) and left the establishment of a more robust method as a future work (mentioned in Section 5).\n\n> Q1: continual pertaining\n\nYes, you are right. Given a base pre-trained model, continual pretraining performs another round of pre-training on new data (Japanese corpus in this study). We will make this clearer in the camera-ready version. Both “continual pretraining” and “continued pretraining” seem acceptable based on our literature survey, but we will also explain this in the camera-ready version.\n\n> Q2: NG expressions\n\nSorry, they have nothing to do with programming languages. NG expressions stand for harmful expressions (adult, violent, aggressive). The reason why we avoided the direct term “harmful” was because this filtering process may remove a large number of non-harmful web pages with tight thresholds.\n\n> Q3: blocklists\n\nWe used the word blocklist as “a list for blocking”, following the name “UT1 blocklist”. We were also reluctant to use the word “black” because some hosts in this list may not be “black”. For example, a hospital website explaining a woman’s maternity is prone to be included in the list.\n\n> Q4: “enjoy” in the Japanese Copyright Act\n\nThank you for pointing out the translation problem. The original term 享受 (kyo-ju) is also translated into \"enjoy\" by the [Japanese government](https://www.japaneselawtranslation.go.jp/ja/laws/view/4207#je_ch2sc3sb5at4). The nuance is that a human consumes the idea of a work (for any purpose, including entertainment, learning, and communication, regardless of the monetary gains of the data). The intention of Article 30-4 is to define an exception for the use case where a computer program consumes works directly for machine learning and information analysis, but a human cannot do that. We will add an explanation in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Although Japanese is not a low resource language itself, it is not one of the highest resources languages in NLP either, so the effort to create a very large corpus is very interesting.\nI particularly liked the thorough explanation of the data collection pipeline (including downloading, filtering, deduplicating, cleansing, etc.), which could be used as an inspiration to create corpora in other languages as well.\nThe resource built seems very relevant, and the results shown over standard benchmarks seem to show it has high quality as well.", "reasons_to_reject": "Some parts of the document are in need of more clarifications, for example the quality and hostname filtering, and the explanation of section 4.1 and associated Table 2.", "questions_to_authors": "- By continual pretraining (or is it continued pretraining?) I understand you take the pretrained model weights and keep on training on your new data. Is this the case? Please make this clear, perhaps indicate some references where the technique is introduced or used.\n- Please explain what NG expressions are. Does is refer to js or other programming languages content?\n- When you refer to \"blocklists\" I think the correct term is \"blacklists\", or is it another concept?\n- I am intrigued by the Japanese Copyright Act mentioned in the ethics statement that says the purpose of the person is not to \"enjoy\" or cause another person to \"enjoy\" the work (quotes are from the text). Is this some translation issue, or what is being referred to as enjoy in this case? Does this mean the use for monetary gains of the data?", "extra_scores": {}}, {"review_id": "7w4MyxypcQ", "reviewer": "Reviewer_uEGP", "summary": "The authors created a novel Japanese corpus for training large language models from Common Crawl archives.\nIt consists of 312B characters, which is substantially larger than previous training corpora for Japanese LLMs, such as CC-100 (25.8B characters), mC4 (239.7 B characters), and OSCAR (74B characters).\nUsing their Japanese web corpus, the authors performed continual pre-training on various LLMs, such as Llama 2 and Mistral, and gained consistent improvements on Japanese benchmark datasets.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your positive review and useful suggestions for our paper. We appreciate the time and effort you have taken to provide feedback. Below, we answer your questions raised.\n\n> R1: only applicable to Japanese\n\nYes, we expected that our paper could receive such a comment. However, our motivation was to build a large corpus specialized for Japanese because the quality of multilingual corpora does not meet our expectation of spending a lot of money for continual pre-training of Japanese LLMs. We also explained our stance in Section 6 (Limitations) that some ideas in this paper could be helpful for other languages and that researchers in each country should put an effort into building a corpus for the country.\n\n> R2: whether their corpus would be effective if they train an LLM from scratch\n\n> Q1: the quantity and quality necessary to train an LLM from scratch\n\n(We merge responses for the rejection reason and question)\n\nWe did not try to build an LLM from scratch for various reasons:\n\n+ The ultimate goal of this project was to explore methods for building open LLMs that archive high performance in Japanese.\n+ Based on the Chinchilla law, we need 205.1 billion tokens (for 10B models) and 1.5 trillion tokens (for 67B models) as the training data. The constructed corpus was sufficient for training 7B models from scratch but probably insufficient for 70B models.\n+ Having said that, we need to mix an English corpus for building Japanese LLMs anyway because English corpora are about 10 times larger than Japanese ones and because we also need multilingual applications (e.g., English-Japanese translation and cross-lingual summarization).\n+ If we mix the training data of Japanese and English, continual pre-training is a reasonable choice because we can omit the large computation for English data.\n+ A lot of Japanese companies released a variety of Japanese LLMs trained from scratch when we started this project (last fall). We wanted to try an alternative approach for building high-performance Japanese LLMs.\n\nAs an academic paper, we agree with your opinion that we want to see the performance of the LLM trained on the corpus from scratch. However, we do not have the computation resources for this at this moment, although our corpus is also ready for use for training Japanese LLMs from scratch.\n\n> Q2: Abbreviations of BL and BW\n\nThank you for the valuable comment. We will avoid these abbreviations in the body of text in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors have created a novel open Japanese web corpus comparable in size and quality to publicly available English web corpora.\nIt will significantly contribute to the advancement of Japanese LLMs compared to training LLMs using the Japanese portion of a multilingual corpus.\n\nThe authors proposed a lightweight language detection method to balance the speed and quality of Japanese text extraction.\n\nUsing the Japanese web corpus, the authors performed continual pre-training on more than ten publicly available LLMs, and they extensively compared their performance using representative Japanese evaluation benchmarks, including llm-jp-eval and lm-evaluation-harness.", "reasons_to_reject": "Many aspects of the proposed method, such as filtering based on character type and length, are only applicable to Japanese.\n\nThe evaluation was only when the authors performed continual pre-training on existing LLMs using the Japanese web corpus.\nThey do not demonstrate whether their corpus would be effective if they train an LLM from scratch.", "questions_to_authors": "I want your current estimates of whether the Japanese web corpus has the quantity and quality necessary to train an LLM from scratch.\n\nThis is not a question, but a comment.\nUsing shorthands such as BL (billion letters) and BW (billion words) in Table 1 is okay.\nHowever, the authors should avoid their use in the body of the paper.", "extra_scores": {}}, {"review_id": "UiJJ12XLc8", "reviewer": "Reviewer_KvfY", "summary": "The paper presents a corpus building project to build a large Japanese web corpus for LLMs. Using Common Crawl WARC as a base is a good idea but a lot of other Japanese related corpora are driven by Japanese academics in their silo networks that has yet been tapped in modern LLM training, e.g. NICT LLM https://www.nii.ac.jp/en/news/release/2023/1020.html and https://www.aozora.gr.jp/#main and more on https://github.com/llm-jp/awesome-japanese-llm?tab=readme-ov-file. Also, references to those work is lacking in this paper (but it's understandable since a lot of the work done in those research labs are disseminated in Japanese only). \n\nThe steps taken in the data processing are clearly described, esp. the language ID and ngram filter process. As for the \"quality filtering\" process of using character sets (Section 3.2), it might be that the filtering by character sets have implicitly also restrict the domain that the model will be learning from, e.g. there's a lot of Japanese BBS (bulletin board sites) / forums where substrings like \"kaomoji\" might be used e.g. ٩(ˊᗜˋ*)و  https://detail.chiebukuro.yahoo.co.jp/qa/question_detail/q11297327146 and https://komachi.yomiuri.co.jp/ Lastly, there are also many nuances in cleaning Japanese texts e.g. half-width / full-width characters and furigana mixed into the texts. \n\nIt would be nice to have some breakdown of how much each of the filtering process taken out, esp. how much of the final text is made up of the Wikipedia dump.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your positive review and useful suggestions for our paper.\n\n> R1: No reference to other Japanese compilation work\n\nThank you for the suggestion. We will comment on the references you raised one by one.\n\n+ [NICT LLM](https://www.nii.ac.jp/en/news/release/2023/1020.html) (actually, this was released by NII but not by NICT). Because this is a concurrent work, the lack of this citation cannot be a reason for rejection. After the submission of our paper, we heard that their paper was (probably) accepted at some refereed international conference. We will check and cite their work in the camera-ready version.\n+ https://www.aozora.gr.jp/#main Aozora-bunko mostly includes old novels with expired copyrights (published 50+ years ago). Because these novels include many old-fashioned words and kanji usages, we did not consider Aozora-bunko for training Japanese LLMs. In addition, Aozora-bunko is not an effort to build a corpus for training Japanese LLMs. Thus, we don’t think the lack of this citation can be a reason for rejection.\n+ https://github.com/llm-jp/awesome-japanese-llm?tab=readme-ov-file We've been aware of this nice website that lists Japanese LLMs. However, as far as we checked, this website does not include a list of Japanese corpora. So again, we do not think the lack of this citation can be a reason for rejection. In addition, we want to emphasize that some strong LLMs listed on the website were used for the performance comparison in Table 2.\n\n> R2: Lacking some breakdown\n\nYou can find the breakdown at the end of Sections 3.1 to 3.5. This cannot be a reason for rejection, either.\n\n> R3: depth of ablation (how much data affects a handful model)\n\nWe believe that an important value of pre-training corpora is to provide data that works for any model and purpose. This is why we showed the effectiveness on different strong models.\n\n> Suggestions\n\nThank you for the useful suggestions. We will reflect these in the camera-ready version.\n\n> Q1: Performance on XL-Sum\n\nThis is explained in the second paragraph of Section 4.3. Before continual pre-training, we added Japanese vocabulary to Llama 2 and Mistral 7B tokenizers (mentioned in Section 4.1). This treatment degrades the downstream performance because the embeddings of newly added tokens were not trained well. In contrast, we didn’t expand the vocabulary of the Mixtral 8x7B tokenizer and observed an improvement in XL-Sum. We will elaborate on this in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Details on data cleaning and data creation is precious and these paper should be out in the public more to make LLM training more transparent and open.", "reasons_to_reject": "Personal preference to accept papers on data creation and processing (thus the 6 score)  but I'll say it's a weak accept, so reasons to reject includes:\n\n- No reference to other Japanese compilation work not using common crawl and web crawled data (see summary comments above)\n- Lacking some breakdown on how each much data was filtered out in each step\n- In data compilation study, depth of ablation (how much data affects a handful model) is more preferable then breath (how much of the performance the data affects different many LLMs). Priorities of computation usage would be better if it's covering depth.", "questions_to_authors": "Suggestion\n=====\n\n- Please add the breakdown of how much data is filtered out in each of the process and if possible sub-processes in the data creation. \n- Increase the spacing between lines in Table 2, esp when there are dotted or solid lines. \n- Cherry pick some examples where the model performs better after training with the data presented in this paper \n- If possible, please reference these other LLMs too https://huggingface.co/tokyotech-llm/Swallow-70b-instruct-hf and https://huggingface.co/Rakuten/RakutenAI-7B\n\nQuestions\n=====\n\n- Is there reason or particular examples for XL-Sum where the 7B models performs worse with the data in this paper? Why did it happen? Is it because the data after the cleaning process inherently (not on purpose) filtered out specific domain or specific length of the texts?", "extra_scores": {}}, {"review_id": "fSvYEQxCaW", "reviewer": "Reviewer_ytbp", "summary": "The paper constructed a high-quality Japanese web corpus which was extracted from Common Crawl dataset for the training of Japanese large language models. Compared with existing Japanese text corpus which was developed overseas as a part of multilingual corpora, the corpus in this paper was specially developed for Japanese and enjoys better quality. The paper also demonstrates a Japanese text collection and cleaning pipeline and empirically analyze the improvement that the newly collected Japanese web corpus can bring to Japanese large language models.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and effort in reviewing our paper. However, we received the reasons for rejection, probably due to misunderstanding and oversight of the [review guideline](https://colmweb.org/ReviewGuide.html) and the paper. We hope that our response addresses your questions, misunderstandings, and concerns. \n\n> R1: limited innovations and technical depth\n\nProcessing 63B web pages and continual pre-training on at least two 7B, a 13B, an 8x7B, and a 70B models cannot be done without technical depth. Unlike other kinds of studies, the top priority of corpus-building is not presenting an innovative method but making the best effort to deliver high-quality and useful data.\n\n> Q1: existing Japanese corpora are good enough\n\nThis is not true. The filtering method in mC4 is insufficient; thus, other English corpora were proposed after mC4. Furthermore, mC4 considered no aspect of Japanese for quality judgment.\n\n> R2: The motivation is not convincing. They can be easily resolved.\n\n> Q2: What is the exact problems of the existing corpus\n\nAmong CC-100, mC4, and OSCAR in Table 1, only mC4 can be a candidate for building Japanese open LLMs because the other two are too small (the numbers of tokens are less than 100B tokens, the size we used for the continual pre-training). However, mC4 uses WET as the data source, which includes unrecoverable errors. We quote Penedo et al. (2023):\n\n```\nhowever, in line with previous works (Gao et al., 2020; Rae et al., 2021), we found WET files to include undesirable navigation menus, ads, and other irrelevant texts.\n```\n\nHTML markups provide useful hints for removing navigation menus and ads, but this information is lost in WET files. This is a critical issue of WET files and mC4 that cannot be resolved later. We will elaborate on this in the extra page allowed in the camera-ready version.\n\nThe size of our cleaned corpus (312 billion letters) is larger than that of uncleaned mC4 (240 billion letters) in Table 1, which is also an advantage of our study. We also want to emphasize that Table 2 demonstrates that our corpus was superior to mC4 cleaned for Japanese (\"+ llm-jp\").\n\n> R3: Any techniques specifically designed for Japanese\n\nSections 3.1, 3.2, 3.4, and 3.5 were specially designed for Japanese.\n\n> R4: The presentation should be further improved\n\nCould you please suggest which part should be improved? For example, should “Llama 2 7B” be replaced with “Llama-2-7b” (as par with the model name)?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper developed a high-quality Japanese text corpus which will be beneficial to Japanese language model training. \n2. The paper also provides a text data collecting and cleaning pipeline which is not limited to Japanese but can also be applied to any other non-English language. \n3. The improvement that the developed Japanese corpus brought on language models looks very impressive.", "reasons_to_reject": "1. The paper has limited innovations and technical depth. The paper proposed very little novel techniques or innovations and its data collecting and cleaning pipeline looks straight forward and only involves data engineering components such as rule-based quality filtering (in which the rules are not hard to design). \n2. The motivation of this paper is not convincing. Although the paper claims that all existing Japanese text corpora are just part of some multilingual corpora and suffer from unsatisfactory quality, there are no concrete evidence to support this argument. In addition, the paper does not provide any concrete problems of the existing Japanese corpus and the reason why they have such problems. Even if the existing Japanese corpora have some problems, I am afraid that they can be easily resolved given that the data collecting and cleaning pipeline is so straight forward. \n3. Although the paper claims that it is dedicated to develop a Japanese-only text corpus, I can hardly identify any techniques proposedin this paper which is specifically designed for Japanese. \n4.  The presentation should be further improved. For example, the term \"Llama 2 7B\" looks wired and looks like the name of some model at the first glance. However, the model's name is \"Llama 2\" and the number of parameters is \"7B\".", "questions_to_authors": "1. Could you please demonstrate some cases in the existing Japanese corpus whose quality is not satisfactory? It seems that from Table 1, each existing Japanese corpus also employed deduplication and data cleaning and they are supposed to be good enough. \n2. Please also demonstrate the exact problems of the existing Japanese corpus and the reason why they have such problems and the reason why the problems cannot be easily tackled.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper describes a procedure to create a large scale Japanese corpus for training large language models.

Strengths:
1.  All reviewers like the contribution of this resource. 
2. The procedure is described with details. 
3. Continual pre-training results show the valuable of this resource. 

Weakness:
1. Some steps are specific to Japanese. 
2. Lacking direction assessment of corpus quality comparing to other resources. 

Overall, though the paper is not particularly novel, it will make a significant contribution with the creation of this resource.
Decision: Accept