Building a Large Japanese Web Corpus for Large Language Models

Authors: naoaki  okazaki
kakeru  hattori
hirai  shota
hiroki  iida
masanari  ohi
kazuki  fujii
taishi  nakamura
mengsay  loem
rio  yokota
sakae  mizuki
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 23:59:04

Abstract: Open Japanese large language models (LLMs) have been trained on the Japanese portions of corpora such as CC-100, mC4, and OSCAR. However, these corpora were not created for the quality of Japanese texts. This study builds a large Japanese web corpus by extracting and refining text from the Common Crawl archive (21 snapshots of approximately 63.4 billion pages crawled between 2020 and 2023). This corpus consists of approximately 312.1 billion characters (approximately 173 million pages), which is the largest of all available training corpora for Japanese LLMs, surpassing CC-100 (approximately 25.8 billion characters), mC4 (approximately 239.7 billion characters) and OSCAR 23.01 (approximately 74 billion characters). To confirm the quality of the corpus, we performed continual pre-training on Llama 2 7B, 13B, 70B, Mistral 7B v0.1, and Mixtral 8x7B as base LLMs and gained consistent (6.6-8.1 points) improvements on Japanese benchmark datasets. We also demonstrate that the improvement on Llama 2 13B brought from the presented corpus was the largest among those from other existing corpora.

Main paper: 
 introduction: 
 ChatGPT, released by OpenAI in late 2022, established a milestone toward achieving generalpurpose artificial intelligence. Research topics using large language models (LLMs) attract much attention including chain of thoughts (Wei et al., 2022; Kojima et al., 2022; Wang et al., 2023a; Trivedi et al., 2023) , instruction tuning (Wang et al., 2023b; c) , evaluation (Suzgun et al., 2023; Zheng et al., 2023) , hallucination (Rawte et al., 2023; Zhang et al., 2024) , bias (Ladhak et al., 2023; Feng et al., 2023) , detecting generated text (Mitchell et al., 2023) watermarking (Kirchenbauer et al., 2023) , poisoning (Wan et al., 2023) , detecting pre-training data (Shi et al., 2024) , unlearning (Yao et al., 2023) , and efficient inference (Dettmers et al., 2023) .While there are various motivations, such as raising the level of research and development in natural language processing, elucidating the mechanisms of LLM intelligence, the security risks of relying on a handful of foreign companies, and achieving responsible artificial intelligence, several Japanese companies and universities have actively been developing open LLMs that achieve good performance on Japanese text. However, in many cases, the quality of the training data for building Japanese LLMs was not satisfactory because they were mostly developed overseas as a part of multilingual corpora.Table 1 lists representative corpora used for training LLMs. Some popular corpora such as Pile-CC (Gao et al., 2020) , ROOTS (Laurenc ¸on et al., 2023) , Dolma (Soldaini et al., 2023) , and RedPajama-Data-v2 1 cannot be used for training Japanese LLMs, including no Japanese text. Table 1 : Representative corpora that can be used to pre-train LLMs. BL, BW, BT, and TT stand for billion letters, billion words, billion tokens, and trillion tokens, respectively. Extracted from the corresponding papers, numbers in English size are not directly comparable because of the difference in units. The number of ClueWeb22 may include text in other languages. Numbers in Japanese size are in Unicode characters and are directly comparable. Text, Langdet, Dedup, and Clean explain methods for text extraction, language detection, deduplication, and cleaning. WET in text extraction indicates that the corpus uses the text extraction results distributed by Common Crawl in WET format.Therefore, CC-100 (Wenzek et al., 2020) , mC4 (Xue et al., 2021) , and OSCAR 23.01 (Abadji et al., 2022) are candidates for training Japanese LLMs, including a certain amount of Japanese text and released with permissive licenses. However, the quality of Japanese text in these corpora is unsatisfactory because they contain noise in the HTML-to-text conversion (Common Crawl WET format) and because they have incorporated no special efforts to improve text quality for Japanese. Although we recognize the importance of efforts in building multilingual corpora, it is difficult to build a useful and reliable Japanese corpus without the knowledge of Japanese language. Concurrently to our work, Enomoto et al. (2024) explored filtering methods for Japanese web corpora, but this study did not demonstrate the effectiveness of the filtering methods on generative LLMs.Therefore, this paper explores a method to construct a large-scale, high-quality Japanese web corpus that can be used for training Japanese LLMs. The presented method includes lightweight language detection that boosts the processing speed of text extraction in the target language. This strategy is applicable not only to Japanese but also to other languages with less text than English. In addition, we specially design a filtering method to find good-quality Japanese text. The corpus developed in this study is made from 21 snapshots of Common Crawl (from CC-MAIN-2020-40 to CC- MAIN-2023-23) , and the size of the corpus after cleaning is 173,350,375 pages and 312,093,428,689 characters. To confirm the quality of the corpus, we perform continual pre-training 2 on Llama 2 7B, 13B, 70B (Touvron et al., 2023) , Mistral 7B v0.1 (Jiang et al., 2023) , and Mixtral 8x7B Instruct (Jiang et al., 2024) as base LLMs. Experimental results demonstrate that continual pre-training consistently improves the base model's performance by 6.6-8.1 points on Japanese benchmark datasets. We also demonstrate that the improvement on Llama 2 13B brought from the presented corpus was the largest among those from other existing corpora. The models trained on the presented corpus are available on the web site 3 and Hugging Face 4 . The implementations of this study are available on GitHub 56 . 
related work: 
 A number of large corpora were built from Common Crawl 7 archives. Common Crawl is a non-profit organization that crawls websites and provides their archives. The crawled data was initially distributed in ARC format 8 , but since the summer of 2013, it has been stored in Web ARChive (WARC) 9 and Web Text (WET) formats. According to Statistics of Common Crawl Monthly Archives 10 , the total amount of accessible archives (data crawled since 2013 till 2023) is 251,325,355,174 pages. Compact Language Detector 2 (cld2) 11 estimated that about 5 percent of these web pages are written in Japanese.Table 1 summarizes the corpora derived from Common Crawl (except for ClueWeb22, where documents are sampled from a commercial search engine). All permissive corpora that contain Japanese text rely on the data in WET format (HTML pages were converted to text by Common Crawl). The advantage of processing data in WET format is the reduction of processing time and data transfer size. Wenzek et al. (2020) proposed a pipeline to extract multilingual text from Common Crawl to train a multilingual BERT model (XLM-R) (Conneau et al., 2020) , and released its implementation (CCNet) 12 and data (CC-100) 13 . Xue et al. (2021) constructed mC4, a multilingual extension of C4 (Raffel et al., 2020) to train mT5, a multilingual variant of T5. Abadji et al. (2022) released OSCAR 23.01 with adult content filtering (based on an n-gram language model) and duplicate removal (based on locality sensitive hashing) added to the previous release.However, the corpus construction procedure often introduces aggressive filtering rules because the WET data usually includes irrelevant text, e.g., noises in HTML-to-text conversion, JavaScript codes, navigation menus, and footers. For example, C4 (Raffel et al., 2020) had to introduce filtering rules such as "remove lines with the word JavaScript" (to remove JavaScript code) and "remove pages with curly brackets {}" (because curly brackets are rarely used in a natural language but often used in a programming language). Therefore, we use WARC instead of WAT as the source format of Common Crawl archives.We follow the design principle of RefinedWeb (Penedo et al., 2023) : scale first (avoid labour intensive human curation process), strict deduplication (respect the value of deduplication for large language models), and neutral filtering (avoid undesirable biases introduced by machine-learning-based filtering). The construction procedure of RefinedWeb has much in common with MassiveWeb (Rae et al., 2021) . Collecting web pages with their own web crawler, Rae et al. (2021) used Google's SafeSearch filter to remove harmful content, and extracted text based on the HTML DOM structure. They also removed documents that contained a lot of repetitions, low-quality text, or duplicated information. In this study, we extend their pipelines to extract high-quality Japanese documents efficiently.We also considered using ClueWeb22 (Overwijk et al., 2022) as a source for building a large Japanese corpus. Although ClueWeb22 has nice and unique properties such as real distribution (web pages are extracted by the crawler of a commercial search engine and follow the distribution of web search activities), large-scale quality content (content extraction pipeline based on the search engine's production-quality content understanding system), and rich information (annotations of content structure, in-links/out-links of web pages, visual features of contents, etc), we could not use ClueWeb22 in this study because an LLM trained on ClueWeb22 cannot be released under an open license. 
method: 
 Figure 1 shows the corpus building pipeline. This pipeline is roughly divided into the following three stages: (1) Extracting Japanese text from WARC files in Common Crawl (Section 3.1); (2) Selecting Japanese text carefully with quality filtering (Section 3.2), deduplication (Section 3.3), and host filtering (Section 3.4); (3) Cleaning extracted text (Section 3.5).Common Crawl snapshots are stored as buckets in Amazon S3 and can be accessed via S3 or the web server14 . We extract HTML content from WARC data using the WARCIO library15 (Step 1, Figure 1 ). We will defer to the explanation of Step 2 until later. We then apply Trafilatura16 (Barbaresi, 2021) for extracting text from HTML markups (Step 3). Step 4 detects the language of a text based on a linear binary classifier (see Appendix A.1 for the detail) and extracts Japanese text only.Because about 5% of the entire Common Crawl is written in Japanese, it is possible to reduce the processing time of Steps 3 and 4 by 95% if we can target Japanese web pages only. However, in order to improve the accuracy of language detection, it is desirable to remove HTML markup before applying the language detection. Therefore, we should apply textStep 1Step 2Step 3Step 4Step 5Step 6Step 7Step 8Step 9 63,352,266,406 pages in Common Crawl (2020 -2023) This makes the processing time 15 times faster Extracting text from HTML (by using Trafilatura) 2, 686, 080, 919 pages (3, 684, 320, 219, 695 characters)   646, 238, 066 pages (1, 202, 868, 044, 631 characters)   185, 766,463 pages (353,191,872,805 characters) 173,350,375 pages (312,674,522,606 characters) Normalize Japanese punctuations into "、" and "。" 173,350,375 pages (312,093,428,689 characters) Figure 1 : The pipeline for building a large Japanese web corpus. extraction first (Step 3), followed by language detection (Step 4). To make matters worse, text extraction takes more processing time than language detection; we cannot reduce the processing time in this processing order. This is the motivation behind Step 2: we select web pages that are highly likely to be in Japanese in a rapid manner without text extraction. A web page proceeds to Steps 3 and 4 only when one of the following two conditions is met: (1) the HTML declares the language as Japanese, for example, <html lang="ja">; (2) the precise Japanese detection (Section A.1) recognizes the content within <title> tag as Japanese.To evaluate the validity of this rule, we compared the results of rapid and precise language detection. More specifically, assuming that the result of the precise language detection is correct, we measured the accuracy of rapid language detection on 10 WARC files 17 (394,192 pages, 12,052,992,707 bytes in total with gzip compression). The precision, recall, and F1 score were 0.888, 0.967, and 0.926, respectively. This result indicates that although rapid Japanese language detection may discard about 3.3% of Japanese web pages, the relatively high precision reduces the processing time for Steps 3 and 4 for non-Japanese web pages. In a benchmark experiment using 1 CPU of Intel Xeon Gold 6130 processor (2.1 GHz), the processing time for Steps 1-4 was about 15 times faster than Steps 1, 3, and 4 (without rapid language detection) because Step 2 removes a number of web pages from the target. This stage obtained 2,686,080,919 pages and 3,684,320,219,695 Japanese characters from 21 snapshots of Common Crawl (from CC-MAIN-2020-40 to CC-MAIN-2023-23).This process (1) removes web pages with many repetitions, (2) selects web pages that contain good-quality Japanese text, and (3) removes web pages that may contain harmful expressions. For processing (1) , we adopted the rules of Rae et al. (2021) to remove documents with many duplicated contents (see Appendix A.2).As for processing (2) , no standard has been established for assessing the quality of Japanese text. Therefore, we designed several rules for (2) selecting Japanese text of good quality. This study removes a text that satisfies either of the following conditions as irrelevant for LLMs (thresholds in parentheses). For example, Rule 2 ensures that a text includes a certain amount of function words such as ga (subject case marker), wo (object case marker) ni (position case maker roughly corresponding to to in English). Rule 3 rejects a web page containing a lot of product or service names (e.g., found in SEO sites), which are often expressed in katakana characters. Rule 7 removes a web page that looks like an RSS feed (a collection of snippets of other web pages). These rules were adjusted manually as the authors examined the text to be deleted and extracted.Although RefinedWeb used UT1 blocklist18 for removing harmful content, its coverage for Japanese pages may be insufficient. In order to remove web pages that may contain harmful expressions (3), we manually created a list of "NG expressions," which may indicate harmful (e.g., adult, violent, aggressive) contents. If the percentage of characters that match NG expressions is greater than 0.05, we exclude the text from the corpus.This quality filtering reduced the size of the corpus to 646,238,066 pages (1,202,868,044,631 characters) . Examining the text before and after the quality filtering, we observe that web pages that may be unuseful for training LLMs (e.g., e-commerce sites) have disappeared.Because Common Crawl visits and crawls the same website multiple times, the archive includes web pages that are identical or similar because of minor modifications or reproductions. Lee et al. (2022) reported that deduplication, removing duplicated text from the corpus, not only reduces memorization of LLMs but also improves pre-training efficiency.Therefore, we performed deduplication using MinHash (Broder, 1997) in a similar manner to Lee et al. (2022) . When using MinHash for deduplication, it is common to create r buckets, where each bucket is a concatenation of b MinHash values, compare r pairs of buckets of two documents, recognize the two documents as a duplicate if any of bucket pairs are identical.In this study, we adopted a setting where b = 20, r = 40 so that a pair of documents with a Jackard coefficient19 of 0.9 can be approximately detected as a duplicate with a probability of 92.5%. When a pair of documents was recognized as a duplicate, we kept the one crawled recently and removed the older one. In the snapshot used in this experiment, the nonduplicate rate of web pages collected between March and June 2023 ranged from 77.8 to 87.9 percent, while the non-duplicate rate of web pages collected before February 2023 dropped to less than 40 percent, and the rate of web pages collected around 2020 was less than 20 percent. This deduplication process reduced the corpus size to 185,766,463 pages (353,191,872,805 characters) .Even after the quality filtering in Section 3.2, we found some irrelevant content in the corpus. Therefore, we created a block list20 consisting of hostnames that met the following criteria.1. Included in the UT1 blocklist.2. Percentage of pages containing the name of a dating site (greater than 0.001).3. Percentage of pages containing NG expressions (greater than 0.005).4. *wikipedia.orgSections 3.1 to 3.4 process text at the document level, i.e., filter out irrelevant documents without altering the text within a document. In this study, to avoid an unexpected side effect, we minimally edit the text: punctuation normalization (replacing Western-style punctuations with Japanese-style ones) and footer trimming (removing footers that were left by Trafilatura). Refer to Appendix A.3 for the details. The process does not delete web pages, but the number of characters decreased from 312,674,522,606 to 312,093,428,689. 
models and training data: 
 In order to evaluate the usefulness of the presented corpus, we conducted continual pretraining of popular LLMs that excel in English22 . We used Llama 2 7B, 13B, 70B, Mistral 7B v0.1, and Mixtral 8x7B Instruct v0.1 as base models and performed continual pre-training of these base models on the presented corpus to examine whether the corpus improves their Japanese knowledge and skill. The training data for continual pre-training was a mixture of the presented corpus, Japanese Wikipedia, RefinedWeb, and arXiv component in The Pile23 (Gao et al., 2020) . Specifically, we prepared approximately 104.9 billion tokens of training data, assuming a sequence length of 4,096 tokens and a batch size of 1,024 with 25,000 steps for continuous pre-training. The ratio of Japanese to English tokens was set to 9:1, with 5% of the training data being the English text from RefinedWeb, 5% being The Pile's arXiv paper text (English), and the remaining 90% being Japanese text. The breakdown of the Japanese text was about 1.6 billion tokens of Japanese Wikipedia, and the presented Japanese corpus occupied the rest. We also used AlgebraicStack (Azerbayev et al., 2024) for Mistral's continual pre-training. For Mixtral, we used both AlgebraicStack and The Vault (Manh et al., 2023) 24 . We used a different ratio (72:28) of Japanese to English tokens for continual pre-training on Mixtral 8x7B Instruct25 .We did not change the architecture of base LLMs; the embedding sizes of tokens, hidden layers, feed-forward layers, the number of attention heads, and the number of layers were unchanged from the base models in continual pre-training. Before continual pre-training, we added Japanese vocabulary to Llama 2 and Mistral 7B tokenizers. The total number of vocabulary was 43,176 for LLMs based on Llama 2 and 42,800 for those based on Mistral 7B. We employed AdamW (Loshchilov & Hutter, 2019) , with hyperparameters set to β 1 = 0.9, β 2 = 0.95, ϵ = 1.0 × 10 -8 . We used re-warming and re-decaying the learning rate (Ibrahim et al., 2024) with 1,000 warm-up steps. For Llama 2, the maximum learning rate was set at 1.0 × 10 -4 with a decay rate of 1/30, while for Mistral and Mixtral, it was 2.0 × 10 -5 with a decay rate of 1/10. The batch size was 1,024. We used 0.1 for weight decay and 1.0 for gradient clipping. In addition, we used Flash Attention (Dao et al., 2022) to improve computational and memory efficiency.In order to examine the impact of a corpus on the performance of LLMs, we performed continual pre-training on ClueWeb22 and llm-jp-corpus v1.0.126 as well as the presented corpus.We also compared our LLMs to others with similar numbers of parameters: CyberAgentLM2 7B (CALM2 7B, a Japanese LLM trained from scratch); Japanese- We used llm-jp-eval27 and lm-evaluation-harness28 as Japanese evaluation benchmarks. llmjp-eval is a benchmark consisting of tasks for multiple choice (MC) question answering, open question answering (QA), reading comprehension (RC), and natural language inference (NLI) tasks. JCommonsenseQA (Kurihara et al., 2022) is employed for MC, JEMHopQA (Ishii et al., 2023) and NIILC (Sekine, 2003) for QA, JSQuAD (Kurihara et al., 2022) for RC. We decided to exclude the NLI dataset from the benchmark because the distribution of ground-truth NLI labels is highly imbalanced, which made the evaluation unstable29 . For lm-evaluation-harness, we used Japanese subsets of XL-Sum (Hasan et al., 2021) for the automatic summarization task and MGSM (Shi et al., 2023) for the arithmetic reasoning task, respectively. In addition, WMT 2020 (Barrault et al., 2020) was used for the Japanese-English and English-Japanese machine translation.Table 2 reports the performance of LLMs on the Japanese benchmark datasets. The row "+ this study" shows the performance of the LLM after we applied continual pre-training to the base model on the presented corpus. The rows "+ this study" demonstrate that the presented corpus consistently improves the performance of the base models via continual pre-training by 6.6-8.1 points on average. The models built by this study outperform other models built from scratch or continual pre-training, establishing the state-of-the-art performance in each model size (7B, 13B, 8x7B, and 70B). This fact shows the usefulness of the presented corpus in building LLMs that excel in Japanese.We can compare different Japanese corpora in continual pre-training of Llama 2 13B. All corpora (ClueWeb22, llm-jp-corpus v1.0.1, and presented corpus) improved the base model's performance by 5.0-7.0 points, which verifies the effectiveness of the continual pre-training. The presented corpus yielded the best improvement (7.0 points) of all corpora; in particular, it drastically enhances the performance of question answering (JCom, JEMHop, NIILC), reading comprehension (JSQuAD), and arithmetic reasoning (MGSM). Although we did not incorporate a special effort to improve these tasks, the model acquired knowledge about Japan and the Japanese language from the presented corpus. In contrast, the presented corpus did not improve the summarization (XL-Sum) task. This trend is observed across other base models, except for Mixtral 8x7B Instruct, potentially indicating that adding Japanese vocabulary may have a detrimental effect on this task. In addition, we found that the continual pre-training on the presented corpus improved the performance in English-Japanese translation but degraded that in the reversed direction (Japanese-English translation). We will explore mitigation of this phenomenon in the future, e.g., by changing the ratio of Japanese to English tokens in the training data and promoting language transfer using an English-Japanese parallel corpus during continual pre-training. 
*.5ch.net: 
 Although the UT1 blocklist includes some Japanese websites, we introduced Criteria 2 and 3 to improve the coverage. The authors manually adjusted the thresholds for the percentage21 . We applied Criterion 4 because we planned to use Wikipedia text extracted from the dump. This filtering process resulted in a corpus of 173,350,375 pages (312,674,522,606 characters).In  and 70B ). We also observed that the improvement on Llama 2 13B brought from the presented corpus was the largest among those from other existing corpora.Future directions include efforts towards the safety of LLMs, such as reducing harmful generations (e.g., discrimination, exclusion, toxicity, hallucination). Currently, we only use the lists of NG expressions and hostnames, but it is desirable to establish more robust filtering methods to remove harmful text for pre-training Japanese LLMs. In addition, although our study focused on the continual pre-training setting, we want to evaluate the presented corpus by training Japanese LLMs from scratch. Although we evaluated the LLMs in downstream tasks such as question-answering and summarization, it is questionable whether this can measure the "general intelligence" of an LLM. At the same time, training an LLM on a pre-training corpus requires huge computations. Therefore, we want to explore a lightweight method for assessing the effectiveness of pre-training corpora without building LLMs. 
limitations: 
 We could not present an ablation study of each step in Figure 1 because of a large amount of computational resources required to train LLMs. Instead, we reported the performance of each step of the corpus construction procedure in Section 3.This study focuses on Japanese, and the construction of corpora for other languages is outside the scope. For English, it may be a good strategy to make use of good quality corpora that have already been developed. Although some ideas in this paper could be helpful in building corpora for other languages, for example, reducing processing time with rapid language detection, we believe that it would be better for researchers from different countries to share the task of building corpora for their own, as assessing text quality requires the knowledge of the language. 
ethics statement: 
 Article 30-4 of the Copyright Act in Japan permits to use of a copyrighted work without the permission of the copyright holder as long as the use is not a person's purpose to personally enjoy or cause another person to enjoy the work. This is a quote from the Japanese translation of Article 30-4 of Copyright Act30 :It is permissible to exploit a work, in any way and to the extent considered necessary, in any of the following cases, or in any other case in which it is not a person's purpose to personally enjoy or cause another person to enjoy the thoughts or sentiments expressed in that work; provided, however, that this does not apply if the action would unreasonably prejudice the interests of the copyright owner in light of the nature or purpose of the work or the circumstances of its exploitation: (snip)The nuance of enjoy probably needs clarification. Enjoying a work roughly corresponds to consuming the idea of a work (for any purpose, including entertainment, learning, and communication, regardless of the monetary gains from the work). The intention of Article 30-4 is to define an exceptional use case where a computer program consumes works directly for machine learning and information analysis, whereas a human cannot do that.This provides the justification for building the presented corpus and LLMs. To the best of our knowledge, we have taken every measure to keep the corpus as non-toxic and unbiased as possible, but we are unaware of any direct ethical consequences caused by the LLMs trained on the presented corpus. 
reproducibility statement: 
 All models developed in this study (continual pre-training on Llama 2 7B, 13B, 70B, Mistral 7B v0.1, and Mixtral 8x7B Instruct) have already been released on Hugging Face. The bench-mark datasets used in this study are also publicly available. Therefore, it is straightforward to reproduce our experimental results reported in Table 2 .Following the pipeline of RefinedWeb (Penedo et al., 2023) , we built a language detector with a linear discriminator using character n-grams as features. We used multilingual Wikipedia texts as the training data for the linear discriminator, and the feature space was constructed from the training data with character unigrams, bi-grams, and trigrams that satisfy one of the following criteria.1. Within the top 400,000 occurrences in the training data for all languages.2. Within the top 400,000 occurrences in the training data of Japanese.3. Within the top 100,000 occurrences in Chinese training data.4. Within the top 10,000 occurrences in the training data for each language.If only Criterion (1) was used, there would be a possibility that features related to Japanese characters would not be obtained sufficiently. Therefore, Criterion (2) aimed to obtain features specific to Japanese. Criterion (3) made it easier to distinguish between Japanese and Chinese, which share Chinese characters. We aimed to stabilize the detection results for texts in languages with insufficient training data using Criterion (4). The total number of distinct features was 821,484.For training the linear discriminator, we used the dump of Wikipedia CirrusSearch 31 (as of June 12, 2023) . In training the language detector, we reduced the amount of training data to 1/20th of the dump because the total amount was too large for training the discriminator. We sampled 100,000 instances in the remaining data for development and evaluation sets.We trained an L2-regularized L2-loss Support Vector Machine on the training data using LIBLINEAR 32 2.46. The regularization coefficient was set to C = 10 empirically by the search on the development set. We confirmed that the performance of this classifier was quite good: 0.996 F1 score on the test set and 0.989 F1 score on whatlang-corpora 33 .The removal of documents that contain many repetitions aims to filter out documents that do not seem to have useful information and to prevent the behavior of LLMs from repeatedly generating the same words. Specifically, we follow the rules of Rae et al. (2021) and remove a document when the percentage of any of the following exceeds the threshold (shown in parentheses).https://www.together.ai/blog/redpajama-data-v2https://github.com/facebookresearch/cc nethttps://huggingface.co/datasets/cc100https://data.commoncrawl.org/https://github.com/webrecorder/warciohttps://trafilatura.readthedocs.io/CC-MAIN-20230527223515-20230528013515-0000?.warc.gz from 0 to 9 in CC-MAIN-2023-23.https://dsi.ut-capitole.fr/blacklists/In this study, features were constructed with 5-grams of characters.We used the word blocklist as "a list for blocking," following the name "UT1 blocklist." We are reluctant to use the word "black list" because some hosts in this list may not be "black." For example, a hospital website explaining a woman's maternity is prone to be included in the list.We placed emphasis on recall at the expense of precision to reduce harmful behaviors of trained models. This design results in including a lot of non-harmful websites in the block list.It may be straightforward to evaluate the corpus by training LLMs from scratch, but we chose continual pre-training because the broader goal of this effort was to build high-performance LLMs that excel in Japanese under the limited amount of computing budget. We can assess the quality of the corpus by checking performance gains from the base models and also comparing the performance with other models.We expected that the RefinedWeb and arXiv would help maintain the base LLMs' English knowledge and skill during continual pre-training.We changed this ratio as an attempt to improve the performance in English.https://github.com/llm-jp/llm-jp-corpushttps://github.com/llm-jp/llm-jp-eval v1.0.0https://github.com/Stability-AI/lm-evaluation-harness/tree/jp-stable commit #9b42d41An LLM may obtain a high score only if the model happens to predict the majority label without understanding and solving the NLI task.https://www.japaneselawtranslation.go.jp/ja/laws/view/4207#je ch2sc3sb5at4 
 
 Appendix:  


Decision: This paper describes a procedure to create a large scale Japanese corpus for training large language models.

Strengths:
1.  All reviewers like the contribution of this resource. 
2. The procedure is described with details. 
3. Continual pre-training results show the valuable of this resource. 

Weakness:
1. Some steps are specific to Japanese. 
2. Lacking direction assessment of corpus quality comparing to other resources. 

Overall, though the paper is not particularly novel, it will make a significant contribution with the creation of this resource.

Reviews: review: paper_summary: paper_summary: This paper presents an effort to create a very large corpus of trustworthy Japanese text that could be used to train LLMs. They describe the method used for obtaining candidate data, cleaning, and deduplicating it. In order to show the quality of the resource, the authors train different LLMs on this data and show that it actually improved the metrics on a benchmark of Japanese NLP tests.
strength_weakness: strength_weakness: Some parts of the document are in need of more clarifications, for example the quality and hostname filtering, and the explanation of section 4.1 and associated Table 2.
questions: questions: - By continual pretraining (or is it continued pretraining?) I understand you take the pretrained model weights and keep on training on your new data. Is this the case? Please make this clear, perhaps indicate some references where the technique is introduced or used.
- Please explain what NG expressions are. Does is refer to js or other programming languages content?
- When you refer to "blocklists" I think the correct term is "blacklists", or is it another concept?
- I am intrigued by the Japanese Copyright Act mentioned in the ethics statement that says the purpose of the person is not to "enjoy" or cause another person to "enjoy" the work (quotes are from the text). Is this some translation issue, or what is being referred to as enjoy in this case? Does this mean the use for monetary gains of the data?
paper_summary: paper_summary: The authors created a novel Japanese corpus for training large language models from Common Crawl archives.
It consists of 312B characters, which is substantially larger than previous training corpora for Japanese LLMs, such as CC-100 (25.8B characters), mC4 (239.7 B characters), and OSCAR (74B characters).
Using their Japanese web corpus, the authors performed continual pre-training on various LLMs, such as Llama 2 and Mistral, and gained consistent improvements on Japanese benchmark datasets.
strength_weakness: strength_weakness: Many aspects of the proposed method, such as filtering based on character type and length, are only applicable to Japanese.

The evaluation was only when the authors performed continual pre-training on existing LLMs using the Japanese web corpus.
They do not demonstrate whether their corpus would be effective if they train an LLM from scratch.
questions: questions: I want your current estimates of whether the Japanese web corpus has the quantity and quality necessary to train an LLM from scratch.

This is not a question, but a comment.
Using shorthands such as BL (billion letters) and BW (billion words) in Table 1 is okay.
However, the authors should avoid their use in the body of the paper.
paper_summary: paper_summary: The paper presents a corpus building project to build a large Japanese web corpus for LLMs. Using Common Crawl WARC as a base is a good idea but a lot of other Japanese related corpora are driven by Japanese academics in their silo networks that has yet been tapped in modern LLM training, e.g. NICT LLM https://www.nii.ac.jp/en/news/release/2023/1020.html and https://www.aozora.gr.jp/#main and more on https://github.com/llm-jp/awesome-japanese-llm?tab=readme-ov-file. Also, references to those work is lacking in this paper (but it's understandable since a lot of the work done in those research labs are disseminated in Japanese only). 

The steps taken in the data processing are clearly described, esp. the language ID and ngram filter process. As for the "quality filtering" process of using character sets (Section 3.2), it might be that the filtering by character sets have implicitly also restrict the domain that the model will be learning from, e.g. there's a lot of Japanese BBS (bulletin board sites) / forums where substrings like "kaomoji" might be used e.g. ٩(ˊᗜˋ*)و  https://detail.chiebukuro.yahoo.co.jp/qa/question_detail/q11297327146 and https://komachi.yomiuri.co.jp/ Lastly, there are also many nuances in cleaning Japanese texts e.g. half-width / full-width characters and furigana mixed into the texts. 

It would be nice to have some breakdown of how much each of the filtering process taken out, esp. how much of the final text is made up of the Wikipedia dump.
strength_weakness: strength_weakness: Personal preference to accept papers on data creation and processing (thus the 6 score)  but I'll say it's a weak accept, so reasons to reject includes:

- No reference to other Japanese compilation work not using common crawl and web crawled data (see summary comments above)
- Lacking some breakdown on how each much data was filtered out in each step
- In data compilation study, depth of ablation (how much data affects a handful model) is more preferable then breath (how much of the performance the data affects different many LLMs). Priorities of computation usage would be better if it's covering depth.
questions: questions: Suggestion
=====

- Please add the breakdown of how much data is filtered out in each of the process and if possible sub-processes in the data creation. 
- Increase the spacing between lines in Table 2, esp when there are dotted or solid lines. 
- Cherry pick some examples where the model performs better after training with the data presented in this paper 
- If possible, please reference these other LLMs too https://huggingface.co/tokyotech-llm/Swallow-70b-instruct-hf and https://huggingface.co/Rakuten/RakutenAI-7B

Questions
=====

- Is there reason or particular examples for XL-Sum where the 7B models performs worse with the data in this paper? Why did it happen? Is it because the data after the cleaning process inherently (not on purpose) filtered out specific domain or specific length of the texts?
paper_summary: paper_summary: The paper constructed a high-quality Japanese web corpus which was extracted from Common Crawl dataset for the training of Japanese large language models. Compared with existing Japanese text corpus which was developed overseas as a part of multilingual corpora, the corpus in this paper was specially developed for Japanese and enjoys better quality. The paper also demonstrates a Japanese text collection and cleaning pipeline and empirically analyze the improvement that the newly collected Japanese web corpus can bring to Japanese large language models.
strength_weakness: strength_weakness: 1. The paper has limited innovations and technical depth. The paper proposed very little novel techniques or innovations and its data collecting and cleaning pipeline looks straight forward and only involves data engineering components such as rule-based quality filtering (in which the rules are not hard to design). 
2. The motivation of this paper is not convincing. Although the paper claims that all existing Japanese text corpora are just part of some multilingual corpora and suffer from unsatisfactory quality, there are no concrete evidence to support this argument. In addition, the paper does not provide any concrete problems of the existing Japanese corpus and the reason why they have such problems. Even if the existing Japanese corpora have some problems, I am afraid that they can be easily resolved given that the data collecting and cleaning pipeline is so straight forward. 
3. Although the paper claims that it is dedicated to develop a Japanese-only text corpus, I can hardly identify any techniques proposedin this paper which is specifically designed for Japanese. 
4.  The presentation should be further improved. For example, the term "Llama 2 7B" looks wired and looks like the name of some model at the first glance. However, the model's name is "Llama 2" and the number of parameters is "7B".
questions: questions: 1. Could you please demonstrate some cases in the existing Japanese corpus whose quality is not satisfactory? It seems that from Table 1, each existing Japanese corpus also employed deduplication and data cleaning and they are supposed to be good enough. 
2. Please also demonstrate the exact problems of the existing Japanese corpus and the reason why they have such problems and the reason why the problems cannot be easily tackled.

Comments: title: Re: Official Comment by Reviewer KvfY
Reminder
Re: Official Comment by Reviewer QC9E
Initiating the discussion
Initiating the discussion
comment: Dear Reviewer KvfY,

Thank you for the additional suggestion. We could not include this table in the initial submission due to the limitation of space, but we will include the table in the extra one page of the camera-ready version.

As the discussion period is nearing its end, we are also keen to know if our response has addressed your other concerns (references and depth of ablations). We would be grateful if you could reconsider the rating if appropriate.

Best,

Authors
Dear Reviewer ytbp,

As the discussion period is nearing its end, we kindly remind you to review our response if you haven't had the chance yet. We are keen to know if our response has addressed your concerns and would be grateful if you could reconsider the rating if appropriate. If there are any further questions or clarifications needed, we would be more than happy to provide additional information.

Thank you very much for your time and consideration.

Authors
Additional suggestion on R2 and breakdown, summarize it in a nice table in the appendix, it'll help a lot. And breaking down 3.4 further by the sub-step would be very helpful esp. understanding how much of the 40B characters removed were from *wikipedia.org
Thank you for the reply to the response. We will include the definition in the revised manuscript.
Thanks for your response. I was not aware of the "NG expression" label, and perhaps other readers will not be aware either, so please include a definition in the revised manuscript.
We sincerely thank the reviewer for the valuable suggestions. We hope our response has thoroughly addressed all concerns and that the reviewer can consider improving their score based on our response. There is nothing we want to add to the response, but we are also happy to discuss with the reviewer if they still have a concern.
We sincerely thank the reviewer for the valuable suggestions. We hope our response has thoroughly addressed all concerns and that the reviewer can consider improving their score based on our response. There is nothing we want to add to the response, but we are also happy to discuss with the reviewer if they still have a concern.