IVUL-KAUST/MOLE huggingface dataset:
Dataset: JaFIn
Description: JaFIn is a Japanese financial instruction dataset that was manually curated from various sources, including government websites, Wikipedia, and financial institutions.
Language: jp
Script: mixed
Domain: wikipedia, web pages
Form: text
Size: 1490 sentences
Provider: Hokkaido University, University of Tokyo
Tasks: instruction tuning, question answering
License: CC BY-NC-SA 4.0
Year: 2024
Authors: Kota Tanabe, Masahiro Suzuki, Hiroki Sakaji, Itsuki Noda
Affiliations: Hokkaido University, University of Tokyo
Abstract: We construct an instruction dataset for the large language model (LLM) in the Japanese finance domain. Domain adaptation of language models, including LLMs, is receiving more attention as language models become more popular. This study demonstrates the effectiveness of domain adaptation through instruction tuning. To achieve this, we propose an instruction tuning data in Japanese called JaFIn, the Japanese Financial Instruction Dataset. JaFIn is manually constructed based on multiple data sources, including Japanese government websites, which provide extensive financial knowledge. We then utilize JaFIn to apply instruction tuning for several LLMs, demonstrating that our models specialized in finance have better domain adaptability than the original models. The financial-specialized LLMs created were evaluated using a quantitative Japanese financial benchmark and qualitative response comparisons, showing improved performance over the originals.
Dataset: Japanese Fake News Dataset
Description: The first Japanese fake news dataset, annotated with a novel, fine-grained scheme. It goes beyond factuality to include disseminator's intent, harm, target, and purpose, based on 307 news stories from Twitter verified by Fact Check Initiative Japan.
Language: jp
Script: mixed
Domain: news articles, social media
Form: text
Size: 307 documents
Provider: SANKEN Osaka University, NAIST
Tasks: fake news detection
License: CC BY-NC-ND 4.0
Year: 2022
Authors: Taichi Murayama, Shohei Hisada, Makoto Uehara, Shoko Wakamiya, Eiji Aramaki
Affiliations: SANKEN Osaka University, NARA Institute of Science and Technology
Abstract: Fake news provokes many societal problems; therefore, there has been extensive research on fake news detection tasks to counter it. Many fake news datasets were constructed as resources to facilitate this task. Contemporary research focuses almost exclusively on the factuality aspect of the news. However, this aspect alone is insufficient to explain "fake news," which is a complex phenomenon that involves a wide range of issues. To fully understand the nature of each instance of fake news, it is important to observe it from various perspectives, such as the intention of the false news disseminator, the harmfulness of the news to our society, and the target of the news. We propose a novel annotation scheme with fine-grained labeling based on detailed investigations of existing fake news datasets to capture these various aspects of fake news. Using the annotation scheme, we construct and publish the first Japanese fake news dataset. The annotation scheme is expected to provide an in-depth understanding of fake news. We plan to build datasets for both Japanese and other languages using our scheme. Our Japanese dataset is published at https://hkefka385.github.io/dataset/fakenews-japanese/.
Dataset: JMultiWOZ
Description: JMultiWOZ is the first Japanese language large-scale multi-domain task-oriented dialogue dataset. It contains 4,246 conversations spanning six travel-related domains: tourist attractions, accommodation, restaurants, shopping facilities, taxis, and weather. It provides dialogue state annotations for benchmarking dialogue state tracking and response generation.
Language: jp
Script: mixed
Domain: web pages
Form: text
Size: 52405 sentences
Provider: Nagoya University
Tasks: other
License: CC BY-SA 4.0
Year: 2024
Authors: Atsumoto Ohashi, Ryu Hirai, Shinya Iizuka, Ryuichiro Higashinaka
Affiliations: Nagoya University
Abstract: Dialogue datasets are crucial for deep learning-based task-oriented dialogue system research. While numerous English language multi-domain task-oriented dialogue datasets have been developed and contributed to significant advancements in task-oriented dialogue systems, such a dataset does not exist in Japanese, and research in this area is limited compared to that in English. In this study, towards the advancement of research and development of task-oriented dialogue systems in Japanese, we constructed JMultiWOZ, the first Japanese language large-scale multi-domain task-oriented dialogue dataset. Using JMultiWOZ, we evaluated the dialogue state tracking and response generation capabilities of the state-of-the-art methods on the existing major English benchmark dataset MultiWOZ2.2 and the latest large language model (LLM)-based methods. Our evaluation results demonstrated that JMultiWOZ provides a benchmark that is on par with MultiWOZ2.2. In addition, through evaluation experiments of interactive dialogues with the models and human participants, we identified limitations in the task completion capabilities of LLMs in Japanese.
Dataset: Japanese Web Corpus
Description: A large-scale Japanese web corpus created from 21 Common Crawl snapshots (crawled between 2020 and 2023). The corpus consists of approximately 312.1 billion characters from 173 million pages. 
Language: jp
Script: mixed
Domain: web pages, public datasets
Form: text
Size: 173350375 documents
Provider: Tokyo Institute of Technology
Tasks: language modeling
License: unknown
Year: 2024
Authors: Naoaki Okazaki, Kakeru Hattori, Hirai Shota, Hiroki Iida, Masanari Ohi, Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Rio Yokota, Sakae Mizuki
Affiliations: Tokyo Institute of Technology
Abstract: Open Japanese large language models (LLMs) have been trained on the Japanese portions of corpora such as CC-100, mC4, and OSCAR. However, these corpora were not created for the quality of Japanese texts. This study builds a large Japanese web corpus by extracting and refining text from the Common Crawl archive (21 snapshots of approximately 63.4 billion pages crawled between 2020 and 2023). This corpus consists of approximately 312.1 billion characters (approximately 173 million pages), which is the largest of all available training corpora for Japanese LLMs, surpassing CC-100 (approximately 25.8 billion characters), mC4 (approximately 239.7 billion characters) and OSCAR 23.10 (approximately 74 billion characters). To confirm the quality of the corpus, we performed continual pre-training on Llama 2 7B, 13B, 70B, Mistral 7B v0.1, and Mixtral 8x7B Instruct as base LLMs and gained consistent (6.6-8.1 points) improvements on Japanese benchmark datasets. We also demonstrate that the improvement on Llama 2 13B brought from the presented corpus was the largest among those from other existing corpora.
Dataset: J-CRe3
Description: A Japanese multimodal dataset containing egocentric video and dialogue audio of real-world conversations between a master and an assistant robot at home.
Language: jp
Script: mixed
Domain: other
Form: text
Size: 11000 images
Provider: RIKEN
Tasks: other
License: CC BY-SA 4.0
Year: 2024
Authors: Nobuhiro Ueda, Hideko Habe, Yoko Matsui, Akishige Yuguchi, Seiya Kawano, Yasutomo Kawanishi, Sadao Kurohashi, Koichiro Yoshino
Affiliations: Kyoto University, Guardian Robot Project, R-IH, RIKEN, Tokyo University of Science, Nara Institute of Science and Technology, National Institute of Informatics
Abstract: Understanding expressions that refer to the physical world is crucial for such human-assisting systems in the real world, as robots that must perform actions that are expected by users. In real-world reference resolution, a system must ground the verbal information that appears in user interactions to the visual information observed in egocentric views. To this end, we propose a multimodal reference resolution task and construct a Japanese Conversation dataset for Real-world Reference Resolution (J-CRe3). Our dataset contains egocentric video and dialogue audio of real-world conversations between two people acting as a master and an assistant robot at home. The dataset is annotated with crossmodal tags between phrases in the utterances and the object bounding boxes in the video frames. These tags include indirect reference relations, such as predicate-argument structures and bridging references as well as direct reference relations. We also constructed an experimental model and clarified the challenges in multimodal reference resolution tasks.
Dataset: JSUT
Description: The corpus consists of 10 hours of reading-style speech
data and its transcription and covers all of the main pronun-
ciations of daily-use Japanese characters.
Language: jp
Script: mixed
Domain: wikipedia, public datasets
Form: audio
Size: 10 hours
Tasks: speech recognition
License: CC BY-SA 4.0
Year: 2017
Authors: Ryosuke Sonobe, Shinnosuke Takamichi, Hiroshi Saruwatari
Affiliations: University of Tokyo
Abstract: Thanks to improvements in machine learning techniques including deep learning, a free large-scale speech corpus that can be shared between academic institutions and commercial companies has an important role. However, such a corpus for Japanese speech synthesis does not exist. In this paper, we designed a novel Japanese speech corpus, named the "JSUT corpus," that is aimed at achieving end-to-end speech synthesis. The corpus consists of 10 hours of reading-style speech data and its transcription and covers all of the main pronunciations of daily-use Japanese characters. In this paper, we describe how we designed and analyzed the corpus. The corpus is freely available online.
Dataset: Japanese Word Similarity Dataset
Description: A Japanese word similarity dataset (JWSD) containing 4,851 word pairs with human-annotated similarity scores. The dataset includes various parts of speech (nouns, verbs, adjectives, adverbs) and covers both common and rare words, designed for evaluating Japanese distributed word representations.
Language: jp
Script: mixed
Domain: public datasets
Form: text
Size: 4851 sentences
Provider: Tokyo Metropolitan University
Tasks: word similarity
License: CC BY-SA 3.0
Year: 2018
Authors: Yuya Sakaizawa, Mamoru Komachi
Affiliations: Tokyo Metropolitan University
Abstract: An evaluation of distributed word representation is generally conducted using a word similarity task and/or a word analogy task. There are many datasets readily available for these tasks in English. However, evaluating distributed representation in languages that do not have such resources (e.g., Japanese) is difficult. Therefore, as a first step toward evaluating distributed representations in Japanese, we constructed a Japanese word similarity dataset. To the best of our knowledge, our dataset is the first resource that can be used to evaluate distributed representations in Japanese. Moreover, our dataset contains various parts of speech and includes rare words in addition to common words.
Dataset: STAIR Captions
Description: A large-scale image caption dataset in Japanese, based on the COCO dataset. It contains 820,310 Japanese captions for 164,062 images, collected via crowdsourcing.
Language: jp
Script: mixed
Domain: captions, public datasets
Form: text
Size: 820310 sentences
Provider: The University of Tokyo, National Institute of Informatics
Tasks: image captioning
License: CC BY 4.0
Year: 2017
Authors: Yuya Yoshikawa, Yutaro Shigeto, Akikazu Takeuchi
Affiliations: The University of Tokyo, National Institute of Informatics
Abstract: In recent years, automatic generation of image descriptions (captions), that is, image captioning, has attracted a great deal of attention. In this paper, we particularly consider generating Japanese captions for images. Since most available caption datasets have been constructed for English language, there are few datasets for Japanese. To tackle this problem, we construct a large-scale Japanese image caption dataset based on images from MS-COCO, which is called STAIR Captions. STAIR Captions consists of 820,310 Japanese captions for 164,062 images. In the experiment, we show that a neural network trained using STAIR Captions can generate more natural and better Japanese captions, compared to those generated using English-Japanese machine translation after generating English captions.
Dataset: Arukikata Travelogue
Description: A Japanese text dataset with over 31 million characters, comprising 4,672 domestic and 9,607 overseas travelogues from the Arukikata website. It was created to provide a shared resource for research, ensuring transparency and reproducibility in analyzing human-place interactions from text.
Language: jp
Script: mixed
Domain: web pages
Form: text
Size: 14279 documents
Provider: Arukikata Co., Ltd.
Tasks: other
License: custom
Year: 2023
Authors: Hiroki Ouchi, Hiroyuki Shindo, Shoko Wakamiya, Yuki Matsuda, Naoya Inoue, Shohei Higashiyama, Satoshi Nakamura, Taro Watanabe
Affiliations: NAIST, JAIST, NICT, RIKEN
Abstract: We have constructed Arukikata Travelogue Dataset and released it free of charge for academic research. This dataset is a Japanese text dataset with a total of over 31 million words, comprising 4,672 Japanese domestic travelogues and 9,607 overseas travelogues. Before providing our dataset, there was a scarcity of widely available travelogue data for research purposes, and each researcher had to prepare their own data. This hinders the replication of existing studies and fair comparative analysis of experimental results. Our dataset enables any researchers to conduct investigation on the same data and to ensure transparency and reproducibility in research. In this paper, we describe the academic significance, characteristics, and prospects of our dataset.
Dataset: JTubeSpeech
Description: A large-scale Japanese speech corpus collected from YouTube videos and their subtitles. It is designed for both automatic speech recognition (ASR) and automatic speaker verification (ASV) tasks, containing over 1,300 hours for ASR and 900 hours for ASV.
Language: jp
Script: mixed
Domain: social media
Form: audio
Size: 1300 hours
Provider: The University of Tokyo, Technical University of Munich, Tokyo Metropolitan University, Carnegie Mellon University
Tasks: speech recognition, speaker identification
License: Apache-2.0
Year: 2021
Authors: Shinnosuke Takamichi, Ludwig Kürzinger, Takaaki Saeki, Sayaka Shiota, Shinji Watanabe
Affiliations: The University of Tokyo, Technical University of Munich, Tokyo Metropolitan University, Carnegie Mellon University
Abstract: In this paper, we construct a new Japanese speech corpus called "JTubeSpeech." Although recent end-to-end learning requires large-size speech corpora, open-sourced such corpora for languages other than English have not yet been established. In this paper, we describe the construction of a corpus from YouTube videos and subtitles for speech recognition and speaker verification. Our method can automatically filter the videos and subtitles with almost no language-dependent processes. We consistently employ Connectionist Temporal Classification (CTC)-based techniques for automatic speech recognition (ASR) and a speaker variation-based method for automatic speaker verification (ASV). We build 1) a large-scale Japanese ASR benchmark with more than 1,300 hours of data and 2) 900 hours of data for Japanese ASV.
Dataset: JCoLA
Description: JCoLA (Japanese Corpus of Linguistic Acceptability) consists of 10,020 sentences annotated with binary acceptability judgments. The sentences are manually extracted from linguistics textbooks, handbooks, and journal articles, and are split into in-domain and out-of-domain data.
Language: jp
Script: mixed
Domain: books
Form: text
Size: 10020 sentences
Provider: The University of Tokyo
Tasks: linguistic acceptability
License: custom
Year: 2023
Authors: Taiga Someya, Yushi Sugimoto, Yohei Oseki
Affiliations: The University of Tokyo
Abstract: Neural language models have exhibited outstanding performance in a range of downstream tasks. However, there is limited understanding regarding the extent to which these models internalize syntactic knowledge, so that various datasets have recently been constructed to facilitate syntactic evaluation of language models across languages. In this paper, we introduce JCoLA (Japanese Corpus of Linguistic Acceptability), which consists of 10,020 sentences annotated with binary acceptability judgments. Specifically, those sentences are manually extracted from linguistics textbooks, handbooks and journal articles, and split into in-domain data (86 %; relatively simple acceptability judgments extracted from textbooks and handbooks) and out-of-domain data (14 %; theoretically significant acceptability judgments extracted from journal articles), the latter of which is categorized by 12 linguistic phenomena. We then evaluate the syntactic knowledge of 9 different types of Japanese language models on JCoLA. The results demonstrated that several models could surpass human performance for the in-domain data, while no models were able to exceed human performance for the out-of-domain data. Error analyses by linguistic phenomena further revealed that although neural language models are adept at handling local syntactic dependencies like argument structure, their performance wanes when confronted with long-distance syntactic dependencies like verbal agreement and NPI licensing.
Dataset: JESC
Description: JESC is a large Japanese-English parallel corpus covering the underrepresented domain of conversational dialogue. It consists of more than 3.2 million examples, making it the largest freely available dataset of its kind.
Language: multilingual
Script: mixed
Domain: captions, TV Channels
Form: text
Size: 3240661 sentences
Provider: Stanford University, Rakuten Institute of Technology, Google Brain
Tasks: machine translation
License: CC BY-SA 4.0
Year: 2018
Authors: Reid Pryzant, Youngjoo Chung, Dan Jurafsky, Denny Britz
Affiliations: Stanford University, Rakuten Institute of Technology, Google Brain
Abstract: In this paper we describe the Japanese-English Subtitle Corpus (JESC). JESC is a large Japanese-English parallel corpus covering the underrepresented domain of conversational dialogue. It consists of more than 3.2 million examples, making it the largest freely available dataset of its kind. The corpus was assembled by crawling and aligning subtitles found on the web. The assembly process incorporates a number of novel preprocessing elements to ensure high monolingual fluency and accurate bilingual alignments. We summarize its contents and evaluate its quality using human experts and baseline machine translation (MT) systems.
Dataset: JDocQA
Description: JDocQA is a large-scale Japanese document-based QA dataset, comprising 5,504 documents in PDF format and 11,600 annotated question-and-answer instances. It requires both visual and textual information to answer questions, and includes multiple question categories and unanswerable questions to mitigate model hallucination.
Language: jp
Script: mixed
Domain: web pages
Form: images
Size: 11600 sentences
Provider: Nara Institute of Science and Technology, RIKEN, ATR
Tasks: question answering
License: CC BY-SA 3.0
Year: 2024
Authors: Eri Onami, Shuhei Kurita, Taiki Miyanishi, Taro Watanabe
Affiliations: Nara Institute of Science and Technology, RIKEN, ATR
Abstract: Document question answering is a task of question answering on given documents such as reports, slides, pamphlets, and websites, and it is a truly demanding task as paper and electronic forms of documents are so common in our society. This is known as a quite challenging task because it requires not only text understanding but also understanding of figures and tables, and hence visual question answering (VQA) methods are often examined in addition to textual approaches. We introduce Japanese Document Question Answering (JDocQA), a large-scale document-based QA dataset, essentially requiring both visual and textual information to answer questions, which comprises 5,504 documents in PDF format and annotated 11,600 question-and-answer instances in Japanese. Each QA instance includes references to the document pages and bounding boxes for the answer clues. We incorporate multiple categories of questions and unanswerable questions from the document for realistic question-answering applications. We empirically evaluate the effectiveness of our dataset with text-based large language models (LLMs) and multimodal models. Incorporating unanswerable questions in finetuning may contribute to harnessing the so-called hallucination generation.
Dataset: Jamp
Description: Jamp is a Japanese Natural Language Inference (NLI) benchmark focused on temporal inference. It was created using a template-based approach, generating diverse examples by combining templates derived from formal semantics test suites with a Japanese case frame dictionary, allowing for controlled distribution of inference patterns.
Language: jp
Script: mixed
Domain: public datasets
Form: text
Size: 10094 sentences
Provider: The University of Tokyo
Tasks: natural language inference
License: CC BY-SA 4.0
Year: 2023
Authors: Tomoki Sugimoto, Yasumasa Onoe, Hitomi Yanaka
Affiliations: The University of Tokyo, The University of Texas at Austin
Abstract: Natural Language Inference (NLI) tasks involving temporal inference remain challenging for pre-trained language models (LMs). Although various datasets have been created for this task, they primarily focus on English and do not address the need for resources in other languages. It is unclear whether current LMs realize the generalization capacity for temporal inference across languages. In this paper, we present Jamp, a Japanese NLI benchmark focused on temporal inference. Our dataset includes a range of temporal inference patterns, which enables us to conduct fine-grained analysis. To begin the data annotation process, we create diverse inference templates based on the formal semantics test suites. We then automatically generate diverse NLI examples by using the Japanese case frame dictionary and well-designed templates while controlling the distribution of inference patterns and gold labels. We evaluate the generalization capacities of monolingual/multilingual LMs by splitting our dataset based on tense fragments (i.e., temporal inference patterns). Our findings demonstrate that LMs struggle with specific linguistic phenomena, such as habituality, indicating that there is potential for the development of more effective NLI models across languages.
Dataset: RuBQ
Description: RuBQ is a Russian knowledge base question answering (KBQA) dataset that consists of 1,500 Russian questions of varying complexity along with their English machine translations, corresponding SPARQL queries, answers, as well as a subset of Wikidata covering entities with Russian labels.
Language: multilingual
Domain: web pages, wikipedia
Form: text
Size: 1500 sentences
Provider: JetBrains Research, ITMO University, Ural Federal University
Tasks: named entity recognition, machine translation
License: CC BY-SA 4.0
Year: 2020
Authors: Vladislav Korablinov, Pavel Braslavski
Affiliations: ITMO University, Saint Petersburg, Russia, JetBrains Research, Saint Petersburg, Russia, Ural Federal University, Yekaterinburg, Russia
Abstract: The paper presents RuBQ, the first Russian knowledge base question answering (KBQA) dataset. The high-quality dataset consists of 1,500 Russian questions of varying complexity, their English machine translations, SPARQL queries to Wikidata, reference answers, as well as a Wikidata sample of triples containing entities with Russian labels. The dataset creation started with a large collection of question-answer pairs from online quizzes. The data underwent automatic filtering, crowd-assisted entity linking, automatic generation of SPARQL queries, and their subsequent in-house verification.