NIRANTAR: Continual Learning with New Languages and Domains on Real-world Speech Data

Authors: Tahir Javed, Kaushal Santosh Bhogale, Mitesh M Khapra
Primary area: datasets and benchmarks
Keywords: continual learning, speech, recognition, datasets, indian languages, multilingual asr

Abstract:
We present Nirantar based on a large-scale effort to collect extempore and conversational speech data from participants spanning 22 languages across diverse locations in India. Given the extensive number of languages and locations involved, data is collected in incremental batches. Each batch introduces new languages, new domains (locations), or both, creating a practical playground for continual learning (CL). Nirantar contains a total of  3250 hours of human-transcribed speech data covering 208 Indian districts across 22 languages, with 1720 hours newly released as a part of this work. The data inflow and resulting multilingual multi-domain episodes are based on real-world data collection rather than simulated episodes commonly found in existing CL datasets. In particular, the amount of data collected and the number of languages and domains involved are not uniform across episodes, reflecting a practical and real-world continual learning scenario. This dataset serves as a playground for training and evaluating CL approaches in three different scenarios: Language-Incremental (LIL), Domain-Incremental (DIL), and the novel Language-Incremental Domain-Incremental Learning (LIDIL), which has not been studied before. To establish the dataset's usefulness, we evaluate several existing CL approaches within these scenarios. Our findings indicate that the behaviour of these algorithms varies across the three scenarios, emphasizing the need for detailed independent studies of each.

Decision: reject
Reviewer scores: 5, 8, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730540674603

thread_type:
official_review

<thread_object>
creator: reviewer_ynaf

<content>
summary:
The paper introduce Nirantar, a large-scale dataset designed for continual learning (CL) in ASR, spanning 22 Indian languages and over 3,250 hours of human-transcribed speech. This dataset includes diverse domains, represented by 208 districts across India, to model real-world multilingual, multidomain CL challenges.

Nirantar's primary contributions include establishing 3 CL scenarios—Language-Incremental Learning (LIL), Domain-Incremental Learning (DIL), and Language-Incremental Domain-Incremental Learning (LIDIL). The LIDIL scenario is novel, introducing both new languages and domains across episodes, a setup that reflects real-world data inflow patterns and has not been previously explored in CL.

To validate the dataset's utility, the authors benchmark several existing CL techniques (including Elastic Weight Consolidation, Experience Replay, and Memory-Aware Synapse) across the 3 scenarios. Experiment results suggest potential areas for improvement in CL approaches for multilingual, multidomain ASR.

strengths:
### Originality
The Nirantar dataset presents a valuable resource for the speech research communities. With 22 Indian languages and over 3,250 hours of human speech, including 1,720 hours newly released as part of this work, and human-annotated transcriptions, the dataset spans a wide variety of topics, domains, and conversational scenarios. The paper introduces a novel Language-Incremental Domain-Incremental Learning (LIDIL) scenario, expanding beyond the traditional Language-Incremental (LIL) and Domain-Incremental (DIL) scenarios that form the three core scenarios of the Nirantar dataset. The inclusion of LIDIL sets a new benchmark for future studies in multilingual and multidomain CL under real-world scenarios.

### Quality
The paper is structured-well and offer a clear overview of the data collection, scenarios, and experimental evaluations on existing CL approaches.

### Clarity
The authors present a clear breakdown of 3 CL scenarios.

### Significance
The dataset’s breadth across 22 Indian languages and 208 districts makes it a valuable resource for developing speech models adaptable to language and domain shifts.

weaknesses:
The paper does not report pre-CL performance metrics, such as Word Error Rate (WER,) for individual episodes across languages and domains, which limits understanding of Nirantar’s distinctiveness as a CL dataset. Specifically, existing ASR datasets like GigaSpeech 2 could be reorganized to create a comparable multi-domain, multilingual LIDIL dataset, so to strengthen its case as a novel CL dataset, additional evidence is needed. For instance, including initial statistics per language, domain, and episode would provide insights into each batch's difficulty across diverse linguistic and regional characteristics, highlighting its uniqueness as a new ASR CL dataset. Additionally, including initial ASR WER would help clarify Nirantar's quality as an ASR dataset.

questions:
### Dataset Quality: 
1. Could you provide details on the quality assurance procedures and any pre-CL performance metrics, such as initial Word Error Rate (WER), to validate transcription accuracy across languages/domains/episodes? Understanding these methods and statistics would clarify the dataset’s reliability and distinctiveness for continual learning applications.

2. Could you elaborate on the procedures to ensure topic and domain diversity across districts, and how this diversity is measured or validated in different episodes? These insights would help demonstrate the dataset’s capacity to represent unique linguistic and regional characteristics, supporting its role as a comprehensive ASR CL dataset.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702410306

thread_type:
official_review

<thread_object>
creator: reviewer_orb3

<content>
summary:
This paper introduces Nirantar, a dataset designed for studying continual learning methods.
The dataset comprises batches of data for various Indian languages and Indian districts.
Various continual learning methods are analyzed on the language incremental, domain incremental and language incremental domain incremental setups.

strengths:
* Continual learning is an important (as it relates to data efficient learning) and unsolved problem.
* The dataset seems valuable for studying continual learning and also beyond that as it supplements existing multilingual ASR corpora with mid/high/low resource data from Indian languages.
* Experimentation is reasonably thorough and tests different CL methods in various setups.

weaknesses:
* It's unclear whether the design of the dataset makes it unique to test CL methods. For example, a similar scheme could be adopted by taking the various releases over time from Common Voice (the domains would not be obtained in a straightforward manner though). The paper does not really demonstrate that the dataset provides value over  taking an existing dataset and slicing it into episodes.
* Related to the previous point, if the methodology to collect the data is the same as for IndicVoices, the contribution for this paper becomes less clear (other than adding more volume).
* A more minor weakness: the writing could be improved, for example by justifying all empirical choices (see clarification questions)

questions:
* 269: “we select the 11 languages having the largest number of hours in Table 1”. Why is the largest number of hours chosen?
* 309: “We create timelines of length τ = 11”. Why was 11 chosen?
* 377: “Average MER: Match Error Rate”. Why not use WER or CER?

presentation: the citation formatting is unusual and can be double checked/fixed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709182440

thread_type:
official_review

<thread_object>
creator: reviewer_xyqq

<content>
summary:
This paper introduces Nirantar, a dataset designed to facilitate continual learning (CL) through real-world, large-scale speech data from 22 Indian languages. Data is gathered incrementally across diverse languages and domains, making it distinct from typical CL datasets that rely on simulated episodes. 

Nirantar support 3 key CL scenarios: Language-Incremental (LIL), Domain-Incremental (DIL), and the novel Language-Incremental Domain-Incremental Learning (LIDIL), which has not previously been studied. The authors’ evaluation of several existing CL algorithms in these scenarios reveals that algorithmic behavior varies significantly across them, suggesting the need for dedicated analyses tailored to each scenario.

strengths:
The dataset’s design closely mirrors real-world CL requirements, providing a valuable resource for research on multilingual, multi-domain, and incremental learning.

The inclusion of both widely used CL scenarios (LIL, DIL) and the novel LIDIL scenario represents a significant contribution, opening new avenues for research.

Evaluations of CL approaches provide useful initial insights, underscoring the complexity and variability of performance across the dataset’s scenarios.

weaknesses:
Figures 3-5 in the report show fluctuating trends, which might indicate an imbalance in the amount of data available for each language. This imbalance could impact the model's performance on specific low-resource languages, potentially limiting its effectiveness in some areas.


While the evaluations provide a starting point, a deeper exploration of algorithmic adaptations or optimizations specifically for LIDIL could enhance the study’s practical impact.

questions:
Given the natural variations in data size per language and domain, what strategies are recommended to handle data imbalance in the CL models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730722075540

thread_type:
official_review

<thread_object>
creator: reviewer_jqup

<content>
summary:
The paper introduces Nirantar, a new benchmark for continual learning (CL) in automatic speech recognition (ASR) that deals with real-world challenges of adding new languages and domains over time. Nirantar is based on a large dataset of 3,250 hours of speech across 22 Indian languages and 208 districts (domains), collected in phases that mimic real-world, incremental data collection.

The main contributions of the paper are:

1) New Data: The paper relies on existing dataset, but contributes an additional 1720 hours of newly collected speech data. 

2) Real Scenarios: the benchmark provides realistic settings for continual learning, including Language-Incremental Learning (LIL), Domain-Incremental Learning (DIL), and a novel Language-Incremental Domain-Incremental Learning (LIDIL), explored in ASR for the first time.

3) Comprehensive Benchmarking: The paper evaluates several popular CL methods (e.g., Experience Replay, Elastic Weight Consolidation) across all scenarios, offering a thorough comparison of CL approaches.

4) Open-Source Contributions: All resources from this work will be made publicly available to encourage further research in multilingual and multi-domain continual learning for ASR.

In summary, Nirantar provides a valuable benchmark that reflects real-world data collection and introduces new challenges for continual learning in speech recognition.

strengths:
- **Novel Contribution:** The work is original, not only contributing newly collected data, but including a novel Continual Scenario for ASR: Language-Incremental Domain-Incremental Learning (LIDIL). This scenario provides a more realistic view of real-world multilingual and multi-domain environments compared to many CL datasets that rely on synthetic or structured data, making its impact relevant beyond the specific languages covered.
Comprehensive Evaluation: The evaluation is extensive, covering popular continual learning methods like Experience Replay, Elastic Weight Consolidation, and Memory-Aware Synapse, along with meaningful metrics. The benchmark also evaluates CL methods across three distinct scenarios (LIL, DIL, and LIDIL), offering a detailed and multi-faceted analysis that allows for a nuanced understanding of model behavior in varying incremental learning conditions.

- **Clarity and Structure:** The paper is structured logically, with clear definitions and explanations of the continual learning scenarios and metrics. The description of data collection, experimental setups, and training procedures is transparent, with steps and design choices explained in detail. This clarity makes it easier for researchers to replicate or build upon the work.

- **High Impact for Low-Resource ASR:** The benchmark is valuable for advancing low-resource ASR research, especially for underrepresented languages like Indian languages. By focusing on multilingual, multi-domain continual learning, the benchmark tackles unique challenges in low-resource ASR.

- **Open Access:** the open release of the dataset and resources under a permissive license further promotes research and collaboration in the field, supporting wider accessibility of multilingual ASR technology.

weaknesses:
- **Limited Baselines:** While the paper evaluates CL methods from replay-based and regularization-based approaches, it excludes architecture-based methods arguing they are impractical for real-world settings as they add parameters for each new language and domain, leading to excessive complexity as episodes increase (lines 328-332). However, this overlooks recent advances in lightweight architecture-based methods like language-specific adapters, which have improved ASR performance even in domains harder than the training data [1]. Including such methods could provide valuable baselines and insights into scalable alternatives that balance parameter efficiency and adaptability.

- **Discussions on Cross-Lingual Transfer:** The dataset is highly diverse, covering 22 Indian languages and 208 districts, but this diversity is underexplored in the analysis. A more granular breakdown of metrics would allow a clearer view of knowledge transferability across closely related languages and domains. For instance, including subgroup analyses by geographic distance, language family, or dialectal variations could highlight the impact of linguistic and regional proximity on model performance. Such details would enhance understanding of the effectiveness of CL techniques in low-resource settings. For example, on line 465, the authors attribute a performance decline to the introduction of a Tibeto-Burman language into the dataset predominantly composed of Indo-Aryan and Dravidian languages. Expanding on this, with metrics grouped by language family or linguistic characteristics, would provide more actionable insights into cross-language transfer. Additionally, including dataset attributes like vocabulary overlap, phonetic variations, or language family classification would better showcase the dataset’s uniqueness and clarify the transferability of findings to other low-resource multilingual datasets.

References

[1] Ferraz, T. P., Boito, M. Z., Brun, C., & Nikoulina, V. (2024). Multilingual Distilwhisper: Efficient Distillation of Multi-Task Speech Models Via Language-Specific Experts. ICASSP 2024.

questions:
1) Could you provide details on how human subjects were compensated and whether explicit consent was obtained for both research and potential industrial use of their voices? Including protocols and consent guidelines in the appendix would be helpful.

2) I recommend providing sample data in the appendix to illustrate the dataset.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730763025316

thread_type:
official_review

<thread_object>
creator: reviewer_6bxs

<content>
summary:
Nirantar is a large-scale dataset featuring 3,250 hours of human-transcribed speech across 22 languages from 208 districts in India, with 1,720 hours newly released in this work. Collected in incremental batches introducing new languages and locations, Nirantar creates a real-world continual learning (CL) setting that supports training and evaluation in Language-Incremental (LIL), Domain-Incremental (DIL), and the novel Language-Incremental Domain-Incremental Learning (LIDIL) scenarios. Initial evaluations show that CL approaches behave differently across these scenarios, highlighting the dataset's utility for diverse CL research.

strengths:
- The dataset encompasses a diverse range of languages across India.
- They offer three distinct continual learning settings: Language-Incremental (LIL), Domain-Incremental (DIL), and the novel Language-Incremental Domain-Incremental Learning (LIDIL).
- The paper is well-written and easy to follow, with a comprehensive analysis included.

weaknesses:
- The first two settings, Language-Incremental and Domain-Incremental, have been explored in previous studies, including CL-mARS, which covers various CL methods in multilingual contexts (https://arxiv.org/pdf/2310.16931), and works by Sadhu et al. (https://www.isca-archive.org/interspeech_2020/sadhu20_interspeech.pdf), Chang et al. (https://arxiv.org/abs/2104.01616), and Li et al. (https://arxiv.org/abs/2302.01496), which investigate incremental domain setups. In light of these prior studies, the contribution here feels mostly incremental in these areas.

- The dataset focuses exclusively on languages from India, which provides rich diversity, though it could be beneficial to include an analysis of language relationships and similarities to understand their potential impact on performance. Expanding the dataset with additional languages from public datasets like FLEURS or CommonVoice, especially those with minimal similarity, would further strengthen the study.

- Additionally, task order is determined randomly, yet exploring the effects of task ordering on performance or providing an analysis that considers language similarity, task order, and performance metrics could add valuable insights.

- Finally, while the study examines rehearsal-based and regularization-based CL methods, incorporating architecture-based approaches, such as Progressive Neural Networks (PNN), Piggyback (PB), and Learning to Prompt (L2P), could offer a more comprehensive view, as highlighted in prior literature.

questions:
Refer to weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1732987327120

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Gentle reminder to all reviewers

comment:
We would like to sincerely thank all the reviewers for their thoughtful feedback and constructive suggestions. We have made every effort to address the comments and improve the manuscript. We are also grateful to reviewers **JqUP** and **oRB3** for their prompt replies to our initial comments. As the deadline for the author-reviewer discussion phase (**Dec 2nd**) is approaching, we kindly request reviewers **6Bxs**, **XYQq**, **YNAF**, and **oRB3** to check our responses and let us know if any further clarifications are needed from our side. If all the concerns are addressed we request the reviewers to appropriately increase their scores.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733226174861

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal Summary

comment:
We sincerely appreciate the reviewers for their insightful and constructive feedback. Their suggestions, including exploring different task orderings and integrating adapters, have been instrumental in enhancing the quality and depth of our manuscript. Below, we present a summary of the key points addressed in our rebuttal, as highlighted by the reviewers:
 
 - **(6Bxs, YNAF)**: Highlighted limitations of existing datasets (e.g., FLEURS, Common Voice, GigaSpeech 2) for studying DIL and LIDIL scenarios as they lack domain information. Presented a comparative analysis, showing Nirantar as the only dataset with real-world, manually transcribed, and episodic data designed for Continual learning (CL). [[Table** 3]](https://anonmyous.objectstore.e2enetworks.net/nirantar-related-datasets.png)
 
 - **(6Bxs)**: Addressed randomness in task order by including results for two additional permutations for the LIDIL scenario, showing consistency across different orderings. [[Figures** 7 to 9]](https://anonmyous.objectstore.e2enetworks.net/nirantar-permute-lidil.png)
 
 - **(6Bxs, JqUP)**: Conducted experiments with adapters in the LIL setting, showing their effectiveness in preventing catastrophic forgetting during training. However, we still caution against the possibility of parameter bloat in scenarios when a large number of adapters are added (e.g., in DIL and LIDIL scenarios) and the limited scalability of adapters compared to ER-based approaches. [[Figure** 10]](https://anonmyous.objectstore.e2enetworks.net/nirantar-lil-adapters.png)
 
 - **(JqUP)**: Explored cross-lingual transfer in LIDIL scenario for Indo-Aryan and Dravidian languages, revealing stable and better learning for Dravidian languages. [[Figures** 11 to 13]](https://anonmyous.objectstore.e2enetworks.net/nirantar-crosslingual-lidil.png)
 
 - **(JqUP)**: Clarified detailed participant instructions, consent and guidelines, which shall be duly added to the final version of the manuscript.
 
 - **(XYQq)**: Addressed the comment on "fluctuating trends", explaining that they reflect challenges of continual learning (CL) methods, such as adapting to significantly different new languages (e.g., Manipuri in LIL scenario) rather than mere data imbalance.
 
 - **(XYQq)**: Discussed strategies for addressing data imbalance.
 
 - **(6Bxs, oRB3)**: Highlighted Nirantar's unique contributions, including the novel LIDIL setting, real-world episodic designs, longer episodic sequences, and evolving evaluation benchmark that distinguish it from existing datasets. Emphasized that Nirantar's focus on CL, novel IL scenarios, and tailored evaluation frameworks mark it as a distinct advancement beyond IndicVoices.
 
 - **(oRB3)**: Justified empirical choices in LIL, DIL, and LIDIL scenarios and clarified the use of Match Error Rate (MER) over WER/CER for consistent evaluation. Additional plots for WER and CER will be included in the final manuscript.
 
 - **(YNAF)**: Provided statistics for language, domain, and episodes [[Table** 4]](https://anonmyous.objectstore.e2enetworks.net/nirantar-dataset-detailed.png) and demonstrated diversity in episodes through evolving vocabulary and domains. [[Figures** 14 to 15]](https://anonmyous.objectstore.e2enetworks.net/nirantar-domain-vocab-evolution.png)
 
 - **(YNAF)**: Added details of multi-level QA processes for data collection and transcription to ensure minimal errors and included pre-CL performance metrics using language-specific models. [[Table** 4]](https://anonmyous.objectstore.e2enetworks.net/nirantar-dataset-detailed.png)
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces Nirantar, a large-scale dataset and benchmark for continual learning (CL) in automatic speech recognition (ASR), focusing on real-world challenges of incremental data collection. Nirantar comprises 3,250 hours of human-transcribed speech spanning 22 Indian languages and 208 districts (domains), with 1,720 hours newly contributed in this work. The work claimed following contribution: 1) a dataset collected in a realist scenario mimicking real-world multilingual and multidoma in CL challenges, where audios were collected incrementally across languages and locations. The dataset support various CL scenarios including Language-Incremental Learning (LIL), Domain-Incremental Learning (DIL), and Language-Incremental Domain-Incremental Learning (LIDIL); 2) initial benchmarking the dataset on several existing CL methods (e.g., Experience Replay, Elastic Weight Consolidation, Memory-Aware Synapse) across these scenarios; authors also promised to make the datasets along with evaluation benchmarks publicly available


Strength of this paper
- Built a dataset with diversity in domains/languages and reflecting real-world multilingual and multi-domain continual learning (CL) challenges, supporting incremental learning and adaptation across diverse languages and domains The dataset allows exploration of a new scenario, Language-Incremental Domain-Incremental Learning (LIDIL), beyond traditional Language-Incremental (LIL) and Domain-Incremental (DIL) settings. LIDIL offers a more realistic view of multilingual, multi-domain challenges and new research problems.
- Authors also conduct benchmarking and evaluation to evaluates several established CL methods, including Experience Replay, Elastic Weight Consolidation, and Memory-Aware Synapse across various scenarios (LIL, DIL, and LIDIL), understand the characteristics of the datasets, and provide analysis and insight for future research
- The dataset and resources will be released under CC-BY-4.0 license, encouraging collaboration and broader accessibility in multilingual ASR research. The dataset will be a valuable resource for the speech research community, setting a new benchmark for multilingual, multidomain, and incremental learning in automatic speech recognition (ASR), supplementing existing multilingual ASR corpora , and advancing research in various problems such as low-resource ASR


Weakness of this paper

Several reviewers raised few concerns/limitations of this paper. By addressing these limitations, the paper could strengthen its experiment and expand impact.

- Scope of the dataset and generalizability: the dataset exclusively focuses on Indian languages, which provides rich diversity but limits its generalizability. Expanding the dataset with additional languages from resources like FLEURS or CommonVoice, especially those with minimal similarity to Indian languages, would improve its applicability and relevance. The paper has limited exploration in several important aspects, e.g.,  cross-lingual transfer or language relationships to have subgroup analyses based on geographic distance, language family, dialectal variations, vocabulary overlap, or phonetic variations, and investigating algorithmic adaptations or architecture-based approaches like Progressive Neural Networks (PNN), Piggyback (PB), and Learning to Prompt (L2P).
- Experiment settings could be improved: the task order is determined randomly, but analyzing the effects of task order and incorporating language similarity into task sequencing could add valuable insights; the dataset’s imbalance in data availability across languages may skew results and it's worthwhile to have some discussion about the effect.
- Concerns about novelty: the dataset’s design does not convincingly show the added value of Nirantar and its uniqueness for testing CL methods, as similar schemes could be applied to other datasets, such as phased releases of CommonVoice. Many of the works were also built upon existing efforts such as IndicVoices, Language-Incremental (LIL) and Domain-Incremental (DIL) setting, with some incremental changes.