Training Task Experts through Retrieval Based Distillation

Authors: Jiaxin Ge, Xueying Jia, Vijay Viswanathan, Hongyin Luo, Graham Neubig
Primary area: foundation or frontier models, including LLMs
Keywords: Retrieval, Distillation, Task-Expert

Abstract:
One of the most reliable ways to create deployable models for specialized tasks is to obtain an adequate amount of high-quality task-specific data. However, for specialized tasks, often such datasets do not exist. Existing methods address this by creating such data from large language models (LLMs) and then distilling such knowledge into smaller models. However, these methods are limited by the quality of the LLMs output, and tend to generate repetitive or incorrect data. In this work, we present Retrieval Based Distillation (ReBase), a method that first retrieves data from rich online sources and then transforms them into domain-specific data. This method greatly enhances data diversity. Moreover, ReBase generates Chain-of-Thought reasoning and distills the reasoning capacity of LLMs. We test our method on 4 benchmarks and shows that our method significantly improves performance by up to 10.76% on SQuAD, 1.37% on MNLI, and 1.94% on BBH.

Decision: reject
Reviewer scores: 3, 5, 5

Reviews and discussion:
<thread>
creation date:
1730440796673

thread_type:
official_review

<thread_object>
creator: reviewer_xczu

<content>
summary:
The paper introduces ReBase, a method retrieves from online sources and transforms them into sample-style datas. Using the online datas ,the method can enhances data diversity and also generates the CoT datas by distilling LLMs. The Experimental results show the method improves performance on SQuAD, MNLI and BigBench-Hard.

strengths:
The motivation is practical and significant, offering a cost-effective solution for obtaining high quality domain-specific datas using online datas.

Paper demonstrates the performance improvement of the ReBase method in multiple benchmarks, such as achieving 7.8%, 1.37%, and 1.94% performance improvements on SQuAD, MNLI, and BigBench Hard datasets, respectively. These results demonstrate the effectiveness of the method.

weaknesses:
Although ReBase performs well on specific benchmark tests, its generalization ability on other types of tasks has not been fully validated. Because the three datasets involved in the article are too simple and early, there is a risk of label exposure. Meanwhile, these three datasets are all generic datasets and do not match the domain specific mentioned in the article.

For  specific domain and unseen types of data, online data may not have similar data, and enhancing through retrieval may not bring real benefits.

Although the ReBase method improves data diversity, there is insufficient discussion in the article about the potential data bias and quality issues that may be introduced during retrieval and conversion processes

questions:
How does the ReBase method generalize across different types of tasks and domains? Is there a plan to validate the effectiveness of this method on other types of tasks?

How to ensure data quality and reduce potential biases during the process of retrieving and converting data? Is there a mechanism to monitor and correct these deviations?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730517085771

thread_type:
official_review

<thread_object>
creator: reviewer_pxv3

<content>
summary:
The paper proposes a retrieval-based task-specific data construction method aimed at improving the quality of task-specific data generated by large language models (LLMs). Specifically, a set of databases was first constructed from Hugging Face. Then, a retrieval method was designed to obtain data based on instructions and examples. The retrieved data is subsequently transformed using LLMs and prompts to make it directly usable for training. The authors test their method on 4 benchmarks and results show that the method significantly improves performance by up to 7.8% on SQuAD, 1.37% on MNLI, and 1.94% on BigBench-Hard.

strengths:
- Data Diversity: ReBase enhances data diversity by retrieving data from multiple online sources
- Reasoning Distillation: ReBase introduces a Chain-of-Thought transformation phase, allowing smaller models to be trained using the reasoning processes generated by larger models, which is particularly useful for reasoning tasks.
- Performance Improvement: In multiple benchmark tests, ReBase has demonstrated significant performance improvements compared to traditional data generation methods.

weaknesses:
- The LLM used in the text is Llama3-8B. Have you tried any other models (different size of params, different model families) ?
- Processing large amounts of data into datastore and retrieve from them can be computational costly, esp. computing cosine similarities. Any efforts to make such computation less so?
- Have there been any assessments of the data quality after TRANSFORMATION, such as issues like hallucinations, the CoT is complete and correct, etc.?

questions:
- Could the experiments be conducted on more baseline LLMs? so to verify its effectiveness on a broader scale (different model sizes, different model families)
- Propose some methods to improve the efficiency of the proposed methods.
- Evaluate the quality of the data after the transformation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730726871903

thread_type:
official_review

<thread_object>
creator: reviewer_5o5m

<content>
summary:
A retrieval-augmented synthetic data generation framework for strong-to-weak distillation.

strengths:
S1. The paper presents the efficacy of retrieval augmentation for new data generation.

S2. The proposed method demonstrates improvement over baseline methods.

weaknesses:
W1. Missing important related work "Retrieval-Augmented Data Augmentation for Low-Resource Domain Tasks"[1].

W2. Usage of the HuggingFace datasets may cause contamination if not explicitly taken care of.

W3. The choice of teacher model is unclear as the ability varies for tasks such as coding or math. How is coding or math correctness assessed for the synthetic samples?

W4. It is vague how the data sources are selected.


[1] Seo, Minju, et al. "Retrieval-augmented data augmentation for low-resource domain tasks." arXiv preprint arXiv:2402.13482 (2024).

questions:
Please see the weaknesses.
</content>
</thread_object>

</thread>


Metareview: