Training Task Experts through Retrieval Based Distillation
Authors: Jiaxin Ge, Xueying Jia, Vijay Viswanathan, Hongyin Luo, Graham Neubig


Reviewer paper summaries: This paper presents Retrieval Based Distillation (ReBase), a method that retrieves data from rich online sources and transforms it into domain-specific data, enhancing data diversity and incorporating Chain-of-Thought reasoning. It tackles the challenge of creating high-quality, task-specific datasets for training deployable models, especially when large, specialized datasets are unavailable.
Summary of strengths: + This paper includes thorough experiments and ablations, highlighting the effectiveness of the proposed method.
+ This paper is well-structured, with clear sections and logical flow, making it easy to follow and understand.
Summary of weaknesses: - **Insufficient robustness and adaptability of data retrieval**. The ReBase method relies heavily on the quality of external data sources. The poor or irrelevant source data may cause bad retrieval results, which threatens the quality of the generated data. At the same time, the existing cosine similarity calculation method may not be accurate enough in some cases, and deep learning models (TF-IDF weighting) or even LLMs can be used for semantic similarity evaluation.
- **Reasonableness of weighting strategy**. The weighting strategy (line 178) uses a simple average to balance the weights of the query score, answer score, and dataset score. This strategy assumes that these three scores are equally important to the final relevance. However, in reality, the relevance of different tasks and datasets may not be balanced. We could draw inspiration from self-attention, dynamically adjusting weights based on the relative values of query, answer, and dataset scores. 
- **Reasonableness of the selection of baseline**. The selected comparison methods (Prompt2Model, Synthesized Data, and Few-Shot Prompting) cover some aspects of data generation and retrieval. However, ReBase uses multiple data sources for retrieval, which itself increases the diversity and richness of the data, while the baseline may only rely on a single data source or a simple generation process, which may not be comparable in terms of data quality and diversity. At the same time, ReBase and the selected baseline methods have obvious differences in application scenarios, goals, and complexity of the methods in some aspects. For example, Few-Shot Prompting has not been fine-tuned, and direct comparison may not be fair enough.
- **Theoretical analysis**. The effectiveness of ReBase is mainly through experiments. The paper does not provide in-depth theoretical proof to explain why this method is effective. There is no theoretical model constructed to describe how the data in the retrieval and transformation process better performance.
Comments, suggestions and typos: Please see the weaknesses above.
Overall assessment scores: 2