Fine-Tuning Language Models with Collaborative and Semantic Experts
Authors: Jiaxi Yang, Binyuan Hui, Min Yang, Jian Yang, Lei Zhang, Junyang Lin, Chang Zhou


Reviewer paper summaries: This work observes that the direct training of all instruction data of different fields cannot guarantee better performance than individual training, which signifies the current SFT suffers from conflicts and capability loss. To this end, this work introduces Collaborative and Semantic Experts (CoE) to first train an individual MoE for each field (Expert Training), and then design a routing algorithm to integrate field-related MoE into a unified CoE (Experts Leveraging). Empirical studies across multiple benchmarks show that CoE can outperform the traditional SFT.
Summary of strengths: 1. the paper is easy to follow and the motivation is clearly stated
2. the experiment includes multiple types of tasks
3. the proposed CoE outperforms individual experts and SFT, which validates its effectiveness
Summary of weaknesses: 1. the novelty seems incremental.  Collaborative and Semantic Experts is adapted from MoE and the main difference is the new routing strategy to integrate individual experts
2. lack essential baselines. The main baseline models are *Expert-General, Expert-Math, and Expert-Coding*. If I understood correctly, these models are trained on respective datasets. First, these models only access a fraction of the entire datasets and therefore the comparison to CoE is not fair.  Second, other competitors such as Branch-Train-Mix and DARE mentioned in the related work should be included for properly valuing the contribution of the proposed method
3. more backbone models should be included. Only LLaMA-7B is evaluated in the experiments. Although this point is mentioned in the limitation section, experiments on other LLMs can make the argument more solid
Comments, suggestions and typos: * can you explain the motivation and intuition at the beginning of Section 4.3
Overall assessment scores: 2.5