Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Safety Self-Alignment

Authors: Zhili Liu, Yunhao GOU, Kai Chen, Lanqing HONG, Jiahui Gao, Fei Mi, Yu Zhang, Zhenguo Li, Xin Jiang, Qun Liu, James Kwok
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: mixture of experts, lora, chain of thoughts, LLM safety

Abstract:
As the capabilities of large language models (LLMs) have expanded dramatically, aligning these models with human values presents a significant challenge. Recent studies demonstrate that powerful LLMs can achieve self-alignment by either correcting their initial unsafe responses or autonomously ranking answers without human intervention. In this work, we identify two key limitations: first, they rely on the assumed emergent capabilities of LLMs, and second, they discard all intermediate reasoning steps when aligning the model with updated answers. To address these challenges, we propose a novel self-alignment method that utilizes a Chain of Thought (CoT) approach, termed AlignCoT. This method encompasses stages of Question Analysis, Answer Guidance, and Safe Answer production. It is designed to enable LLMs, even smaller and weaker models like 7B LLMs, to produce high-quality, safe responses. Furthermore, we introduce the Mixture of insighTful Experts (MoTE) architecture, which applies mixture of experts to enhance each component of the AlignCoT process, markedly increasing alignment efficiency. The MoTE approach not only outperforms existing methods in aligning LLMs with human values but also highlights the benefits of using self-generated data, revealing the dual benefits of improved alignment and training efficiency.

Decision: reject
Reviewer scores: 5, 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1730428247534

thread_type:
official_review

<thread_object>
creator: reviewer_tqaf

<content>
summary:
This paper proposes a self-alignment method that utilizes a Chain of Thought approach (AlignCoT) to address two challenges: 1. Recent studies rely on the assumed emergent capabilities of LLMs.  2. Recent studies discard all intermediate reasoning steps, when aligning the model with updated answers.

strengths:
This paper shows the promising performance of AlignCoT and MoTE based on Alpaca-7B and Wizard-Vicuna-Uncensored 7B.

weaknesses:
1. In Table 1, it is unclear if the results are based on Alpaca-7B or Wizard-Vicuna-Uncensored 7B.
2. For evaluation, it would make this paper stronger if the authors could provide more details about how human annotators provide a final verification for precise results. For example, what are the annotation guidelines provided to annotators, how disagreements between annotators were resolved, how human annotators are picked, and how many human annotators are in the verification process?
3. It would be better if the authors could show if their method has the same effects in bigger LLMs (such as picking some LLMs with 13B or even larger).
4. The metrics, help, harm, and harm-help are too vague and may be sensitive to human annotators. It would make this paper stronger if the authors could provide better metrics. For example, how the authors deal with the annotator bias or subjectivity in current metrics.

questions:
Check the weaknesses section above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730579997000

thread_type:
official_review

<thread_object>
creator: reviewer_ak5g

<content>
summary:
The paper introduces AlignCoT a self-alignment method for generating dataset that facilitating the safe responses of LLMs. Furthermore, the paper also proposes Mixture of Insightful Expert (MoTE) that extends a mixture of experts (MoE) framework with a share expert to facilitate the exchange of knowledge across different stages of AlignCoT. The two contributions are relatively orthogonal, but the sufficient experiments are conducted to demonstrate the necessity of both methods on HH-RLHF benchmark.

strengths:
1. The paper demonstrates comprehensive experiments, draws comparison with enough baselines, and carefully examines multiple design choices in both AlignCoT and MoTE. 
2. The paper is very-well organized and the presentation is clear.

weaknesses:
1. The benefit of the shared expert in MoTE is a bit unclear. It sounds like a mixture of MoE with single expert LoRA. The authors mentioned information exchange. This is very generic, does MoTE extends to other self-alignment methods?
2. The novelty from each method is relatively limited. For example, AlignCoT is just one of CoT frameworks. A large amount of CoT frameworks have been explored these days. Also MoTE sounds like a mixture of MoE with single expert LoRA. Although, tricks like step skipping and the process to streamline the training of MoTE have been explored. The overall novelty of MoTE compared to MoE is relatively limited.

questions:
1. Do author expect MoTE to be extended to other self-alignment methods that involves multiple experts? Could the authors provide more insight on why shared expert can improve the performance?
2. I am a bit confused by single-step inference (line 161) and figure 2. Why the paper aims for improving the single-step inference? Single_Sequence dataset is crafted for Single-step Inference setting, but MoTE can even achieve better performance. Could the author elaborate more on this result?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730613034050

thread_type:
official_review

<thread_object>
creator: reviewer_s7t5

<content>
summary:
This paper presents a method to improve the safety alignment of LLMs using AlignCoT, a Chain-of-Thought (CoT) based prompting technique that structures datasets with intermediate reasoning steps, and MoTE, a multi-LoRA approach where different parts of the sequence embedding are processed by specialized LoRA modules. The authors train and evaluate their approach on the Safe-RLHF and HH-RLHF datasets, assessing both helpfulness and harmlessness in single-step and multi-step inference settings.

strengths:
- The paper provides detailed ablation studies, clearly analyzing the contributions of each component, including AlignCoT and MoTE, to the overall performance.

weaknesses:
- The formulations in Section 3, including Equation 1, are incomplete and should be revised with more standard notations. For instance, rather than using 𝑥 for context-response pairs, (𝑥,𝑦) should be used, and the source (policy/distribution) of 𝑥 should be clarified. Additionally, the justification for modeling CoT as a joint probability distribution between different steps is not clear. If this formulation is kept, parts of Appendix A should be integrated into the main text for clarity.

- The related work section lacks coverage of self-alignment methods and other fine-tuning approaches based on CoT. A dedicated subsection comparing AlignCoT with previous work is needed to clarify its novelty.

- Despite the point above, Section 3 provides minimal technical discussion. The formulation could be more effectively replaced with a figure illustrating the structural prompting process.

- The "Efficient training of MoTE" section offers little novelty, as its function appears similar to existing masked attention techniques.

- The paper's clarity needs improvement. For example:
  - It is unclear which 7B LLM is referred to in Line 177. Is it Alpaca-7B?
  - The metrics used in Figure 2 are not specified or explained in the text.

- Important details about the methods used in training and evaluation should be moved from Appendix B to the main text.

- The paper's reliance on Alpaca-7B as a baseline is concerning, especially given the rapid advancements in LLM alignment and training. Using an older model might mean addressing issues that have already been mitigated in more recent models. This raises the question of whether the improvements demonstrated are still relevant in the context of modern LLMs. It would be more convincing to evaluate the method on current models like Llama-3-8B, Llama-2-7B, or Mistral-7B, which better reflect the state-of-the-art in language model performance and alignment challenges.

- The evaluation presented in Table 1 is somewhat narrow, relying heavily on PKU-SafeRLHF and HH-RLHF, which are closely related datasets, which limits the evaluation's breadth. Cross-testing results should be included, and other benchmarks like XSTest, WildChat, or win rate comparisons against Llama-Guard or Perspective API (less recommended) should be considered.

- Unless I overlooked it, the code for the proposed methods has not been provided. This makes it difficult to verify the results or apply the techniques described in the paper.

questions:
See Weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730773844615

thread_type:
official_review

<thread_object>
creator: reviewer_dhul

<content>
summary:
This paper proposes a new AlignCoT dataset, which includes the original question, question analysis, answer guidance, and final safe answer by prompting the model with Chain of Thought (CoT). A new SFT method is also introduced, based on the AlignCoT dataset. This method incorporates multiple LoRA matrices corresponding to different parts: question, question analysis, answer guidance, and answer. Through comparative analysis, MoTE demonstrates better alignment efficacy compared to benchmark alignment techniques.

strengths:
Strengths:

1. The innovative idea of using multiple LoRA experts.

2. Leveraging CoT and introducing AlignCoT.

weaknesses:
Weaknesses:

1. The writing could be improved.

2. The MoTE method is heavily reliant on the AlignCoT dataset, making it difficult to extend to other datasets.

questions:
I am unclear about the relationship between \(x\), \(x_a\), \(x_g\), \(y_{\text{cot}}\), and MoTE. From my understanding, this paper uses a CoT prompt to produce the question analysis first, followed by answer guidance, and finally the safe answer. Additionally, the paper uses this generated data as fine-tuning data for SFT on this model. Is the data generated by the model used to retrain the same model?

- Line 128: It is not clear whether the question analysis, answer guidance, and final safe answer are produced by the same model, or if question analysis and answer guidance can be generated by other models.

- Line 128: Where does the training data (\(x_a\), \(x_g\), \(y_{\text{cot}}\)) for SFT come from?

- Figure 2: Some subfigure titles have a period while others do not. Additionally, the x-axis is not clearly labeled.

- Lines 182-210: Should this part be located in the Experiments section? It seems to be an analysis of the method.

- Line 185: An introduction to the metrics would be helpful. In Table 1, there is no explanation for "Help," "Harm," and "Harm-Help."

- Line 367: What fine-tuning datasets are used for the baselines? \(D_a\), \(D_g\), \(D_{\text{ans}}\), etc.? If it is \(D_{\text{ans}}\), since MoTE is based on the AlignCoT dataset, this could lead to an unfair comparison.
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary**: This paper proposes MoTE (Mixture of insighTful Experts), a framework that combines the CoT reasoning method with a Mixture of Experts (MoE) architecture for improving safety alignment in LLMs. The authors introduce AlignCoT, a structured dataset that includes intermediate reasoning steps such as question analysis, answer guidance, and safe response generation, allowing fine-tuning through self-generated data. The MoTE framework builds on this by assigning specific LoRA experts to each step in the AlignCoT process, introducing shared experts and step-skipping for training efficiency. 

**Strengths**:
- Using multiple LoRA experts is a novel idea
- Detailed ablation studies
- The paper is very-well organized and the presentation is clear

**Weaknesses**:
- Lack of clarity in the writing: Most reviewers asked for clarifications on the method and pointed out that the writing could be improved, to an extent where the paper would benefit from a heavy full-on revision. Moreover, Reviewer `S7t5` asked for extended coverage I the related work section in order to clarify the approach’s novelty compared to prior work.
- Limited novelty: Reviewers `S7t5` and `aK5g` complain about limited novelty — AlignCoT is just one of many CoT frameworks, novelty of MoTE compared to MoE is relatively limited, and moreover the benefit of the shared expert in MoTE is a bit unclear compared to MoE.
- Poor baseline comparison: Multiple reviewers asked for comparisons against newer more powerful models that better reflect the state-of-the-art in language model performance and alignment challenges, and are concerned about whether the results would hold in those cases.
- Limited reproducibility/generalization capabilities: Multiple reviewers pointed out ways in which the method may not be easy to replicate or generalize— reliance on the AlignCoT dataset, no code provided, narrow evaluation on PKU-SafeRLHF and HH-RLHF.

**Recommendation**:
Overall, the paper presented an interesting idea of using multiple LoRA experts and had detailed ablation studies, but unfortunately there are big concerns regarding writing clarify, novelty, inappropriate baseline comparison, and limited generalization. After the rebuttal, reviewers agree that the paper would benefit from a significant revision. As such, I vote to Reject this paper.