Reward Steering with Evolutionary Heuristics for Decoding-time Alignment
Authors: Chia-Yu Hung, Navonil Majumder, Ambuj Mehrish, Soujanya Poria


Reviewer paper summaries: This paper introduces a decoding-time alignment technique that utilizes Instruction Mutations and Reward-guided Beam Search to enhance a trained LLM's alignment with Reward Model scores.
Summary of strengths: 1. The method is train-free, potentially reducing training costs.
Summary of weaknesses: 1. This paper is poorly organized. For instance, the method name "DARWIN" is abruptly introduced for the first time in line 108.

2. The experimental results are not sufficiently convincing:

    2.1. AlpacaEval and MT-Bench are not considered challenging benchmarks. Results on these benchmarks do not adequately demonstrate competitiveness with preference optimization methods.

    2.2. In AlpacaEval, the proposed method performs similarly to or worse than preference optimization methods, such as DPO. In MT-Bench, it performs worse than the simplest baseline, Best-of-N. The method does not consistently achieve significant improvement.

3. Compared to classical decoding methods, the proposed method has higher time complexity. Increasing cost during the inference stage, rather than the training stage, is disadvantageous for practical deployment.

4. Furthermore, this paper focuses on decoding-time alignment, which is inherently insufficient for achieving performance comparable to train-time alignment. Increasing research efforts are directed towards iterative and online training-time alignment to achieve deeper and closer preference alignment. Consequently, the novelty of this work is minor, and its contribution to the community is limited.
Comments, suggestions and typos: N/A
Overall assessment scores: 2