COrAL: Order-Agnostic Language Modeling for Efficient Iterative Refinement

Authors: Yuxi Xie, Anirudh Goyal, Xiaobao Wu, Xunjian Yin, Xiao Xu, Min-Yen Kan, Liangming Pan, William Yang Wang
Primary area: foundation or frontier models, including LLMs
Keywords: autoregressive large language modeling, decoding, iterative refinement

Abstract:
Iterative refinement has emerged as an effective paradigm for enhancing the capabilities of large language models (LLMs) on complex tasks. However, existing approaches typically implement iterative refinement at the application or prompting level, relying on autoregressive (AR) modeling. The sequential token generation in AR models can lead to high inference latency. 
To overcome these challenges, we propose **C**ontext-Wise **Or**der-**A**gnostic **L**anguage Modeling (COrAL), which incorporates iterative refinement directly into the LLM architecture while maintaining computational efficiency. Our approach models multiple token dependencies within manageable context windows, enabling the model to perform iterative refinement internally during the generation process. Leveraging the order-agnostic nature of COrAL, we introduce sliding blockwise order-agnostic decoding, which performs multi-token forward prediction and backward reconstruction within context windows. This allows the model to iteratively refine its outputs in parallel in the sliding block, effectively capturing diverse dependencies without the high inference cost of sequential generation.
Empirical evaluations on reasoning tasks demonstrate that COrAL improves performance and inference speed, respectively, achieving absolute accuracy gains of $4.6$\% on GSM8K and $4.0$\% on LogiQA, along with inference speedups of up to $3.9\times$ over next-token baselines. Preliminary results on code generation indicate a drop in pass rates due to inconsistencies in order-agnostic outputs, highlighting the inherent quality--speed trade-off.

Decision: reject
Reviewer scores: 3, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730517940150

thread_type:
official_review

<thread_object>
creator: reviewer_lrim

<content>
summary:
The authors propose a new decoding method, called CORAL, which can speed up the decoding process and maintain (or upraise) the performance of the model in some tasks. CORAL has 2 parts: prediction and verification. The experiment shows that the verification part can help the model to generate more accurate results. CORAL also designed a strategy named "multi-forward" to speed up the decoding process (although it may hurt the performance). The result shows that the CORAL is useful in math problems but is useless in the code generation task.

strengths:
1. The topic of the paper is interesting, transformer-based model do have the problem of slow decoding speed. 

2. It make a good balance between the speed and the performance.

weaknesses:
1. The improvement of the CORAL is not generalizable enough. It only works well in the some math/logic problems but not in the code generation task.

2. Although the speed of the decoding process is improved, it needs to use more GPU memory (and "waste" some computation because of verification and multi-forward) to achieve this. So it is not friendly to equipment that most people use.

questions:
In eq.8 entropy is always positive, so -H(x) is always negative and exp(-H(x)) is always less than 1. So min(a,a*exp(-H(x))) is always a*exp(-H(x)).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730662675128

thread_type:
official_review

<thread_object>
creator: reviewer_oenx

<content>
summary:
The authors introduce a novel decoding strategy combining autoregressive modelling with ROBERTA-like order agnostic refinement. Given a partial sequence, they predict multiple tokens ahead, which they subsequently refine using ROBERTA-like denoising autoencoder. The authors see performance improvements on GSM8K and LogiQA and poor performance on code generation.

strengths:
* the authors propose an interesting paradigm and show that it has promise for reducing computational cost and enhancing performance in certain settings
* the method is applicable to autoregressive pretrained language models and seems to improve their performance in certain settings
* the authors provide a quite extensive ablation study for their method
* the paper contains some beautiful figures such as figure (2) and (3). Even though Figure (2) is a little bit unclear to me. Why are there seemingly different offsets for the refinements and why is there not much visual seperation inbetween forward prediction and refinement?

weaknesses:
* pseudo-code for Algorithm 1 is provided without walking through the pseudo-code
* in the experimental section the baselines are not described in enough detail, just AR. the proposed method requires finetuning, are the AR baselines also finetuned on the tasks?
* the by-far-best performance is achieved using the w\o multi-head prediction ablation, which is not the proposed method and thus weird. I assume this variant suffers from increased computational cost compared to the proposed method. It would be interesting to compare this ablation with a method from the related work that has a similar computational cost.
* comparison to refinement methods from the related work is missing
* a somewhat non-standard notation for expected values is used. their subscripts seem to be used much like in summations, but usually subscripts at an expected value are used to indicate over which distribution the expectation is taken: e.g., equation (1) and equation (3)

questions:
It would be really interesting to check how much performance is lost by starting from a pretrained model as compared to full training a method employing coral from scratch. Do you think that some performance is left on the table because you start from a pretrained model? 

In the main result part, to increase my rating I would like to see a comparison to other interative refinement methods that have a similar computational cost as the w/o multi-token prediction variant of the proposed method and also a more detailed description of the autoregressive baseline.

Suggestion: Maybe it would be a good idea to incorporate an application in which this method shines. E.g., by looking into domains that can benefit from the order-agnostic aspect such as protein language modelling.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692218214

thread_type:
official_review

<thread_object>
creator: reviewer_dasr

<content>
summary:
The paper proposes COrAL(Context-Wise Order-Agnostic Language Modeling), a novel architecture for language modeling that enhances efficiency in iterative refinement, aiming to reduce inference latency in large language models (LLMs). Traditional autoregressive models, which generate text sequentially, struggle with efficiency due to the natural linear time complexity in inference. COrAL incorporates iterative refinement directly into the model, allowing multi-token generation and backward reconstruction within manageable context windows. This order-agnostic approach enables simultaneous forward and backward decoding within sliding context windows, effectively accelerating inference and improving performance on reasoning tasks. Empirical tests show significant improvements in both accuracy and inference speed, demonstrating COrAL's promise in capturing diverse token dependencies without the high latency typical of AR models. However, challenges remain, such as reduced performance in code generation due to output consistency issues, indicating areas for further refinement.

strengths:
- Improved Efficiency and Performance:  COrAL’s order-agnostic framework allows simultaneous forward and backward processing, significantly reducing inference latency compared to traditional autoregressive models. Compared to the ablated baselines, empirical results on datasets like GSM8K and LogiQA demonstrate notable accuracy gains, confirming the model’s effectiveness in complex reasoning tasks.
- Scalably Adaptable from Existing Models: By using context-wise modeling and target-aware positional encoding, COrAL manages to enhance dependency capture without substantially increasing computational resources, making it feasible for deployment in large-scale applications, even with existing large language models with only minor adaptation.

weaknesses:
- Lack of survey of some (maybe kind of obsolete yet important) existing methods: This method resembles Scheduled Sampling in multiple aspects, yet it severely lacks the acknowledgement of this method (no citation nor even mentioning). It shares many ideas and practices with SS, necessitating a deeper analysis on the connection and differences between the method. For example, I'd recommend the authors to emphasize the capability of the proposed method on semi-parallel, refinitive generation, whereas SS was originally only proposed for improvements of performance in sequential generation. 
- Lack of deeper discussion on the theoretical insights: I appreciate the authors' awesome work in presenting and delivering the empirical results, but I presume it would appeal the community more if some insightful conclusions can be presented alongside the experiment observations.

questions:
The clarity of the paper is good, it's easy for people to follow generally. I don't have further questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730694478287

thread_type:
official_review

<thread_object>
creator: reviewer_u9ey

<content>
summary:
This paper proposes Context-Wise Order-Agnostic Language Modeling (COrAL), which incorporates iterative refinement directly into the LLM architecture while maintaining computational efficiency. Empirical evaluations on reasoning tasks demonstrate that COrAL improves performance and inference speed, and results on code generation indicate a drop in pass rates due to inconsistencies in order-agnostic outputs, highlighting the inherent quality–speed trade-off.

strengths:
- This paper is well-writen and easy to follow. 
- The performance on logical reasoning tasks are good.

weaknesses:
- I think this paper is similar to the other type of works, i.e., speculative decoding, what the difference between them? 
- The noverty is limited, since the specific ways for iterative refinements, the training methods to learn correction, are borrowed from previous works. 
- The significant one: this method seems to only work in specific tasks, the logical reasoning tasks in this paper. However, we always focus on the generalization of current language models, i.e., the competitive on a wide range of tasks.

questions:
- If the way to generate tokens in the first step is different from that in the process of iterative refinements? Are there any better methods to generate draft tokens.
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary:** 

The paper introduces COrAL, a framework designed to integrate iterative refinement directly into LLMs while maintaining computational efficiency. COrAL addresses limitations of autoregressive models, such as high inference latency and sequential dependency, by modeling token dependencies within context windows in an order-agnostic manner. It combines forward multi-token prediction and backward reconstruction to enable efficient sliding block-wise decoding, achieving improvements in both accuracy and inference speed. 

**Strength:**
- COrAL’s order-agnostic framework enables simultaneous forward and backward processing.
- The method enhances dependency modeling with minimal computational overhead.
- The authors conduct extensive ablation studies, showcasing the robustness of their approach in balancing speed and performance.

**Weakness:**
- The paper lacks discussion and citations for related works.
- While the empirical results are robust, the paper could benefit from deeper theoretical discussions to provide insights beyond experimental observations.
- While COrAL excels in math and logic tasks, it struggles with code generation, limiting its general applicability.