Title: Revisiting Random Generation Order: Ordinal-Biased Random Training for Efficient Visual Autoregressive Models
Primary area: generative models
Keywords: visual auto regressive model

Abstract: We observe an interesting Ordinal Asymmetry phenomenon when training visual autoregressive (AR) generators with randomized generation paths: early tokens, due to limited context, suffer higher losses and primarily capture global structure, while later tokens, with richer context, incur lower losses and refine local detail. This suggests that later tokens correspond to easier roles and targets with richer context from a curriculum perspective, emphasizing that learning these first can accelerate training before gradually incorporating harder early tokens. Therefore, we propose Ordinal-biased Random Training (ORT), a simple strategy that first biases loss weights toward later tokens—easier targets with richer context—and then gradually anneals to uniform weighting, ensuring that earlier, harder tokens are also learned. Specifically, we implement ORT with an ordinal focal loss that assigns position-dependent weights; the schedule is controllable and can emphasize either early or late tokens. In practice, ORT shows a striking sudden convergence: gradient norms collapse sharply during the middle of randomized-path training, providing clear evidence that late-biased weighting accelerates early-stage optimization. Experiments on ImageNet-256 with RAR validate our analysis: ORT halves the randomized training phase (200→100 epochs, 2× faster) while maintaining FID comparable to the 400-epoch RAR-XL baseline.