Title: Moment- and Power-Spectrum-Based Gaussianity Regularization for Text-to-Image Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Normal Distribution

Gaussian Distribution

Text-to-Image Models
Abstract: We propose a novel regularization loss that enforces standard Gaussianity, encouraging samples to align with a standard Gaussian distribution. This facilitates a range of downstream tasks involving optimization in the latent space of text-to-image models.
We treat elements of a high-dimensional sample as one-dimensional standard Gaussian variables and define a composite loss that combines moment-based regularization in the spatial domain with power spectrum-based regularization in the spectral domain. Since the expected values of moments and power spectrum distributions are analytically known, the loss promotes conformity to these properties. To ensure permutation invariance, the losses are applied to randomly permuted inputs. Notably, existing Gaussianity-based regularizations fall within our unified framework: some correspond to moment losses of specific orders, while the previous covariance-matching loss is equivalent to our spectral loss but incurs higher time complexity due to its spatial-domain computation. 
We showcase the application of our regularization in generative modeling for test-time reward alignment with a text-to-image model, specifically to enhance aesthetics and text alignment. Our regularization outperforms previous Gaussianity regularization, effectively prevents reward hacking and accelerates convergence.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n1, This paper proposes a unified regularization framework for latent optimization in text-to-image generative models. Authors combine moment-based regularization in the spatial domain with power-spectrum-based regularization in the frequency domain to better enforce standard Gaussianity of latent variables.\n\n2. This approach is motivated by the limitations of existing regularizers (e.g., KL, norm, or kurtosis penalties), which may fail to eliminate residual structures or correlations in the latent space. By encouraging the latent to match both the marginal moment statistics and the expected power spectrum of white Gaussian noise, the method prevents reward hacking and improves optimization stability during test-time reward alignment. \n\n3, Experiments on aesthetic and text-alignment tasks with a one-step text-to-image model show that the proposed method achieves higher reward objectives while maintaining better image quality compared to existing approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n\n1. The method is well-motivated and theoretically grounded, and very good written.\n\n2. Reward hacking is well known issue and important topic.\n\nWeakness:\n\n1. This paper is mainly evaluated only on a single text-to-image model (FLUX) with limited prompts. Broader evaluation on other architectures or diffusion models would strengthen the paper.\n\n2. It would help if authors can discuss more regard whether matching higher-order moments could yield further improvements or not.\n\n3. It assumes that optimizing the latent alone suffices to align rewards, but i didnt see it explore interactions with jointly optimizing text or conditioning variables, which might be relevant in practice.\n\nQUESTIONS:\nWould the method generalize to other text-to-image model except FLUX, multi-step diffusion models or text-to-video models? It would be good to see the discussion.\n\nIs there more quantitative / qualitative results that support the conclusion? Right now, the experiments are very toy.\n\nLIMITATIONS:\n1.it would be good to expand on how latent regularization might propagate biases from the pre-trained generator.\n\n2. Only on a single text-to-image model (FLUX) with limited prompts are mainly evaluated.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel regularization loss function to enforce standard Gaussianity in the latent space of text-to-image generative models. The method combines moment-based regularization in the spatial domain with power spectrum-based regularization in the spectral domain, treating high-dimensional samples as collections of i.i.d. standard Gaussian variables. The authors demonstrate that several existing Gaussianity regularization methods (KL divergence, kurtosis, norm-based regularization) can be viewed as special cases of their unified framework. In the spectral domain, they leverage the fact that the power spectrum of i.i.d. Gaussian samples follows a chi-square distribution to design corresponding regularization terms. The method shows superior performance in test-time reward alignment tasks for text-to-image models, effectively preventing reward hacking and accelerating convergence.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n+ The paper provides a unified theoretical framework that encompasses multiple existing Gaussianity regularization methods under the moment-matching paradigm, with Theorem 1 providing a rigorous characterization of standard Gaussian distributions via moment conditions.\n\n+ The combination of spatial and spectral domain regularization is innovative, particularly the use of DFT coefficient distributions to design spectral domain regularization.\n\n+ Compared to PRNO's O(D²) complexity, the method achieves O(D log D) time complexity through FFT, significantly improving efficiency.\n\n+ The method demonstrates superior performance on both aesthetic image generation and text-aligned image generation tasks, with detailed ablation studies.\n\nWeaknesses:\n\n+ While the paper provides theoretical foundations for moment conditions and spectral properties, it lacks deeper theoretical analysis of why dual-domain regularization is necessary.\n\n+ Key parameters like λpower=25.0 lack theoretical justification or systematic hyperparameter analysis.\n\n+ The authors acknowledge that their loss function value alone cannot reliably indicate how closely a latent vector matches a true standard Gaussian distribution, which is a significant limitation.\n\n+ Experiments are primarily conducted on the FLUX model, with insufficient validation of generalization to other generative models.\n\nQUESTIONS:\n+ How were key parameters like λpower=25.0 chosen? Can you provide parameter sensitivity analysis or adaptive parameter selection methods?\n\n+ Given the acknowledged limitation that loss function values cannot reliably indicate Gaussianity, could you design better Gaussianity evaluation metrics? This is important for practical applications.\n\n+ How does the method perform on other generative models (e.g., Stable Diffusion, DALL-E)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- Authors propose a spatial and spectral regularization loss that enforces gaussianity in latents used for reward alignment in text to image models. \n- In the spatial domain, authors treat a N dimensional gaussian as a collection of 1-D iid gaussian values and utilize the moments of the samples to define a moment-based regularization loss. Authors show that the commonly used KL-divergence, Kurtosis and Norm regularizations are special cases of this spatial domain gaussianity regularization loss.\n- Authors observed that while the latent can have correct marginals in the spatial domain but there can be a mismatch in the spectral domain. To correct this, authors propose another regularization loss in the spectral domain to enforce gaussianity. This is done by a loss that aligns the empirical power spectrum with its expected distribution i.e. $\\chi_2$ distribution.\n- Quantitative results on Aesthetic image generation and text-Aligned image generation show the effectiveness of the proposed regularizations in reward alignment.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The unification of existing methods under the spatial regularization proposed in this work is very informative and useful to the community. \n- Qualitative and Quantitative results show the effectiveness of the proposed method.\n\n**Weaknesses**\n- The introduction and abstract was very hard to read. After reading the abstract, as a reader, I wasn't sure what was the application of the current method. Authors talk briefly about their method in the abstract but what is not clear to me was \"Why is any of this necessary?\". \nAuthors claim their method is useful for a range of downstream tasks but only provide results for reward alignment in text to image models. Then, a better way to organize the paper and the abstract would be to first state that for reward alignment, latents are optimized, and it is important to keep them close to the gaussian manifold. The paper would read very well if it is rewritten as a latent regularization loss for reward alignment instead of its current form. \n- Authors claim gaussianity is important and propose a spatial regularization term but end up using 1st and 2nd moments only in the loss. Why not enforce more terms since matching more moments implies getting much closer to the gaussian manifold.\n\nQUESTIONS:\n- In L314 authors claim that their method achieves higher rewards and outperforms baselines with fewer updates. Please provide experimental evidence to support this statement.\n- Authors only focus on text to image models for their downstream task. I would recommend rewriting the paper to focus directly on that and propose the two regularizations instead of the way it is written right now for me to improve my ratings. In its current form, I find the paper very hard to follow.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe Gaussian distribution plays a central role in optimizing diffusion-based models. In particular, when optimizing diffusion for reward alignment, it is important to incorporate a regularization technique to minimize reward hacking and other negative effects of optimization. This paper investigates existing regularization techniques using the structural properties of the standard Gaussian distribution and proposes a novel regularization method that incorporates information from both spatial and spectral domains. The paper performs experiments on reward alignment in a one-step image generation model and text-aligned image generation. The results show that, compared to baselines such as no regularization, KL divergence, kurtosis, ReNO, and PRNO, the proposed regularization technique outperforms them all.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nWhen performing reward alignment, it is crucial to regularize models to prevent reward hacking and degradation in performance. This paper introduces a novel, simplified formulation of alignment, synthesizing existing work by demonstrating that most current approaches are optimizing a subset of moments of a Gaussian distribution that have a general form, as presented. Not only is the paper well-written, but it also clarifies how the proposed idea relates to previous methods, making the unification clear. \n\nWeakness:\n\nThe paper appears to lack clear ablation studies demonstrating how sensitive the $\\mathcal{K}$ or $\\lambda_\\text{power}$ parameters are in practice. For example, presenting ablation experiments with different \\mathcal{K} and \\lambda_\\text{power} values would offer valuable insight into which factor has the greatest impact. Additionally, since the observation indicates that KL matches the first two moments, it's important to clarify whether the primary contribution of the experiments is in adding the constraint within the spectrum domain, given \\mathcal{K} is set to either 1 or 2.\n\nQUESTIONS:\n- How sensitive is \\mathcal{K} in the joint objective?\n- Did you experiment with moments higher than 2?\n- How sensitive are the experiments to \\lambda_\\text{power}?\n- Is the joint objective just a combination of KL and PRNO? If so, can you elaborate on why regularization is needed in both the spectral and spatial domains simultaneously?\n- Could you explain how spectral and spatial domain regularization work together?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-07"}