Kati8/ICLR_2023_2025 huggingface dataset:
authors: Cheng Lu
reviews: 
Comments: This work proposed a set of improved training techniques to stabilize the training of continuous-time consistency models, including new consistency function formulations, new network architectures and new training objectives. With these new training techniques, the proposed method called sCMs outperformed all previous consistency models in terms of one-step and two-step FIDs.
Strengths: - This paper is very well-written and easy to read. 
- This work proposed a new diffusion formulation, called TrigFlow, that unifies EDM and Flowing Matching, and also simplifies the analysis of continuous-time consistency models. 
- It provided a thorough analysis of the training stability of continuous-time consistency models, from the perspective of network architecture, training objective and diffusion process parameterization.
- Experiments on CIFAR-10, ImageNet-64 and ImageNet-512 demonstrate the effectiveness of the proposed method and the scalability of continuous-time consistency models.
Weaknesses: - Although I really like the improvements of continuous-time consistency models, which could fundamentally eliminate the discretization error in discrete-time consistency models, it comes with more time and memory costs related to JVP computation in the loss function. To this end, this work introduces JVP of Flash Attention to reduce the costs, which is great. Still, there may be a considerable gap between the continuous-time and discrete-time consistency models. I wonder if the paper can provide a more detailed comparison between sCMs and the previous discrete-time consistency models - ECMs, in terms of the training convergence and memory cost. 
- There is no explanation for the phenomenon that sCT performs better than sCD on CIFAR-10 and ImageNet-64, but sCTs performs worse on ImageNet-512. Any intuition of why sCT suffers from increased variance at larger scales? 
- There are no ablation study results on “Adaptive Double Normalization” except for claiming it “removes its instability in CM training”.
- In Figure 5b, it looks like “w/o adaptive weighting” achieves better two-step FIDs than “w/ adaptive weighting” and very similar one-step FIDs to “w/ adaptive weighting”. Why do we need adaptive weighting?
- In Figure 5c, do discrete-time CMs have a constant number of time steps $N$ or a timestep schedule up to the maximum number of steps $N$? If it is the former one, it seems to be a bit unfair to discrete-time CMs because the scheduling of time steps is very important to them. Does it make more sense to compare with the best-performing discrete-time CMs?
- In Figure 7, does the paper apply TTUR proposed by DMD2 (Yin et al. 2024a)? From the DMD2 paper, TTUR improves the performance of VSD. Thus, a comparison with VSD + TTUR is more convincing.
- In Figure 7, sCDs condition the consistency network on the guidance scale $s$. I wonder if VSD also condition the generator on the guidance scale, for a consistent evaluation setting? 
- A minor issue: In line 266, should it be $c_{\text{noise}}(t) = \frac{1}{4} \log(\sigma_d \tan t) $?
Recommendation: 8
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: The authors propose improvements to the consistency models generative paradigm and named their new method sCM. Specifically they -vastly- improve the FID for consistency models with the introduction of several new ideas to both stabilize and simplify continuous consistency models. 

My understanding is the main claim of simplification for sCM comes from the simplification of EDM (Kerras et al.) normalizing design, resulting in $c_{in}=c_{skip}=1$ which in turns simplifies the continuous expression of consistency models. Another simplification is the combination of both EDM and Flow Matching concepts into their method which they call TrigFlow. Yet another simplification, not claimed as such by authors, is the use of vanilla L2 loss compared to Huber/LPIPS use in previous iterations of consistency models. This last simplification has the additional benefit to be more probabilistically grounded.

There are 3 main proposed ideas to stabilize the training of consistency models:
1. Identity-time transformation as a replacement to the log-transformation from EDM
2. Fourier embedding of the time dimension are replaced by positional embeddings
3. AdaGN is modified to also normalize the conditioning inputs for scale and bias.
More ideas are also proposed in the training objective to stabilize training, namely: tangent normalization and tangent warmup. It is my understanding that the adaptative weighing is the same as in EDM.

The paper also provides ample ablations to demonstrate the effects and the reasoning motivating these 3 proposed improvements.
Strengths: The paper is very well grounded mathematically and experimentally. The analysis is based on understanding the causes of training instabilities by decomposing the loss, validating each component experimentally and proposing changes to solve the root causes.
The mathematics while greatly simplified are still pretty complex and the paper shines in its clarity to make the logical reasoning easy to follow.
The experimental results are also outstanding resulting in very significant gains, essentially taking consistency models within 10% of the SOTA for diffusion models.
Weaknesses: 1. The limitations of the method are not very clear to me besides the fact that it's still 10% worse than SOTA for diffusion.
2. The section on positional embeddings (line 269 and on) lacks details to be fully understandable without having to read another paper. Maybe beefing up that section would make the paper more self-contained.
3. I found all figure very useful with the exception of Figure (3) which I felt did not add much value.

I also noticed a typo (definitely not affecting my score), just leaving it there for authors to fix their manuscript:
- Line 362: "cause instability" => "causes instability"
Recommendation: 10
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 4
reviews: 
Comments: The authors introduce an enhanced approach to consistency models called sCM, delivering substantial performance improvements measured by FID scores. The core contributions involve multiple refinements to stabilize and streamline continuous consistency models. The primary simplification mechanism stems from adopting the normalized design framework from EDM (Kerras et al.), yielding $c_{in}=c_{skip}=1$, which reduces complexity in the continuous formulation of consistency models. Additionally, the integration of EDM and Flow Matching methodologies under the TrigFlow framework represents another important simplification. A third simplification involves switching to standard L2 loss instead of previously used Huber/LPIPS losses, offering improved probabilistic grounding. For training stabilization, three major components are introduced: (1) identity-time transformation as an alternative to EDM's log-transformation, (2) replacement of Fourier embeddings with positional embeddings for temporal encoding, and (3) enhanced AdaGN that normalizes both scale and bias of conditioning inputs. Additional stabilization mechanisms include tangent normalization and tangent warmup techniques. The adaptive weighting scheme appears consistent with EDM's approach. Comprehensive ablation studies effectively validate each proposed component and justify the design choices.
Strengths: The work demonstrates exceptional mathematical rigor coupled with thorough experimental validation. The methodology systematically identifies training instability sources through loss decomposition, validates proposed solutions empirically, and addresses underlying causes methodically. The mathematical framework, despite significant simplification from prior work, maintains appropriate complexity while excelling in presentation clarity that facilitates understanding of the logical progression. Experimental outcomes are particularly impressive, narrowing the gap to state-of-the-art diffusion model performance to approximately 10%, representing a meaningful advancement for consistency models.
Weaknesses: 1. The practical limitations of the approach remain somewhat ambiguous, with the primary constraint appearing to be the persistent 10% performance gap relative to leading diffusion model approaches. 2. The positional embedding discussion (beginning at line 269) lacks sufficient detail for complete comprehension without consulting external references, and expanding this section would enhance the paper's self-sufficiency. 3. Most figures provide valuable insights; however, Figure 3 appears to contribute limited additional information. Additionally, a grammatical correction needed: Line 362 should read 'causes instability' rather than 'cause instability'.
Recommendation: 10
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 4