Question: Where is the density estimation result for longrun EBM?
Evidence: Prior work has suggested that persistent learning is the most effective method for learning a valid EBM density. Furthermore, works that learn a valid density have avoided rejuvenation because the incorporation of newly rejuvenated samples into the persistent bank ensures that EBM updates will always include samples that are not at the steady-state. However, persistent learning without rejuvenation has shortcomings mentioned in Section 3. We present hypothesized conditions for learning a valid density that motivate the design of our MCMC initialization:
• After a certain point in training, all samples used to update the EBM must be approximate steadystate samples of the current model p(x; θ). • Persistent samples that are newly rejuvenated (up to about 50K Langevin steps since rejuvenation, and possibly many more) cannot be approximate steady-state samples for any known rejuvenation sources, including data, generators, and noise. • Persistent samples that have undergone sufficiently many lifetime Langevin updates for a model whose weights are changing very slowly can be approximate steady-state samples.
Figure: Visualization of our longrun initializatin procedure. Newly rejuvenated samples must remain the the burnin bank until they have approach the model steady-state, at which point they move to the update bank to be used for model gradients.
Both the second and third point are corroborated by prior work as well as our own observations. The third point means that persistent states updated with shortrun Langevin can eventually act as longrun Langevin samples if the optimizer learning rate is small, because the EBM samples in previous timesteps are essentially samples from the current EBM.
Learning a valid density that includes rejuvenation while satisfying the conditions above requires separating the newly rejuvenated samples from samples that are used to update the EBM. This leads us to introduce two persistent image banks: one for newly rejuvenated samples, and one for samples that will be used to update the EBM. Samples in the newly rejuvenated bank that have been updated sufficiently many times will eventually replace samples from the bank used to update the EBM, at which point newly rejuvenated states will be added to the burn-in bank. Figure shows a visualization of the MCMC initialization method. As in Section 3, we will use a pre-trained generator to efficiently obtain high-quality rejuvenated samples so that the generation process is fully synthetic, although data samples could be used as well. Our goal is to preserve the sample quality for an arbitrary number of MCMC steps. We note that this is a sufficient but not necessary condition for learning a valid EBM density. Nonetheless, stable sampling is an important step towards rigorous probabilistic EBMs. Even with our improved initialization, we find that extremely longrun trajectories of 1 million or more MCMC steps still tend to oversaturate, although to a much lesser degree. To further stabilize the appearance of extremely longrun Langevin samples, we include prior energy terms in the model. Our longrun EBMs have the form
where U (x; θ) is the model whose weights are updated, U 0 (x; θ 0 ) is a prior EBM with fixed weights θ 0 and σ is a parameter controlling the strength of a Gaussian prior. We used a prior EBM in a shortrun manner. The role of the prior EBM is to provide some stability but also to provide a tendency to oversaturate at longer trajectories so that the current EBM learns to correct oversaturation. The Gaussian prior is meant to discourage unbounded activations outside of the image hypercube. Further discussion is in Appendix H. We find that including both of these terms significantly improves the ability to learn quality synthesis over long trajectories.
Table 4: FID for 5K samples after 100K Langevin and 1M Langevin steps. FID remains stable over long trajectories.****Figure 13: Longrun samples at 100,000 steps and extremely longrun samples at 1 million steps
for EBMs trained on three datasets. Our initialization is able to preserve a high degree of realism over the first 100K steps, and a reasonable degree of realism over very long trajectories. While oversaturation and distortion is noticeable for some samples using 1M steps, many samples have reasonable appearance and there is high diversity. Our method makes significant progress towards aligning longrun samples with high-quality samples from training to ensure that the model is a valid
density
Answer: Our density estimation results are the longrun FID scores in Table 4 and visualizations of longrun samples in Figure 11. Please refer to the general statement for our explanation of why we believe these are an appropriate evaluation to investigate density modeling.