CHIMLE: Conditional Hierarchical IMLE for Multimodal Conditional Image Synthesis
Authors: Shichong Peng
Seyed Alireza Moazenipourasil
Ke Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: Mode-covering Generative Model
Diverse Conditional Image Synthesis
Implicit Maximum Likelihood Estimation (IMLE)

Abstract: A persistent challenge in conditional image synthesis has been to generate diverse output images from the same input image despite only one output image being observed per input image. GAN-based methods are prone to mode collapse, which leads to low diversity. To get around this, we leverage Implicit Maximum Likelihood Estimation (IMLE) which can overcome mode collapse fundamentally. IMLE uses the same generator as GANs but trains it with a different, non-adversarial objective which ensures each observed image has a generated sample nearby. Unfortunately, to generate high-fidelity images, prior IMLE-based methods require a large number of samples, which is expensive. In this paper, we propose a new method to get around this limitation, which we dub Conditional Hierarchical IMLE (CHIMLE), which can generate high-fidelity images without requiring many samples. We show CHIMLE significantly outperforms the prior best IMLE, GAN and diffusion-based methods in terms of image fidelity and mode coverage across four tasks, namely night-to-day, 16x single image super-resolution, image colourization and image decompression. Quantitatively, our method improves Fréchet Inception Distance (FID) by 36.9% on average compared to the prior best IMLE-based method, and by 27.5% on average compared to the best non-IMLE-based general-purpose methods. More results and code are available on the project website at https://niopeng.github.io/CHIMLE/.

CHIMLE: Conditional Hierarchical IMLE for Multimodal Conditional Image Synthesis
Shichong Peng', Alireza Moazeni Ke Li
APEX Lab
School of Computing Science
Google
Simon Fraser University
seyed alireeee_aiieeeeee oow.
Abstract
A persistent challenge in conditional image synthesis has been to generate diverse output images from the same input image despite only one output image being observed per input image. GAN- methods are prone to mode collapse, which leads to low diversity. To get around this, we leverage Implicit Maximum Likeli- hood Estimation (IMLE) which can overcome mode collapse fundamentally. IMLE uses the same generator as GANs but trains with different, non-adversaiial objective which ensures each observed image has generated sample nearby. Un- fortunately, to generate high-fidelity images, prior IMLE-based methods require large number of samples, which is expensive. In this paper, propose new method to get around this limitation, which we dub Conditional Hierarchical IMLE (CHIMLE), which can generate high-fidelity images without requiring many sam- ples. We show CHIMLE significantly outperforms the prior best IMLE, GAN and cuffusion-based methods in terms of image fidelity and mode coverage across four tasks, namely night-to-day, 16x single image super-resolution, image colourization and image decompression. Quantitatively, our method improves Fréchet Incep- tion Distance (FID) by 36.9% on average compared to the prior best IMLE-bass method, and by 27.5% average compared to the best non-MMLEbbased general- purpose methods. More results and code are available on the project website at nttps: github. 10/CHIMLE///H
Introduction
Impressive advances in image synthesis have been made by generative models y 30 83 15
78
BI 38)
Generative models are probabilistic models; in the context of image synthesis, they aim to learn the probability distribution over natural images from examples of natural images. Whereas unconditional generative models learn the marginal distribution over images, conditional generative models learn the conditional distribution over images given conditioning input, such class label, textual description or image.
Conditional generative modelling is more challenging than unconditional generative modelling, due to the need ensure consistency between the conditioning input and the generated output while still ensuring image fidelity and diversity. As the conditioning input becomes more specific, there are fewer images in the training set that are consistent with the conditioning input, resulting in weaker supervision. So, in the order of decreasing supervision are class, text and iage-condiiionall generative models in class conditioning, many images correspond to each class; text conditioning, multiple images at times share the same textual description; in image conditioning (c.g., grayscale image as the input and a colour image as the output), typically only one output image that corresponds to each input image is observed (e.g., only way colour grayscale image is observed). We
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

cIMLE
Select
Generator
Latent Code
i. Divide
CHIMLE
Select
ii. Conquer
Generator
Combine
(top row) and the proposed method, CHIMLE (bottom row). Whereas cIMLE samples many latent codes and selects the one that produces the best result, CHIMLE divides the latent code by dimension into components, selects the best sample for each component and iteratively combines the selected components to form the overall latent code. As shown, the sample from CHIMLE contains more fine-grained details than the one from eIMLE.
Figure 1: Overview of the prior best IMLE-based method, cIMLE 33
will focus the most challenging of these settings, namely image conditioning. Regardless of the supervision, the goal of conditional generative models to learn the full distribution over all possible images consistent with the input.
Attaining this goal is challenging, especially when the supervision is weak, and so popular existing methods such variational autoencoders (VAEs) 4S 770 and generative adversarial nets (GANs) 126 27 focus on generating a point estimate of the conditional distribution, such as the mean (VAE) or an arbitrary mode (GAN). Such methods known unimodal methods, whereas methods that aim to generate samples from the full distribution are known as multimodal methods, since they enable samples to be drawn from many modes there exist.
Extending popular unconditional generative models such as VAEs, GANs and diffusion models to the inage-conditional multimodal setting has proven to be challenging. VAEs suffer from posterior where the encoder ignores the conditional input, leading the decoder to produce the conditional mean, which typically corresponds
collapse TX 577
blurry image. GANs suffer from mode where the generator tends to ignore the latent noise, leading the generator to produce the same mode of the conditional distribution for the same input regardless of the latent noise. As a result, other equally valid modes (e.g., other possible ways to colour a grayscale image) cannot be generated. Recent and concurrent work [42] 6 extends diffusion models this setting, but they can produce both too little and too much diversity by creating spurious modes and dropping modes. We dub this problem mode forcing and include analysis of its cause in the appendix.
collapse 37 105
recent method [53] takes a different approach by extending an alternative generative modelling technique known as Implicit Maximum Likelihood Estimation (IMLE) 522 to the conditional synthesis setting. At a high level, IMLE uses a generator like GANs, but instead using an adversarial objective which makes each generated sample similar to some observed image, IMLE uses a different objective that ensures each observed image has some similar generated samples. Hence, the generator can cover the whole data distribution without dropping modes. We refer interested readers to 53 for more details on the algorithm.
However, as shown in the top row of Figure I the output image from cIMLE 5 lacks fine details. For cIMLE to generate high quality samples, there must sample out m samples generated during training that is similar to the observed image. This necessitates large unfortunately, generating large number of samples computationally expensive. So we are faced with tradeoff: do improvements in sample quality have come at the expense of sample efficiency?

---

In this paper, we show that there is way around this conundrum. The idea is to generate samples in a clever way such that the best sample about as similar the observed image as if large number of samples had been generated, without actually generating that many samples. We propose three ideas: partitioning of the latent code, partial evaluation of latent code components and iterative construction of latent code as shown in the bottom row of Figure TT known as Conditional Hierarchical IMLE (CHIMLE)
These three ideas give rise to novel method We demonstrate that CHIMLE significantly outperforms the prior best IMLE-based method [53 in terms of both fidelity and diversity across four challenging tasks, namely night-to-day, 16x single image super-resolution, image colourization and image decompression. Moreover, show that CHIMLE achieves the ttate-of-the-an results in image fidelity and mode coverage compared to leading general-purpose purpose multimodal and -pecifc methods, including both GAN-base and diffusion-based methods.
2 Method
2.1 Preliminaries: Conditional Implicit Maximum Likelihood Estimation (cIMLE)
In ordinary unimodal synthesis, the model is a function fo parameterized by 0 that maps the input to the generated output. To support multimodal synthesis, one can add a latent random variable as an input, so now fa takes in both the input x and a latent code z drawn from a standard Gaussian A I) and produces which adds a discriminator network that tries to tell apart the observed image y and the generated output y. The generator is trained to make its output y seem real possible to the discriminator. Unfortunately, after training, fx z) often produces the same output for all values of z because of mode collapse. Intuitively, this happens because making ý real as possible would push it towards the observed image y. so the generator tries to make its output similar to the observed image y for all values of z. As result, naīvely applying cGANs to the problem of multimodal synthesis is difficult.
image y output. To train such network, we can use conditional GAN (cGAN),
Conditional IMLE (cIMLE) [ proposes an alternative technique for training the generator network fo. Rather than trying to make all outputs generated from different values of z similar to the observed image y. only tries to make some of them similar to the observed image y. The generator is therefore only encouraged to map value of to the observed image y. allowing other values of map to other reasonable outputs that are not observed. This makes it possible to perform multimodal synthesis. Also, unlike cGANs, cIMLE does not use discriminator and therefore obviates adversarial training, making training more stable. The cIMLE training objective is:
(x (1.))) (())))((())) ),0(0)
min E 2000 ((0.))
min
(1)
JE m)
where d(::.) :) is distance metric, m is hyperparameter, and x()) and (() are the 1th input and observed image in the dataset. During each iteration of the training loop, a pool of m samples are generated from fo for each input x6)) The cIMLE algorithm then selects the closest sample for each observed image (0) to optimizes 0.
2.2 Conditional Hierarchical IMLE (CHIMLE)
In order to generate high-quality images using cIMLE, we need some of the m generated samples to be similar to the observed image y during training. Achieving this requires large m, but generating samples is expensive. This forces a tradeoff between the number of samples and the mode modelling accuracy, which is less than ideal.
The key observation is that the gradient of the loss depends solely on the closest sample for each observed image, as shown in Eqn. | all other samples that are not selected do not contribute. Therefore, we could efficiently search for a latent code z that would produce a sample of a comparable degree of similarity to y as one that would be selected from larger pool z", could achieve high sample quality without actually generating that many samples.
To solve search problems, time-tested paradigm in computer science is entails dividing the problem into simpler sub-problems, conquering each sub-problem and combining the solutions from the solved sub- sub-problem. In the context of searching for latent code that
which
3

---

simulates the quality of z", we propose dividing the latent code into components, evaluating each latent code component and combining the latent code components iteratively.
In the ensuing discussion, we will consider a common paradigm of image synthesis models 68 37 13 39
where the model processes the image at I. different scales. We can feed different dimensions of the latent code as inputs to different layers operating at different resolutions, thereby controlling the variations of content different resolutions.
divide it into Division of Latent Code To reduce the difficulty of the original problem, sub-pru broblrrbsems. Each of these sub-problems should be a simpler instance of the original problem. We propose dividing the dimensions of the latent code RR" into L different components S {zi)_ where x, E & and Σ k; = n, which amounts to dividing the latent space into mutually orthogonal subspaces. Each component corresponds to the dimensions that serve as input to layers operating at a particular resolution.
Partial Evaluation of Latent Code Components To determine how promising a latent code com- ponent is. we need way to evaluate partially constructed latent code, with only some components, indexed by Sr C SS, being realized with concrete values. We propose using partial output Yp that depends only on the currently constructed components Z.s., and evaluating Yp against part of the observed image Yp that only contains details resolutions modelled by zs.. So instead of computing d(ý: y), we compute Yp). By doing so, we can evaluate the quality of zs. more accurately. since yp does not depend on the unconstructed components ss.. To realize this, we add output heads at the L different resolutions that the model operates to produce the partial outputs y, and downsample the observed image y to the corresponding resolutions to produce the partial observed images yp. The values of the partially realized components that result in the greatest similarity to Yp is the final selected component, i.c., x = arg min p(yp,
Methed
Numbor of Sumples Nosded
175
CHIMBE
150
dMLE
Algorithm 1 Conditional Hierarchical IMLE Al-
100
gorithm for a Single Data Point
88 1t
Require: One input image L increasing resolu- tions {x the set of partial observed images
[yr] L at corresponding resolutions and the gener- ator fo(..) that produces L outputs at correspond- ing resolutions
0.16
0318
0.320
0.322
0.334
0.326
LPIPS Diszzace
Figure 2: Generated sample quality comparison between the proposed Conditional Hierarchical IMLE (CHIMLE) and cIMLE [53]. The plot shows the number of generated samples needed to reach the same distance (as measured by LPIPS) to the observed image with CHIMLE cIMLE. A smaller number of generated samples means bet- ter efficiency in finding sample that is close the observed image. The result is averaged over ten independent runs. As shown, CHIMLE con- sistently outperforms cIMLE by requiring signif- icantly fewer samples. Also, the more stringent the required LPIPS distance gets, the greater the disparity between the number of samples needed by cIMLE vs. CHIMLE.
for = 1 to do
Randomly generate m i.i.d. latent codes for the e component 2 ,...,2
300 + 1st output from fo fo((x1...., x1), (zi..... zi (2)) Vj
z] <- arg miny d(y), Vj E (I.....m.
end for
return (z) 14
11-1
Iterative Combination of Latent Code Components To maximize efficiency, the solution to one sub-problem should reveal some information on which other sub-problems to solve. So, we need to decide on which sub-problems to solve first and how to use their solutions to decide on which sub-problem to solve next. We leverage the fact that solving for the appearance at lower resolutions (c.g., finding out that the coarse structure of apple is red) can inform on the probable appearances

---

Code
Global Stream
Code
Global Stream
Code
Global Stream
Code
Code
Code
Local Stream
Local Stream
Local Stream
Module
Mapping Network
Skip Connection To Following Module(s)
AdaIN
Addition
Code
Code
Conv
Upsample
Conv
Conv
Sigmaid
RRDB
RRDB
RRDB
Figure 3: Our model consists of multiple modules, each of which operates on 2x the resolution of the previous Each module contains global stream and local stream. The local stream consists of sequence of ceiiual-in-residuue dense blocks (RRDB). The global stream consists of a mapping network which applies adaptive instance normalization (AdaIN) to the output of each RRDB block.
at higher resolutions (e.g., the fine details of the apple are likely red as well). We propose starting the construction the latent code from the component operating at the lowest resolution to the highest, ie. from to 2,,, where z are assumed to be ordered by resolution in increasing order. At given resolution / {1... L). we set components at lower resolutions to the values selected in previous steps zi..... z to construct žş, (zi ......... 2i-1, zi), and evaluate the quality of zs using the partial evaluation method proposed above.
Putting everything together, we obtain the Conditional Hicrarchical IMLE (CHIMLE) algorithm. which is detailed Alg. for the special case of one input image. We validate the effectiveness of CHIMLE by comparing the number of generated samples CHIMLE needs to obtain the same level of similarity to the observed image as cIMLE. As shown in Figure 2 CHIMLE consistently requires fewer samples than cIMLE, demonstrating a significant improvement in sample efficiency.
2.3 Model Architecture
As shown Figure D our architecture contains a sequence of modules, each of which handles an input image of particular resolution and outputs image whose resolution is doubled along each side. We downsample the input image repeatedly by half to obtain input images at different resolutions, which fed into different modules. As mentioned in Sect. 2.2 we divide the dimensions of the latent code into components and feed each to a different module. Note that this architecture generalizes to varying levels of output resolution, since can simply add more modules for high-resolution outputs. We add intermediate output heads to each module and add supervision to the output of each module. We choose LPIPS [0I as the distance metric used in the IMLE objective function.
For the design of each module, we follow the paradigm of prior work on modelling global and local image features 59 10]
58 74 and design module that comprises of two branches, local processing branch and a global processing branch. The local processing branch takes in the conditional image input, a latent code with the matching resolution the image and the output from the previous module if applicable. The global processing branch takes in a fixed-sized latent code input and produces scaling factors and offsets for the different channels of the intermediate features in the local processing branch. For more implementation details, please refer to the appendix. Due to the hierarchical nature the architecture and its role the implicit model in IMLE (also known as the "generator" in GAN parlance), we dub this architecture the Tower Implicit Model (TIM).
3 Experiments
Baselines To validate our main contribution of this paper, namely improving the output fidelity of IMLE-based methods, compare to the leading IMLE-based method, cIMLE [53]. As
5

---

Input
NDM
6
BicycleGAN 1005
cIMLE
CHIMLE (Ours)
Input
MoNCE 95
RFB-ESRGAN 76
cIMLE 53
CHIMLE (Ours)
Input
MoNCE h5
InstColorization N9
cIMLE F
CHIMLE (Ours)
Input
BicycleGAN 1005
DrCNN 97
eIMLE 53
CHIMLE (Ours)
Figure 4: Qualitative comparison of our method (CHIMLE)) and selected baselines on night-to-day (1 row), 16x super-resolution (2nd row), image colourization (3 row) and image decompression (4th row). each row, we show the results from the best performing strong task-specific baseline (if applicable), the best prior IMLE-based method cIMLE (53) and CHIMLE. Better viewed zoomed-in. As shown, CHIMLE correctly preserves the details of the input while generating fine-grained details in the output, which compares favourably to the baselines.
baseline,
secondary comparison, demonstrate potential impact in broader context and compare method to the leading multimodal general-purpose conditional image synthesis methods according to recent survey (55), including four GAN-based methods, such BicyleGAN [105) MSGAN [60]
DivCo (56) and MoNCE [55. and two diffusion-barte methods, such as DDRM 42 and NDM 6
As tertiary comparison, also compare method popular tpek-sp methods based on leaderboard rankings challenges and recent survey papers 196
Tasks We apply our method to four different conditional image synthesis tasks, namely night-to- day, 16x single image super-resolution. image colourization and image decompression. We pick night-to-day because is classic task in the multimodal general-purpose conditional image synthesis literature 37 105]. We pick the remaining tasks because they are more challenging than typical tasks considered in the literature evidenced by the availability of task-specific methods tailored each.
Training Details We use four- module TIM model (described in Sect. 2.3 for all tasks. The input each module is downsampled from the full resolution input to resolution the module operates on. We trained all models for 150 epochs with mini- batch size of I using the Adam optimizer on an NVIDIA V100 GPU. Details for datasets are included in the appendix.
Evaluation Metrics We evaluate the visual fidelity of generated output images using the Fréchet Inception Distance (FID) 3C and Kernel Inception Distance (KID) 7 We also measure the diversity of the generated output images since it is crucial objective of multimodal conditional image

---

synthesis. One metric to measure diversity is the LPIPS diversity score (not to be confused with the LPIPS distance metric) [051]. which is the average LPIPS distance between different output samples for the same input. Unfortunately it has limitations, because it may favour set of samples where To get around this, we use hüthfullnessweighhee variance (FwV) (3 instead, which is the average LPIPS distance dLPIPS between the output samples and the mean, weighted by the consistency with the original image measured by a Gaussian kernel
some are of poor fidelity as shown Figure 5
(y(1)) ,, where wi; = exp ( --dLrus (=d (y (ylt.)) ,
M -- ux (1J)
(2)
2n
where ((()) is the 7th generated sample from the nth imput. y(0) is the mean of all generated samples from the 1th input. yb) is the 7th observed image. The kernel bandwidth parameter trades off the importance of consistency vs. diversity.
LPIPS Diversty
Faihilinew--eighredd Varience (o-0.2)1
Samples
Group
0.079
0.0045
Group 2
0.217
0.0034
Figure 5: Comparisons of LPIPS diversity score and seaaffuiness-weighhe variance two groups of sample images. Although Group 2 achieves a better LPIPS diversity score (shown on the left), some of the samples have poor visual fidelity. Group I contains samples that are both diverse (zoom in for difference in the seeds on the surface of the strawberry) and of high visual fidelity, which is preferred by the faithfulness- weigghte variance (shown on the left).
We measure how well the modes in the model distribution and the true distribution match along two axes, mode accuracy (precision) and mode coverage (recall). We measure the former with the Fi/s score component of the PRD metric 771, which is heavily weighted towards precision, and the latter with the F& score component of the PRD metric [ which is heavily weighted towards recall, and the inference-vin cetereecce-ia-optinnee via-optimization (IvO) metric [61, which measures the final mean squared error
(MSE) between the the observed image and the best output image that can be generated (i.e., where the latent code is optimized w.r.t. MSE while keeping model weights fixed). In the conditional image synthesis setting, IvO is more sensitive metric than the Fs score in PRD, since it would detect cases where the output samples for a given input are not diverse but the output samples for different inputs are diverse in aggregate. In other words, IvO measures mode coverage the conditional distribution, whereas the FsS score measures mode coverage in the marginal distribution. In light of the large number of noise samples in NDM (2 for each of the 1000 diffusion steps) which all need to be optimized in calculating IvO, we omit it due to the lack of sufficient computational resources.
3.1 Quantitative Results
We compare the perceptual quality and output diversity in Table ! As shown, CHIMLE outperforms best prior IMLE-based method, cIMLE, general- pprppse conditional image synthesis methods and task-specific methods in terms of FID and KID across all tasks. CHIMLE also outperforms all multimodal baselines in terms of FwV across all tasks. These results show that CHIMLE can produce state-of-the-art.
more realistic and diverse images than the baselines, thereby setting
We compare the mode accuracy and coverage in Table 2 As shown, CHIMLE outperforms or achieves comparable performance to cIMLE and other multimodal general-purpose baselines in
'(exp (-d. -)/o²² is considered a Gaussian kernel rather than Laplacian kernel because LPIPS distance is defined squared Euclidean distance in feature space.)
7

---

Night-todaay
Super Resolution (SR)
Colourization (Col)
Image Decompressio (DC)
FID
4 PuV
FID
PuV
FID
PuV
FID
KID L PvV
General-Purpue Purpuse Methods:
GAN-buzed:
BAcyieAA NDa
179.08
132.35
0.23
67.17
58.25
0.84
53.33
31.44
7.44
87.35
20.24
0.29
MSGAN
176.35
27. 75.75
DieGa
179.85
136.44
0:22
66.97
54.86
0.65
63.82
34:44
12.49
92. 76
0:25
MoNCF
232.10
217.79
0.18
47:07
95.31
0.68
27:67
9.55
18.30
113.21
27. 36
1.12
DDRM
258.69
170.72
167.50
132.53
0.22
118.90
106.51
43.98
16.21
14.39
136.32
02.64
Task Spocitic Methods:
REB-ESRGA ESRGAN (SR) 7d
12.90
8.01
InnCooooizationn
DeCNN (DC)
100.38
48.05
IMLE based Methods:
cIMLE [5]
166 28
125. 15
0.47
26.42
14.75
5.22
63.00
19.3 30
14.65
101
30.11
3.14
CHIMLE (Ourz)
90.10
4.54
12.12
Table I: Comparison of fidelity of generated images, measured by the Fréchet Inception Distance (FID) and Kernel Inception Distance (KID), and diversity, measured by wettffulness-wiighte variance (FwV). We show the results by our method (CHIMLE) and the leading IMLE-based, task-specific baselines and general-purpos
conditional image synthesis baselines. Lower values of FID/KID are variance shows more variation in the generated
better and a higher value of faithfulness-weighhe samples that are faithful the observed image. For VwV, we show the results for the bandwidth parameter a = 0.2 as it achieves good balance between consistency and diversity. The KID and the FwV are shown on the scale of 10 "_" indicates metric not applicable. We compare favourably relative to the baselines.
terms of PRD scores. In addition, CHIMLE significantly outperforms cIMLE and other multimodal general-purpose baselines in terms of IvO. These results show that CHIMLE achieves better mode accuracy and coverage of the true distribution compared to the baselines.
3.2 Qualitative Results
We show the qualitative comparison of our method (CHIMLE) and selected baselines various tasks in Figure 4 As shown, CHIMLE generates high-quality results, like the high contrasts in the building and the bushes in night-to-day (1 row), the sharp appearance of the wing pattern of the butterfly wing and the flowers in the background (2nd row), the vibrant yet detailed colouring of the parrot (3rd row) and the high-frequency details in the sculpture (4th row). For all methods, we show videos of different samples generated from the same input on the project website and in the supplementary materials. Due mode collapse, GAN-based baselines incorporate regularizers that improve the diversity of samples at the expense of their fidelity, resulting in more varied but less realistic samples as shown in the videos. Diffusion-based baselines suffer from mode forcing (see appendix or with little diversity. The prior best IMLE-based method (cIMLE) avoids spurious samples but falls short in image fidelity. In contrast, CHIMLE generates samples that are both diverse and of high fidelity. Figure shows the qualitative comparison of the best reconstruction of the observed image found by different methods using IvO 6.. which tries to approximately invert each model by optimizing over the latent code input to find a generated image closest to the observed image. As shown, the reconstruction of the CHIMLE model is the closest to the observed image and achieves almost pixel-level accuracy. In addition, we found that for our model, IvO can consistently converge to high-quality sample, whereas it would diverge quite often for the baselines, suggesting that inverting our model much easier than the baselines. This demonstrates that CHIMLE successfully covers the mode corresponding to the observed image.
[ for more details) and generate spurious samples that are either excessively diverse
3.3 Ablation Study
We incrementally remove (1) iterative combination (IC), (2) partial evaluation (PE), (3) latent code division (CD). As shown in Figure 7 each is critical to achieving the best results. Notably, we see an average increase in terms of FID by 7.7% after removing IC, by another 7.8% after removing PE and by another 12. 12.8% after removing CD, thereby validating the importance of each of our contributions.
8

---

Nighesindyy
Sepe-Rrachhinn (SR)
Cabemtaation (CH)
Image Deormpendo (DC)
Piys (Prectcoo)
1L (Rocel)
Flue (Proeiioo)
Piye (Procisio))
Floe (Parcision)
(Recal))
(Recall)
(Rocal)
GaNdased
0.30
pbiCa
0.46
59.50
0.30
0.0)
10.08
1.99
0.65
6:38
1.08
MoNCX 21
0.81
0.13
18:10
6 0.01
9.55
0.35
16.33
0.71
0.3
0.73
13.10
10008:00 0.79
0.90
0.76
20.00
0.10
0.12
151/001
0.23
0.73
11.81 0.00
3.04
13.70 0.51
0-37
5.41 0.01
IMLE
<IMLE [0]
0.58
1.4
3,50
0.42
6,75 0.444
0.92
0.64
0.73 0.00
6.91
0.74
1,50
CHIM12/00rr)
0.97
0..8
01306 22 0.004
0.30
0.7%
1:49
10111
0.97
10.7
01332 32
0..1
0.95
0.83
0.31
0.10
Table 2: Comparison of mode accuracy, measured by Fi /8 score of the PRD metric [71], and mode
coverage, measured by Fs score of the PRD metric [ and the inference- -vi- via-optimization metric (IvO) [6I, between the model distribution and the true distribution by our method (CHIMLE) and the leading IMLE-based and general general-purpose purpose conditional image synthesis baselines. higher value of the Fg score shows better mode coverage (recall) and higher value of the F1/8 shows better mode lower value of IvO shows better mode coverage since indicate that the model can generate samples close to the observed images. For IvO, we show the average result across independent runs on the scale of 10 As shown, CHIMLE outperforms or achieves comparable performance to the baselines.
accuracy (precision).
BicycleGAN 100
MSGAN 60
DivCo 56
MoNCE 95
DDRM 4
cIMLE S3
CHIMLE (Ours)
Observed Image
Figure 6: Qualitative comparison of best reconstructions of the observed image found by optimizing over the latent code input using the method of Inference via Optimization (IvO) 6ol. The optimization starts by feeding random latent code input to each method and minimizes the reconstruction loss from the model output the observed image by optimizing the latent code input. As shown, the CHIMLE model allows for significantly better reconstruction of the observed image than the other methods, which demonstrates that the mode corresponding the observed image successfully covered by CHIMLE.
Related Work
Ceneral-Purposs Conditional Image Synthesis Conditional image synthesis generate images from conditioning input. The input can come various forms, such a class label 62 a textual description [69. or an image [37] The latter is sometimes known as image-to-image translation, which is the setting we consider. General image-conditional methods can be paired or unpaired. In the paired setting, each image in the source domain corresponds to an image in the target domain (37 93 86 75). In the unpaired setting, images in the source domain may not correspond 67] Whereas these previous methods are all
to images in the target domain [044] 92
55
16
unimodal, other methods aim for the more challenging task of multimodal synthesis in the paired
51 17 Since many of the aforementioned works use GANs as their generators which suffer from mode collapse, some works aim address this issue by introducing mode seeking terms 60 adding contrastive losses
setting 105 24 22 and in the unpaired setting 34
50
(56 95 33
Another

---

SR
DC
Full Model
16.01
73.69
No IC
17.06
81.24
No IC, PE
18.53
87.95
No IC, PE, CD
21.13
101.38
Full Model
No IC
No IC, PE
No IC, PE, CD
Figure 7: Qualitative (left figure) and quantitative (right table) of Fréchet Inception Distance (FID) comparison as we gradually remove (1) iterative combination (IC), (2) partial evaluation (PE), (3) latent code division (CD) on two tasks, 16x super-resolution (SR) (first row in the figure) and image decompression (DC) (second row in the figure). As shown in the figure, the generated image contains less fine details (as in SR, where the strawberry seeds become less pronounced) or colour saturation (as in DC) as we remove each component, which is also reflected by the FID shown in the table.
line of work uses diffusion models 42 6]. These methods can both generate spurious modes and drop modes (due to the problem of mode forcing, which we analyze in appendix D) and need to devote part of the model capacity to reconstruct the information encoded in the conditioning input from pure noise and retain it over the course of diffusion. this paper, we focus on the paired image-conditions multimodal setting, and sidestep mode collapse by building recent generative modelling technique, conditional IMLE [53].
Task-Specific Approaches There is large body of work on super-resolution, most of which
consider upscaling factors of 2 - 4x. See [91, 63 89] for comprehensive surveys. Many methods
regress to the high-resolution image directly and differ widely in the architecture (3 4
47
80 28
102
19
54
Various conditional GANs have also been developed for the problem
72
66
90
76
For image colourization, recent survey R provides detailed overview of different
methods. Some methods 14 combine it with the input to produce the output. Other methods [36 48
12 ICC I|I use deep networks to directly predict the colour and 21
79] take advantage of features from different levels or branches in the network for colour prediction. There is relatively little work on image decompression to knowledge.
[03
4 targets DCT-based compression methods and explicitly models the quantization errors of DCT coefficients. More work was done on learned image compression [2] |I which changes the encoding of the compressed image itself. Another related area is image denoising, see [82] for a survey of deep learning methods. Many of the methods
uses ResNet 29 backbone or other variants of deep CNNs 997 98 99 85
5 Discussion and Conclusion
Limitation Our method can only generate output images that are similar to some training images. Take colourization as an example- there were no green apple in the training data, then our model would not be able to generate green apples.
Societal Impact The TIM architecture adopts a modular design, its capacity could be expanded by adding more modules. While it may produce more refined results, training such a huge model may result increased greenhouse gas emissions if the electricity used is generated from fossil fuels.
Conclusion In this paper, we developed an improved method for the challenging problem of multimodal conditional image synthesis based on IMLE. We addressed the undesirable tradeoff between sample efficiency and sample quality of the prior best IMLE-based method by introducing a novel hierarchical algorithm. We demonstrated achieves the state-of-ther te-thhe-rtt art results in image fidelity without compromising diversity or mode coverage.
wide range of tasks that the proposed method
Acknowledgements This research was supported by NSERC, WestGrid and the Digital Research Alliance of Canada. We thank Tristan Engst, Yanshu Zhang, Saurabh Mishra, Mehran Aghabozorgi and Shuman Peng for helpful comments and suggestions, and Kuan-Chieh Wang, Devin Guillory, Tianhao Zhang and Jason Lawrence for feedback on an early draft of this paper.
10

---

References
[1] Eirikur Agustsson, Michael T'schannen, Fabian Mentzer, Radu Timofte, and Luc Van Gool. Generative adversarial networks for extreme learned image compression. In Proceedings of the IEEE International Conference on Computer Vision, pages 221-231, 2019.
[2] Eirikur Agustsson, Michael T'schnnen,, Fabian Mentzer, Radu Timofte, and Luc Van Gool. Extreme learned image compression with gans. In CVPR Workshops, volume 1, page 2, 2018.
[3] Saeed Anwar, Muhammad Tahir, Chongyi Li, A. Mian, F. Khan, and A. W. Muzaffar. Image colorization: A survey and dataset. ArXiv, abs/2008. 10774, 2020.
[4] Yuval Bahat and Tomer Michaeli. Explorable decoding compressed images. arXiv preprint arXiv:2006. 20X332000009333,, 2020.
[5] S. Baker and T. Kanade. Limits on super-resolution and how to break them. IEEE Transactions on Puttern Analysis and Machine Intelligence, 24(9): 167-1183, 2002.
[6] Georgios Batzolis, Jan Stanczuk, Carola-Bibiane Schönlieb, and Christian Etmann. Non- uniform diffusion models. arXiv preprint 12Xiv:220770978666 2022.
[7] Mikolaj Binkowski, Danica J. Sutherland, Michal Arbel, and Arthur Gretton. Demystifying mmd gans. ArXiv, abs/1801 181.01141,, 2018.
[8] Konstantinos Bousmalis, Silberman, David Dohan, Erhan, and Dilip Krishnan. Un- supervised pixel-level domain adaptation with generative adversarial networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 95-104, 2017.
[9] Andrew Brock, J. Donahue, and K. Simonyan. Large scale gan training for high fidelity natural image synthesis. ArXiv, abs/1809.1 1096, 2019.
[10] Bingyi Cao, Andre F. de Araújo, and Jack Sim. Unifying deep local and global features for image search. In ECCV, 2020.
[11] Yun Cao, Zhiming Zhou, Weinan Zhang, and Yong Yu. Unsupervised diverse colorization via generative adversarial networks. ArXiv, abs/ abs/ 172.000076674 2017.
[12] Fabio Maria Carlucci, Paolo Russo, and Barbara Caputo. (de))co: Deep depth colorization. IEEE Robotics and Automation Letters, 1-2386-2393, 2018.
[13] Qifeng Chen and Vladlen Koltun. Photographic image synthesis with cascaded refinement networks. In IEEE International Conference on Computer Vision (ICCV), volume page 2017.
[14] Zezhou Cheng, Qingxiong Yang, and Bin Sheng. Deep colorization. 2015 IEEE International Conference on Computer Vision (ICCV), pages 415-423, 2015.
[15] R. Child. Very deep vaes generalize autoregressive models and can outperform them on images. ArXiv, abs/201 1. 10650, 2020.
[16] Yunjey Choi, Min-Je Choi, Munyoung Kim, Jung- Woo Ha, S. Kim, and Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8789-8797, 2018.
[17] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Wos Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8185-81 8194, 2020.
[18] Bin Dai, Ziyu Wang, and David Paul Wipf. The usual suspects? reassessing blame for vae posterior collapse. In ICMI., 2020.
[19] Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang. Second-order atten- tion network for single image super-resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11065-11074, 2019.

---

[20] Duc-Tien Dang-Nguyer Nguyen, Cecilia Pasquini, Valentina Conotter, and Giulia Boato. Raise: A raw images dataset for digital image forensies. Proceedings of the 6th ACM Multimedia Systems Conference, pages 219-224, 2015.
[21] Aditya Deshpande, Jiajun Lu, Mao-Chuang Yeh, Min Jin Chong, and David Alexander Forsyth. Learning diverse image colorization. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2877-2885, 2017.
[22] J. Donahue, Philipp Krähenbühl, and Trevor Darrell, Adversarial feature learning. ArXiv, 2017.
[23] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning a deep convolutional network for image super-resolution. In European conference on computer vision, pages 199. Springer, 2014.
[24] Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Alex Lamb, Martín Arjovsky, Olivier Mastropictro, and Aaron C. Courville. Adversarially learned inference. ArXiv, abs/ 1606.00704, 2017.
[25] Vincent Dumoulin, Ethan Perez, Nathan Schucher, Florian Strub, Harm de Vries, Aaron C. Courville, and Yoshua Bengio. Feature-wise transformations. Distill, 2018.
[26] Ian Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.
[27] Michael U Gutmann, Ritabrata Dutta, Samuel Kaski, and Jukka Corander. Likelihood-free inference via classification. arXiv preprint arXiv::407, 1407.4981, 2014.
[28] Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita. Deep back-projection networks for super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1664-1673, 2018.
[29] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[30] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in neural information processing systems, pages 6626-6637, 2017.
[31] Jonathan Ho, Ajay Jain, and P. Abbeel. Denoising diffusion probabilistic models. ArXiv, abs/2006. 1 11239, 2020.
[32] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-666 -65511, 2020.
[33] Xueqi Hu, Xinyue Zhou, Qiusheng Huang, Zhengyi Shi, Li Sun, and Qingli Li. Qs-attn: Query- selected attention for contrastive learning in i2i translation. arXiv preprint 27X44:2203.08483, 2022.
[34] X. Huang, Ming- Yu Liu, Serge J. Belongie, and Kautz. Multimodal unsupervised image-to- image translation. ArXiv, abs/1804. 2018.
[35] Xun Huang and Serge J. Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. 2017 IEEE International Conference on Computer Vision (ICCV), pages 1510-1519, 2017.
[36] S. lizuka, Edgar Simo-Serm and H. Ishikawa. Let there be color! ACM Transactions on Graphics (TOG), 35:1 - 11, 2016.
[37] Phillip Isola, Jun-Yan Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125-1134, 2017.
12

---

[38] Alexia Ahiicoer-Marttnnee Ke Li, Rémi Taillefe,,, Tal Kachman, and Ioannis Mitliagkas. Gotta go fast when generating data with score- score-based models. arXiv preprint #xXv:2105.14080 2021.
[39] Tero Karras, S. Laine, and Timo Aila. A bayee- generator architecture for generative ad- versarial networks. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4396-44900 2019.
[40] Tero Karras, S. Laine, Miika Aittala, Janne Hellsten, J. Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8 8107-8116, 16, 2020.
[41] Tero Karras, Samuli I Laine, and Timo Aila, A style-hased generator architecture for generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[42] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restora- tion models. In Advances in Neural Information Processing Systems, 2022.
[43] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1646-1654, 2016.
[44] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1442.6990, 2015.
[45] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. 2013.
[46] Pierre- Yves Laffont, Zhile Ren, Xiaofeng Tao, Chao Qian, and James Hays. Transient attributes for high-level understanding and editing of outdoor scenes. ACM Transactions on Graphics (TOG), 33:1 11, 2014.
[47] Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. Deep laplacian pyramid networks for fast and accurate super-resolution. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5835-5843, 2017.
[48] Gustav Larsson, M. Maire, and Gregory Shakhnarovich. Learning representations for automatic colorization. ArXiv, 000166666366... 2016.
[49] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. In CVPR, volume page 4, 2017.
[50] Hsin-Ying Ying Lee, Hung-Yu Yu Tseng, Jia-Bin Huang, Maneesh Kumar Singh, and Ming-Hsuan Yang. Diverse image-to-image translation via disentangled representations. In ECCV, 2018.
[51] Hsin-Ying Lee, Hung- Yu Tseng, Qi Mao, Jia-Bin Huang, Yu-Ding Lu, Maneesh Kumar Singh, and Ming-Hsuan Yang. Drit++: Diverse image-to-image translation via disentangled representations. International Journal of Computer Vision, pages 1-16, 2020.
[52] Ke Li and Jitendra Malik. Implicit maximum likelihood estimation.
arXiv preprint
arXiv: 09087, 2018
[53] Ke Li Shichong Peng*, Tianhao Zhang* and Jitendra Malik. Multimodal image synthesis with conditional implicit maximum likelihood estimation. International Journal of Computer Vision, May 2020.
[54] Zhen Li, Jinglei Yang, Zheng Liu, Xiaomin Yang, Gwanggil Jeon, and Wei Wu. Feedback network for image super-resolution. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 3867-3876, 2019.
[55] Ming- Yu Liu, Breuel, and J. Kautz. Unsupervised image-to-image translation networks. In NIPS, 2017.
13

---

[56] Rui Liu, Yixiao Ge, Ching Lam Choi, Xiaogang Wang, and Hongsheng Li. DivCo: : Diverse
conditional image synthesis via contrastive generative adversarial network. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16372-16381, 2021.
[57] James Lucas, G. Tucker, Roger B. Grosse, and Mohammad Norouzi. Don't 't blame the elbo! linear vae perspective on posterior collapse. In NeurIPS, 2019.
[58] Zhiming Luo, Akshaya Kumar Mishra, Andrew Achkar, Justin A. Eichel, Shaozi Li, and Pierre- Marc Jodoin. Non-local deep features for salient object detection. 2017 IEEE Conference on Computer Vision and Pattern Recognition ( (CVPR), pages 1593-66014 2017.
[59] Xuezhe Ma, Xiang Kong, Shanghang Zhang, and Eduard H. Hovy. Decoupling global and local representations via invertible generative flows. In ICLR, 2021.
[60] Qi Mao, Hsin- Hsin-Ying Lee, Hung-Yu Yu Tseng, Siwei Ma, and Ming-Hsuan Yang. Mode seeking generative adversarial networks for diverse image synthesis. 20/9 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 1429-1437, 2019.
[61] Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. In ICLR, 2017.
[62] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014.
[63] Kamal Nasrollahi and Thomas B. Moeslund. Super-resolution: a comprehensive survey. Machine Vision and Applications, 1011442-4688 2014.
[64] Augustus Odena, Christopher Olah, and Jonathon Shiens. Conditional image synthesis with auxiliary classifier gans. In International conference on machine learning, pages 2642-2651, 2017.
[65] Yingxue Pang, Jianxin Lin, Tao Qin, and Zhibo Chen. Image-to-image translation: Methods and applications. ArXiv, abs/2110.08622, 2021.
[66] Seong-Jin Park, Hyeongseok Son, Sunghyun Cho, Ki-Sang Hong, and Seungyong Lec, SRfeat: Single image super-resolution with feature discrimination. In Proceedings of the European Conference on Computer Vision (ECCV), pages 455, 2018.
[67] Taesung Park, Alexei A. Efros, Richard Zhang, and Man Zhu. Contrastive learning for unpaired image-to-image translation. In European Conference on Computer Vision, 2020.
[68] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.0643 2015.
[69] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak
Lee. Generative adversarial text to image synthesis. arXiv preprint (rX.vv1605.00336, 2016.
[70] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and variational inference in deep latent gaussian models. In International Conference on Muchine Learning, 2014.
[71] Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. In NeurlPS, 2018.
[72] Mchdi S. M. Sajjadi, Bernhard Schölkopf. and Michael Hirsch. Enhancenet: Single image super-resolution through automated texture synthesis. 2017 IEEE International Conference on Computer Vision (ICCV), pages
2017.
[73] Tim Salimans and Diederik Kingma. Weight normalization: A simple reparameterizatioo to accelcrate training of deep neural networks. ArXiv, Ab2/1602.,7886. 2016.
[74] Sungyong Seo, Jing Huang, Hao Yang, and Yan Liu. Interpretable convolutional neural networks with dual local and global attention for review rating prediction. Proceedings of the Eleventh ACM Conference Recommender Systems, 2017.
14

---

[75] Tamar Rott Shaham, Michaël Gharbi, Richard Zhang, Eli Shechtman, and Tomer Michaeli. Spatially-adaptive pixelwise networks for fast image translation. 2021 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 14877-14886, 2021.
[76] Taizhang Shang, Qiuju Dai, Shengchen Zhu, Tong Yang, and Yandong Guo. Perceptual extreme super resolution network with receptive field block. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1778-1787, 2020.
[77] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep un- supervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 1256-2265 PMLR, 2015.
[78] Yang Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. ArXiv, abss//90.005600 2019.
[79] Jheng- theng-Wei Su, Hung kuo Chu, and Jia-Bin Huang. Instance-aware image colorization. 2020 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 7965- 7974, 2020.
[80] Ying Tai, Jian Yang, and Xiaoming Liu. Image super-resolution via deep recursive residual network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3147-3155, 2017.
[81] Yaniv Taigman, A. Polyak, and Lior Wolf. Unsupervised cross-domain image generation. ArXiv, abbs//611..22000 2017.
[82] Chunwei Tian, Yong Xu, Lunke Fei, and Ke Yan. Deep learning for image denoising: A survey. ArXiv, abs/1810.050522, 2018.
[83] Arash Vahdat and J. Kautz. Nvae: A deep hierarchical variational autoencoder. ArXiv, abs/2007.03899) 2020.
[84] Lijun Wang. Huchuan Lu, Xiang Ruan, and Ming-Hsuan Yang. Deep networks for saliency detection via local estimation and global search. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3183-319 2015.
[85] Tianyang Wang, M. Sun, and Kaoning Hu. Dilated residual network for image denoising. ArXiv, 0aos//708.00473 2017.
[86] Ting-Chun Wang, Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8798-8807, 2018.
[87] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Chen Change Loy, Yu Qiao, and Xiaoou Tang. Esrgan: Enhanced super-resolution generative adversarial networks. CoRR, abs/1809.00m 01889000211.. 2018.
[88] Yifan Wang, Federico Perazzi, Brian McWilliams, Alexander Sorkine- Horkung. Olga Sorkine- Hornung, and Christopher Schroers. fully progressive approach single-image super- resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion Workshops, pages 864-873, 2018
[89] Zhihao Wang, Jian Chen, and Steven CH Hoi. Deep learning for image super-resolution: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
[90] Xiangyu Xu, Deqing Sun, Jinshan Pan, Yujin Zhang, Hanspeter Pfister, and Ming-Hsuan Yang. Learning super-resolve blurry face and text images. In Proceedings of the IEEE international conference on computer vision, pages 251-260, 2017.
[91] Chih- Chih-Yuan Yang, Chao Ma, and Ming-Hsuan Yang. Single-image super-resolution: A bench- mark. In Proceedings of European Conference Computer Vision, 2014.
[92] Zili Yi, Hao Zhang, P. Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image-to-iman translation. 2017 IEEE International Conference on Computer Vision (ICCV), pages 2868-2876, 2017.
15

---

[93] Donggeun Yoo, Namil Kim, S. Park, Anthony S. Pack, and In-So Kweon. Pixel-level domain transfer. ArXiv, abs/1603.07444 2016.
[94] Yuan Yuan, Siyuan Liu, Jiawei Zhang, Yongbing Zhang, Chao Dong, and Liang Lin. Un- supervised image super-resolution using cycle-in-cycle generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 701-710, 2018.
[95] Fangneng Zhan, Jiahui Zhang, Yingchen Yu, Rongliang Wu, and Shijian Lu. Modulated contrast for versatile image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[96] K. Zhang, Shuhang Gu, Radu Timofte, Taizhang Shang, Qiuju Dai, Shengchen Zhu, Tong Yang, Yandong Guo, Younghyun Jo. Sejong Yang. Seon Joo Kim, Lin Zha, Jiande Jiang, Xinbo Gao, Wen Lu, Jing Liu, Kwangjin Yoon, Taegyun Jeon, Kazutoshi Akita, Takeru Ooba, Norimichi Ukita, Zhipeng Luo, Yuehan Yao, Z. Xu, Dongliang He, Wenhao Wu, Yukang Ding, Chao Li, Fu Li. Shilei Wen. Jianwei Li. Fuzhi Yang, Huan Yang, Jianlong Fu, Syung-Hoon Kim, JacHyun Back, J. C. Ye, Yuchen Fan, Thomas S. Huang, Junyeop Lee, Bokyeung Lee, Jungki Min, Gwantae Kim, Kanghyu Lee, Jaihyun Park, Mykola Mykhailych, Haoyu Zhong, Yukai Shi, Xiaoju Yang. Zhijing Yang. Liang Lin, Tongtong Zhao, Jinjia Peng, Huibing Wang. Zhi Jin, Jiahao Wu, Yifu Chen, Chenming Shang, Huanrong Zhang, Jeongki Min, S HrishikeshP., Densen Puthussery, and V JijiC. Ntire 2020 challenge on perceptual extreme super-resolution: Methods and results. 2020 JEEE/CVF Conference on Compuler Vision and Pattern Recognition Workshops (CVPRW), pages 2045-2057, 2020.
[97] K. Zhang, W. Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep enn for image denoising. IEEE Transactions on Image Processing, 26:3142-3155 2017.
[98] K. Zhang, W. Zuo, and Lei Zhang. Ffdnet: Toward fast and flexible solution for cnn-based image denoising. IEEE Transactions on Image Processing, 27:4608 46222 2018.
[99] Kai Zhang, W. Zuo, and Lei Zhang. Learning a single convolutional super-resolution network for multiple degradations. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3262-3271, 2018.
[100] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European Conference on Computer Vision, pages 649-666. Springer, 2016.
[101] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[102] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. CoRR, abs/1802.08897 2018.
[103] Jiaojiao Zhao, J. Han, Ling Shao, and Cees G. M. Snoek. Pixelated semantic colorization. International Journal of Computer Vision, 128:818-28 88:34, 2019.
[104] Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. 2017 IEEE International Conference on Computer Vision (ICCV), pages 2242-2251 2017.
[105] Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A. Efros, O. Wang, and E. Shechtman. Toward multimodal image-to-image translation. In NIPS, 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
16

---

(b) Did you describe the limitations of your work? [Yes] See Section S for details.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 5 for details.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results... results.
(a) Did you state the full set of assumptions of all theoretical results? [N/A] No theoretical
(b) Did you include complete proofs of all theoretical results? [N/A] No theoretical results.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main perimental results (either in the supplemental material or as a URL.)? [Yes] See the appendix and the project website.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] Each experiment is computationally expensive. but we during the course of developing the method, found it to be quite robust different seeds. We believe that this is because our method optimizes a non-adversaria objective, which makes the training stable. We demonstrate the training stability exhibited by our method in Sect. C of the appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 3 for details.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Existing assets
cited.
(b) Did you mention the license of the assets? [N/A] Assets used in this research carry no license/access restrictions.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
No new assets.
(d) Did you discuss whether and how consent obtained from people whose data you's 're using/curating? [N/A] No new assets.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] No personally identifiable information in the data.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] No crowdsourcing.
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A] No crowdsourcing.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] No crowdsourcing.
17


Reviews:
[{"review_id": "baCBlUW6kv", "reviewer": "Reviewer_UYaB", "summary": "This work proposes a new method Conditional Hierarchical IMLE to get around limitation of requirements of a large number of samples to generate high-fidelity images.\nThe proposed CHIMLE is shown to improve generated image fidelity, with a clear reduction in Fréchet Inception Distance compared to the prior best IMLE-based method.", "questions": "See above comments.", "limitations": "The limitations and  social impact have been discussed in the manuscript.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\nTo generate samples in a way such that the best sample is about as similar to the observed image as if a large number of samples had been generated without actually generating that many samples,\nthe author proposes several methods, including partitioning the latent code, partial evaluation of latent code components, and iterative construction of latent code. These ideas give rise to a the method of Conditional Hierarchical IMLE or CHIMLE. \nThis work demonstrates that CHIMLE significantly outperforms the prior best IMLE-based method in terms of both fidelity and diversity on a variety of tasks. Besides, they also show that CHIMLE achieves superior image fidelity and mode coverage compared to leading general-purpose multimodal and task-specific methods.\n\n\n\nWeaknesses:\n1. In the experiments, what backbones are adopted in CHIMLE for different tasks, e.g., colorization, super resolution. If the comparison fair as different methods use different generator structure, e.g., BicycleGAN, MSGAN.\n2. Some proper metrics should be used for different tasks, e.g., PSNR for super-resolution.\n3. Please check the citation forms, I find some citations are in false format, e.g.:\nYingchen Yu Rongliang Wu Shijian Lu Fangneng Zhan, Jiahui Zhang. Modulated contrast for versatile image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YRp69Ep-Qn-", "reviewer": "Reviewer_VssD", "summary": "This paper aims to generate diverse and high-fidelity images in conditional image synthesis with a relatively small number of samples. Compared with the previous method conditional IMLE that counters the mode collapse, this paper improves the tradeoff between sampling efficiency and quality by introducing a diver-and-conquer selection mechanism by dimensions in the latent space. The proposed strategy brings better fidelity as well as the diversity of the generated results.", "questions": "1. As expressed in the weakness part, I have the following concern: does the hierarchical conditioned generation process deprecate the parallelism of this method? \n2. And, how does the sampling efficiency connect to the proposed strategy that uses divide-and-conquer?\n3. It is unclear whether the improved network architecture or the CHIMLE method contributes more to the SOTA performance. Can this be verified in new ablation studies that adopt the same generator architecture while using cIMLE, or adopt cIMLE's RRDB structure while using CHIMLE?", "limitations": "The author has addressed the limitations and potential negative societal impact at the end of this paper. ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n+ The proposed divide-and-conquer strategy for latent code selection is very interesting. The division of latent code by dimensions is simple yet effective, and the partial evaluation of latent code components can well address the sub-problems with a modified criterion. When combing the latent code components, this method also considers the correlation between them and proposes the solve the sub-problems from low to high resolutions.\n+ The intuition behind this proposed method is clearly explained. \n+ The experimental results show great improvement on multiple tasks (16x super-resolution, night-to-day image translation, image colorization, extreme compression artifacts reduction)\n\nWeakness:\n- The divide-and-conquer algorithm aims to solve conceptually difficult problems with improved efficiency and parallelism. However, since this method requires constructing latent code at lower resolutions first, parallelism cannot be achieved. Besides, although the adoption of this time-tested algorithm into latent space sampling is very interesting, the author does not discuss too much about why high efficiency/low cost is vital for this problem. While the author does mention the sampling efficiency of CHIMLE is better than cIMLE, it is not very clear how it connects to the proposed strategy.\n- This paper changes both network structure and the sampling strategy, while the experimental results do not include such ablation studies that disentangle the two. In the IMLE paper, the authors at least compare the SRIM and BicycleGAN with the same generator architecture to show the effectiveness of the proposed method, which could serve as a good example for this paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "ND3Oh4C5ssH", "reviewer": "Reviewer_LstA", "summary": "This paper presents a novel way of applying a non-adversarial image generation technique called IMLE to image-to-image problems.\nNamely, the authors propose to split a latent code into several parts corresponding to different image scales. These sub-codes are selected coarse-to-fine: First, the code corresponding to the lowest resolution is found among the randomly sampled candidates. Afterward, this latent sub-code gets frozen, and subsequent parts of the code are chosen. As demonstrated in the paper, this hierarchical approach achieves the same quality as the baseline, with a lower number of generated samples.\n\nAccording to the reported results of the evaluation, the presented approach outperforms several general-purpose image-to-image models, as well as some task-specific methods.", "questions": "I ask the authors to address the concerns listed above.", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. Strengths.\n\n    This paper proposes an interesting idea of incorporating multi-scale techniques into the IMLE-based generative model.\n    This sound quite reasonable and it's a pleasure to see that this trick helps to reduce the number of required candidate samples, which is a common downside of IMLE.\n     Moreover, the obtained results show that in the considered setting the presented solution outperforms even the task-specific baselines, which proves that CHIMLE may be useful for applications.\n     Also, the authors have submitted the code, which is a good service for the community. \n\n1. Weaknesses.\n\n    * The explanation is sometimes unnecessarily wordy: while the method finally just employs the multi-scale search of latent codes, it is presented in a much more general way, although only one way of latent division is considered. \n    * The evaluation uses F-measures computed from precision and recall. As shown in [1], these metrics struggle with issues, and it is more common to compute so-called Improved Precision and Recall [1].\n    * Unfortunately, the resolution of generated images is not specified explicitly in the paper. Therefore, I am not sure if this approach may be scaled up to the image size of 1K. I ask the authors to provide the image size the model operates. \n    * Why were not models with diverse outputs considered as task-specific baselines? E.g. for the case of super-resolution, SRFlow [2] may be a suitable one. This paper presents a novel way of applying a non-adversarial image generation technique called IMLE to image-to-image problems.\n\n\n1. References\n\n    [1] Kynkäänniemi et al. Improved Precision and Recall Metric for Assessing Generative Models. NeurIPS 2019.\n\n    [2] Lugmayr et al. SRFlow: Learning the Super-Resolution Space with Normalizing Flow. ECCV 2020.\n\n1. Post-rebuttal comments.\n\n    I thank the authors for their feedback. It has addressed my main concerns. Therefore, I am inclined to increase the score.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3YnjSVpya-", "reviewer": "Reviewer_WPok", "summary": "The paper focuses on \"one-shot\" (so the SOTA autoregressive transformer and diffusion models are not considered for the comparison in this paper) multi-modal conditional generative tasks. The proposed method is based on the IMLE algorithm. The authors reduce the computational cost (in the training stage) of the previous conditional IMLE framework by a divide and conquer strategy. Experiments are conducted on night-to-day, monocular image super-resolution, image colorization, image decompression, and layout-to-image tasks to validate the effectiveness of the proposed method over the previous conditional IMLE scheme.", "questions": "Please see Weakness.", "limitations": "The authors address the limitations and potential social impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n1. The paper is well-written, and the idea of divide and conquer for improving cIMLE is sound.\n2. The authors provide a clear introduction to the previous cIMLE method, which is very helpful for readers unfamiliar with this field.\n3. According to extensive experimental results, the proposed method synthesizes more realistic and diverse images compared to several baseline approaches.\n*Weakness*\n1. Several metrics are proposed to evaluate the realism and diversity of the generated images, e.g., precision and recall. Can the authors provide more intuitions why these metrics are not used in measuring the diversity in conditional generative modeling?\n- Kynkäänniemi et al., \"Improved Precision and Recall Metric for Assessing Generative Models.\"\n2. Recent data augmentations can significantly improve the performance of GAN-based approaches, e.g., ADA, especially in the limited training data case. Can the authors comment on 1) how the proposed IMLE-based performs against the GAN-based methods trained with ADA and 2) how the proposed algorithm performs in the limited training data case?\n- Karras et al., \"Training Generative Adversarial Networks with Limited Data.\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept