DEEAPR: Controllable Depth Enhancement via Adaptive Parametric Feature Rotation
Authors: Hanul Shin
Youngchan Song
Soo Min Kang
Venue: Submitted to ICLR 2023 (2023)

Abstract: Understanding depth of an image provides viewers with a better interpretation of the 3D structures within an image. Photographers utilize numerous factors that can affect depth perception to aesthetically improve a scene. Unfortunately, controlling depth perception after the image has been captured is a difficult process as it requires accurate and explicit depth information. Also, defining a quantitative metric of a subjective quality (i.e., depth perception) is difficult which makes supervised learning a great challenge. To this end, we propose DEpth Enhancement via Adaptive Parametric feature Rotation (DEEAPR), which modulates the perceptual depth of an input scene using a single control parameter without the need for explicit depth information. We first embed content-independent depth perception of a scene by visual representation learning. Then, we train the controllable depth enhancer network with a novel modulator, parametric feature rotation block (PFRB), that allows for continuous modulation of a representative feature. We demonstrate the effectiveness of our proposed approach by verifying each component through an ablation study and comparison to other controllable methods.

Under review as a conference paper at ICLR 2023
DEEAPR: CONTROLLABLE DEPTH ENHANCEMENT VIA ADAPTIVE PARAMETRIC FEATURE ROTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Understanding depth of an image provides viewers with a better interpretation of the 3D structures within an image. Photographers utilize numerous factors that aesthetically improve scene. Unfortunately, con-
can affect depth perception trolling depth perception after the image has been captured is difficult process it requires accurate and explicit depth information. Also, defining a quantitative metric of subjective quality (i.e., depth perception) is difficult which makes su- pervised learning a great challenge. To this end, we propose DEpth Enhancement via Adaptive Parametric feature Rotation (DEEAPR), which modulates the per- ceptual depth an input scene using a single control parameter without the need for explicit depth information. We first embed content-independent depth percep- tion of a scene by visual representation learning. Then, we train the controllable depth enhancer network with novel modulator, parametric feature rotation block (PFRB), that allows for continuous modulation of a representative feature. We demonstrate the effectiveness of our proposed approach by verifying each compo- nent through an ablation study and comparison to other controllable methods.
INTRODUCTION
Enhancing depth perception in 2D images exhibits more realistic image content the viewers allows for better interpretation of the 3D scene structure. The human visual system uses vari- cty of cues to inference depth information, such as disparity that arises from binocular vision or mo- tion (Kim et al. (2016). Depth cues in single static images, such as occlusion, shading, or blur, are referred to pictorial depth cues (O'Shea et al.) 1997p. Many artists and photographers utilize pic- torial depth cues by adding synthetic effects to in- crease impression of depth in still images. For example, amplifying the defocus blur triggers the depth-of-focus cue, where the objects in the range of focus appear sharp and those further away ap- pear blurry. However, enhancing depth perception of an image by manipulating depth cues without explicit depth information of scene is challeng- ing task. Moreover, depth perception of a scene is subjective quality that varies from image to image, which difficult to learn in supervised manner.
Shellwv
deg
Eacoder
Degneentation space
stallow
Contrcllbble Depth Eclsncer
Figure 1: Illustration of DEEAPR framework.
Our final goal to modulate the perceptual depth of input scene using single control parameter without the need for explicit depth information. While the traits of depth perception make supervised training very difficult, we rely on the recent success of unsupervised visual representation learning that considers high dimensional space in which the degree of similarity is inversely correlated with the distance between instances. As the first step, embed content-independent depth perception of scene onto the representation space

---

Under review as a conference paper at ICLR 2023
by combining contrastive learning (Hadsell et al. 2006) and metric learning (Balntas et al. 2016). The representation space of the encoder then bridges the image space and the control parameter axis. Instead of relying full supervision, we aim to train the direction of depth enhancement during depth enhancer training. Fig. Iillustrates our DEEAPR framework combining visual representation learning and a controllable neural network. We first train depth representation encoder in an unsupervised manner, then train the direction of depth perception enhancer such that appropriate change would be induced in the image domain when the depth representative feature is modulated.
Our contributions are three-fold. First, we propose a novel strategy to learn visual representation space of style disentangled from image contents. Second, we present a controllable neural network that enhances the depth perception of an input image according to a single control parameter. Lastly, we present a novel modulator, parametric feature rotation block (PFRB), that allows for continuous modulation of feature representation while preserving its norm.
2
RELATED WORKS
2.1 VISUAL REPRESENTATION LEARNING
Recent success of unsupervised representation learning in natural language processing has moti- vated the computer vision community to pursue learning a manifold of natural images. To learn an effective feature space for downstream tasks, many works utilized contrastive loss (Hadsell et al.) 2006) to discriminate highly correlated instances (i.e., positive pairs) from others. In general, var- tous content-preserrvie
data augmentation is applied to the same image independently in order to generate positive pairs (Chen et al., 2020a). All other examples obtained from different images are considered negative samples, and the objective is to minimize the distance between positive pairs with respect to myriads of negative samples. Momentum Contrast (MoCo) (He et al., 2020b) is representative upproach which aims build a dynamic dictionary encoded by mhoving-averager momentum encoder. Furthermore, DASR (Wang et al.) (021) adapts MoCo to learn a content- independent degradation representation for the blind super-resolution task. Our strategy similar to MoCo v2 (Chen et al.) (200b) and partly motivated from DASR in that we aim to train content- independent representation of depth perception via contrastive learning. We use triplet margin loss Balntas et al. 2016)) additional regularizer to disentangle depth perception from the image content.
2.2 DEPTH PERCEPTION ENHANCEMENT
Interpretation of 3D scene structure from single static images can be obtained from various pictorial depth cues, such as occlusion, relative size, linear perspective, aerial perspective, texture gradients, depth-of-focus and shading (Cipiloglu al. 2010). These depth cues are utilized in conventional methods by for example, enhancing local contrast (Ritschel et al. 2008 Lee et al. (0144), slightly darkening or lightening the background partly occluded objects (Luft et al.] 2004), or applying shading and/or shadows (Vergne et al. 2011] Lopez-Moreno et al. 201 1p. While these methods require explicit use of depth information the pixel level, Hel-Or et al. (017)) extract reflectance derivatives of an image to manipulate shading cues. Conventional methods often require precise ma- nipulation of multitude of parameters for plausible and visually pleasing output. More recently, there have been great interest in developing deep learning-bar -baeed approaches to render artificial depth-of- field effect, often referred to as the Bokeh effect. Indeed, through the AIM Boken Challenges (Igna-) tov al. 2019 2020b), numerous approaches created depth-dependent blur effects by using depth information explicitly (Peng et al.) 2202)) or implicitly (Ignatov et al.) 2020a Qian et al. 2020 Dutta et al. (021). Our DEEAPR differs from aforementioned approaches in two major aspects: DEEAPR procudes a depth enhanced image based on a single parameter without explicit depth information.
2.3 CONTROLLABLE NEURAL NETWORK
Conventional deep learning methods learn a deterministic mapping for each task and output single image given an input. Controllable methods, on the other hand, provide flexibility in producing im- ages at continuous levels for modulation over diverse imagery effects. Controllability has appeared in several low-level vision tasks such as style transfer to permit smooth interpolation between differ-
ent artistic styles (Dumoulin et al., 2016; Ghiasi et al. 2017 Huang & Belongie 2017
Wang al.
2

---

Under review as a conference paper at ICLR 2023
Gommiiaa proas of poiiro and modiggii supplo
Sivib prbae
Randen Reaze
Aextrn bilp Ip 000
AMRS (p=3.2)
Diska inlaneer
Radker (p-0.))
Sadio
AMON maze
K+
Ip +022
Rosto
-050
Bodte
m05
Figure 2: (a) Generation process of query, positive and hard negative sample. (b), (c) The three samples have same image content but hard negative sample (b3/c3) has different depth perception from the query (b1/c) and positive sample (b2/c2).
2019b) and image restoration to restore images continuous levels of degradation (Fan al. 2019; He et al. 2019 2020a Kim et al., [021]] Wang et al. 2019a b). Various designs of controllable neu- ral networks have been explored, which include tuning different parameters of normalization layers (Dumoulin et al.)
2016 Ghiasi et al. 2017 Huang & Belongie 2017). applying linear interpolation on parameters of multiple correlated networks (Wang al. 2019b), modifying intermediate layer 2019) and using a network consisting of several fully-connected layers to learn the channel importance for specified task (He et al. 2020a Kim et al., [202). Similarly, [Wang] et al. (2019a) designed neural network consisting of main branch and tuning branch to modu- late the feature of the main branch. While aforementioned approaches applied the control parameter directly to modify the base image generation network, the control parameter in DEEAPR is encoded to modulate the depth representative feature to separate the base network from the variable part, lifting the burden of multitask learning from the base network.
features (He et al.
3 METHODS
The DEEAPR network consists of a depth representation encoder and a controllable depth en- hancer. The representation encoder is responsible for generating the depth representative feature of the input image and the enhancer for generating a depth enhanced image. The representation encoder and the enhancer are trained in two separate stages, which are described in Sec. 3.1 and 3.2 resp.
3.1 LEARNING DEPTH REPRESENTATION
is crucial that the content of an image is preserved in the process depth enhancement. Thus, our goal is to learn feature representative of depth that can be modulated without affecting the contents of a given image. Due to the lack quantitative metric that measures the perception of depth, we devise data-driven approach to extract feature representative of depth via visual representation learning. As current existing contrastive loss-based algorithms are limited in distinguishing content from style (or depth perception in the context of depth enhancement), we introduce hard negative sample, k_, for each query. The hard negative sample shares the same content as the query and the positive sample, but has different style reinforce the encoder embed visual instances according to their style, not content. This is achieved by adding the triplet margin loss (Baintas et al. 2016) as regularization, Cregs to the MoCo v2's framework, as follows:
exp (q k+/T)
Liotal = La + XLreg = - log Σ exp (g kz/T) + À max((g k-) (g k4) + m, 0}. (1)
where K is the dictionary size, is the softmax temperature, À is regularization parameter and m is a margin. Margin m regulates the difference of correlations between query-positive and query-hard negative pairs. The Greg term is crucial for disentangling style from content as it forces the query to push away image patches with same content but different depth perception. We analyze the effect of the regularizer in representation space in Sec. 5.1

---

Under review as a conference paper at ICLR 2023
10 antiocua of DEEAI atwet
Aepeostdie Faridr (boden
Id Nanamaaaiieaerrrnn.... (Prite)
Uplinck 10
GELL
Figure 3: (a) The overall architecture of DEEAPR network and (b), (c), (d) its constituents.
To obtain positive and hard negatives for given query, applied pre- and post-transforma de- pending on their ability to affect the perception of depth. That is, as blur, brightness, contrast and saturation affect depth perception, obtain the query. positive and hard negative samples. Crop, resize, horizontal flip, additive white Gaussian noise and hue, the other hand, have effect depth perception, and thus were ap- plied independently. We applied the synthetic depth enhancement model (described in Sec. 4.1] the query and positive pair or the hard negative sample with equal probability. We provide concrete examples of query, positive and hard negative samples in Fig. 2 and (c).
applied these operations commonly across the image
3.2 LEARNING DEPTH ENHANCEMENT EFFECT
Network architecture. Fig. illustrates the architecture of the DEEAPR network and its con- stituents. The parametric feature rotation blocks (PFRBs) modulate the depth representative feature according to a and provide modified local representations to the depth enhancer. The depth enhancer, which is responsible for enhancing the depth perception to given extent, takes structure of U-Net consisting of N = Downblocks and N Upblocks. The features are down- scaled via strided convolution and upscaled via transposed convolution. As the features from the Downblock and Upblock correspond to blurred features of the input due to the down- and upscaling operations, ensure that the blocks remain connected via the skip connection to retain sharp as well as contrast and brightness information.
architectural
Downblock and Upblock. Both the Downblock and the Upblock reference depth representation, but treat the information in different ways. The Downblock references the global depth representative feature and applies channel attention to its input to selectively downsample Upblock receives two sets of fcatures: one from the previous Upblock (or the last Downblock in case of the first Upblock) and another from the skip-connection. We ensure that spatial variance maintained while combining these features by applying spatially varying scales and biases using spatial feature transform (SFT) layer (Wang et al., 2018). The SFT layer takes the modified local condition.
Each
depth representation corresponding to each resolution
Parametric Feature Rotation Block (PFRB). The local depth representation modulated by hy- perparameter, which controls the strength of the depth enhancement effect. As representative features reside along the unit hypersphere due to the normalization layer, the features are controlled by applying sinusoidal operations the encoded control parameter. That is, the local representation multiplied by cosine and sine of wa + Ø, where w and ø are trainable vectors. Formally,
PFRB σ) = [wos(wa + +) ④ ftocat, sin(wo + Φ) @ ftocat,):
(2)
where @ denotes element-wise multiplication (i.e., Hadamard product) and denotes concatena- tion. Notice that PFRB preserves the norm of output features regardless of the value of a by the Pythagorean trigonometric identity. This ensures that the output feature of PFRB has a local norm of 1, which in turn stabilizes training of the depth enhancer module.
4

---

Under review as a conference paper at ICLR 2023
Self-Supervised Training. Suppose E R 1] denotes the control parameter of depth enhance- ment, where o = 0 implies that the target image is the imput image and o = 1 implies that the target is its depth-enhanced version. Training depth enhancement network continuous levels of E [0, 1] in a fully supervised way would require pairs of an input image and a depth-enhanced image various levels of However, the initial depth perception of input differs from image to image, which leads to different perception of depth enhancement despite using the same parameters. Therefore, we utilize the fact that the direction of change remains consistent (i.e., a + 1, the target image should be perceived deeper) and only aim to train the direction of depth enhancement.
Indeed, when training the DEEAPR network, we sample the control parameter a from {0,1) with equal probability but aim to infer depth enhanced images in between I [0, 1]. Given an image I and its depth-enhanced version O, the depth enhancer network is trained using I.1 loss as reconstruc- tion loss, where the target is the input when 0 and the target is O otherwise. Inferencing depth enhanced images at continuous levels of E [ 1] is made possible despite training at discrete levels of {0, the the sinusoidal functions in the PFRB allow for smooth interpolation between the depth representation features that reside on the unit hypersphere.
4 EXPERIMENTAL SETUP SETUP
4.1 MODELING DEPTII ENHIANCEMENT EFFECT
We considered two processes to synthesize depth enhanced images: contrast manipulation and cir- cular blur effect. Increasing the contrast of near objects enhances volumetric perception, while decreasing the contrast of the background simulates an atmospheric obscuring effect. To model the depth-dependent contrast effect, we modified the standard haze model (Yang & Sun 2018):
(contras:(x) (xx)T(() + + - T(x)) and T(x) = max exp(-n(d(x)) - 6contvast)) (3)
where [(x) is the intensity of scene radiance at position x, A the atmospheric light and d(xx E [0, 1] the normalized scene depth. The standard haze model only simulates the obscuring effect by setting A to some constant value. However, we observed that this model does not account for the contrast enhancement effect of close objects and creates visually unpleasing artifacts in images with low brightness. Correspondingly, we redefined A I + a, where I [7(x) Evx T(x) global average of the scene radiance and a is a constant. In addition, we did not limit the max value of T(x) to 1 to enhance contrast of relatively near objects (i.c., d(x) < &controst).
Synthetic blur is frequently used in photography to manipulate depth perception of a scene. Images captured with shallow offth-of-field have blurred background, making the in-focus objects stand out. We modified the Bokeh effect synthesis in Wadhwa et al.] (2018) and applied a circular blur kernel with its size r linearly increasing with respect depth, d(x). The blur kernel is applied to regions farther away than the distance oatur (i.e.. d(x)= >> Sstur) and the size of the kernel increases with slope s. When applying the kernel, we ignored pixel values whose depth difference exceeded threshold dmaax = 0.1 with respect to the center pixel, x,. In summary, the final output image O(x) was obtained by
( = Mr, E [(x)) [d(x,) d(x)) dmax)) Ex. Ex.Er(e)) (d(x.) -(0)))))mm)))msm Ocontrast (X;), (4)
where [2() is local circular window with radius r(x) = max[s(d(x)) 6,tur).03
Both effects require accurate depth information of scene, d(x). For this purpose, we used pre- trained DPT (Ranitl et al.] 202T) for monocular depth estimation and refined the predicted depth using the fast bilateral solver (Barron Poole 2016).. We observed that depth prediction by DPT was unreliable in regions where depth changed drastically or absolute depth value was large; thus, we integrated these two heuristies into generating a confidence map for the fast bilateral solver.
4.2 TRAINING DETAILS
Representation Learning. We trained the depth representation encoder that takes on the architec- tural structure of ResNet-50 (He et al.] (016) with an additional fully connected layer as suggested in Chen al. (2020 b). Following the experimental setup in Chen et al. (002b)), set 0.07,
5

---

Under review as a conference paper at ICLR 2023
X lrpel sta
342 10
0 120
wa to
Figure 4: t-SNE visualization of representation space, color coded by the strength of depth enhance- ment effect. (b) Features are widely dispersed w/o regularization. (c), (d) As À increases, features at opposite extremes of depth perception are located far away from each other.
K = 65536 and momentum to 0.9. The learning rate was initialized to 0.03 and adjusted with cosine annealing scheduler. The network was trained on V100 GPUs for 2000 epochs with a batch size of 32. We used DIV2K train set (Agustsson & Timofte 2017). Flickr2K (Lim al. (017) and Waterloo Exploration Database (WED) (Ma et al. (017) to obtain a total of 8194 high quality images. In unsupervised representation learning, it important to show variety of strengthening effects to train a rich representation space. Therefore, we randomly sampled each parameter of depth enhancement model uniformly from the following range: a [0,15, 0.3). 1) [R|00.7, 0.9). Scontrast: Obtur E (0.35, 0.5) and & E R[11,33.8)
DEEAPR Training. The DEEAPR network was trained on 8 V100 GPUs for 5000 epochs with batch size of 32 using the same dataset. We used the representation encoder trained at À = 10 and fixed its weights during training of the depth enhancer. The learning rate was initialized to 0.005 then adjusted with cosine annealing scheduler. The parameters of depth enhancement model were fixed for consistency: a 0.225, 17 0.8, &contrast ōstur 0.425 and s 22.44
5 RESULTS
5.1 ANALYZING REPRESENTATION SPACE
In this section, we analyze the role of the proposed triplet margin loss, Creg, defined in equation as a regularization function in learning the depth representative feature space independent of con- tent. The effectiveness of Creg is observed by training the depth representation encoder (Sec. 3.1 at various levels of À E {0, 10, 100}, then performing t-SNE analysis on the global depth repre- sentation features, Satohat, from the DIV2K validation set and generated 100 different depth-enhanced versions for each im- age by applying the depth enhancement model (Sec. 4.1). For visualization purposes, we defined a variable o' to control the numerous parameters that regulate the strength of depth enhancement with a single parameter. That is, the five parameters in the depth enhancer, a, 17, Scontrast, Oster and s, were linearly interpolated between their weakest and strongest values according to of (e.g., y O'mino' + T)max (1 o')), where o' was uniformly sampled from R(0, 1].
randomly selected 10 images specified value of X. Specifically,
Fig. shows the distribution pattern of encoded features in the representation space, where each feature point color coded by the regulation parameter o' (i.e., strength of depth enhancement). At
A 0, the distance between points that correspond to similar values of a' is similar to the distance between points that correspond to extreme values of o', resulting in a widely dispersed scattered plot. In contrast, as À increases, the points that correspond to extreme values are pushed away from each other while those with similar values are pulled towards each other, resulting in a thin band of gradation with extreme values at distant ends.
5.2 DEPTH PERCEPTION ENHANCEMENT
We expect the DEEAPR network to enhance the depth perception of a given image when o and show smooth transition of depth enhancement effect within the range of RR 1). Indeed, Fig.

---

Under review as a conference paper at ICLR 2023
Iegert
Trpur
Figure 5: Output of DEEAPR when [0, 1] showing gradual change of the depth enhancement effect. Results are best viewed electronically with zoom. See supplemental for additional examples.
Lapat::
oc Eayy Cocantidioon
Rascal cate
Figure 6: Six variants of PFRB used in the ablation study. Purple box indicates GELU activation.
shows that DEEAPR enhances depth perception of the input (column (a)) by simulating contrast and blur effect when a = (column (e)). Contrast enhancement of foreground objects (red boxes) strengthens volumetric perception, while the background (blue boxes) becomes blurry and hazy, making the foreground objects further stand out. Furthermore, DEEAPR not only produces plausible outputs when E {0. 1}. but also exhibits smooth transition of effects in range |[0,, 1]. More specifically, close objects (the berries in the first row, trees and buildings in the second row and the car in the third row) show gradual enhancement of contrast as a -> 1 and far objects (leaves in the first row, the bridge in the second row and the trees in the third row) become blurry and have decreased contrast as + 1, producing perceptual depth enhancement effects.
5.3 ABLATION STUDY
In this section, we study the importance of the PFRB design in producing the controllable depth en- hancement effect. To this end, we designed six additional controllable module variants and trained the DEEAPR network using the same pretrained representation encoder and condition as described in Sec. 3.1 and 4.2 resp. (see Fig. 6 for illustration of the six PFRB design variants, (a) to (f)). (a) To observe the effectiveness of modulating depth representative feature using the complementary si- nusoidal functions, we ablated the sine function in the modulator. (b) To observe the importance of norm preservation via the Pythagorean trigonometric identity, we embedded an additional normal- ization term to the cosine function in (a). (c) The importance of modulating the control parameter by affine transformation was sought by ablating 6. We further explored alternatives to modulating the control parameter by using (d) logistic funsstooounctton with trainable midpoint dp and curve steepness k. fully connected layers and (f) residual scaling. (e) and (f) were motivated from the controllable component of CFSNet (Wang et al. 2019a) and CResMD (He et al. 2020a), resp.
To study the effectiveness of each controllable module, we compared the variability of the output according to the control parameter a. For each image in the DIV2K validation set, we first obtained the output at various levels of (c.g., with the pretrained depth representation encoder commonly used across all six variants and the pro- posed DEEAPR model (referred to the baseline) and measured the cosine similarities of encoded features with respect to the encoded feature of the output produced at o 0.
0.0, 0.1, 0.9, 1.0). Next, encoded each output
The cosine similarities of the encoded features of the output is illustrated in Fig. 7 [a.. The proposed DEEAPR model (blue) demonstrates the second largest variability according to o with a smoothly decreasing cosine similarity curve increases. While Fig. ( residual scaling (pink) showed

---

Under review as a conference paper at ICLR 2023
the most discrepancy between 0 and in the cosine similarity curve, visualization of the output results in Fig. (c) revealed that residual scaling failed to reproduce the input at o = 0 and to produce depth enhanced image at >> 0. In addition, the cosine similarity plot revealed that Fig. ([a) w/o sine (orange) and Fig. ( logistic function (purple) failed to produce a variable outpur, which was further confirmed with visible changes in the image domain. While Fig. w/o phi (red) was able to produce a variable output in the encoding space and image space (as shown in Fig. ([a) and (c), resp.), it was less effective than the proposed model. Altogether, our study revealed that using both cosine and sine functions is crucial for modulating features obtained via representation learning
5.4 COMPARISON OTHER CONTROLLABLE METHODS
In this section, we compare DEEAPR to other controllable methods. Specifically, we consider image interpolation, network interpolation (Wang et al. (209b) and two other controllable methods:
CFSNet (Wang et al. 2019a) and CResMD (He et al.) 2020a).
For image interpolation, we obtained an image with smooth variation depth by interpolating between the input image and the depth enhanced image. That is, suppose the DEEAPR network is trained at fixed o = such that the network always enhances the depth perception of the input. Then the image interpolated output with controllable parameter o' E R[00, 1] is defined as follows:
Oing Dimg_interp(I:4 (I; = (1 - o')I + o' I))
(5)
where fo (I, ) is the DEEAPR network with parameters 6. Note that we use of in this section to avoid confusion with which is an input to the DEEAPR network.
For network interpolation, we interpolated network parameters associated with the input image and the depth enhanced image. The network affiliated with the input image was obtained by fine-tuning the trained network, [I, 1), to learn identity mapping (i.e., to reproduce the input image = 1). The network interpolated output was obtained by linearly interpolating the network parameters according to a' as follows:
net interp (I: for (I, where 6' =(1- (1- (1 f + a'0
For CFSNet and CResMD, we referred to the original papers for architecture and trained the net- works under the same condition as DEEAPR.
Similar to Fig. aa), we show the correlation between encoded features of the output at various levels o' R00 1] with respect to the output at o' 0 in Fig. (b)) and the interpolated results at select values of of in Fig. c... The cosine similarity curve indicates that network interpolation (green) does not decrease monotonically as > but forms a v-shaped curve, implying that this method does not offer continuous strengthening effect in range R[0, 1]. Again, the proposed method (blue) shows the largest variability among other controllable methods, as well as visually pleasing output.
CONCLUSION
In this paper, propose controllable neural network that enhances the depth perception of the given image by modulating the depth representative feature obtained from the pretrained depth repre- sentation encoder. To train the content-independent depth representative feature space, we introduce a hard negative sample that shares the same content as the query and the positive sample but differs in depth perception. The representation encoder is regularized with a triplet margin loss, which plays crucial role in disentangling style (i.e., depth perception) from content. The sinusoidal operations the parametric feature rotation block (PFRB) is key to generating controllable depth enhancement effect as they permit smooth variation of representative features residing along the unit hypersphere.
We want to emphasize that our strategy is not limited to controllable depth enhancement but can be extended to other image-to-image translation scenarios; for example, a controllable image restora- tion model can be formulated by training the representation encoder to learn a content-independenn representation of degradation quality and the enhancer to restore the low quality input image. In addition, our unsupervised visual representation learning strategy is effective even when the style is difficult to be measured quantified as focus on learning the style space manifold. Thus, our DEEAPR framework is applicable as a general purpose mmage-to-image translation method with single-parameter controllability.
8

---

Under review as a conference paper at ICLR 2023
study
Comparison other controllable
1.000
0.000
0.500
PP12
0.999
OETARR (oum))
AND pP))
1778 B11er
Lig
0,599
n4t inda
loyers
Rest allingg
ChesMD
Quelitative
CRSNA
CRIO0O
Figure 7: Cosine similarities of encoded features of output at various levels of a with respect to the encoded feature of output at O, comparing (a) different PFRB designs and (b) other controllable methods. (c) Qualitative results of DEEAPR and other methods. All networks (except for image interpolation and network interpolation) are trained with {0, 1) then evaluated in range a E R[0, 1]. Best viewed electronically with zoom. See supplemental for additional examples.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Eirikur Agustsson and Radu Timofte. NTIRE 2017 Challenge on Single Image Super-Resolutiin Dataset and Study. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Work- shops (CVPRW), pp. 126-135, 2017.
Vassileios Balntas, Edgar Riba, Daniel Ponsa, and Krystian Mikolajezyk. Learning Local Feature Descriptors with Triplets and Shallow Convolutional Neural Networks, In British Machine Vision Conference (BMVC), volume 1, pp. 3, 2016.
Jonathan T Barron and Ben Poole. The Fast Bilateral Solver. In European Conference Computer Vision (ECCV), pp. 617-632. Springer, 2016.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. In International Conference on Machine Learning (ICML), pp. 1597-1607. PMLR, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved Baselines with Momentum Contrastive Learning. arXiv preprint arXiv:2003.0420 042977 2020b.
Zeynep Cipiloglu, Abdullah Bulbul, and Tolga Capin. A Framework for Enhancing Depth per- ception Computer Graphics. In Proceedings of the 7th Symposium on Applied Perception in Graphics and Visualization, pp. 141-148, 2010.
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. Learned Representation for Artistic Style. In International Conference on Learning Representations (ICLR), 2016.
Saikat Dutta, Sourya Dipta Das, Nisarg Shah, and Anil Kumar Tiwari. Stacked Deep Multi- Scale Hierarchical Network for Fast Bokeh Effect Rendering from Single Image. In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 2398-2407, 2021.
Qingnan Fan, Dongdong Chen, Lu Yuan, Gang Hua, Nenghai Yu, and Baoquan Chen. Decou- ple Learning for Parameterized Image Operators. In European Conference on Computer Vision (ECCV), 2019.
Golnaz Ghiasi, Honglak Lee, Manjunath Kudlur, Vincent Dumoulin, and Jonathon Shlens. Ex- ploring the Structure of Real-Time, Arbitrary Neural Artistic Stylization Network. In British Machine Vision Conference (BMVC), 2017.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality Reduction by Learning Invari- ant Mapping. In /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), volume 2. pp. 1735-1742, 2006.
Jingwen He, Chao Dong, and Yu Qiao. Modulating Image Restoration with Continual Levels via Adaptive Feature Modification Layers. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Jingwen He, Chao Dong, and Yu Qiao. Interactive Multi-Dimension Modulation with Dynamic Controllable Residual Learning for Image Restoration. In European Conference on Computer Vision (ECCV), 2020a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 770- 778, 2016
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. In IEEE/CVF Conference Computer Viston and Pattern Recognition (CVPR), pp.
2020b.
Hagit Hel-Or, Yacov Hel-Or, and Renato Keshet. Depth-Streech: Enhancing Depth Perception with- out Depth. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2017.
10

---

Under review as a conference paper at ICLR 2023
Xun Huang and Serge Belongie. Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization. In IEEE International Conference on Computer Vision (ICCV), 2017.
Andrey Ignatov, Jagruti Patel, Radu Timofte, Bolun Zheng, Xin Ye, Li Huang, Xiang Tian, Saikat Dutta, Kuldeep Purohit, Praveen Kandula, al. AIM 2019 Challenge on Bokeh Effect Synthesis: Methods and Results. In 2019 IEEE/CVF International Conference Computer Vision Workshop (ICCVW). pp- 3591-3598. IEEE, 2019.
Andrey Ignatov, Jagruti Patel, and Radu Timofte. Rendering Natural Camera Bokeh Effect with Deep Learning. In 2020 IEEE Conference on Computer Vision and Patlern Recognition Work- shops (CVPRW), June 2020a.
Andrey Ignatov, Radu Timofte, Ming Qian, Congyu Qiao, Jiamin Lin, Zhenyu Guo, Chenghua Li, Cong Leng, Jian Cheng, Juewen Peng. et al. AIM 2020 Challenge on Rendering Realistic Bokeh. In European Conference Computer Vision (ECCV), pp. 213-228 Springer, 2020b.
Heewon Kim, Sungyong Baik, Myungsub Choi, Janghoon Choi, and Kyoung Mu Lee. Searching for Controllable Image Restoration Networks. In IEEE International Conference on Computer Vision (ICCV), 2021.
HyunGoo R Kim, Dora E Angelaki, and Gregory C DeAngelis. The Neural Basis of Depth Per- ception from Motion Parallax. Philosophical Transactions of the Royal Society B: Biological Sciences, (7167))):001500555 2016.
Jun-Tae Lee, Chulwoo Lee, Young Sim, and Chang-Su Kim. Depth-Guided Adaptive Contrast Enhancement Using 2D Histograms. In IEEE International Conference on Image Processing (ICIP), 2014.
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep resid- ual networks for single image super-resolution. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW, July 2017.
Jorge Lopez-Moreno, Jorge Jimenez, Sunil Hadap, Ken Anjyo, Erik Reinhard, and Diego Gutierrez. Non-Photorrellittic, Depth-based Image Editing. Computers & Graphics, 35(1):99-111 2011.
Thomas Luft, Carsten Colditz, and Oliver Deussen. Image Enhancement by Unsharp Masking the Depth Buffer. ACM Transactions on Graphics (ToG), 2004.
Kede Ma, Zhengfang Duanmu, Qingbo Wu, Zhou Wang, Hongwei Yong, Hongliang Li, and Lei Zhang. Waterloo Exploration Database: New Challenges for Image Quality Assessment Models. IEEE Transactions on Image Processing (TIP), 26(2):1004-1016 Feb. 2017.
Robert P O'Sha, Donovan G Govan, and Robert Sekuler. Blur and Contrast as Pictorial Depth Cues. Perception, (6(5):599-612) 1997.
Juewen Peng, Zhiguo Cao, Xianrui Luo, Hao Lu, Ke Xian, and Jianming Zhang. BokchMe: When Neural Rendering Meets Classical Rendering. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16283-16292 2022.
Ming Qian, Congyu Qiao, Jiamin Lin, Zhenyu Guo, Chenghua Li, Cong Leng, and Jian Cheng. Bg- gan: Bokeh-Glass Generative Adversarial Network for Rendering Realistic Bokeh. In European Conference on Computer Vision (ECCV), pp. 229-244. Springer, 2020.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision Transformers for Dense Prediction. In IEEE International Conference on Computer Vision (ICCV), pp. 12179-12188, 2021.
Tobias Ritschel, Kaleigh Smith, Matthias Ihrke, Thorsten Grosch, Karol Myszkowski, and Hans- Peter Seidel. 3D Unsharp Masking for Scene Coherent Enhancement. ACM Transactions on Graphics (ToG). 2008.
Romain Vergne, Romain Pacanowski, Pascal Barla, Xavier Granier, and Christophe Shlick. Im- proving Shape Depiction under Arbitrary Rendering. IEEE Transactions on Visualization and Computer Graphics, 2011.
11

---

Under review as a conference paper at ICLR 2023
Neal Wadhwa, Rahul Garg, David E Jacobs, Bryan E Feldman, Nori Kanazawa, Robert Carroll, Yair Movshovitz-Attias Jonathan T Barron, Yael Pritch, and Mare Levoy. Synthetic Depth-of-Field of-lldd with a Single-Cament Mobile Phone. ACM Transactions Graphics (ToG), 37(4):1-13, 2018.
Longguang Wang, Yingqian Wang, Xiaoyu Dong, Qingyu Xu, Jungang Yang, Wei An, and Yu- lan Guo. Unsupervised Degradation Representation Learning for Blind Super-Resolution. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10581-10590, 2021.
Wei Wang, Ruiming Guo, Yapeng Tian, and Wenming Yang. CFSNet: Toward Controllable Feature Space for Image Restoration. In IEEE International Conference on Computer Vision (ICCV), 2019a.
Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. Recovering Realistic Texture in Image Super-Resolution by Deep Spatial Feature Transform. In IEEE/CVF Conference on Computer Vision and Pultern Recognition (CVPR), pp. 606-615, 2018.
Xintao Wang, Ke Yu, Chao Dong, Xiaoou Tang, and Chen Change Loy. Deep Network Interpolation for Continuous Imagery Effect Transition. In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2019b.
Dong Yang and Jian Sun. Proximal Dehaze-Net: A Prior Learning-baset Deep Network for Single Image Dehazing. In European Conference Computer Vision (ECCV), pp- 702-717, 2018.
12


Reviews:
[{"review_id": "yCrWkP5o2_", "reviewer": "Reviewer_MFV5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for enhancing the depth perception of an image. The proposed method first embeds content-independent depth perception of a scene using the visual representation learning technique, and then trains a controllable depth enhancer network based on a parametric feature rotation block (PFRB). Some visual results are shown to verify the proposed method. \n", "summary_of_the_review": "Given that the idea is not very new, the reported results are not visually pleasing, and the evaluation is not sufficient, I would like to give a reject rating at the current stage.", "strength_and_weaknesses": "Strength\n- This paper proposes a strategy to learn the visual representation space of style disentangled from image contents.\n- A controllable neural network with a single control parameter is proposed to enhance the depth perception of the input image.\n- A parametric feature rotation block (PFRB) is introduced to enable continuous modulation of feature representation.\n\nWeaknesses\n- It is unclear to me the advantage of this method over existing methods utilizing explicit depth information. In fact, the ground-truth depth-enhanced images are just generated using the estimated depth maps of the input image. Also, the controllable factor (i.e., sigma) can be utilized in methods using explicit depth as well. \n- Lack of comparison with other depth enhancement methods.\n- There is no numerical result. I understand that quantitative evaluation is difficult, but a user study might be the least that can be provided. \n- The qualitative results are not visually pleasing and unrealistic. There are many obvious errors in the generated images, e.g., in Fig.7(c) regions with different depths have the same blur effects, and regions with similar depths have different blur effects.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "pJaHZkT-M3u", "reviewer": "Reviewer_mEsF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the DEpth Enhancement via Adaptive Parametric feature Rotation (DEEAPR) method to modulate depth information with a single control parameter. They first use visual representation learning to embed content-independent depth perception of a scene, then train a controllable depth enhancement network with a novel modulator (parametric feature rotation block, PFRB), which is for continuous modulation of a representative feature. They verify the effectiveness of proposed components and the whole method via ablated studies and comparisons.", "summary_of_the_review": "The paper lists three contributions but I find them not convincing. This reflects in how they explain the motivations and discuss their differences with existing methods. Hence, at this point, I am on the negative side.", "strength_and_weaknesses": "Comments.\n\nThe paper lists three contributions. The first one is a new method to learn style disentangled visual representation space of image contents. This mostly relates to visual representation learning, but the difference between the proposed method and existing methods is not explained. Sec. 2.1 mentions that it is similar to (Chen et al., 2020b) and is motivated by DASR (Wang et al., 2021), but it is not clear what the exact similarities and differences are. Hence, it is very hard to evaluate this contribution. \n\nBesides, it is not clear how to define the \"content-independent depth perception of a scene\" and why \"content-independent\" relates to this task. How can the combination of (Hadsell et al., 2006) and (Balntas et al., 2016) embed the content-independent depth perception of a scene onto a representation space, and why this bridges the image space and the control parameter axis? Why choose these two methods? How to define the control parameter axis? What is the \"direction\" of depth enhancement?\n\nThe second contribution is a controllable neural network that enhances the depth perception of an image with a single control parameter. Sec. 2.2 says the main differences to existing methods are that DEEAPR uses a single parameter and does not rely on explicit depth information. However, it seems that (Dutta et al., 20) also do not rely on explicit depth information. So the key difference is about using a single parameter. It is not explained why the authors propose to use a single control parameter. Is a single control parameter enough and why?\n\n\nThe third contribution is the parametric feature rotation block (PFRB) for continuous modulation of feature representation. Its motivation and novelty are not explained clearly. For example, (Wang et al., 2019a) have a tuning network to modulate the feature of the main branch. How does the proposed DEEAPR differ from this?\n\nAlmost all methods included for discussion and comparisons are published before/in 2021. More recent works should be included.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ch34VKWE40", "reviewer": "Reviewer_3SWe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a network-based approach to control the depth-of-field (DoF) effect of photographs post-capture. While there are many previous works in artificial Bokeh generation, the authors claimed their novelties lie in 1) learned representation space for the DoF effect, 2) controllable network for the strength of the DoF effect, and 3) a new network block called PFRB that can preserve the norm when modulating the features. ", "summary_of_the_review": "While I appreciate the effort the authors carry out in designing the network and module for the specific problem of post-capture DoF effect rendering which is very useful in computational photography and has a wide-range impact in real life. \n\nMy biggest concern is whether the authors' fundamental motivation, that we need a learnable latent representation space for the DoF effect, is valid. \n\nUsually learning an implicit latent space for image manipulation or enhancement is crucial when the problem can not be easily and explicitly defined by physics-based models or the underlying physical model is too complex or ill-posed  (i.e. manipulating the age of the person in a given image, image style transfer, etc.). However, in the case of DoF or Bokeh effect rendering, the problem is very well and explicitly defined. The DoF is clearly defined as a function of the depth ($u$), camera focal length ($f$), aperture size or F-number ($N$) and circle-of-confusion ($\\mu$), $DoF = \\frac{2 \\mu^2 N C}{f^2}$. Shall or deep DoF (or \"depth perception\" per this paper) can be nicely controlled and rendered with the camera parameter factored into $O(x)$, defined in equation 4.", "strength_and_weaknesses": "### Strengths\n1. Controllable network for different DoF effect levels (though seen in previous work).\n2. Network architecture and PFRB module seem novel.\n### Weakness\n1. Given Bokeh's image formation model and the DoF effect's strength are clearly defined, I have doubt whether learning a latent space for rendering is necessary. \n2. The term \"depth perception\" is ambiguous, but rather DoF is the good latent space that models the background/foreground blur/sharpness contrast. \n4. Lack of results to show the robustness of the proposed network. Only very few images (I counted 4 throughout the paper and supplementary material) are used for visual evaluation. \n5. The shown results image has unnatural artifacts. For example, the bridge image, different portions of the bridge at the same depth are exhibiting different levels of blur.\n5. The depth-based contrast model $O_{contrast}$ in section 4.1, though already redefined from the haze/fog model from prior, is still heavily haze-oriented. For general photos taken under good weather conditions, the proposed method may introduce an unwanted or exaggerated hazy effect.  The plant image in the supplementary material proves this point: as $\\sigma$ increases, the further away background (maybe ~50$cm$) is becoming hazy, whereas the depth-dependant haze effect should hardly be seen in the real world for an object about this distance. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MapIlFKGs7", "reviewer": "Reviewer_4wXg", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a novel method for depth enhancement for single image. They used a single hyperparameters to control the synthetic defocus blurriness. ", "summary_of_the_review": "In summary, even though the paper shows that they can achieve depth enhancement effect in a single image by varying value of a hyperparameter, there's still a very large space for this paper to improve w.r.t. writing and performance. ", "strength_and_weaknesses": "Strength:\n+ The output image can be controlled by adjusting the input hyperparameters\n\nWeakness:\n- The paper is poorly written and extremely hard to follow. The countless grammatical errors, notions used without any references,  and bad paper organization make it almost impossible for readers to understand the paper content. The following listed ones are just a few: What is query, positive sample and hard negative sample mean (Sec. 3.1 and Fig.3)? What is depth representation means? What's the difference between global/local depth representative feature (Sec. 3.2)? How you can rotate the parametric features in PFRB?\n- The technical contribution is marginal. I didn't find significant difference between images of different \\sigma.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject