Bridging Implicit and Explicit Geometric Transformations for Single-Image View Synthesis
Authors: Byeongjun Park
Hyojun Go
Changick Kim
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Single-Image View Synthesis
Transformer

Abstract: Creating novel views from a single image has achieved tremendous strides with advanced autoregressive models. Although recent methods generate high-quality novel views, synthesizing with only one explicit or implicit 3D geometry has a trade-off between two objectives that we call the ``seesaw'' problem: 1) preserving reprojected contents and 2) completing realistic out-of-view regions. Also, autoregressive models require a considerable computational cost. In this paper, we propose a single-image view synthesis framework for mitigating the seesaw problem. The proposed model is an efficient non-autoregressive model with implicit and explicit renderers. Motivated by characteristics that explicit methods well preserve reprojected pixels and implicit methods complete realistic out-of-view region, we introduce a loss function to complement two renderers. Our loss function promotes that explicit features improve the reprojected area of implicit features and implicit features improve the out-of-view area of explicit features. With the proposed architecture and loss function, we can alleviate the seesaw problem, outperforming autoregressive-based state-of-the-art methods and generating an image $\approx$100 times faster. We validate the efficiency and effectiveness of our method with experiments on RealEstate10k and ACID datasets.

Bridging Implicit and Explicit Geometric Transformations for Single-Image View Synthesis
Anonymous Author(s)
Affiliation
Address
email
Abstract
Creating novel views from a single image has achieved tremendous strides with advanced autoregressive models. Although recent methods generate high-quality novel views, synthesizing with only one explicit or implicit 3D geometry has a trade-off between two objectives that we call the "seesaw" problem: preserv- ing reprojected contents and 2) completing realistic out-of-view regions. Also, autoregressive models require considerable computational cost. In this paper, we propose a single-image view synthesis framework for mitigating the seesaw prob- lem. The proposed model is an efficient non- -on-autorreressive model with implicit and explicit renderers. Motivated by characteristies that explicit methods well preserve reprojected pixels and implicit methods complete realistic out-of-view region, we introduce loss function to complement two renderers. Our loss function promotes that explicit features improve the reprojected area of implicit features and implicit features improve the out-of-view area of explicit features. With the proposed architecture and loss function, we can alleviate the seesaw problem, outperforming autoregressive-basee state-of-the-arr methods and generating an image ss 1000 times faster. We validate the efficiency and effectiveness of our method with experiments on RealEstatel00 10K and ACID datasets.
10
11
12
14
16
Introduction
18
Single-image view synthesis is the task of generating novel view images from given single image [5] 47 50 [4]. It can enable the movement of the camera from a photograph and bring an image to 3D, which are significant for various computer vision applications such image editing and animating. To perform the realistic single-image view synthesis in these applications, we can expect that the novel view image has consist of existing objects and unseen new objects from the reference viewpoint. Therefore, for high-quality novel views, the following two goals should be considered: 1) contents of single reference image and 2) generating semantically
20
18 23
38/40
22
24
25
preserving 3D transformed compatible pixels for filling the unseen region. To achieve two goals, explicit and implicit methods have been proposed.
26
27
28 With the recent success differentiable geometric transformation methods 2 31). explicit meth- ods [5] [1 23 55 [57] leverage such 3D inductive biases to guide the view synthesis network to 30 preserve 3D transformed contents, and various generative models are applied to complete the unseen regions. Explicit methods can produce high-quality novel view images in small view changes, where
32 the content the reference viewpoint still occupies a large portion. However, for large view changes,
33 the image quality degraded due a lack of ability generate pixels of the unseen region. To deal
34 with this problem, outpainting with the autoregressive model is exploited to fill unseen regions (3
35 but generating photo-realistic images remains challenge for explicit methods.
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

67
51 In this paper, we present a von-autoregressivv framework for alleviating the seesaw problem. Our FDD on large viewpoint changes
62
SynSin
Better generation
57
0.063 sec/img
0,063 seciing SynSan-
Trade-ufl
47
PixclSynth
Input image
SynSin 500
PixelSynth 39
6.22 sec/img
42
sec/ing
0.056 secting
Gealrree
Better preserve
32
15
PSNR small viewpoint changes
Explicit method
! Implicit method
Warped image
GeoFree 40
Ours
Figure 1: Seesaw problem of explicit and implicit methods. Explicit methods well preserve warped contents but sacrifice to fill unseen pixels (f PSNR on small view change, f FID on large view change). Implicit methods amply fill unseen pixels but fall short of preserving seen contents (4 PSNR on small view change, FID on large view change). Our proposed framework alleviates this seesaw problem and generates an image faster than the state-of-the-art methods.
36 On the other side, implicit methods 38 40 46
less enforce 3D inductive biases and let the model learn the required 3D geometry for view synthesis. Based on the powerful autoregressive trans- former [TO], recent implicit methods learn the 3D geometry from reference image and camera parameters. Implicitly learned 3D geometry allows the model to synthesize diverse and realistic novel view images but fails preserve the contents of the reference image since they reduce 3D inductive biases.
37
38
39
40
41
To up, previous single-image view synthesis methods suffer from trade-off between two objectives: 1) preserve seen contents and 2) generate semantically compatible unseen regions. Figure shows an apparent trade-off that explicit methods well preserve generation of unseen regions and vice versa for implicit methods. Here, we call this trade-off the seesaw problem and emphasize the need for combining solid points of explicit and implicit methods.
43
contents with sacrificing the
45
46
47 Moreover, recent methods often depend on autoregressive models, which generate individual pixels sequentially. Sequential generation causes too slower view synthesis than aon-autorerressive methods, limiting their application areas, such as image animating in real-time. Therefore, we refocus on a fast and efficient non-autoregressivv model for single view synthesis.
49
52 approach aims to design the architecture and loss functions. We design two parallel render blocks
53 which explicitly or implicitly learn geometric transformations from point cloud representations. To bridge explicit and implicit transformations, we propose novel loss function that motivates 55 explicit features improve seen pixels implicit features and implicit features improve unseen pixels of explicit features. Interestingly, observe that proposed loss makes two renderers embed balanced way to create novel 57 discriminative features and allow the model to use both renderers merge the pros of both explicit views. With the proposed architecture and the loss function, and implicit methods, alleviating the seesaw problem. As a result, our non-autoregressive framework better preserve contents, better complete unseen pixels, and generate images 1100 times 61 faster than autoregressive methods. We validate the efficiency and effectiveness of our framework with experiments on the indoor dataset RealEstate 10K [58] and the outdoor dataset ACID 23
59
Related Works
63
64 Novel view synthesis Given multiple images from different viewpoints of scene, novel view synthesis aims to generate novel view images. Traditionally, multi-view geometry utilized for synthesizing novel viewpoints 4
85
13 21
42 59
Recently, deep neural networks have been
67 used rendering [5
29 and several representation for view synthesis such as multi-plane
image ||| 45 58
point cloud depth [4], radiance field 30 49 55 and voxel 25 33 43

---

Implicit Renderer
Embeddings
Global Set Attention
Dverlap Ratch Umbbeddroo
Block
Trandformr
Upsample
ISAB
etobal
M
Input Image Test
MicFFN
flooe
Back progoeo
LSA Layer
Camera Pose
X_X
output Image byy
Cnce!
Overtap Patch
Transformes
Local Set Attention
Block
Upsample
Decoder
concatenation
DepthNet
Encoder
M
ViewNet
Explicit Renderer
Figure 2: An overview of network architecture. Our network takes reference image ref and a relative camera pose T as inputs. The depth estimation network (DepthNet) first predicts a depth map D, and the view synthesis network (ViewNet) generates target image ltgt from Inef. D and T. Specifically,
used for calculating the 3D) world coordinate X and the normalized image coordinate Xigg at the reference viewpoint, which are passed through various positional encoding layers in the encoder (e.g., ógtobal, 8ube and (oeet) to provide the scene structure representations. Encoded features JN are then transformed by both Implicit Renderer and Explicit Renderer with T, Finally, transformed feature map, h; and he. are concatenated generate Itgt by the decoder.
69 Single-image view synthesis is more challenging than general novel view synthesis since a single input image is only available [5] [8| 23
38 40
50 [4]. Explicit methods directly inject 3D 47
71 inductive biases into models. For example, SynSin (50] uses 3D point cloud features with estimated
72 depth from the model, projects to novel viewpoints, and refines unseen pixels with recent generative
73 models [ SynSin works well in small viewpoint changes but degrades in large viewpoint changes due to the lack of generating unseen pixels. To deal with this problem, PixelSynth (9) exploits the autoregressive outpainting model (37] with 3D point cloud representation. Despite using the slow autoregressive model, it cannot generate unseen pixels well. For implicit method, Rombach er al. [40] propose powerful autoregressive transformer. By less enforcing 3D inductive biases, this approach can generate realistic view synthesis and complete the unseen region without explicit 3D geometry. However, its inference time is long due to the autoregressive model, and it fails to preserve seen contents of reference image. We bridge these implicit and explicit methods non-autoregressive architecture, which can outperform autoregressive approaches with fast inference.
75
78
77
78
79
80
81
82
Transformer for point cloud The transformer and self-attention have brought a breakthrough in natural language processing 4848 and computer vision [9]. Inspired by this success, transformer and self-attention networks have been widely applied for point cloud recognition tasks and achieved remarkable performance gain. Early methods utilize global attention for all of the point clouds, resulting in a large amount of computation and inapplicable for large-scale 3D point cloud [24 [2 (33
84
85
86
87
Lee al. (20) propose the SerTransformer module suitable for point cloud due permutation- invariant, which uses inducing point methods and reduces computational complexity from quadratic
88
89
linear in the number of elements. Also, local attention methods utilized to enable scalability [14] [6]. Notably, among local attention methods, Fast Point Transformer [34] which uses voxel hashing-basse based architecture, achieves both remarkable performance and computational efficiency. Global attention may dilute important content by excessive noises most neighbors are less relevant, and local attention may not have sufficient context due their scope. Therefore, Our approaches use both global and local attention to deal with 3D point cloud representation.
90
34
91
93
3 Methodology
95
Given reference image Fref and a relative camera pose T, the goal of single-image view synthesis is 97 to create target image legt with keeping visible contents of Tref and completing realistic out-of-view pixels. To achieve this, we focus mitigating the seesaw problem between explicit and implicit
3

---

methods in terms of the network architecture and the loss function. Figure 2 describes an overview of 99
100
our network architecture. The network consists of two sub-networks, the depth estimation network (DepthNet) and the view synthesis network (ViewNet). Note that the pre-trained DepthNet generates depth map D, which is used for ViewNet to synthesize the photo-realistic ligt-
102
103
3.1
Depth Estimation Network (DepthNet)
104
We train the depth estimation network for explicit 3D geometry since ground-truth depths are not available. Following Monodepth2 [2], our DepthNet is trained in a self-supervised manner from
105
106
relative pose between images is available, we substitute the pose estimation network with the ground-truth relative pose, Then, we train the network on reprojection losses and smoothness losses with auto-masking in their work. After training DepthNet, we fix it during training ViewNet.
monocular video sequences. Because a ground-trun
107
108
109
110
3.2
View Synthesis Network (ViewNet)
111
We design a simple view synthesis network built on architectural innovations of recent transformer models. Specifically, we exploit 3D point cloud representation to consider the relationship between the geometry-aware camera pose information and the input image.
112
113
The encoder aims extract scene representations from feature point cloud of reference image. To deal with point clouds, we design a Global and Local Set Attention (GLSA) block which simultaneously extracts overall contexts and detailed semantics. For efficient input size of transform- ers, Lref E xx33 is encoded into fo IR # W. xC by an overlapping patch embedding (5]]. where C denotes the channel dimension. Then, the homogeneous coordinates p of pixel in fo are mapped into normalized image coordinates Ximg as Ximg(p) (p) = KJ'P. p. where K₁ denotes the camera intrinsic matrix of Jo- Finally, 3D world coordinates of are calculated with depth map D as
Encoder
115
116
117
118
119
120
X (p) D(p) X img (p). Our encoder architecture N stacked GLSA block, and i-th GLSA block
122
receives 11-1,Xim and X and outputs f: with Mix-FFN (51).
Global Set Attention. We utilize Induced Set Attention Block (ISAB) 120 to extract global set attention between the feature point clouds. With positional encoder dgtobal and vector concatenation operator C the global attention of i-th GLSA bock is represented
124
Pgtobart (p) = ISA + 6gtobal (p)).
(1)
126
Local Set Attention. We use a modified Lightweight Self-Attention (LSA) layer 34 for the set attention in r x r local window of each pixel point. Unlike the decomposing relative position of voxels in [34, we decompose the relative position of 3D world coordinates between neighbor pixels using normalized image coordinates as:
127
129
(p) (g) (p) Xxmg (p)) (X (g) Xmmg())) + (Ximg Xg))(g)), (2)
130
where q E (p) is a neighbor set of homogeneous coordinates in a T X r window of p. With decomposition in Eq. 2 we can divide the relative positional encoding into continuous positional encoding spleal and a discretized positional encoding 8yala Then, the computation procedures for local set attention gioca of i-th GLSA block is similar to LSA layer as:
132
135
Ho (p) = fe(p) ① Spaen (X., - Nimg(p)), Sc (p) Sc (p)) Stel (X (Ximg()) - (img(g)))o((t))
where and are MLP-layers, and Sa(a,b) 135 a and b. As pixel coordinates of p and g are all integer, the encoding of Xsmo (p) - Xmmg (q) is hashed over 22 values, resulting in a space complexity reduction from O(HW r2 C) to O(HW C) + O()2.())
a-b ||л|||Б!| computes the cosine similarity between
196
138
Given the scene representations of the reference image, the rendering module learns 3D transformation from the reference viewpoint to the target viewpoint. Motivated by our observations of implicit and explicit methods, we design an Explicit Renderer(FR) and an Implicit Renderer(IR) connected in parallel to bypass the seesaw problem. The structure of the two renderers
Rendering Module
140

---

Implicit Renderer
Out Out-of-view pixels
grad
K, D.
grad
Explicit Renderer
Re Re-projected pivels
Figure 3: An overview of our transformation similarity loss. Two transformed features, h; and he are complemented each other by the transformation similarity loss. Specifically, we first derive out-of-view mask from K, D and T. By using O, two transformation similarity loss, i.c. Lts,in and Lts out, are applied encourage the discriminability of h; and he respectively. To guide the another renderer as intended, we allow the back- proppappeeetted gradients Lis,in only to the reprojected regions of h2, and those of Lis,out only to the out-of-view regions of he-
142
similar; they consist of overlapping patch embedding, GPT architecture [36 and ResNet blocks with upsampling layers. Note that the overlapping patch embedding and upsampling layers are designed for downsampling and upsampling the input feature with the factor of 4, respectively. The major difference between the two renderers is how the relative camera pose T is used for the geometric transformation.
143
144
145
146
147
Explicit Renderer (ER). Given the rotation matrix and translation vector of relative camera pose T, p can be reprojected to the homogeneous coordinates of target viewpoint p' p' K R. (p) + t. The output of encoder fN is warped by splatting operation (31 with optical flow from to p/. Then, warped JN goes through the explicit renderer to produce explicit feature map he.
149
150
151
Implicit Renderer (IR). Unlike the explicit renderer, the implicit renderer uses the camera parameter itself. Instead of embedding 3x4 camera extrinsic matrix, pose information; Translation vector and axis-angle notation [|]l 0) to parameterize rotation matrix
independent 7 parameters to embed
153
154
R. We use positional encoding layer 6pos to embed these parameters and add them to the input of the transformer block. fNv passes through the implicit renderer and outputs implicit feature map h;- Please refer to the supplementary materials for details to compute the axis- axis-angle notation.
155
156
157
Two feature maps from ER and IR, which are denoted as he and h4, are then concatenated before the decoder. We use simple CNN-based decoder by gradually upsampling the concatenated feature map with four ResNet blocks. Instead of generating pixels in an auto-regressive manner, we directly predict all pixels in the one-path, resulting in more than 110 times faster than the -tate-of-the-are autoregressive methods 48 generating images.
Decoder
158
159
160
161
162
3.3
Loss Design for ViewNet
163
Following the previous single-image view synthesis methods 33
also the flsss,, perceptual loss (55 and adversarial loss to learn the network. Specifically, we compute é-lloss and perceptual loss between Ligt and the ground-truth image Igz the target viewpoint. Also, we use the
50
164
185
186
global and local discriminators [19] with Projected GAN [41] structure and hinge loss [22]. We observe that our methods improve the generation performance even through these simple network structural innovations. Furthermore, we introduce a transformation similarity loss Lt, to complement
187
188
189
two output feature maps he and h;-
170
Transformation Similarity Loss As an extension of the existing seesaw problem, he may have better discriminability than h; in reprojected regions, conversely, h; has better delineation of out-of- view regions than he. Therefore, as shown in Fig. 3 we design the transformation similarity loss between he and h;s expecting that by learns to keep reprojected image contests, and h1 also learn to generate realistic out-of-view pixels. Specifically, we use negative cosine similarity function
171
172
175
175
S. for calculating the similarity between two feature maps, and the transformation similarity loss
Lte Ats Aout ous out is formulated as:
5

---

1
Σ (1 0(p)) dettc(p)
Lie,in =
Ep(1-
(4)
Σ O(p) (((detach(t.())).hhbb)))) E, Op)
1
Lis
where (pp) E x denotes an out-of- view mask which is derived from the depth map D and the relative camera pose T. Note that, without detach operations, our transformation similarity loss performs the same as simple negative cosine similarity loss between two feature maps. Thus, we detach gradients back- pack-propagated from Its. to he and gradients from Tus, to h4, because the detach operation allows the components of Tia to be applied to the intended area.
177
178
180
Final Learning Objective Taken together, our ViewNet is trained on the weighted sum of a & Te, perceptual loss Tees an adversarial loss Ladv and a transformation similarity loss Lts- The total loss is then I. = Te, + AcT.e + Iiduo + Lts- We fix d and Xade = 0.1 for all experiments.
183
184
Table 1: Types of baselines and our method. Note that InfNat 23 varies according to the number of steps, so we mark A.
Types
Tatacchnkk
Viewsppanacee
SynSin
PixelSynth
GooiFrre
LockOiside
Ous
Explicn
Implicit Aitonogroxixe
Experimental Results
185
186
4.1
Experimental Settings
187
We now describe experimental settings, and please refer to the supplementary materials for further details about datasets, baselines, and our network architecture.
188
189
Dataset
We used two standard datasets, RealEstate 58 and ACID
23 which are collection
190
of videas mostly captured in indoor and outdoor scenes, respectively. We divided train and test sequences in 40
191
192
To validate the effecttiveness of our framework, we compared our method to previous single-image view synthesis methods: Tatarchenko et al. [46]. Viewappearance [57]. Synsin (50). InfNat [23], PixelSynth (39], GeoFree 40) and LookOutside [38. Table Table[Tbiifii briefly shows whether each method is an explicit, implicit, and autoregressive model. Compared to previous methods, we
Baselines
193
194
195
196
use both explicit and implicit geometric transformations without an autoregressive model.
197
Evaluation Details Because explicit and implicit methods are respectively advantageous in small
198
view change and large view change, methods should be evaluated on several sizes of viewpoint changes for fair comparison. Therefore, we used a ratio of out-of-view pixels over all pixels to quantify view changes, resulting in three splits are categorized into small (20-40%) medium (40-60%) and large (60-80%). Since evaluation datasets do not have ground-truth depth maps, we used depth maps from our pre-trained DepthNet to derive the ratio of out-of-view mask pixels. Finally, we used randomly selected 1,000 image pairs for each test split.
199
201
203
204
We use PSNR on the small split and FID [16] on the medium and large split as evaluation metrics. PSNR traditional metric for comparing images, which is widely used to evaluate consistency,
205
206
Nevertheless, PSNR is poor metric to verify the image quality on large viewpoint changes [39 40
207
Still, it can be a good metric for evaluating the preservation of reprojected pixels on small view changes. Therefore, we use PSNR on the small split to evaluate the ability to preserve seen contents.
208
209
For evaluating images quality of view synthesis, FID is widely used [3 40 50). Especially in
the medium and large split with many out-of-view pixels, FID indicates how well the model fills
211
out-of-view pixels and generates realistic images. We use the PSNR and FID of specific splits as
evaluation metries, but report the PSNR and FID of all splits to show the overall trend.

---

Table Quantitative results on RealEstatel0K and ACID, Image quality is measured by PSNR
and FID for three types of view changes, i.e., Small, Medium and Large. Furthermore, we show the average performance over all view changes at the end. For both datasets, best results each metric
are in bold, and second best are underlined.
Datuset
Methods
PSNK
FIDI
PSNKT
Medium
FID,
PSNKT
Large
FIDI
PSNR
Average
FID.
Tararchenko ef al 40
11.12
258.75
10.90
248.35
10.80
249.24
10.94
232.18
Viewappearance
12.51
142.93
110.84
147.27
12.58
133.68
SynSin [50
15.38
41.75
14.88
43.06
13.96
61.67
14.74
48.83
RealEstaae
58
SynSin-6x
15.17
33.72
14,99
37.28
14.26
48. 29
14.81
39.76
PixelSynth
14.46
37.23
38.39
45.44
13.40
40.35
GeoFree 40
14.16
33.48
13.15
34.21
12.57
35.28
13.29
34.32
LookOutside
12.58
44.87
12.72
43.17
12.11
45.22
12.47
43.75
15.87
32.42
14.65
33.04
13.83
35.26
14.78
33.57
Tatarchenko of al. 46
14.43
148.19
14.20
151.24
14.34
150.47
14.32
149.97
Viewappearance
57
14.46
161.91
13.58
203.19
13.21
218.37
13.75
194,49
SyaSin 50
17.48
55.64
16.49
75.88
16.87
79.04
16.95
70.19
InfNat
(1-step)
15.94
64.32
14.40
90.80
13.65
106.28
14.66
87.13
ACID 3
InfNat
(5-tee)
15.16
64.48
14.79
71.52
65.45
14.95
67.15
PixelSynth
39
15.81
53.38
14.33
63.48
13.53
65.60
14.56
60.82
GeoFFre
53.21
14.24
54.78
14.42
55.64
ours
17.52
42.52
16.54
51,56
15.81
49.28
16.62
47.79
(a) Input Image
(b) Warped Image
(c) SynSin 50
(d) PixelSynth A9)
(e) GenFree
AM1
( Ours
Figure 4: Qualitative Results on RealEstatel0K and ACID. We compare baselines to our method.
The top two rows are from RealEstate 10K, and the bottom two rows are from ACID.
Implementation Details We first resized all images into resolution of 256 256, and normalized
214
RGB value following [39 (0). We trained DepthNet using a batch size 50 for 100k iterations and
ViewNet using batch size 32 for 150k iterations. Training takes about 3 days 4 NVIDIA Geforce
216
RTX 3090 GPUs. We used an Adam (27] optimizer (with 31 = 0.5 and _3 = 0.9) and applied
weight decay of 0.01. We first linearly increased the learning rate from 10 to 3 10 during the
218
first 1.5k steps, and then a cosine-decay learning rate schedule [26] was applied towards zero. In
ViewNet, used 8 GLSA blocks with local window size renderer for all experiments.
5 and 6 transformer blocks in each
220
4.2
Comparison to Baselines
222
now compare our method with the state-of-the-art methods on RealEstate 10K and ACID. Table shows quantitative results for both datasets. The implicit method GeoFree [40] reports lower
7

---

Table 3: Average inference time.
Table 4: Ablation study on Les-
Methods
SynSin
InfNat (5-step)
PixelSyath
Loss Type
Time (s/img)
0.063
1.14
6.22
no La
Lexan
Luxout 14.73
(no detach)
Lex
Methods
GeoFree
LookOutside
Ours
PNSRT FIDL
14.47
14.62
14.59 40.44
14.78
Time (s/img)
9.39
22.13
0.056
40.45
38.05
36,95
33,57
Table 6: Ablation Study on hyperparame-
Table 5: Ablation Study on the Set Attention.
ters of transformation similarity loss.
Set Aticution
Small
Medium
Large
Loss Weight
Small
Medhum
Large
Shooal
Mylobal
PSNRT
13.69
34.07
PSNK
14.61
34.81
PSNKT
13.75
37.63
FID
61
Am
Ava
PSNR
15.75
13.95
FTD]
PSNR
14.65
TID.
14.10
PSNRI
13.11
37
FID.
15.74
32.80
14.61
34.37
13.88
38.68
10
15.48
17.46
14.39
37.46
13.36
221
1588
32.42
14,65
33.04
13.83
35.26
0.1
15.46
34.96
14.37
37.51
13.64
39.51
10
35.03
14.54
15.87
32.42
14.65
33.04
1833
35.26
224
FID in the medium and large split than explicit methods such as SynSin [50] and PixelSynth [39], but its PSNR of the small split is lower. This shows that previous methods are suffered from the seesaw problem. However, our method consistently achieves the highest PSNR in the small split on both datasets, which means our method better preserves reprojected contents than previous methods. Moreover, our method also achieves the lowest FID in all splits on both datasets, and this demonstrates
225
228
227
228
229
that our method generates better quality images with filling compatible pixels regardless of view changes. As observed in [38 39], we note that SynSin and its variant (i.e., SynSin-6x) often produce entirely gray images, resulting they still performing competitive results PSNR of the medium and large split. Considering this, our method stably outperforms previous methods in all splits.
231
233
Also, qualitative results in Fig. Dillustrate that the warped regions are well-preserved and invisible parts are well-completed in our method, whereas explicit methods do not generate realistic images, and an implicit method loses the semantic information of visible contents. Specifically, GeoFree [40] does not preserve the table in the first sample and the ships floating on the sea in the third sample. Also, explicit methods R9 50 either make the entire out-of-view regions in one color or produce a less realistic view than our method.
234
235
236
237
238
239
We confirm that mitigating the seesaw problem by well-bridged explicit and implicit geometric transformations yields high-quality view synthesis, even acquiring a generation speed of about 110 times faster than the previous autoregressive models, as shown in Table 3 The fast generation of novel view images allows our method to be scalable to various real-time mil-time applications.
240
241
242
243
4.3
Ablation Study: Type of Set Attention
244
We design the global and local set attention block to simultancously extract overall contexts and detailed semantics. Therefore, we conducted an ablation study on RealEstate (58] to verify each attention improves the performance of generating novel views. Table B shows the quantitative result
245
246
247
for the type of set attention. Interestingly, our local set attention improves the performance relatively in large view changes, while our global set attention performs well on small view changes. From this result, we conjecture that local and global set attention are more useful for structural reasoning of out of- regions and 3D scene representation of reprojected regions, respectively. Also, significant performance improvement is achieved when both attentions are used.
248
250
251
4.4
Ablation Study: Transformation Similarity Loss
253
The transformation similarity loss Lis is weighted combination of Les,in and Les.out- To understand the effect of each component, we conducted ablation studies of transformation similarity loss on the RealEstate 10K dataset. Table reports the average PSNR and FID of our model by changing various components of L... Results show that combining with gradient stopping operation, Lts,ins and out achieves best results among the five variants. Also, either using Lta.in or Lts, improves the performance and shows that guiding one renderer from the other renderer with the proposed loss function is effective. Notably, transformation similarity loss is not practical when the detach operation is not used. From this result, it is necessary to selectively guide unseen and seen regions by detaching the gradient.
255
257
259
261
We also performed an ablation study on balancing parameter Ain and Aout- Table 6 illustrates the results varying weight of Tas- Results show that the case of Xin = 1, Aout = 1 performs best, As mentioned above, it seems essential to complement each other in balanced way.
263
8

---

Smral
Shatt Langs
Large
barg
Large
IA,603ll/ PPoll
(a) Overall
(b) without Lee
(c) with Lea,in
(d) with Lta,out
(c) with Lts
Figure 5: Histogram of [|bb6))]la/1l1.(() la the small and large split of RealEstate dataset.
285
4.5
Dependency Analysis between Implicit and Explicit Renderers
Our proposed architecture exploits the implicit and explicit renderer and mixes their outputs for decoding view synthesis results. To understand the dependency between two renderers, we analyze the norm of output feature maps. For spatial position p. the norm ratio of two spatial features [r)]l] hs) can represent how much depends on the explicit feature he(p) compared to
267
268
269
270
implicit feature h:(p), For example, if the ratio is large, the model depends the explicit renderer
271
than the implicit renderer at position p. We compare histograms of the norm ratio by changing the components of Lts and data splits as shown Fig.
272
273
Figure 5a depicts that using out and Lia.in tends to make the model more dependent explicit and implicit features, respectively, compared to our method trained without Lts- Furthermore, these tendencies are more apparent in difficult cases (i.e., large split) as shown in Fig. [d [d From our ob- servations, we conjecture that guiding only specific renderer improves the discriminability of that renderer, resulting in the model depending on the improved renderer. Surprisingly, the model trained on combining all components of Lis uses both renderers in a balanced way, and there is less bias in norm ratio even according to data splits as shown in Fig. 5e
275
277
278
279
(a) Input Image
(b) Warp Image
281
282
283
The effectiveness of our transformation similarity loss is firmed by comparing it to our method that is trained without Lts. shosows that our model trained without L,s has some outliers for large view changes despite there being less bias according to data splits. We observe these outliers are derived when the model fails to generate realistic out-of-view regions, especially
285
287
(c) Without T..
(d) With Tex.
Figure 6: Visual ablation study. With- out the transformation similarity loss, our model complete textured out-of view regions but not realistic enough than model trained with the transformation similarity loss.
289
290
challenging settings, such as the net- work having to create novel views for both indoor and outdoor scenes, as shown in Fig. We also confirm that our model
291
292
293
trained with Lis performs well even in extreme cases, inform-
294
ing that Lts improves two renderers to embed discriminative features. Collectively, Lts improves the discriminability of output features from two renderers and
295
296
makes the behavior of the model stable, resulting alleviating the seesaw problem.
5
Conclusion
298
We have introduced a single-image view synthesis framework by bridging explicit and implicit renderers. Despite using autoregressive models, previous methods still suffer from the seesaw problem since they use only one explicit or implicit geometric transformation. Thus, we design two parallel renderers to mitigate the problem and complement renderers with transformation similarity loss. Alleviating the seesaw problem allows the network to generate novel view images better than previous methods, even with non-autoreeeessivve structure. We note that the effectiveness of bridging two renderers can applied in other tasks, such as extrapolation. We believe that our work can prompt refocusing non autoregressive architecture for single-image view synthesis.
300
302
304

---

References
308
[1] Kara-Ali Aliev, Artem Sevastopolsky, Maria Kolos. Dmitry Ulyanov, and Victor Lempitsky. Neural point-based graphics. In European Conference on Computer Vision, pages 696-712. Springer, 2020.
308
309
[2] Wenbo Bao, Wei-Sheng Lai, Chao Ma. Xiaoyun Zhang, Zhiyong Gao, and Ming- Hiugg-ssaan Yang. Depth-aware aware video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3703-3712, 2019.
310
311
312
[3] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. urXiv preprint 1rXiv:1809.1109 2018.
313
314
[4] Shenchang Eric Chen and Lance Williams. View interpolation for image synthesis. In Proceedings of the 20th annual conference on Computer graphics and interactive techniques, pages 179-288, 1993.
316
[5] Xu Chen, Jie Song, and Otmar Hilliges. Monocular neural image based rendering with continuous view control. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4090-4100, 2019.
317
318
319
[6] Paul Debevec, Yizhou Yu, and George Borshukov. Efficient view-dependent image-based rendering with projective texture-mapping. In Eurographics Workshop on Rendering Techniques, pages 105-116. Springer. 1998.
320
321
322
[7] Paul E Debevec, Camillo J Taylor, and Jitendra Malik. Modeling and rendering architecture from photographs: A hybrid geometry-anc and image-based approach. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 11 11-20, 1996.
325
[8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint 118.v01810004805, 2018
328
327
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
328
329
330
[10] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 12873-12883, 2021.
332
[11] John Flynn, Michael Broxton, Paul Debevec, Matthew Valal,, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker. Deepview: View synthesis with learned gradient descent. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2367-2376 2019.
334
335
336
[12] Clément Godard, Oisin Mac Aodha, and Gabriel Brostow. Unsupervised monocular depth estimation with left-right consistency. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 270-279, 2017.
337
338
339
[13] Steven Gortler, Radek Grzeszezuk, Richard Szeliski, and Michael F Cohen. The lumigraph. In Proceedings of the 23rd annual conference Computer graphics and interactive techniques, pages 43-54, 1996.
341
342
[14] Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai- Rii-J Mu, Ralph R Martin, and Shi-Min Hu. Pet: Point cloud transformer. Computational Visual Media, 7(2):17-199)) 1999 2021.
343
344
[15] Peter Hedman, Julien Philip, True Price, Jan-Michael Frahm, George Drettakis, and Gabriel Brostow. Deep blending for free-viewpoint image-based rendering. ACM Transactions on Graphics (TOG), (7 2018.
345
348
347
[16] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
319
350
[17] Yuxin Hou, Amo Solin, and Juho Kannala. Novel view synthesis via depth-guided skip connections. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3119-1128, 2021.
351
352
353
[18] Ronghang Hu, Nikhila Ravi, Alexander C Berg, and Deepak Pathak. Worldsheet: Wrapping the world in a 3d sheet for view synthesis from a single image. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 12528 12537,2377720 2021.
354
10

---

356
[19] Satoshi lizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Globally and locally consistent image completion.
ACM Transactions Graphics (ToG), 36(4):1-14, 2017.
358
[20]
Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-ban based vermutation-invatian neural networks. In International Conference on Machine Learning, pages 3744-3753. PMLR, 2019.
380
[21] Mare Levoy and Pat Hanrahan. Light field rendering. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 42, 1996.
382
[22]
Jae Hyun Lim and Jong Chul Ye. Geometric gan. arXiv preprint arXiv: 170.0094 2017.
384
[23] Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa. Infinite nature: Perpetual view generation of natural scenes from single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14458 14467. 2021.
388
[24] Xinhai Liu, Zhizhong Han, Yu-Shen Liu, and Matthias Zwicker. Point2sqquenee: Learning the shape representation of 3d point clouds with an attention based sequence sequence network. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8778-8785, 2019.
388
369
370
[25] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. arXiv preprint arXiv:1906.0777 2019.
371
372
373
[26]
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, arXiv preprint arXiv: 1608.0393, 2016.
374
375
[27] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
376
10Xiv:171..0100, 2017.
377
[28] Ricardo Bartin-Brualla, Rohit Pandey, Shuoran Yang. Pavel Pidlypenskyi, Jonathan Taylor, Julien Valentin, Sameh Khamis, Philip Davidson, Anastasia Tkach, Peter Lincoln. et al. Lookingood: Enhancing perfor- mance capture with real-time neural re-rendering, arXiv preprint arXiv: 1.111..22 1.05029, 2018.
378
379
380
[29] Moustafa Meshry, Dan B Goldman, Sameh Khamis, Hugues Hoppe, Rohit Pandey, Neah Snavely, and Ricardo Bruuala. Neural rerendering in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6878 6887, 2019.
381
383
[30] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren
Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In European conference on computer vision, pages 405 421. Springer, 2020.
385
[31] Simon Niklaus and Feng Liu. Softmax splatting for video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5437-5446, 2020.
387
[32]
David Novotny, Ben Graham, and Jeremy Reizenstein. Perspectivenet: A scene-consistent image generator for new view synthesis in real indoor environments. Advances Neural Information Processing Systems,
389
380
32, 2019.
391
[33] Kyle Olszewski, Sergey Tulyakov, Oliver Woodford, Hao Li, and Linjie Luo. Transformable bottleneck networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7648- 7657, 2019.
382
393
394
[34] Chunghyun Park, Yoonwoo Jeong, Minsu Cho, and Jaesik Park. Fast point transformer. arXiv preprint urXiv:2112.04702. 2011.
395
396
[35] Taesung Park, Ming- Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially- adaptive normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2337-2346, 2019.
397
398
399
[36] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)9, 2019.
400
401
[37] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelin images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
402
403
[38]
Xuanchi Ren and Xiaolong Wang. Look outside the room: Synthesizing a consistent long- term 3d scene video from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ( (CVPR), 2022.
404
11

---

408
[39] Chris Rockwell, David F Fouhey, and Justin Johnson. Pixelsynth: Generating 3d-consistent experience from single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14104-14113, 2021.
408
[40] Robin Rombach, Patrick Esser, and Björ Ommer. Geometry-free view synthesis: Transformers and no 3d priors. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 18556-14366, 2021.
410
411
412
[41] Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected gans converge faster. Advances in Newral Information Processing Systems, 34, 2021.
413
414
[42] Steven M Seitz, Brian Curless, James Diebel, Daniel Scharstein, and Richard Szeliski. A comparison and evaluation of multi-view stereo reconstruction algorithms. In 2006 IEEE computer society conference on computer vision and pattemn recognition (CVPR 066, volume 1, pages 519-528. TEEE, 2006.
415
417
[43] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Weizstein, and Michael Zollhofer. Deepvoxels: Learning persistent 3d feature embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages
419
2019.
420
[44] Pratul P Srinivasan, Tongzhou Wang, Ashwin Sreelal, Ravi Ramamoorthi, and Ren Ng. Learning to synthesize a 4d rgbd light field from a single image. In Proceedings of the IEEE International Conference on Computer Vision, pages 2243-2251, 2017.
421
422
423
[45] Pratul P Srinivasan, Richard Tucker, Jonathan T Barron, Ravi Ramamoorthi, Ren Ng, and Noah Snavely. Pushing the boundaries of view extrapolation with multiplane images. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 175-184, 2019.
424
426
[46]
Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox. Multi-view 3d models from single images with convolutional network. In European Conference Computer Vision, pages 322-337. Springer, 2016.
428
429
[47] Shubham Tulsiami, Richard Tucker, and Noah Snavely. Layer-structured 3d scene inference via view synthesis. In Pruceedings of the European Conference on Computer Vision (ECCV), pages 302-317, 2018.
430
431
[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
432
439
434
[49]
Qianqian Wang. Zhicheng Wang. Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla Noah Snavely, and Thomas Funkhouser. Ibmet: Learning multi-view amage-bascd rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690 4699, 2021.
435
437
[50] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. Synsin: End-to-end view synthesis from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7467-7477, 2020.
439
440
441
[51] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo., Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information P Processing Systems, 34, 2021.
442
443
444
[52] Saining Xie, Sainan Liu, Zeyu Chen, and Zhuowen Tu. Attentional shapscontextnet for point cloud recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 4606-4615, 2018
[53] Jiancheng Yang, Qiang Zhang, Bingbing Ni, Linguo Li, Jinxian Liu, Mengdie Zhou, and Qi Tian. Modeling point clouds with self-attention and gumbel subset sampling. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3323-3332, 2019.
448
449
450
[54] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578 4587, 2021.
451
452
453
[55] Kai Zhang, Germot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492, 2020.
454
455
[56] Hengshuang Zhao, Li Jiang, Jiay Jia, Philip HS Torr. and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16259- 16269,66,,,,8 2021.
12

---

457
[57] Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A Efros. View synthesis by appearance flow. In European conference on computer vision, pages 286-301. Springer, 2016.
[58] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely, Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint 180.00980 2018.
480
[59] C Lawrence Zitnick. Sing Bing Kang, Matthew Uyttendacle, Simon Winder, and Richard Szeliski. High- quality video view interpolation using a layered representation. ACM transactions on graphics (TOG), 23 (3)6000608, 2004.
462
463
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how answer these questions. For each question, change the default [TODO] to [Yes], [No] or [N/A] - You are strongly encouraged to include a justification to your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
466
487
488
Did you include the license the code and datasets? [Yes]
470
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
471
Did you include the license to the code and datasets? [N/A]] ?
472
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper. please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
473
475
1. For all authors...
478
(a) Do the main claims made the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
477
478
(b) Did you describe the limitations of your work? [Yes] We deal with our limitations in conclusion.
479
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Potential privacy concern is discussed in conclusion.
480
481
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
485
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
484
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
486
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (cither in the supplemental material or as URL)? [Ys]]
487
488
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosea)?
489
[Yes] In section 4 and a supplementary material, training details are presented.
480
(c) Did you report error hars (c.g., with respect to the random seed after running experiments multiple times)? [No] We observe stable performance from ablation study.
492
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] As mention section 4, we use 4 RTX 3090 GPUs.
494
4. If you are using existing assets (e.g.. code, data, models) or curating/releasing new assets...
485
(a) your work uses existing assets, did you cite the ereators? [Yes] Please see supplementary
496
material.
497
(b) Did you mention the license of the assets? [Yes] We use publicly available data for experiments.
498
(c) Did you include any new assets either in the supplemental material URL? [Yes] In the supplementary material, we provide the code.
499
500
(d) Did you discuss whether and how consent was obtained from people whose data you're us-
501
ing/curating? [N/A]
502
(c) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content? [N/A]
503
504
5. If you used crowdsoureing or conducted research with human subjects...
505
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
13

---

507
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "fMBmfBoP9PE", "reviewer": "Reviewer_mgev", "summary": "There are two main paradigms in the task of single-image view synthesis. The explicit methods use the explicit 3D geometry and thus achieve good results on the reprojected pixels, while the implicit methods leverage implicit 3D inductive bias to implicitly learn the 3D geometry and could complete realistic out-of-view image regions. However, none of them can reach both goals. To mitigate such a seesaw problem, this paper bridges both implicit and explicit geometric transformations in a unified framework. Besides, a novel loss is proposed to fuse the geometry features from implicit and explicit branches. The proposed method can render state-of-the-art results on the novel views and achieve about 100x speedup in rendering compared to implicit autoregressive models.", "questions": "In this manuscript, experiments are not comprehensive. Please refer to the questions proposed in the weaknesses.", "limitations": "1. The idea is a little bit straightforward, just like a simple combination of A and B.\n2. Experiments are not comprehensive and not enough to demonstrate the effectiveness of the proposed method.\n3. Some confusing sentences inside:\n- Lines 50 ~ 51: Our approach consists of architecture and loss functions.\n- Lines 159 ~ 160: ... generating photo-realistic images. What's the relationship between rendering speed and image quality. The emphasis on \"photo-realistic\" is unnecessary.\n- Line 48: application area -> application areas\n...", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. Combining the explicit and implicit geometry features in a unified framework is a reasonable and straightforward idea. Since explicit 3D geometry can preserve reprojected image areas and implicit 3D geometry helps to complete out-of-view regions. Meanwhile, the experiments on RealEstate10K and ACID datasets prove the effectiveness of such a combination.\n2. The whole paper is succinct and easy to follow.\n3. The ablation study in Sec. 4.3 & 4.4 demonstrates the necessity of the proposed Transformation Similarity Loss.\n\nCons:\n1. From the main quantitative results in Table 2, the proposed method only shows superiority in terms of the FID metric. Although using both explicit and implicit geometric transformations, this method still works worse than SynSin. It seems that this method doesn't solve the aforementioned trade-off. For the lower FID score, this method just equips with a better imagination ability, which may be boosted with a more powerful adversarial loss. In lines 164~166, the authors mention they use a new GAN loss. How about the performance of SynSin + this new loss? Such a very important ablation study is not included in this manuscript.\n2. The paper only provides the results on RealEstate10k and ACID datasets. How about the results on Matterport3D?\n3. Given the better performance on the FID score, this method should compare with another two crucial baselines, Infinite Nature and Look Outside Room. I notice the authors' clarification in the supp. material. If available, the performance comparisons must be provided.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XmW5blUqBYc", "reviewer": "Reviewer_gwT5", "summary": "The paper proposes a non-autoregressive architecture for single-image view synthesis that combines both explicit and implicit methods. Two parallel renders extract explicit feature maps and implicit feature maps correspondingly, and the proposed transformation similarity loss encourages the consistency between two feature maps in the reprojected region and the out-of-view region. ", "questions": "- In Figure 4, there are check-board artifacts for your method (e.g. the 2nd row), while the autoregressive methods don’t have that. Do you think it can be caused by the upsampling in the decoder? \n", "limitations": "The limitations are discussed in Sec 5.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: The idea of combining the explicit and implicit methods for single-image view synthesis is interesting. Also, the non-autoregressive framework reduces the inference time significantly. Some existing methods like GeoFree[40] also compare both explicit and implicit geometric transformation, but they didn’t combine the advantages of explicit and implicit methods, so the paper is sufficiently novel.\n\nQuality: Overall the proposed method is well-described.\n- As mentioned in the paper, the PSNR is a good metric for evaluating the preservation of reprojected pixels on small changes only. Is it possible to evaluate it for reprojected regions only? In table 2, the Synsin[50] has very competitive PSNR performance (even better for both medium and large split); in Figure 4, the Synsin also shows a better ability to preserve projected contents (e.g. the carpet near the door in 1st row and the droplight in 2nd row), so the explanation in Line 230-232 is not very convincing to me.\n- Maybe it is worth moving the ablation study on the Type of Set Attention in supp. to the main paper since the design of the encoder didn't be discussed in the experiments section, though it seems a novel contribution.\n\nClarity:  The paper is well-written and the motivation is clear. Some parts of the experiments can be explained better:\n- The definition of the explicit methods is not clear to me. For example, the Tatarchenko [46] is defined as an explicit method, but the way they parametrize the pose is very similar to the proposed Implicit renderer (angles only, no explicit warping introduced to the features to generate the image).\n- It’s good to evaluate methods on different sizes of viewpoint changes. Is it possible to also show the qualitative results respectively?  Currently, Figure 4 is not clear since it only has the input source image and doesn’t show the target image, so it’s difficult to know if they are examples of small or large viewpoint changes.\n\nSignificance: The idea of rendering two feature maps in parallel can be general and it is possible to extend it to other frameworks.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "T_siGKYs8w7", "reviewer": "Reviewer_qWX6", "summary": "This paper proposes a new method for single-image view synthesis. The authors point out that existing methods are faced with a \"seesaw\" problem, i.e., a trade-off between preserving observed contents and synthesizing out-of-view regions. To address this issue, the authors propose a method that shares the advantages of both explicit and implicit geometric transformations, where the former is good at preserving existing contents while the latter is good at synthesizing out-of-view contents. A transformation similarity loss is proposed, which motivates the explicitly rendered feature and implicitly rendered feature to learn the best from each other. The pipeline is a feed-forward model without an autoregressive module, thus significantly faster than other autoregressive models. Experiments on two datasets show that the proposed method achieves obvious improvements over other implicit or explicit baselines both qualitatively and quantitatively. Ablation studies are carefully conducted to verify the effectiveness of the proposed transformation similarity loss and the dependency between explicit and implicit renderers.", "questions": "1. I suggest adding InfiniteNature (or a warp+inpainting method) as a baseline if possible.\n\n2. I suggest manually fusing the two features according to the out-of-view mask or sending the mask as an additional input to the final decoder so that the decoder can learn to fuse them.\n\n3. Authors may consider adding some video results, which would be more intuitive for view synthesis tasks.\n\n4. Authors may comment on any weakness listed above if they disagree.", "limitations": "It seems that for reprojected regions, some mismatch to the input image can still be observed in the generated results. Thus, the \"preserving reprojected contents\" objective is still not fully fulfilled and has room for improvement.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. This paper points out the trade-off between explicit and implicit geometric transformation-based methods, and proposes a hybrid model that integrates the best of both, which is new and reasonable. The synergy effects of explicit and implicit renderers are well demonstrated in experiments.  \n\n2. The proposed transformation similarity loss is novel to me. Its effectiveness is clearly verified in Tab.4 and Fig.5, which shows that it helps the explicitly rendered feature and implicitly rendered feature to benefit from each other.  \n\n3. The proposed method shows clear improvements over other baseline approaches in quantitative and qualitative evaluations. Results in Fig.4 show that it better preserves existing contents and synthesizes out-of-view contents.  \n\n\nWeaknesses:\n\n1. I feel that the \"seesaw\" problem is somewhat exaggerated. In my humble opinion, a simpler and more straightforward way to solve this problem is to reproject or warp existing pixels to the target view based on the depth, and then use a generative model to synthesize the out-of-view pixels conditioned on the warped pixels, like inpainting. In this way, it naturally preserve existing contents and synthesize out-of-view contents. InfiniteNature [23] is designed in this way but is not compared. If this design can address the \"seesaw\" problem, then the contribution of this work is not strong. Otherwise, the authors should add some discussion to explain why.   \n\n2. Following the previous point, the authors mentioned in the supplementary that InfiniteNature [23] is not compared because the training code is not available. However, to my knowledge, the testing code is already available online and comparison is possible. For example, even GeoFree [40] compares with InfiniteNature. I suggest adding InfiniteNature (or a warp+inpainting method) as a baseline if there is no strong reason why this cannot be done.  \n\n3. If explicitly rendered feature and implicitly rendered feature are complementary, then why not just manually fuse the two features according to the out-of-view mask or send the mask as an additional input to the final decoder so that the decoder can learn to fuse them?  In this case, the two features do not have to mimic each other. Thus, I doubt if the transformation similarity loss is necessary. I suggest adding this experiment as an ablation study.\n\nSome typos and grammar issues:  \nLine 53: a novel loss function that explicit features improve ... -> a novel loss function that motivates explicit features to improve ...  \nLine 116: should $I_{ref} \\in \\mathcal{R}^{H\\times W \\times C}$ be $I_{ref} \\in \\mathcal{R}^{H\\times W \\times 3}$?  \nLine 135: resulting in a space complexity is reduced from ... -> resulting in a space complexity reduction from ...  \nLine 207: Therefore, We -> Therefore, we  \n\n\nConsidering all the strengths and weaknesses, I rate borderline reject now. I may change my rating if my concerns in weaknesses are addressed.\n\n=============\n\nI have read the authors' feedback. My major concerns on InfiniteNature and feature fusion are well addressed. Thus, I increase my rating to weak accept.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KxXevr4Vj1h", "reviewer": "Reviewer_k3Jg", "summary": "This paper studies the task of single image novel view synthesis. The key message is very clear. Methods with explicit 3D projection mechanism, like SynSin, better preserve observed regions, whereas methods with implicit renderer, like GeoFree, do better at outpainting (synthesizing unseen regions). And using both of them in the same framework can complement and reinforce each other and get the best of both worlds.\n\nThe paper does a very good job at conveying this message, and presents promising results on both indoor and outdoor datasets to demonstrate the benefit of this joint approach.", "questions": "1. Report PSNR both only on seen pixels and on the entire image, if it's manageable during the rebuttal.\n2. Finetune the models of the previous methods on the same dataset, if this is not done currently.\n3. Show more diverse visual comparisons, maybe in the final version.", "limitations": "The paper includes a brief discussion on the limitations. The current quality of the results are still quite limited. It would be helpful to provide some failure examples if any.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n### S1 - Message is clear and makes a lot sense\n- The paper delivers the motivation of using both implicit and explicit rendering in single image NVS in a very clear and concise manner.\n- The benefits of both worlds can be maintained by defining losses that use the better one to guide the other, ie, features from the explicit branch will guide the ones from implicit branch in observed regions, with the gradient of the former being detached, and the other way around in unobserved regions.\n- The resulting model seems to overcome the shortcomings of both approaches, as shown in the comparison results.\n\n### S2 - Clear and concise writing\n- I enjoy reading the paper. It's very clear and coherent.\n\n## Weaknesses\n### W1 - Results are still a bit unsatisfactory\n- First, I was expecting more qualitative comparisons in the supplementary material. There are only four examples shown. They're all good examples with large viewpoint changes. But showing more diverse results will be helpful.\n- Based on the limited comparisons provided, it does not seem that the proposed method can fully preserve the benefits of both implicit and explicit methods. The quality of both observed and unobserved (outpainted) regions seems to be slightly compromised.\n  1. Observed regions do not seem to be preserved very well, seemingly worse than SynSin and PixelSynth on indoor scenes. I think neither of them has been trained on images of outdoor scenery, so I'm not surprised they both fail on scenery images. The PSNR is also worse than SynSin with larger viewpoint changes.\n  2. Outpainted regions also appear worse than GeoFree, although the latter uses an autoregressive framework.\n  3. There seem to be obvious artifacts in the synthesized images, eg the center of the generated image in second row of Fig 4 (dinning table).\n- Are the PSNR computed only on the seen pixels or on the entire image? Reporting both numbers will resolve the question in 1.\n- Have the compared models been finetuned on the same dataset? If not, I do not think the comparison is fair.\n\n### Minor comments\n- Line 116: I suppose $f_0$ has a larger channel dimension than input image $I_{ref}$. Use different symbols for their channel dimensions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject