Pseudo-3D Scene Modeling for Virtual Reality
Using Stylized Novel View Synthesis
Kuan-Wei Tseng∗
National Taiwan University
Tokyo Institute of Technology
Taipei, Taiwan
Tokyo, Japankuanwei@g.ntu.edu.tw

Jing-Yuan Huang∗
Yang-Sheng Chen

Chu-Song Chen
Yi-Ping Hung

National Taiwan University
Taipei, Taiwan
jingyuanhuangg@gmail.com
yeti0193275@gmail.com

National Taiwan University
Taipei, Taiwan
chusong@csie.ntu.edu.tw
hung@csie.ntu.edu.tw

Figure 1: Scenario and method overview. We propose modeling the unreachable area in VR with stylized novel view synthesis
as an alternative to actual 3D model or 360 image. A: users can see a stereoscopic image pair synthesized from the monocular
content in VR. B: the pipeline of ArtNV[Tseng et al. 2022] applied to our system for novel view generalization and stylization.
It integrates depth-based view synthesis and stereoscopic style transfer to generate the left- and right-eye views of VR glasses.

ABSTRACTKEYWORDS

Stylized Novel View Synthesis is an emerging technique that combines style transfer and view synthesis. However, none of the existing works explore their applications in Virtual Reality (VR). Thiswork devises a novel application for stylized novel view synthesis.
We propose to replace actual 3D scene models or 360 images withstylized stereoscopic images for the areas outside the major playarea but are still visible to the user. User study results reveal thatusers can feel 3D sense and tell them from plane texture. Codes andother materials are available at: kuan-wei-tseng.github.io/ArtNV

View Synthesis, Style Transfer, Virtual Reality (VR)CCS CONCEPTS• Computing methodologies → Image processing; Imagebased rendering; Virtual reality.
∗ Both authors contributed equally to this research.

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
SIGGRAPH ’22 Posters, August 07-11, 2022, Vancouver, BC, Canada© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9361-4/22/08.
https://doi.org/10.1145/3532719.3543232

ACM Reference Format:
Kuan-Wei Tseng, Jing-Yuan Huang, Yang-Sheng Chen, Chu-Song Chen,and Yi-Ping Hung. 2022. Pseudo-3D Scene Modeling for Virtual Reality
Using Stylized Novel View Synthesis. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Posters (SIGGRAPH’22 Posters), August 07-11, 2022. ACM, New York, NY, USA, 2 pages. https://doi.org/10.1145/3532719.3543232

1INTRODUCTION

Recent advances in novel view synthesis and stylization have shown
great potential for applications in 3D displays. Given the content
image and a style template, it aims to generate unseen novel views
at specified camera poses with designated artistic styles.
To integrate both view synthesis and neural style transfer, aprimary challenge is the spatial consistency between views. Thatis, if the 2D points on different views correspond to the same 3D
point, they must have the same style transfer. Previous works handle the consistency by training a shared 3D representation in an
implicit [Chiang et al. 2022] or explicit [Huang et al. 2021] manner.
Though achieving great consistency, they require multi-view inputs
and is time consuming. Mu et al. [Mu et al. 2022] propose a 3D
photo stylization model that can generate stylized novel views withSIGGRAPH ’22 Posters, August 07-11, 2022, Vancouver, BC, Canada

Tseng and Huang, et al.

neural style transfer with the following objectives:
L = 𝜆1 L𝑐𝑜𝑛𝑡𝑒𝑛𝑡 + 𝜆2 L𝑠𝑡 𝑦𝑙𝑒 + 𝜆3 L𝑐𝑜𝑟𝑟 ,(1)

where L𝑐𝑜𝑛𝑡𝑒𝑛𝑡 and L𝑠𝑡 𝑦𝑙𝑒 are the content and style losses computed with pretrained VGG-19 models, respectively. The combination coefficients are set as 𝜆1 = 1𝑒6, 𝜆2 = 1, and 𝜆3 = 3000.
Figure 2: First/third-person view of the proposed system.

3

only a content image and a style template as input. However, theirapplicability has not been addressed in VR yet.
Constructing virtual scenes is essential to VR, and it usuallyrequires 3D models or 360 images. Even for unreachable places, westill need to create the scene as the user’s sight can still reach it. In
this work, we propose to use stylized novel views as an alternative
for scene modeling in VR. For the unreachable areas, we can display
synthesized stereoscopic images in the VR glasses to make theusers perceive 3D. Moreover, the neural style transfer can ensure
the reachable and unreachable areas to share the same artisticstyles. In detail, we adopt our previous work, ArtNV [Tseng et al.
2022], as the stylized novel view synthesis model. It is a two-stage
pipeline consisting of view synthesis and geometric-consistent styletransfer stages. Our approach integrates Synsin [Wiles et al. 2020],
a view synthesis model , with online neural style transfer [Gatyset al. 2016] using optical flow-based dense matching to achievestereoscopic consistency. It is noteworthy that our method only
needs a single image as content input. Creators can simply takean arbitrary content image to construct a pseudo-3D scene thathas same style with the texture of interactable objects. User study
experiments reveal that the proposed method allows users to feel
3D senses in a virtual world effectively.

To evaluate our method, we recruit 10 users (5 males and 5 females)
to conduct a user study. We compare our method with that of placing a planar image outside the window with a distance of 2.5m. Theusers will see a scene with five different styles. For each style, weshow the result of our method and the planar-image method in arandom order and ask the users which one is more stereoscopic.
The results show that 9 out of 10 people prefer our method. We
also asked users to explain their criteria. Users commented that
"I can tell whether the image is planar if it does not follow mymovement naturally." (P1, P2, P3) and "I feel dizzy if the image isstereoscopic" (P5, P9, P10). We further asked users whether they feel
discomfort or weird during the experiments. Besides motion sickness, users also responded that they feel some results are defocused.
This is probably because the baseline of stereoscopic images is notcustomized for each testee, which may be improved by pupillary
distance calibration before using our system.

2 DESIGN AND IMPLEMENTATION
2.1 Virtual Scene
We use Unity to construct virtual scenes. The scene is divided into
two parts: reachable area and unreachable area. Without loss ofgenerality, as a example shown in Fig. 1, the play area is a roomof size 4m*4.5m*5.2m with several pieces of furniture and a seethrough window in VR. We use different textures for each piece ofthe furniture, such as cotton quilts and wood-grain tabletops. Styletransfer is performed on these texture maps before attaching themto the furniture. A window is placed on the wall of the room, and its
size fits for most of the users to see the unreachable area. Outside thewindow, the user can see stylized stereoscopic images. Specifically,
we customize the shading method in Unity to render the left andright eye’s images in a quad close to the window, respectively.

2.2

Stylized Novel View Synthesis

We utilize the novel view synthesis and stylization pipeline inArtNV. Given the baseline between two views, we first generate astereo image pair using SynSin, a depth-based novel view synthesis
approach. Next, we estimate the dense optical flow between theimage pair using RAFT [Teed and Deng 2020] with the occlusionmaps obtained by the forward-backward consistency check. We
leverage the dense correspondence to construct a consistency lossL𝑐𝑜𝑟𝑟 for constraining stylized outputs to have similar colors in theoverlapping regions. Finally, we perform image optimization-based

4USER STUDYDISCUSSION AND FUTURE WORKS

We introduce a novel application of stylized novel view synthesis toVR scene modeling. It is an easy-to-implement and low cost alternative to the actual 3D models or 360 images for the unreachable areain the virtual scene. Creators can take a content image and a style
template to achieve congruent user experience on both stereoscopy
and style. As for the future work, we aim to extend our method tomodel widely the visible area to increase its utility.

ACKNOWLEDGMENTS
This work was partially supported by Ministry of Science and Technology in Taiwan (MOST 110-2221-E-002-185-MY2 and MOST 1112218-E-002-020-MBK).

REFERENCES
Pei-Ze Chiang, Meng-Shiun Tsai, Hung-Yu Tseng, Wei-Sheng Lai, and Wei-Chen
Chiu. 2022. Stylizing 3D Scene via Implicit Representation and HyperNetwork. In
Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision(WACV). 1475–1484.
Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2016. Image Style Transfer
Using Convolutional Neural Networks. In Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition (CVPR).
Hsin-Ping Huang, Hung-Yu Tseng, Saurabh Saini, Maneesh Singh, and Ming-Hsuan
Yang. 2021. Learning to Stylize Novel Views. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
Fangzhou Mu, Jian Wang, Yicheng Wu, and Yin Li. 2022. 3D Photo Stylization Learning to Generate Stylized Novel Views from a Single Image. Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Zachary Teed and Jia Deng. 2020. Raft: Recurrent all-pairs field transforms for opticalflow. In Proceedings of the European Conference on Computer Vision (ECCV). Springer,
402–419.
Kuan-Wei Tseng, Yao-Chih Lee, and Chu-Song Chen. 2022. Artistic Style Novel View
Synthesis Based on A Single Image. Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition (CVPR) Workshops.
Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. 2020. SynSin:
End-to-End View Synthesis From a Single Image. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition (CVPR).