Rilievo: Artistic Scene Authoring via
Interactive Height Map Extrusion in VR
Sevinc Eroglu1 , Patric Schmitz1 , Carlos Aguilera Martinez2 ,Jana Rusch, Leif Kobbelt1 and Torsten W. Kuhlen112Visual Computing Institute, RWTH Aachen University
Barcelona School of Informatics, Polytechnic University of Catalonia (UPC)

The authors present a virtual authoring environment for artistic creationin VR. It enables the effortless conversion of 2D images into volumetric 3D
objects. Artistic elements in the input material are extracted with a convenientVR-based segmentation tool. Relief sculpting is then performed by interactively
mixing different height maps. These are automatically generated from the input
image structure and appearance. A prototype of the tool is showcased in an
analog-virtual artistic workflow in collaboration with a traditional painter. It
combines the expressiveness of analog painting and sculpting with the creative
freedom of spatial arrangement in VR.

1Introduction

Due to the recent advent of commodity virtual reality (VR) systems, new formsof artistic creation in immersive environments are being explored. Powerful tools
are being developed, ranging from 3D drawing applications to virtual sculpting
and 3D modeling, such as Google Tilt Brush, Gravity Sketch, Facebook Quilland Oculus Medium. Content creation entirely inside VR is an exciting area thatposes notable interface design challenges, yet there is also high demand fromartists to bring existing 2D creations into virtual environments. While existing
digital content creation workflows can prepare 2D artwork for VR applications,this requires training with geometric modeling tools and considerable manual
work [1].
We present our prototype of an artistic VR authoring environment thatmakes 2D material amenable to presentation in VR with minimal manual effort.
Our goal is to design a simple workflow that targets users without a technicalbackground in digital content creation. Traditionally trained painters, ordinary
users, as well as children that want to explore their creativity in VR are among
our target audience. In our application, planar 2D artwork is interactively
extruded using height maps extracted from the drawing.
Another ongoing effort is to match the creative possibilities of digital creation
with the expressiveness of traditional painting using mixed materials. Populardrawing applications incorporate complex models of paint interaction based on
physical properties, and research investigates novel algorithms and interaction
techniques that emulate the analog creation process [2, 3, 4]. Eroglu et al.
1

Figure 1: Top left: “twisted and urban”, 190 x 190 cm, mixed media on canvas( c Jana Rusch, 2019). Top center: an interpretation created in VR. Other:artworks created with our system. ( c Sevinc Eroglu, 2020)
recently presented a drawing tool for VR inspired by the analog paper marbling
technique [5].
We propose an artistic workflow at the interface between classical paintingand VR that combines the expressive power of analog painting with the capabilities of digital creation. Traditional paintings and relief sculptures on canvas aredigitized and inserted into a 3D composition. This hybrid art form unites theprecise handcraft of analog painting and sculpting with the creative freedom of
spatial arrangement inside a 3D virtual authoring environment. Figure 1 shows
artworks that were created with our system.

2

Scene Authoring in Virtual Reality

We developed an intuitive VR authoring environment to create a virtual 3D
composition from 2D material. Our system enables an artists to define theirindividual collection of compositional elements based on images and, optionally,
relief height maps. To create the elements, the artist first isolates desired partsof the provided image using a segmentation tool. Instances of these elements
are dragged into the scene from a palette and arranged spatially using direct
manipulation for rigid transformations and a bimanual controller gesture forscaling.
To give each element instance its own an individual 3D appearance, its surface is extruded by mixing different height profiles with a novel direct manipulation technique we call the “drawbar” metaphor. It enables simplified virtual
sculpting without knowledge about mesh processing or 3D modeling by dragging out “drawbar” handles. Plausible height profiles are inferred from theinput image or user-provided graphics. Relief height maps that were manuallysculpted in an analog process and digitized using a 3D scanner can be used, as
we demonstrate in section 3.
2

Figure 2: Top: 3D user interface for segmentation, feature extraction andheight map generation. Bottom: a segmentation result, the extracted featureedges, and the inferred height map. (Flower image c Sam Oth, CC BY-SA-2.5)( c Sevinc Eroglu, 2020)
In the following, we detail the segmentation, feature-based height map extraction and interactive extrusion steps.

2.1

Segmentation and Height Map Extraction

To create a compositional element, the artist first defines a region in the input
image that contains the desired structure. To reduce manual labor and speed
up the artist’s workflow, we perform a segmentation based on the GrabCutalgorithm [6]. The artist sparsely labels regions as foreground or backgroundpixels by drawing onto the input image with a variable-size brush. The algorithm
then infers the optimal labeling of foreground and background by solving an
iterated graph cut problem. Figure 2 shows our interactive segmentation tool.
The resulting segmentation mask serves as the basis for our first heightmap. We perform a distance transform on the outline, which results in a heightprofile that increases with distance from the object contours. Since many objects
exhibit a profile that bulges outward from the center and falls off toward thesides, this yields a plausible basic 3D shape in many cases.
In addition to the height profile inferred from the contour, we enable theartist to add higher-frequency detail. To this end, we perform edge detection on3

the input image using a Sobel filter for intensity gradient estimation, followedby non-maximum suppression and hysteresis thresholding, similar to the Cannyedge detection algorithm [7]. On the resulting edge image, we apply a distancetransform, yielding a height map that grows with the distance from salient
image features. We provide the artist with a simple user interface to adjust
the hysteresis thresholds until the desired feature edge density is achieved. The
resulting edge image and corresponding height map are updated interactively
while the artist adjusts the sliders, enabling a fast and intuitive workflow.
Lastly, we generate height maps based on the luminance intensity in the
input image or any user-provided image file. This enables the artist to overlay
the height profile with the object’s own apparent brightness, or emboss arbitrary
structure from regular or stochastic input textures, such as brick walls, cellstructures or marble, into the element’s final extruded height profile.

2.2

Interactive Height Map Extrusion

Figure 3: Top: 3D user interface for interactive height map extrusion using“drawbars”. Bottom left: element palette. Bottom center and right: closeup of the beveling tool. (Flower image c Sam Oth, CC BY-SA-2.5) ( c SevincEroglu, 2020)
Our tool presents a palette of compositional elements as a menu that is
attached to the nondominant hand. It shows the image from which the object4

was created next to the generated height maps. A submenu enables the selectionof additional height maps supplied by the artist. After an element is selected,
a preview floats above the menu, which can then be instantiated by dragging it
into the scene.
We provide a height map extrusion panel based on the “drawbar” metaphor
to create an individual 3D appearance for each element. It is inspired by vintage
tone wheel organs, where the final sound is mixed from individual registers bypulling out sliding drawbars. The surface is displaced symmetrically in opposing directions to create a closed surface mesh that can be observed from any
direction.
In the extrusion panel, the height maps are laid out in an arc shape as shownin Figure 3. Each height map is depicted as a grayscale image and features adrawbar handle that is dragged outward to perform an extrusion. The individual
grayscale images are displaced to visualize the relative extrusion depth of eachcomponent. A large grayscale image in the center shows the final mixture for
additional visual reference. We offer an option to hide the preview image tocreate an unobstructed view of the element. For non-smooth height maps,
the opposing surfaces can detach visually. While this can be a desired effect,
we provide an option to connect the parts by automatically adding appropriate
faces. While sculpting the height map mixture using the drawbars, the element’s
3D shape is continuously updated.
The height maps generated via distance transforms vary linearly with distance to contours or feature edges. This results in a pyramidal shape of theheight profile, with a sharp ridge at the medial axis. We offer a “beveling” tool
to add positive or negative curvature to the slope of the height profile. Thisenables the artist to continuously vary the profile from sharp and pointy tosmooth and round, which Figure 3 illustrates.

3Analog Painting and Sculpting for Rilievo

In our hybrid analog and virtual workflow, graphical elements that are drawn
on canvas serve as the starting point for the virtual art piece. The painters canmake use of any traditional painting technique that they are accustomed to, likedrawing with pencils or chalk, blending oil paint and watercolor and working
arbitrary materials into the painting to achieve the desired appearance.
In a second process step, a height field is sculpted underneath a projected image of the painting using clay, spackling paste or other formable materials. This
enables the artist to work with familiar painting techniques and afterward augment the piece with a height profile. To achieve this, we take a high-resolution
photograph of the painting, rectify and project it onto a ground plane.
Finally, a structured light scanner captures a 3D model of the sculpted relief.
It is aligned with the photograph and rendered as a depth buffer image toproduce a height map that conforms to the original painting. The resultingheight map and the photograph of the painting then serve as input for thevirtual composition process. Figure 4 illustrates the analog creation process.
We will present results of this work to the public in the form of a museum
exhibition. For this, we are collaborating with the artist Jana Rusch, who istransitioning her current series of works Inner Green Fields into different 3Dcompositions [8]. Her analog paintings will be presented alongside the digital5

Figure 4: Analog relief sculpting underneath the projected painting. ( c Sevinc
Eroglu, Photo: Patric Schmitz, 2020)
creations, with the virtual scene overlaid on the real canvases as augmentedreality. Her works examine the forms, structures and speeds of human living
spaces and their evolution over time. In the painting of Jana Rusch, the green
field becomes a vision of the urban, a basis for discussion of the urban structuresof the future.

4User Feedback

The informal feedback that we received from users of our system was quite
promising. Students and colleagues from our institute, some with experiencein VR and some without, found the tool simple to use and engaging. Playful experimentation with different input textures and height profiles resulted inmany fascinating visual results. After a short demonstration of the availablecontrols, people used the system without further assistance. In our judgment,
the tool proved very useful for the tasks and the target audience that we envisioned. We plan to evaluate this further in a formal user study with a morewidespread sample of our audience, particularly including children without any
technical background. Furthermore, we believe that the creation of an art piece
inside an immersive environment, in which it will ultimately be experienced by
the recipient, has significant artistic potential. Our collaborating artist statedthat authoring inside VR gives her an unprecedented artistic perspective regarding spatial arrangement that directly relates to her body proportions andmovement. While previous research has shown the potential of scene authoring
within VR [9], we will evaluate our system in an expert user study with artists
and designers that have experience with digital content creation tools.

65Outlook

The authoring environment we presented is a work in progress and there are
numerous improvements planned for the future. We will enable the placementof light sources alongside improved visual quality via physically based rendering(PBR) with normal maps generated from the height fields. We will enable drawing of albedo color and PBR material maps, such as roughness and reflectivity,
directly onto the element surface. For the analog-virtual creation process thatwe sketched, estimating these material properties from a real canvas by a lightstage setup is a promising prospect.
A major goal is to overcome the static nature of the compositions and integrate a possibility to modulate the scene’s visual appearance over time or basedon external control inputs. To this end, we are developing a UI widget for what
we call “source-target modulation”. It enables quick and direct mapping of a setof input sources to modulation targets in the scene, with a direct manipulationinterface to adjust value ranges and the slope of the mapping function.
For all planned interface extensions, the main challenge will be to comeup with suitable 3D user interface metaphors that strike a balance between
expressiveness of the tool and accessibility by our target audience. This is avery compelling interface design challenge that we are eager to take on in thefuture.

References[1] Laura Kottlowski. From 2d to 3d to vr in less than a year. SIGGRAPH,2019.
[2] Peter Vandoren, Luc Claesen, Tom Van Laerhoven, Johannes Taelman,
Chris Raymaekers, Eddy Flerackers, and Frank Van Reeth. Fluidpaint: aninteractive digital painting system using real wet brushes. In Proceedings ofthe ACM International Conference on Interactive Tabletops and Surfaces,
pages 53–56, 2009.
[3] Luc Claesen, Peter Vandoren, Tom Van Laerhoven, Andy Motten, Domien
Nowicki, Tom De Weyer, Frank Van Reeth, and Eddy Flerackers. Smartcamera soc system for interactive real-time real-brush based digital paintingsystems. In 2010 18th IEEE/IFIP International Conference on VLSI andSystem-on-Chip, pages 247–252. IEEE, 2010.
[4] Jiho Yeom and Geehyuk Lee. Designing a user interface for a painting
application supporting real watercolor painting processes. In Proceedingsof the 10th asia pacific conference on Computer human interaction, pages
219–226, 2012.
[5] Sevinc Eroglu, Sascha Gebhardt, Patric Schmitz, Dominik Rausch, andTorsten Wolfgang Kuhlen. Fluid sketchingimmersive sketching based on
fluid flow. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pages 475–482. IEEE, 2018.
[6] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. ”grabcut” interactive foreground extraction using iterated graph cuts. ACM transactions
on graphics (TOG), 23(3):309–314, 2004.
7

[7] John Canny. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence, (6):679–698, 1986.
[8] Jana Rusch. Inner green fields. www.innergreenfields.com/. (Accessedon 03/11/2020).
[9] Telmo Zarraonandia, Paloma Dı́az, Alvaro Montero, and Ignacio Aedo. Exploring the benefits of immersive end user development for virtual reality.
In International Conference on Ubiquitous Computing and Ambient Intelligence, pages 450–462. Springer, 2016.

8