# A Data Driven Approach to Generate Realistic 3D Tree Barks

Aishwarya Venkataramanan, Antoine Richard, Cédric Pradalier

<figure><img src="image_1.png" /></figure>

To cite this version:

Aishwarya Venkataramanan, Antoine Richard, Cédric Pradalier. A Data Driven Approach to Generate Realistic 3D Tree Barks. Graphical Models, 2022, 123, pp.101166. 10.1016/j.gmod.2022.101166. hal-05077303

HAL Id: hal-05077303

https://hal.science/hal-05077303v1

Submitted on 21 May 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A Data Driven Approach to Generate Realistic 3D Tree Barks

Aishwarya Venkataramanan,<fnref n="1" /> Antoine Richard,<fnref n="2" /> Cédric Pradalier<fnref n="3" />

## Abstract

3D models of trees are ubiquitous in video games, movies, and simulators. It is of paramount importance to generate high quality 3D models to enhance the visual content, and increase the diversity of the available models. In this work, we propose a methodology to create realistic 3D models of tree barks from a consumer-grade hand-held camera. Additionally, we present a pipeline that makes use of multi-view 3D Reconstruction and Generative Adversarial Networks (GANs) to generate the 3D models of the barks. We introduce a GAN referred to as the Depth-Reinforced-SPADE to generate the surfaces of the tree barks and the bark color concurrently. This GAN gives extensive control on what is being generated on the bark: moss, lichen, scars, etc. Finally, by testing our pipeline on different Northern-European trees whose barks exhibit radically different color patterns and surfaces, we show that our pipeline can be used to generate a broad panel of tree species' bark.

## 1 Introduction

Tree modeling is an important aspect of computer graphics: its wide range of applications in computer-generated scenes and video games makes it a key-element of today's digital scene. Furthermore, the forestry industry, where this project originates from, could benefit from high quality synthetic data to increase the performances of data-driven, non-destructive quality assessment techniques.

The computer graphics literature consists of various methods to generate high quality trees, but only a handful focuses on modeling the tree barks realistically [19, 38]. The difficulties in generating realistic tree barks arise

<footnote n="1">¹Georgia Institute of Technology, Atlanta, USA, Present address:Université de Lorraine, CNRS, LIEC, F-57000 Metz, France</footnote>

<footnote n="2">²Georgia Institute of Technology, Atlanta, USA</footnote>

<footnote n="3">³Georgia Tech Lorraine, CNRS IRL 2958, F-57070 Metz, France</footnote>
from the complex appearance of the bark surfaces and color. Within a same
tree specie, the barks often present a wide variety of patterns and intricate
details. Consequently, realistic bark modeling is an active field of research.
In the literature, the majority of the works [4, 9, 25, 26, 35, 39] use traditional
3D modeling methods to generate synthetic tree barks. In contrast to the
classical modeling methods, we explore a deep-learning-based methodology
to generate high quality 3D models of tree barks. More precisely, we propose
a methodology to generate realistic tree barks using 3D Reconstruction and
deep Generative Adversarial Networks (GANs) [8]. 3D Reconstruction is
used to generate tree barks of real trees, while GANs are trained on real tree
barks and used to generate synthetic barks. The GANs used here generate
tiles that need to be merged together to form a continuous map that is then
rolled onto a tree trunk. Because of the rolling and the tiling, there tends
to be a discontinuity at the place where the tiles meet. We alleviate this
using a GAN based tiling method. Furthermore, GANs offer a lot of control
on how the bark is generated, allowing the users to add scars, cut branches,
moss and many other details easily. All in all, the GANs proposed here allow
the users to modify the texture and features of the trees trunks to suit their
needs easily and realistically. Our proposed pipeline can be divided into the
following two steps:

* 3D reconstruction of real barks to obtain their geometry and color.

* GANs to generate intricate bark details and colors in order to create synthetic tree barks.

Using our method, we generate barks of Oak, Beech and Robinia trees that
have strongly contrasting structures, demonstrating the broad application
scope of our method. Furthermore, we showcase the robustness of our
method by training the GANs on a dataset acquired under various light-
ing conditions. Our experimental results show that our approach generates
realistic looking features for trees with both smooth and deeply ridged sur-
faces. When provided with smooth tree bark models with no surface features,
this method could be used to generate thousands of realistic looking 3D bark
models for several types of trees. Overall, our contributions are as follows:

* We propose a new pipeline for generating 3D models of realistic tree barks using 3D Reconstruction and GANs. This method offers extensive control over the generated bark allowing to synthesize more features other than bark patterns such as moss, scars, etc.

* We demonstrate a method to automatically acquire labelled dataset to perform deep-learning based background removal on our trees.
* We introduce a GAN network: Depth-Reinforced-SPADE, that can generate multi-modal outputs and also generates the surface geometry information and the color of the barks simultaneously.

To the best of our knowledge, this is the first attempt at generating realistic tree barks using deep learning.

Code is available in: https://github.com/vaishwarya96/Tree-barks-generation.
git and dataset in https://www.dropbox.com/scl/fo/19mktx02p49eq2vxx1bf1/
h?dl=0&rlkey=ghsaqiznjw5kgdnig4tot2jw5

# 2 Related Work

## 2.1 3D Modeling of Trees and Tree Barks

Numerous works have focused on generating complete tree models, including the branches and the leaves. Amongst these, [39] uses tree-cuts from real tree models to generate synthetic trees and [4,26,35] use sketch-based modeling. [9,25] use a procedural approach, but they do not model the bark. Many works [9, 20, 25, 29, 35] model trees or plants from images. [21] uses a combination of GAN and procedural modeling to generate tree models. While these methods have been successful in generating the overall structures of the trees, they do not model the realistic bark textures which is important for identifying the different tree species. Thus, our focus in this study is to develop a tree-specie specific bark generator.

[6, 18, 36] model cracks and basic structures, but they are not capable of imitating the bark patterns of a specific species. [2] proposes to use X-ray images of a real bark to generate Maple bark structures. While this method is interesting, it is cumbersome and requires X-ray images of trees which are hard to come by. [38] uses texton analysis to generate barks from a single photograph, and [19] uses procedural modeling to generate barks. Contrary to these methods, we use deep-learning to generate realistic tree barks. Learning based approach facilitates the users in generating several unique barks without much of manual intervention. Additionally, the approach presented here allows the user to control what will be generated on the trees' barks.

## 2.2 Generative Adversarial Networks

The GAN introduced in [8] uses neural networks to generate the images. However, using neural networks to generate images are limited to simple
datasets. To solve this, DCGAN [30] incorporates Deep Convolutional Neural Networks (CNN) in the generator and discriminator to generate complex images. Using deep CNNs boosted the quality of the generated images compared to using neural networks. Since then, novel loss functions [1, 23] and deep CNN architectures [14–16] have been evolving to stabilize the training, and to generate higher quality images. For instance, [14] could achieve high resolution images of human faces by training a model on images of progressively increasing resolution.

With the introduction of pix2pix [12], conditional GANs have attracted wide-spread attention due to their ability to generate good-quality images from an input label. Pix2pix is an image-to-image translation network, where the input label is an image and the network generates an image corresponding to the input. Various image-to-image translation networks have been proposed in [5, 27, 37, 40, 42] since the success of pix2pix. CycleGAN [42] uses a cycle consistency loss to perform image-to-image translation on an unpaired set of training data. StarGAN [5] makes use of mask vector method to translate images from one domain to another, when the training data is from two different domains. Self-Attention GAN [40] adds self-attention blocks to the generator and discriminator to generate fine details from all feature locations. Pix2pixHD [37] uses a multi-scale generator and discriminator to generate high-resolution images from input images, while SPADE [27] uses a spatially adaptive normalization to retain the semantic information throughout the generator network while generating photo-realistic images from semantic maps.

Additionally, as part of our pipeline, we have to tile the images generated by the GANs to obtain a continuous image. The state-of-the-art GAN network for tiling images is TileGAN [7]. However, TileGAN requires the images to be generated by Progressive GAN [14], and it tends to modify large parts of the original images to be tiled, which is not desirable for our objective. Hence, we developed a GAN based on in-painting for our objective.

# 3 Generating Tree Barks

In the coming sections, we explain the pipeline to generate realistic tree barks from sequences of images. The block diagram of our pipeline is given in Fig. 1. Our pipeline is divided into two stages:

* In the first stage, we acquire the dataset, perform background suppression on the images, conduct 3D reconstruction, and extract the surface geometry and the color of the bark.
* In the second stage, we use GANs to generate the synthetic surface geometry and bark colors and provide control over the generation of the moss, scars, lichens, etc.

<figure><img src="image_3.png" /><figcaption>Figure 1: Our proposed pipeline for generating 3D models of real and synthetic trees</figcaption></figure>

## 3.1 3D Reconstruction, Surface Geometry and Color Extraction

### 3.1.1 Dataset Acquisition

As shown in Fig. 1, the first step in the pipeline is to capture image sequences of trees. To perform 3D reconstruction and triangulation, the image sequences of trees must be captured from multiple camera positions. To make our method affordable and as general as possible, we rely on consumer-grade monocular cameras. Initially, while testing with smartphone cameras, motion blur was introduced due to jerking of the hand, which deteriorated the quality of the 3D reconstruction. Hence, we use the DJI Osmo pocket, an inexpensive gyro-stabilized camera. Gyro-stabilization counteracts the camera shakes, and gives sharper and better quality images.
The videos were acquired by slow movement of the camera to avoid motion blur. First, the vertical portions of the trees were captured, followed by a slight horizontal rotation of the camera. This sequence was repeated to cover the entire circumference of the tree.

Since we capture the images using a single monocular camera, the global scale information of the tree is not preserved in the final 3D point cloud. To get the scale information, we attach square markers of 3 by 3 cm onto parts of the trees while capturing the video. We place the markers onto the part of the tree near the ground so that it does not obstruct the mid portion of the trunk which is required for extracting the surface geometry and color information. Once the 3D point cloud of the reconstructed tree is obtained, the known marker dimension is used to scale the tree accordingly.

### 3.1.2 Background Suppression

To perform the reconstruction, we rely on Pix4DMapper [28]: a feature-based 3D reconstruction tool. With this type of tool, removing the background reduces the number of features points, that not only makes the reconstruction faster but also more accurate. Background segmentation reduces the risk of confusing the object of interest with the associated background. This is especially true in images, since they are 2D projections of the 3D world with both object and background ending up on the same plane. Hence, we developed a semi-supervised learning method to remove the background portions of the images and retain only the tree barks. In this method we first create a coarse dataset using traditional computer vision methods, and then use this dataset to train a Deep Neural Network. The advantage of using standard computer vision algorithm to make a coarse dataset is that no manual labeling is required. Considering that our images were acquired using a gyro-stabilized camera, we can assume that the transition between the consecutive frames is gradual and smooth. Under this assumption, we developed a mechanism to automatically label the images as bark (foreground) or non-bark (background), using optical flow.

To remove the background portions from the images, we use PSPNet [41], a deep learning segmentation network. It is first trained on the CMU dataset [31], and fined-tuned on our automatically acquired dataset.

From each of the consecutive image frames, we obtain the horizontal and vertical flow vectors and use these to calculate the magnitude and direction of the flow. As our images are acquired using a gyro-stabilized camera, we can hypothesize the following: when the camera translates along the tree, the direction of the vector is vertical and the magnitude of the flow
vectors is greater in the foreground than in the background, seeing that the objects closer to the camera move faster than the far-away objects. On the contrary, when the camera rotates, the magnitude of the flow is greater in the background since the far-away objects move faster than the objects closer to the camera. A sample of the flow magnitude when the camera translates and rotates is shown in Fig. 2.

<figure><img src="image_4.png" /><figcaption>Figure 2: Magnitude of optical flow calculated on video sequences: when the camera translates, the magnitude of the flow is greater in the foreground; (b) when the camera rotates, the magnitude of the flow is larger in the background</figcaption></figure>

We use an adaptive threshold on the magnitude of the flow vectors to mask the background portions in the images. We calculate the magnitude of the maximum flow vector for each frame and compare it with the other flow vectors for the same frame. When the camera translates vertically, and the magnitude of the flow vector is greater than 50% of the maximum magnitude, we label those regions as the foreground and the other regions to be the background. The adaptive threshold method worked best when the camera translates and failed for the rotation case. Hence we use the labels when the camera translates and ignore when it rotates. For a more robust method to perform background suppression than optical flow, we train a deep learning segmentation network.

After training, the images extracted from all the trees' videos are sent through the network. The output from the segmentation network is a label map with two classes: bark-regions and non-bark regions. Sometimes, the segmentation network detects trees in the background and retains them. To remove those trees, we find the largest connected region, which is the tree bark to be reconstructed. This region is retained while the others are masked. Finally, we dilate the final mask to include the edges of the tree
<figure><img src="image_5.png" /><figcaption>Figure 3: Pipeline used for suppressing the background regions in the tree images. The images are passed through a segmentation network to remove the background portions of the images. Then the largest contour is detected and retained. Finally the mask obtained is used to acquire tree bark images with the background portions erased.</figcaption></figure>

bark and apply the resulting mask on to the original images to suppress the background regions, as shown in Fig. 3.

If we had acquired the images with a 3D camera, such as Kinect or Realsense, this process would not have been necessary. However, these cameras are cumbersome and require taking a laptop or small embedded computer to record the trees, which is impractical in forests. Furthermore, the RGB and depth images acquired by these cameras are of low quality, particularly in outdoor conditions. Hence, if one wanted to extract the foreground objects using depth information, we would favor a small 3D laser-scanner, such as a Livox Mid-40, which we could remap to the cameras pixels.

### 3.1.3 3D Reconstruction

The images, with the background portions suppressed are given to Pix4dMapper [28] as it is faster and more accurate than Colmap [32, 33] to generate a dense 3D point cloud of the tree.
### 3.1.4 Surface Geometry and Color Extraction

Once the 3D dense point cloud of the tree is obtained, we extract the surface geometry and the bark color from the point cloud. To do this, we approximate the 3D point cloud of the tree to a series of circles stacked vertically. In the cylindrical coordinates, we discretize the point cloud along the height and angle. The height is discretized in steps of 1 mm and angle in steps of 0.5°. For each of the circles obtained from the discretized height, we obtain the center point using a circular least-square fit.

To extract the surface geometry information, we construct a 2D map in polar coordinates, with the radius measured from the center point of the tree as a function of the height and the angle. Since the height and angle are discretized, there may be many points corresponding to a particular value of height and angle. Thus, we take the mean value of the radius of the points to construct the radius map. The mean radius is calculated as a Gaussian weighted sum of the distance between the point and the center of the cell in the map divided by the sum of weights.

Similarly, to extract the color of the bark, we calculate the mean values of R, G and B as a function of height and angle, where instead of the radius, the R, G, B values of the point are used.

Sometimes, there may be cells in the map where there is no point. This is much more prevalent in the case of oak and robinia trees as the ridges on the bark surface creates occlusions which are not reconstructed in the point cloud. As the bark of the beech tree is much smoother, the occlusions are minimal. The presence of cells with no point creates empty regions in the final map. To avoid this, we construct a multi-level image pyramid. The lowest level in the pyramid consists of the original resolution map, and the highest level consists of map with resolution $1/2^7$ of the original map. For each level in the pyramid, if there are cells with no point, the radius and RGB values are interpolated from the map of the next level up. Fig. 4 demonstrates the need for interpolating. Fig. 4(a) is the oak bark before interpolation. The black regions in the image are due to empty cells. After interpolation, the empty cells are filled with values from the higher level in the pyramid, which can be seen in Fig. 4(b).

## 3.2 Synthetic Bark Surface and Color Generation

In this section, we will discuss the second stage of our pipeline: generating synthetic surface and color of the barks using conditional GAN(cGAN). For each tree specie, we train a separate GAN that specialises to the particular
<figure><img src="image_6.png" /><figcaption>Figure 4: Example of an oak bark (a) before interpolation, (b) after interpolation</figcaption></figure>

tree type and generates the fine bark details corresponding to that tree species. For training the GAN we smooth the ground truth real barks until no fine bark details are present. We provide the smooth bark as input to the GAN and the network learns to generate the fine bark details on this by using the real barks with fine details as ground truth. Later, the user could provide maps of smooth tree trunks to the network and use the trained GAN models to generate thousands of synthetic barks.

To test our approach, we use empirical mathematical models to generate thousands of smooth tree bark structures. These mathematical models are approximations of large varieties of the trees that have cylinder-like structure and do not accurately represent any tree in particular.

### 3.2.1 Generating Smooth Approximate Radius Maps

The smooth surface of the bark represents the overall structure of the tree trunk without any finer structure details. The map consists of radius values $r$ of the tree trunk as a function of height $z$ and angle. To generate these maps, initially, we create a 2D map with an exponentially decaying radius $r_0$.

$$r_0 = ax^z \qquad (1)$$

where, $x$ is a floating point number between the range 0.8 and 0.9, and $a$ is a random value chosen between 0.1 and 0.9. To approximate the wavy structure of an actual bark, we add a series of sinusoids of varying amplitude
and frequencies to $r_0$.

$$r = r_0 + \sum_{i=1}^{n} b_i \sin(\omega_i \theta_i) \qquad (2)$$

where, $n$ is an integer chosen randomly between 1 and 3, $b_i$ is the amplitude of the $i^{th}$ sinusoid. $\omega_i = 2\pi f_i$ where $f_i$ is a value between 20 and 100. This determines the frequency of the generated sinusoids. $\theta_i$ is chosen based on $\omega_i$ to ensure that $r$ is 2$\pi$-periodic. The amplitudes and frequencies of the sinusoids, and the value of $a$ are randomly chosen in order to generate unique smooth tree barks every time. The parameter values are hand-tuned for each tree species based on the visual observations of the tree bark. For instance, the beech trees are mostly cylindrical and hence $b_i$ is set to a value close to zero (range of 0 to 0.005). For oaks, $b_i$ has a higher range of values (0.01 to 0.05). Similarly, one could adjust the range of radius values to fit to the tree type. For example, palm trees are mostly thin and so will lie in the lower range of the radius. The number of parameters to be tuned are limited (radius, amplitude of sinusoid) and so we could easily find the suitable range of values of these parameters and use the generic model for our testing. While we developed the smooth trunk models and parameters empirically, one can also use one's own models and use the pipeline to generate the fine barks details.

Our GANs were trained using the real tree bark models that were smoothed, but still works well when tested on the empirically developed bark models which shows that the GANs are robust to the input smooth barks and can generalise well. While they work well for most of the trees with cylindrical-like structure, it would fail on non-cylindrical barks such as the buttress root trees.

### 3.2.2 Generating Bark Surface and Color Using GANs

In this section, we explain the architecture that generates the surface and color maps from the smooth surface maps. Apart from the ability to generate barks, we also want to give the user the ability to generate moss, lichen, scars, and any other particularities that would be labelled in the dataset, which we provide through semantic labels. Each label value in the semantic labels indicate the location of moss, lichen and scars.

The main drawback of using most of the state-of-the-art cGANs for bark generation is that they generate a single image corresponding to the input. Most of these networks lack multi-modality i.e, they generate a deterministic output for an input label. Ideally the network must generate multiple outputs
for identical inputs to add variability in the bark patterns generated. Thus, to generate the surface and color maps using these networks, one must use a two-stage pipeline: first input the smooth surface maps and obtain the surface maps with the generated details. Second, use the generated surface maps and semantic labels as input to another network and obtain the color maps. It is necessary to use the generated surface maps as input in order to ensure multi-modality in the generated colors. When the networks are provided with identical semantic labels, it generates deterministic output. Therefore, to obtain unique color maps, one must add variability in the input, which we do through the surface maps. The uniqueness of surface maps is ensured through the unique input smooth surface maps.

However, as the input surface maps pass through the deep layers of the color map generator, the surface information gets lost which results in the color maps lacking the necessary sharpness in texture. Additionally, the color generator network is dependent on the semantic labels(moss, lichens, etc), while the surface generator is independent of it. This necessitates the need for two different architecture types for generating the surface and color maps. As a result, we developed an architecture based on SPADE [27] called "Depth-Reinforced-SPADE". The network takes two inputs: a smooth surface map and a semantic map, and outputs a precise surface map and a color image at the same time.

<figure><img src="image_7.png" /><figcaption>Figure 5: Encoder architecture of the generator in Depth-Reinforced-SPADE</figcaption></figure>

The architecture of Depth-Reinforced-SPADE is made up of a main generator and two discriminator networks. The main generator consists of an encoder and two decoders for generating the surface and color maps. The encoder is made of ResNet blocks [10]. The input to the encoder is the smooth surface map. The encoder compresses the input into a concise latent space, which preserves the input smooth surface bark information, and also helps in achieving multi-modality in the generated outputs. The decoder uses the latent space obtained from the encoder to generate the accurate surface maps and the corresponding color. Rather than using a single de-
<figure><img src="image_8.png" /><figcaption>Figure 6: Decoder architecture of the generator in Depth-Reinforced-SPADE</figcaption></figure>

coder to generate the surface and color, we separate the generation. This
is for two reasons: first, the surface and color generation are two separate
tasks. Using a single decoder to generate them resulted in unstable training.
Second, the color maps are dependent on semantic labels whereas the surface
maps are independent of them. Thus, the two decoders are architecturally
different. They take in the latent space as input. The surface generator con-
sists of ResNet blocks whereas the color generator consists of SPADE ResNet
blocks [27]. The SPADE ResNet blocks take in the semantic labels and the
output channels from the surface generator's ResNet blocks. This is done
to reinforce the surface information onto the color generator to enhance the
texture of the generated color maps. Since both the surface and semantic
labels are passed through the SPADE ResNet layers, the information from
surface maps can overwhelm the semantic labels. To maintain a balance
between both the information, we pass the surface generator's feature maps
through convolution layers with an output of 16 channels and append its
output to the input semantic map. Without this convolution which acts as a
surface regularizer, the color generator was not able to leverage the semantic
input fully, leading to an uncontrolled growth of moss and lichen all over
the generated bark. The network architecture of Depth-Reinforced-SPADE
is shown in Fig. 5 and Fig. 6.
<figure><img src="image_9.png" /><figcaption>Figure 7: Our GAN architecture for tiling the images</figcaption></figure>

To obtain high quality outputs, we use separate discriminator networks for the surface and color maps that specialise in discriminating the surface and the colors respectively. the architecture of the discriminator networks are identical to the one used in pix2pixHD [37].

We train the network with the same loss functions used in SPADE [27], namely, the GAN loss, feature matching loss, perceptual loss [13] and KL-Divergence Loss.

### 3.2.3 Tiling the Images and Creating 3D Meshes

The surface and color maps obtained from the GANs are parts of a full tree bark. To build the full structure of a tree trunk, the images must be combined together to form a full map. When the image tiles are combined, there will be discontinuities at the places where the images are joined. The discontinuities in the surface maps appear as bumps in the 3D models of the tree barks, while the discontinuities in the color maps results in abrupt change in bark colors. Thus these discontinuities must be removed to obtain a continuous map. The existing GAN-based tiling method [7] modifies large parts of the images and it requires the images to be generated by Progressive GAN [14]. This cannot be used for our objective because we do not rely on Progressive GAN to generate our images. Also, it is preferred that only a minimal portion of our generated images are modified. Thus, for tiling
the images, we used an in-painting based method, where the discontinuities
in the tiled image are masked and a conditional GAN network is used to
generate a continuous image by filling only the masked regions.

To tile the images, we combine four image tiles into a single larger image. The combined images will have discontinuities at the places where the images are joined. Before feeding the images to our tiling network, we mask 15 pixels from each side of the discontinuities and fill the masked areas with the mean values of the image. Thus, contrary to the previous method, only the masked regions are modified by the GAN. To prevent the GAN from modifying the whole image we constrain its generation space to the gap regions only. This was done to prevent it from learning a trivial copy task that can be implemented separately. Doing so drastically improved the quality of the generated images and increased its convergence speed. This process can be seen in Fig. 7.

The network is a modified pix2pix architecture [12], where the generated image is copied with the unmasked pixels from the input image. Thus, the unmasked portions from the input images are by-passed from the generator and the resulting images are given to the discriminator. This way, we can ensure that the generated images are of the same quality as the original images. Because we are copying the unmasked portions of the original images on to the generated images, we can be assured that the GAN only fills the masked regions without modifying the other image parts. This tiling technique is applied to both the surface and color images to obtain the full maps. Once we obtain the full surface and color maps, we construct a 3D models of the tree bark using VTK [34].

# 4 Experiments

## 4.1 Dataset

We conducted experiments on our tree bark dataset. As of today it contains 3 tree-species: Oak, Beech, and Robinia (Locust).

To build this dataset we recorded videos of different trees. In total, we acquired videos of: 20 oak trees, 15 beech trees, and 18 robinia trees. These videos were captured with a DJI Osmo camera at 29.99fps in full HD resolution (1920x1080). All the trees were captured from local forests/streets under natural lighting, with automatic exposure. Fig. 8 shows some of the captures from the video frame of oak trees. Each video consists of approximately 7000 frames. Since we use a gyro-stabilized camera, the transition between the consecutive frames are smooth and there are no major changes
<figure><img src="image_10.png" /><figcaption>Figure 8: Samples of oak images with different illuminations.</figcaption></figure>

between consecutive frames. Most of the consecutive frames contain redundant information and so, the images are extracted from every tenth frame of the whole video sequence. This way, the number of images for reconstruction are in the range of a few hundreds, which could be handled efficiently by the photogrammetry software: Pix4D Mapper. For better reconstruction results, we recommend using hand held gimbals and a professional grade camera with controlled brightness, although as we demonstrate here, satisfying results can still be achieved without any of those.

The videos of the oak trees were acquired on a sunny day and most of the oaks in our dataset consists of regions with shadows and parts illuminated by sunlight and sun glares. Because of this, the 3D reconstruction and the color maps extracted from the reconstructed point clouds consists of regions with varying color intensities. Considering this, we acquired the videos of the beech and robinias on an overcast day. The bark colors were free from drastic changes due to shadows. The contrast in between these two datasets allowed us to evaluate how our approach fared in challenging situations, and helped us evaluate the reliability of our pipeline.

Our approach requires semantic label maps for training. The semantic maps for the tree barks were obtained by hand labelling each of the bark images. We used an online labelling tool, Labelbox: https://labelbox.com. Each of the bark color images were labelled into four categories, and are represented with the following colors in the semantic maps:

1. Bark - yellow

2. Defect/scar - orange

3. Moss - red

4. Lichens - blue

The dataset used to train the GANs consists of maps of size 256 x 256. To generate these images, we take the large surface and color maps and take crop out of them with a stride of 16. This allowed us to artificially augment
the number of samples and let us train complex GANs with a fairly limited
amount of data. Once the images are acquired, the entire set of surface/depth
maps were normalized to the range 0 to 65535. Before feeding all the images
to the network, they are instance normalized to the range [-1,1].

## 4.2 Baseline

4.2.1 Generation

To evaluate the generation quality of our network to other GANs we com-
pare our Depth-Reinforced SPADE with two state-of-the-art GAN architec-
tures: SPADE [27] and pix2pixHD [37]. SPADE and pix2pixHD can generate
photo-realistic images from input semantic maps. To synthesize the surface
and color maps for bark generation from pix2pixHD and SPADE, we use a
two-stage pipeline. First, we provide the smooth surface map as input to
the network and obtain the generated surface map. Next, we train another
network, where we provide the generated surface map and semantic label as
input and obtain the color map. (1) For pix2pixHD, we provide the smooth
surface map through the encoder and obtain the generated surface maps.
Next, we concatenate the generated surface maps and semantic labels and
pass it through the encoder of another pix2pixHD network and obtain the
color maps. (2) For SPADE, since the network has been designed to work
with semantic labels, we provide the input surface maps through the encoder
and semantic labels through the SPADE ResNet blocks to obtain the surface
maps. Next, we provide the generated surface map through another encoder
and decoder of SPADE to obtain the color maps.

Instead of generating the surface and color maps using two independent
networks for pix2pixHD and SPADE, one could use a single encoder and
decoder network to generate surface and color maps with the outputs con-
catened. However, synthesizing both the surface and color maps at the
same time using a single generator (encoder and decoder) and discriminator
resulted in the training becoming very unstable. This is due to the inherent
nature of the GANs where the generator and discriminator compete against
each other and are thus unstable while training. Concatenating two different
types of images (surface maps and color maps) for generation results in the
network not learning and resulted in non-convergence.

4.2.2 Tiling

To evaluate the results of our tiling technique, we compare ourselves to Edge-
Connect [24]. It consists of two networks: the first network adds edges in
the areas to be in-painted so that there is a continuation of the edges from
the surrounding areas. The second network fills the color in the in-painting
areas. Finally, before saving the images, they copy the areas other than the
in-painted ones to the generated images.

# 5 Results

## 5.1 3D Reconstruction and Surface Geometry Extraction

After acquiring the images and performing background suppression on them,
they were used to obtaining 3D reconstructed trees using Pix4DMapper.
Examples of 3D point-clouds of an oak, beech, and robinia trees are shown
in Fig. 9.

<figure><img src="image_11.png" /><figcaption>Figure 9: 3D point clouds of oak, beech, and robinia trees</figcaption></figure>

Once the 3D point clouds of the trees are acquired, the radius and color
information are extracted from them. The surface maps consists of the ge-
ometry of the tree bark in polar coordinates, where, the map contains the
radius values from the center points as a function of the height of the tree
and the angle. This is similar to cutting open a 3D cylinder and flattening
it as a plane. Examples of the maps obtained for an oak, beech, and robinia
trees are shown in Fig. 10.

## 5.2 Architecture

In this section, we compare the results obtained from pix2pixHD, SPADE
and Depth-Reinforced SPADE for the generated surface and color maps.
<figure><img src="image_12.png" /><figcaption>Figure 10: (a) Real surface and color map of an oak tree (b) Surface and color map of a beech tree (c) Surface and color map of a robinia tree. The colorbar shows the radius values of the barks in metres.</figcaption></figure>

<table><caption>Table 1: Quantitative evaluation of the different architectures for surface map generation. The values here are the FID scores, where a lower score indicates better performance.</caption><thead><tr><td>Datasets</td><td>pix2pixHD [37]</td><td>SPADE [27]</td><td>Ours</td></tr></thead><tbody><tr><td>Oak</td><td>65.873</td><td>97.592</td><td><b>50.432</b></td></tr><tr><td>Beech</td><td>55.372</td><td>89.320</td><td><b>52.868</b></td></tr><tr><td>Robinia</td><td>55.043</td><td>75.534</td><td><b>45.096</b></td></tr></tbody></table>

### 5.2.1 Surface-Map Generation

Fig. 11 shows the generation results of the surface generators for pix2pixHD, SPADE and Depth-Reinforced SPADE. The synthetic maps generated by pix2pixHD are blurry. When the SPADE network was provided with the semantic labels, it distorted the generated surface maps. The semantic labels
<figure><img src="image_13.png" /><figcaption>Figure 11: Different methods to generate surface and color tiles for oak, beech and robinia barks. The inputs are the semantic labels and the smooth surface maps. Under pix2pixHD and SPADE: First columns are the generated surface maps. Second columns are the color maps generated when only the semantic labels are provided as input. Third column is when both semantic labels and smooth surface maps are provided as input.</figcaption></figure>
<table><caption>Table 2: Quantitative evaluation of the different architectures. Higher accuracy, mIOU and lower FID indicate better performance. The datasets are highly imbalanced which explains the low mIOU results of the segmentation networks.</caption><thead><tr><th rowspan="2">Datasets</th><th colspan="3">pix2pixHD [37]</th><th colspan="3">SPADE [27]</th><th colspan="3">Ours</th></tr><tr><th>Acc</th><th>mIOU</th><th>FID</th><th>Acc</th><th>mIOU</th><th>FID</th><th>Acc</th><th>mIOU</th><th>FID</th></tr></thead><tbody><tr><td>Oak</td><td>87.88</td><td>22.49</td><td>183.09</td><td>89.06</td><td><b>22.87</b></td><td>85.17</td><td><b>89.17</b></td><td>22.57</td><td><b>49.21</b></td></tr><tr><td>Beech</td><td>50.77</td><td>15.68</td><td>113.02</td><td>51.73</td><td>15.57</td><td>65.273</td><td><b>52.23</b></td><td><b>15.70</b></td><td><b>54.45</b></td></tr><tr><td>Robinia</td><td>N/A</td><td>N/A</td><td>715.72</td><td>N/A</td><td>N/A</td><td>164.19</td><td>N/A</td><td>N/A</td><td><b>57.51</b></td></tr></tbody></table>

<table><caption>Table 3: Number of parameters in the network. Note: For pix2pixHD and SPADE, the values reported here is the total number of parameters for training both the surface and the color generator.</caption><thead><tr><th>Architecture</th><th>Number of parameters</th></tr></thead><tbody><tr><td>pix2pixHD</td><td>366M</td></tr><tr><td>SPADE</td><td>249.8M</td></tr><tr><td>Depth-Reinforced SPADE</td><td><b>207.6M</b></td></tr></tbody></table>

interfered with the input smooth surface in the deep layers of the decoder network. Whereas, the surface maps from Depth-Reinforced SPADE are sharper than the other methods. This is because, the KL-Divergence loss preserves the input information in the latent space. Moreover, there is no interference with the semantic labels, this allows the network to generate finer bark structures compared to the other two methods. This is the main reason for separating the surface and color generation in the Depth-Reinforced SPADE architecture. The synthetic Robinia trees surface-maps are particularly convincing, with realistic patterns that spawn the entire generated tile. Overall the results of the surface maps look good, for each tree species our GAN is capable of generating specific patterns from a smooth surface. This can also be verified from the quantitative metrics shown in Table. 1. We calculate the Fréchet Inception Distance (FID) [11] score between the generated images and the ground truth images, and a lower score indicates better performance.

### 5.2.2 Color-Map Generation

Fig. 11 shows some generation results for the oak, beech and robinia trees. For pix2pixHD and SPADE we have shown two versions of color map generation. In the first version, we provide only the semantic labels as input.
This is to see how the network behaves when provided with identical inputs. In the second version, we provided the surface maps along with the semantic labels as input to verify that surface maps can help the network generate multi-modal color maps. The fourth column and seventh column in Fig. 11 are the color maps generated for pix2pixHD and SPADE respectively, when only the semantic labels are provided as input to the network. Based on the results, pix2pixHD fails to generate correct outputs with a constant label map. This is a well known issue of pix2pixHD that the SPADE architecture addresses. Even though SPADE solves the problem encountered with pix2pixHD, SPADE is deterministic: given a label, the output of the generator will always be the same, which is demonstrated in column seven for the robinia barks. To make these networks generate varying outputs, one must introduce significant variability in the input. Thus, when these networks were provided with the surface maps, they started to generate multi-modal outputs as shown in columns five and eight of Fig. 11. In the case of Depth-Reinforced SPADE, since we generate the surface maps and let it guide the color generator, we always get unique color maps. Furthermore, if one has a close look at the color maps synthesized by our GAN the results look sharper, since we use the generated surface maps to reinforce the color maps which enhances the texture of the generated maps and make them look sharp. More results can be seen in appendix D, E, and F.

To compare the different network quantitatively we use the same metrics as [27]: the pixel accuracy, the mean intersection over union (mIOU), and the Fréchet Inception Distance (FID) [11] score. The semantic segmentation which gives the pixel accuracy and mIoU scores is applied on the generated images and compared to the labels used to generate the image. DeepLab V3+ [3] is used to perform the semantic segmentation. As shown in Table 2, the results provided by our architecture are superior with a lower FID across all datasets, a slightly better accuracy and a comparable mIoU. Please note that on the robinia trees we could not evaluate the mIoU or the accuracy as there is only one label.

### 5.2.3 Controlling the Bark Synthesis
<figure><img src="image_14.png" /><figcaption>Figure 12: Different generation results where we draw unusual shapes on beech barks.</figcaption></figure>

Using our GAN we can bend mother nature to our will, allowing us to painlessly draw familiar objects on the bark as illustrated in Fig. 12. In this figure we chose to draw the Stanford bunny and a heart on the bark of a beech tree. This allows us to demonstrate the benefits of using GANs to generate tree barks: using GANs let us change the trees easily and realistically. The main drawback of using GANs is that to correctly generate some labels they have to appear quite often in the dataset on which they were trained. In our case we could not reproduce those results with the oak or robinia trees as their dataset had too few occurrences of moss or lichen.

Finally, we also compare the architecture between pix2pixHD, SPADE and Depth-Reinforced SPADE in Table. 3. For pix2pixHD and SPADE, we calculate the total number of parameters for generating the surface and the color. Since the surface and color generator of Depth-Reinforced SPADE share the encoder, the number of parameters is reduced. This in turn reduces the computational cost and reduces the time for training.

## 5.3 Tiling the Images

The generated images from the GAN networks were tiled using our modified pix2pix architecture. An example of a continuous image obtained by combining four smaller color images of an oak bark is shown in Fig. 13. Fig. 13(a) consists of an image with four images combined together. This creates a discontinuity in the regions where they are joined. The discontinuities were masked as shown in Fig. 13(b). As it can be seen, only a small
part surrounding the discontinuities are masked, and only these regions are filled by the GAN. Since we use a copying task in the architecture, where the unmasked regions are retained, the majority of the parts in the original images are retained and the quality of the original images are also maintained. Finally, after tiling, the image is shown in Fig. 13(c). The GAN has filled the masked regions with suitable values so as to generate a continuous image.

Fig. 14 shows a situation where the state of the art in-painting method Edge-Connect [24] fails to join the different tiles. It can be seen that, although the network generates the right edge patterns, it generates radius values with a range higher than the actual values. Lastly, as the copy task is performed on the generated images, the difference in the radius values is clearly visible. However, in our proposed architecture, the copy task is performed before feeding the images to the discriminator. Hence, the generator is forced to fill the missing parts with the correct range of radius, and so it generates a continuous map.

## 5.4 Construction of the 3D Meshes

After the tiled radius and the corresponding color maps were obtained, they were used to construct 3D meshes. The 3D models of oak, beech and robinia trees without and with the color are shown in Fig. 15 and Fig. 16. From the 3D mesh without color, the finer structural details of the tree barks can be observed on the real and the GAN generated meshes. The GANs have captured the nuances on the oak, beech and robinia barks, and the synthetic 3D models look as realistic as the original 3D model. Since there is more control over the bark color generated, it results in realistic looking 3D models. More results can be seen in appendix D, E and F. To demonstrate the robustness of the models to the input smooth bark geometry, we have generated the barks with $b_i = 0$ and for three different input radius values. The radius values are 0.20, 0.50 and 0.80, on which the network has successfully generated the fine bark patterns and color as shown in Fig. 16.

To assess the quality of our generated 3D models of the tree barks, a survey was conducted, where we had a randomly mixed up set of oak and beech trees and we asked our participants to differentiate the real barks (3D meshes of the barks reconstructed from the real tree images) from the generated barks. 160 participants contributed in the survey, and were given unlimited time to differentiate the barks. Among our participants 34 were forestry experts. At the beginning, the participants were shown samples of real and generated barks from both the oak and beech barks. Our survey
<figure><img src="image_15.png" /><figcaption>Figure 13: (a) The tiled image with discontinuities (b) Image after the discontinuities are masked (c) Continuous image obtained from the tiling network</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>Figure 14: Example of a situation where Edge-Connect [24] fails</figcaption></figure>
<figure><img src="image_17.png" /><figcaption>Figure 15: Synthetic and real barks</figcaption></figure>

consisted of 12 samples, 6 in oak and 6 in beech. Based on the survey
analysis the precision and recall of the participants to detect real trees was
calculated. The results are given in Table. 4. We recall that: *precision* =
<figure><img src="image_18.png" /><figcaption>Figure 16: Synthetic oak, beech and robinia barks generated with $b_i = 0$ and with three different radius values in the smooth bark. The value of $a$ in Eq. 1 for bark 1 is 0.20, bark 2 is 0.50 and bark 3 is 0.80. Irrespective of the radius values used, the network is able to generate realistic barks and colors.</figcaption></figure>

$TP/(TP + FP)$, and $recall = TP/(TP + FN)$.

From Table. 4 we can see that the precision of the participants is around 55%, which shows that a large portion of them mistook our synthetic trees for real trees. Additionally, the recall which evaluates the ability of the classifier to find all the positive samples, is close to 0.6, which also indicates that the participants have trouble differentiating the real barks from the synthetics one. All in all this results prove that our GAN generated trunks are realistic enough to fool around 40% of people, expert and layman alike.

# 6 Limitations and Future Work

As of today, our work faces some limitations. The first limitation comes from the very nature of our approach. Because we are using a data-driven method-
<table><caption>Table 4: Precision and recall for the detection of real 3D models of oak and beech trees. Higher is better.</caption><thead><tr><th rowspan="2"></th><th colspan="2">laymans</th><th colspan="2">experts</th></tr><tr><th>Precision</th><th>Recall</th><th>Precision</th><th>Recall</th></tr></thead><tbody><tr><th>Oak</th><td>0.54</td><td>0.60</td><td>0.51</td><td>0.56</td></tr><tr><th>Beech</th><td>0.60</td><td>0.63</td><td>0.55</td><td>0.59</td></tr><tr><th>Overall</th><td>0.57</td><td>0.615</td><td>0.53</td><td>0.585</td></tr></tbody></table>

ology, we need a significant amount of data to train our neural network on. This is particularly true with the robinia and oak trees, which had very little moss and lichen on them, making the network incapable of synthetizing the Stanford bunny. The second limitation in this work is related to the tiling techniques that could be improved on: the junction between the tiles are not perfectly smooth over the height. New methodologies, such as SRFlow [22] or GLOW [17] could be investigated. Yet, this tiling problem could also be partially solved by acquiring the trees under controlled illumination. This would give uniform colors and illumination across the dataset making the GANs generation much more homogeneous on the color side. Furthermore, higher quality recording hardware would definitely provide higher resolution color and surface maps and hence sharper GAN results. Our method works well on trees with cylindrical structure, however it can't handle the case of non-cylindrical trees such as those with buttress roots. In an effort to increase the realism of our method, future work will focus on applying different style to generate barks of different age: allowing to generate a broader diversity of tree barks. Finally we will collect a larger dataset with more trees and more species.

# 7 Conclusion

In this work, we propose a novel pipeline to generate realistic tree barks from a set of images. Deviating from the conventional methods, we use a data-driven generation method relying on deep neural networks to generate the tree barks. Once a deep neural network is trained, there is not much of manual intervention required to generate new barks. This is in contrast to traditional methods which requires manual tuning of parameters to generate barks. Additionally, our method gives extensive control to the users to generate additional features such as moss, scars, etc. We first created an efficient method using self-supervised learning to suppress the background
portions of the image sequences that yields better quality tree models, and
at the same time, reconstructs the trees faster. We also proposed a GAN
architecture called Depth-Reinforced-SPADE. This GAN takes as an input
both a label map and a smooth surface. From these two inputs, it gener-
ates a detailed bark surface and the color map simultaneously. Instead of
using two separate GANs to generate the surface/depth and color, Depth-
Reinforced-SPADE can generate both simultaneously, thus saving time and
computational resources. We also presented a method to tile smaller images
obtained from our GANs to produce a continuous map of the surface and
color of the tree barks. Our method was successfully tested on trees with
smooth and ridged barks, which yielded high-quality tree barks, implying
that our method could be easily extended to other tree bark types to obtain
genuine looking barks. Finally, we demonstrated that our GANs enable users
to synthesize bark tiles with user-defined inputs like a bunny or a heart. The
only limitation being the diversity of the datatset the GAN was trained on.

8 Acknowledgement

This research was made possible with the support from the French National
Research Agency, in the framework of the project WoodSeer, ANR-19-CE10-
011.
References

[1] M. Arjovsky, S. Chintala, and L. Bottou, "Wasserstein gan," *arXiv preprint arXiv:1701.07875*, 2017.

[2] J. Bloomenthal, "Modeling the mighty maple," *ACM SIGGRAPH Computer Graphics*, vol. 19, no. 3, pp. 305-311, 1985.

[3] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, "Encoder-decoder with atrous separable convolution for semantic image segmentation," in *Proceedings of the European conference on computer vision (ECCV)*, 2018, pp. 801-818.

[4] X. Chen, B. Neubert, Y.-Q. Xu, O. Deussen, and S. B. Kang, "Sketch-based tree modeling using markov random field," in *ACM SIGGRAPH Asia 2008 papers*, 2008, pp. 1-9.

[5] Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo, "Star-gan: Unified generative adversarial networks for multi-domain image-to-image translation," in *Proceedings of the IEEE conference on computer vision and pattern recognition*, 2018, pp. 8789-8797.

[6] B. Desbenoit, E. Galin, and S. Akkouche, "Modeling cracks and fractures," *The Visual Computer*, vol. 21, no. 8-10, pp. 717-726, 2005.

[7] A. Frühstück, I. Alhashim, and P. Wonka, "Tilegan: synthesis of large-scale non-homogeneous textures," *ACM Transactions on Graphics (TOG)*, vol. 38, no. 4, pp. 1-11, 2019.

[8] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial nets," in *Advances in neural information processing systems*, 2014, pp. 2672-2680.

[9] J. Guo, S. Xu, D.-M. Yan, Z. Cheng, M. Jaeger, and X. Zhang, "Realistic procedural plant modeling from multiple view images," *IEEE transactions on visualization and computer graphics*, 2018.

[10] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in *Proceedings of the IEEE conference on computer vision and pattern recognition*, 2016, pp. 770-778.
[11] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” in *Advances in neural information processing systems*, 2017, pp. 6626–6637.

[12] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in *Proceedings of the IEEE conference on computer vision and pattern recognition*, 2017, pp. 1125–1134.

[13] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in *European conference on computer vision*. Springer, 2016, pp. 694–711.

[14] T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of gans for improved quality, stability, and variation,” *arXiv preprint arXiv:1710.10196*, 2017.

[15] T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, 2019, pp. 4401–4410.

[16] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, “Analyzing and improving the image quality of stylegan,” *arXiv preprint arXiv:1912.04958*, 2019.

[17] D. P. Kingma and P. Dhariwal, “Glow: Generative flow with invertible 1x1 convolutions,” in *Advances in neural information processing systems*, 2018, pp. 10215–10224.

[18] J. Kratt, M. Spicker, A. Guayaquil, M. Fiser, S. Pirk, O. Deussen, J. C. Hart, and B. Benes, “Woodification: User-controlled cambial growth modeling,” in *Computer Graphics Forum*, vol. 34, no. 2. Wiley Online Library, 2015, pp. 361–372.

[19] P. Laitoch, “Procedural modeling of tree bark,” Bachelor thesis, Charles University, Prague, 2018.

[20] C. Li, O. Deussen, Y.-Z. Song, P. Willis, and P. Hall, “Modeling and generating moving trees from video,” *ACM Transactions on Graphics (TOG)*, vol. 30, no. 6, pp. 1–12, 2011.
[21] Z. Liu, K. Wu, J. Guo, Y. Wang, O. Deussen, and Z. Cheng, "Single image tree reconstruction via adversarial network," *Graphical Models*, vol. 117, p. 101115, 2021.

[22] A. Lugmayr, M. Danelljan, L. Van Gool, and R. Timofte, "Srflow: Learning the super-resolution space with normalizing flow," *arXiv preprint arXiv:2006.14200*, 2020.

[23] X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. Paul Smolley, "Least squares generative adversarial networks," in *Proceedings of the IEEE International Conference on Computer Vision*, 2017, pp. 2794–2802.

[24] K. Nazeri, E. Ng, T. Joseph, F. Z. Qureshi, and M. Ebrahimi, "Edge-connect: Generative image inpainting with adversarial edge learning," *arXiv preprint arXiv:1901.00212*, 2019.

[25] B. Neubert, T. Franken, and O. Deussen, "Approximate image-based tree-modeling using particle flows," in *ACM SIGGRAPH 2007 papers*, 2007, pp. 88–es.

[26] M. Okabe, S. Owada, and T. Igarashi, "Interactive design of botanical trees using freehand sketches and example-based editing," in *ACM SIGGRAPH 2006 Courses*, 2006, pp. 18–es.

[27] T. Park, M.-Y. Liu, T.-C. Wang, and J.-Y. Zhu, "Semantic image synthesis with spatially-adaptive normalization," in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, 2019, pp. 2337–2346.

[28] S. Pix4D, "Pix4d mapping 4.1 user manual," *Pix4D SA: Lausanne, Switzerland*, 2017.

[29] L. Quan, P. Tan, G. Zeng, L. Yuan, J. Wang, and S. B. Kang, "Image-based plant modeling," in *ACM SIGGRAPH 2006 Papers*, 2006, pp. 599–604.

[30] A. Radford, L. Metz, and S. Chintala, "Unsupervised representation learning with deep convolutional generative adversarial networks," *arXiv preprint arXiv:1511.06434*, 2015.

[31] T. Sattler, W. Maddern, C. Toft, A. Torii, L. Hammarstrand, E. Stenborg, D. Safari, M. Okutomi, M. Pollefeys, J. Sivic, et al., "Benchmarking 6dof outdoor visual localization in changing conditions," in
*Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, 2018, pp. 8601–8610.

[32] J. L. Schonberger and J.-M. Frahm, “Structure-from-motion revisited,” in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, 2016, pp. 4104–4113.

[33] J. L. Schönberger, E. Zheng, J.-M. Frahm, and M. Pollefeys, “Pixelwise view selection for unstructured multi-view stereo,” in *European Conference on Computer Vision*. Springer, 2016, pp. 501–518.

[34] W. J. Schroeder, B. Lorensen, and K. Martin, *The visualization toolkit: an object-oriented approach to 3D graphics*. Kitware, 2004.

[35] P. Tan, T. Fang, J. Xiao, P. Zhao, and L. Quan, “Single image tree modeling,” *ACM Transactions on Graphics (TOG)*, vol. 27, no. 5, pp. 1–7, 2008.

[36] B. Tao, Z. Changshui, and S. Wei, “A multi-agent based approach to modelling and rendering of 3d tree bark textures,” in *European Conference on Artificial Life*. Springer, 2003, pp. 572–579.

[37] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catan-zaro, “High-resolution image synthesis and semantic manipulation with conditional gans,” in *Proceedings of the IEEE conference on computer vision and pattern recognition*, 2018, pp. 8798–8807.

[38] X. Wang, L. Wang, L. Liu, S. Hu, and B. Guo, “Interactive modeling of tree bark,” in *11th Pacific Conference on Computer Graphics and Applications, 2003. Proceedings*. IEEE, 2003, pp. 83–90.

[39] K. Xie, F. Yan, A. Sharf, O. Deussen, H. Huang, and B. Chen, “Tree modeling with real tree-parts examples,” *IEEE transactions on visualization and computer graphics*, vol. 22, no. 12, pp. 2608–2618, 2015.

[40] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena, “Self-attention generative adversarial networks,” *arXiv preprint arXiv:1805.08318*, 2018.

[41] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing network,” in *Proceedings of the IEEE conference on computer vision and pattern recognition*, 2017, pp. 2881–2890.

[42] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networkss,” in *Computer Vision (ICCV), 2017 IEEE International Conference on*, 2017.
# A Learning Objective

We use the same loss functions as pix2pixHD [37], namely the GAN loss, feature-matching loss, perceptual loss and the KL-divergence loss.

## A.1 GAN Loss

The objective of the cGAN is to optimize the minmax loss function of the GAN. The GAN loss for the surface generator is given by

$$L_{GAN_s} = E_{(s)}[\log(D_s(x, s))] + E_{(x)}[\log(1 - D_s(x, G_s(x)))] \quad (3)$$

where s is the real surface map. D_s is the prediction from the surface discriminator. x is the input smooth surface map, G_s(x) is the surface map generated from the surface generator. Similarly, the GAN loss of the color generator is given by:

$$L_{GAN_c} = E_{(l,c)}[\log(D_c(l, c))] + E_s[\log(1 - D_c(l, G_c(l, x)))] \quad (4)$$

where l is the input semantic label, c is the real color image, G_c(l, c) is the color map generated from the color generator. D_c(l, c) is the prediction from the discriminator when provided with the semantic labels and the real color image. D_c(l, G_c(l, x)) is the prediction from the discriminator when provided with the semantic labels and the generated color image. The total GAN loss is the sum of the surface and color GAN loss

$$L_{GAN} = L_{GAN_s} + L_{GAN_c} \quad (5)$$

## A.2 Feature Matching Loss

The feature-matching loss is calculated between the real and synthesized images. We extract the features of the real and synthetic images from the multi-layer discriminator and calculate the L1 distance between the extracted features. The feature-matching loss for the surface generator is

$$L_{FM_s} = E_{(s)} \sum_{i=1}^{T} \frac{1}{N_i} [||D_s^{(i)}(x, s) - D_s^{(i)}(x, G_s(x))||_1] \quad (6)$$

where T is the number of layers in the discriminator and N_i is the number of elements in each layer. Similarly, the feature-matching loss for the color generator is

$$L_{FM_c} = E_{(l,c)} \sum_{i=1}^{T} \frac{1}{N_i} [||D_c^{(i)}(l, c) - D_c^{(i)}(l, G_c(l, x))||_1] \quad (7)$$
The total feature-matching loss is

$$L_{FM} = L_{FM_s} + L_{FM_c} \qquad (8)$$

### A.3 Perceptual Loss

We use a pre-trained VGG network to get pair-wise features of the real and synthetic images at different layers of the network and calculate the L1 distance between the maps. This is given by $L_p$.

### A.4 KL-Divergence Loss

The KL-Divergence Loss is calculated for the encoder of the network. This is given by

$$L_{KD} = D_{KL}[Q(z|x) || P(z|x)] \qquad (9)$$

where x is the input smooth surface map, z is the latent space obtained from the encoder. P is the original distribution and Q is the simpler estimation of it.

The total loss is the sum of all the loss functions

$$L_{total} = L_{GAN} + L_{FM} + L_p + L_{KD} \qquad (10)$$

## B Training Details

### B.1 pix2pix

We used the original parameters proposed in [12] for our experiments the code can be found here https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix. git. The learning rates of the generator and discriminator were 0.0002. We used the Adam solver with $\beta_1 = 0$ and $\beta_2 = 0.999$, with a batch size of 8. The experiments were conducted on an 8 GB Nvidia GeForce GTX 1080.

### B.2 pix2pixHD

We used the original parameters of [37] for training the network. The learning rate was 0.0002, and we used an Adam optimizer with $\beta_1 = 0.5$ and $\beta_2 = 0.999$. The code for network can be found here: https://github.com/NVIDIA/pix2pixHD.git. The experiments were conducted on a 11 GB Nvidia GeForce GTX 1080Ti, with a batch size of 4.
## B.3 SPADE and Depth-Reinforced-SPADE

We used the original parameters proposed in [27] for our experiments, the code used to train the SPADE GANs can be found here: https://github.com/NVlabs/SPADE.git. The learning rates of the generator and discriminator were 0.0001 and 0.0004 respectively. We used the Adam solver with $\beta_1 = 0$ and $\beta_2 = 0.999$. The experiments on the tree bark dataset were conducted on a 11 GB Nvidia GeForce GTX 1080Ti, with a batch size of 4.

## B.4 Edge-Connect

The code for training the network can be found in: https://github.com/knazeri/edge-connect.git. The learning rate of the generator is 0.0001. We used the Adam solver with $\beta_1 = 0$ and $\beta_2 = 0.999$. The experiments on the tree bark dataset were conducted on a 11 GB Nvidia GeForce GTX 1080Ti, with a batch size of 4.

# C Effect of limiting depth passed to color generator.

To study the effect of convolution layers that limit the passage of depth information between the depth and color generator in our decoder architecture, we conduct an experiment on the Oak bark dataset. When all the layers from the depth generator are transferred to the color generator unrestricted, the semantic information gets suppressed by the depth, as seen in Figure 17. Thus, the network generates the moss even on the regions specified for the bark. Whereas when the depth information is transferred selectively using the convolution layers, the network is able to generate the moss and bark at their respective locations without overlapping. Thus, the convolution layers ensure that there is a balance between the semantic and depth information, which ensures good quality generation.
<figure><img src="image_19.png" /><figcaption>Figure 17: Effect of limiting the number of channels passed between the depth generator and the color generator. Left: Input label, indicating moss and bark. Center: When no convolution layer is used, the semantic information is suppressed by the depth. Right: When a convolution layer is used, both the depth and the semantic information are used to generate the bark.</figcaption></figure>
D Method results: Oak

<figure><img src="image_20.png" /><figcaption>Figure 18: Point clouds</figcaption></figure>
<figure><img src="image_21.png" /><figcaption>Figure 19: Color Maps and Surface Maps</figcaption></figure>
<figure><img src="image_22.png" /><figcaption>Figure 20: Generated Tiles</figcaption></figure>

<figure><img src="image_23.png" /><figcaption>Figure 21: Synthetic Trunks</figcaption></figure>
E Method results: Beech

<figure><img src="image_24.png" /><figcaption>Figure 22: Point clouds</figcaption></figure>
<figure><img src="image_25.png" /><figcaption>Figure 23: Color Maps and Surface Maps</figcaption></figure>
<figure><img src="image_26.png" /><figcaption>Figure 24: Generated Tiles</figcaption></figure>

<figure><img src="image_27.png" /><figcaption>Figure 25: Synthetic Trunks</figcaption></figure>
F Method results: Robinia

<figure><img src="image_28.png" /><figcaption>Figure 26: Point clouds</figcaption></figure>
<figure><img src="image_29.png" /><figcaption>Figure 27: Color Maps and Surface Maps</figcaption></figure>
<figure><img src="image_30.png" /><figcaption>Figure 28: Generated Tiles</figcaption></figure>

<figure><img src="image_31.png" /><figcaption>Figure 29: Synthetic Trunks</figcaption></figure>