Title: LEARNING TO GENERATE 3D TRAINING DATA THROUGH HYBRID GRADIENT

Abstract: Synthetic images rendered by graphics engines are a promising source for training deep networks. However, it is challenging to ensure that they can help train a network to perform well on real images, because a graphics-based generation pipeline requires numerous design decisions such as the selection of 3D shapes and the placement of the camera. In this work, we propose a new method that optimizes the generation of 3D training data based on what we call "hybrid gradient". We parametrize the design decisions as a real vector, and combine the approximate gradient and the analytical gradient to obtain the hybrid gradient of the network performance with respect to this vector. We evaluate our approach on the task of estimating surface normal and depth from a single image. Experiments on standard benchmarks show that our approach can outperform the prior state of the art on optimizing the generation of 3D training data, particularly in terms of computational efficiency.

1. INTRODUCTION:

Synthetic images rendered by graphics engines have emerged as a promising source of training data for deep networks, especially for vision and robotics tasks that involve perceiving 3D structures from RGB pixels (Butler et al., 2012; Yeh et al., 2012; Varol et al., 2017; Ros et al., 2016; McCormac et al., 2017; Xia et al., 2018; Chang et al., 2017; Kolve et al., 2017; Song et al., 2017; Richter et al., 2016; 2017; Zhang et al., 2017; Li & Snavely, 2018) . A major appeal of generating training images from computer graphics is that they have a virtually unlimited supply and come with high-quality 3D ground truth for free.

Despite its great promise, however, using synthetic training images from graphics poses its own challenges. One of them is ensuring that the synthetic training images are useful for real world tasks, in the sense that they help train a network to perform well on real images. Ensuring this is challenging because a graphics-based generation pipeline requires numerous design decisions including the selection of 3D shapes, the composition of scene layout, the application of texture, the configuration of lighting, and the placement of the camera. These design decisions can profoundly impact the usefulness of the generated training data, but have largely been made manually by researchers in prior work, potentially leading to suboptimal results.

In this paper we address the problem of automatically optimizing a generation pipeline of synthetic 3D training data, with the explicit objective of improving the generalization performance of a trained deep network on real images.

One idea is black-box optimization: we try a particular configuration of the pipeline, use the pipeline to generate training images, train a deep network on these images, and evaluate the network on a validation set of real images. We can treat the performance of the trained network as a black-box function of the configuration of the generation pipeline, and apply black-box optimization techniques. In fact, recent work by Yang & Deng (2018) has explored this exact direction. They use genetic algorithms to optimize the 3D shapes used in the generation pipeline. In particular, they start with a collection of simple primitive shapes such as cubes and spheres, and evolve them through mutation and combination into complex shapes, whose fitness is determined by the generalization performance of a trained network. They show that the 3D shapes evolved from scratch can provide more useful training data than manually created 3D CAD models. 

Hybrid Gradient Approximate Gradient Analytical Gradient:

Real images

Figure 1: Our "hybrid gradient" method. We parametrize the design decisions as a real vector β and optimize the function of performance L with respect to β. From β to the generated training images and ground truth, we compute the approximate gradient by averaging finite difference approximations. From training samples X to L, we compute analytical gradients through backpropagation with unrolled training steps.

The advantage of black-box optimization is that it makes no assumption about the function being optimized as long as it can be evaluated. As a result, it can be applied to any existing function including advanced photorealistic renderers. On the other hand, black-box optimization is computationally expensive-knowing nothing else about the function, it needs many trials to find a good update to the current solution. In contrast, gradient-based optimization can be much more efficient by assuming the availability of analytical gradients, which can be efficiently computed and directly correspond to good updates to the current solution, but the downside is that analytical gradients are often unavailable, especially for many advanced photorealistic renderers.

In this work, we propose a new method that optimizes the generation of 3D training data based on what we call "hybrid gradient". The basic idea is to make use of analytical gradients where they are available, and combine them with black-box optimization for the rest of the function. Our hypothesis is that hybrid gradient will lead to more efficient optimization than black-box methods because it makes use of the partially available analytical gradient.

Concretely, if we parametrize the design decisions as a real vector β, the function mapping β to network performance L can be decomposed into two parts: (1) from design parameters β to the generated training images X, and (2) from the training images X to the network performance L. The first part often does not have analytical gradients, due to the use of advanced photorealistic renderers. We instead compute an approximate gradient by averaging finite difference approximations along random directions (Mania et al., 2018) . For the second part, we compute analytical gradients through backpropagation-with SGD training unrolled, the performance of the network is a differentiable function of the training images. Then we combine the approximate gradient and the analytical gradient to obtain the hybrid gradient of the network performance L with respect to parameters β, as illustrated in Fig. 1 .

A key ingredient of our approach is representing design decisions as real vectors of fixed dimensions, including the selection and composition of shapes. Yang & Deng (2018) represent 3D shapes as a finite set of graphs, one for each shape. This representation is suitable for a genetic algorithm but is incompatible with our method. Instead, we propose to represent 3D shapes as random samples generated by a Probabilistic Context-Free Grammar (PCFG) (Harrison, 1978) . To sample a 3D shape, we start with an initial shape, and repeatedly sample a production rule in the grammar to modify it. The (conditional) probabilities of applying the production rules are parametrized as a real vector of a fixed dimension.

Our approach is novel in multiple aspects. First, to the best our knowledge, we are the first to propose the idea of hybrid gradient, i.e. combining approximate gradients and analytical gradients, especially in the context of optimizing the generation of 3D training data. Second, the integration of PCFG-based shape generation and hybrid gradient is also novel.

We evaluate our approach on the task of estimating surface normal and depth from a single image. Experiments on standard benchmarks show that our approach can outperform the prior state of the art on optimizing the generation of 3D training data, particularly in terms of computational efficiency.

Generating 3D training data Synthetic images generated by computer graphics have been extensively used for training deep networks for numerous tasks, including single image 3D reconstruction (Song et al., 2015; Hua et al., 2016; McCormac et al., 2017; Janoch et al., 2011; Yang & Deng, 2018; Chang et al., 2015) , optical flow estimation (Mayer et al., 2018; Butler et al., 2012; Gaidon et al., 2016) , human pose estimation (Varol et al., 2017; Chen et al., 2016) , action recognition (Roberto de Souza et al., 2017) , natural language modeling (Johnson et al., 2017) , and many others (Qiu et al., 2017; Martinez-Gonzalez et al., 2018; Xia et al., 2018; Tobin et al., 2017; Richter et al., 2017; 2016; Wu et al., 2018) . The success of these works has demonstrated the effectiveness of synthetic images.

To ensure the relevance of the generated the training data to real world tasks, a large amount of manual effort has been necessary, particularly in acquiring 3D assets such as shapes and scenes (Chang et al., 2015; Janoch et al., 2011; Choi et al., 2016; Xiang et al., 2016; Hua et al., 2016; McCormac et al., 2017; Song et al., 2017) . To reduce manual labor, some heuristics have been proposed to automatically generate 3D configurations. For example, Zhang et al. (2017) design an approach to use entropy of object masks and color distribution of the rendered image to select sampled camera poses. McCormac et al. (2017) simulate gravity for physically plausible object configurations inside a room.

Prior work has also performed explicit optimization of 3D configurations. For example, Yeh et al. (2012) synthesizes layouts with the target of satisfying constraints such as non-overlapping and occupation. Jiang et al. (2018) learns a probabilistic grammar model for indoor scene generation, with parameters learned using maximum likelihood estimation on the existing 3D configurations in SUNCG (Song et al., 2017) . Similarly, Veeravasarapu et al. (2017) tunes the parameters for stochastic scene generation using generative adversarial networks, with the goal of making synthetic images indistinguishable from real images. Qi et al. (2018) synthesize 3D room layouts based on humancentric relations among furniture, to achieve visual realism, functionality and naturalness of the scenes. However, these optimization objectives are different from ours, which is the generalization performance of a trained network on real images.

The closest prior work to ours is that of Yang & Deng (2018) , who use a genetic algorithm to optimize the 3D shapes used for rendering synthetic training images. Their optimization objective is the same as ours, but their optimization method is different in that they do not use any gradient information. Similarly, Meta-Sim (Kar et al., 2019 ) also tries to optimize 3D parameters with REINFORCE towards better task generalization performance. However, it does not backpropagate analytical gradients from the meta-objective, so their algorithm is still black-box estimation by mulitple trials.

Unrolling and backpropogating through network training One component of our approach is unrolling and backpropagating through the training iterations of a deep network. This is a technique that has often been used by existing work in other contexts, including hyperparameter optimization (Maclaurin et al., 2015) and meta-learning (Andrychowicz et al., 2016; Ha et al., 2017; Munkhdalai & Yu, 2017; Li & Malik, 2017; Finn et al., 2018) . Our work is different in that we apply this technique in a novel context: it is used to optimize the generation of 3D training data and it is integrated with approximate gradients to form hybrid gradients.

Hyperparameter optimization Our method is connected to hyperparameter optimization in the sense that we can treat the design decisions of the 3D generation pipeline as hyperparameters of the training procedure.

Hyperparameter optimization is typically approached as black-box optimization (Bergstra & Bengio, 2012; Bergstra et al., 2011; Lacoste et al., 2914; Brochu et al., 2010) . Since black-box optimization does not assume knowledge about the function being optimized, it requires repeated evaluation of the function, which is expensive in this case because it contains the process of training and evaluating a deep network. In contrast, we combine analytical gradients from backpropagation and approximate gradient from generalized finite difference for more efficient optimization.

3. PROBLEM SETUP:

Suppose we have a probabilistic generative pipeline. We use a deterministic function, f (β, r) to represent the sampling operation. This function f takes the real vector β and the random seed r as input. An image and its 3D ground truth are computed by evaluating the function f (β, r). By choosing n different random seeds r, we obtain a dataset of size n for training:

EQUATION

Then, a deep neural network with initialized weights w 0 is trained on the training data X, with the function train(w 0 , X) representing the optimization process and generating the weights of the trained network.

The network is then evaluated on real data X with a validation loss l eval to obtain a generalization performance L:

EQUATION

Combining the above two functions, L is a function of β, and the task is to optimize this value L with respect to the parameters β.

As we mentioned in the previous section, black-box algorithms typically need repeated evaluation of this function, which is expensive.

4. APPROACH:



4.1. GENERATIVE MODELING OF SYNTHETIC TRAINING DATA:

We decompose the function f (β, r) into two parts: 3D composition and rendering.

3D composition Context-free grammars have been used in scene generation (Jiang et al., 2018; Qi et al., 2018) and in parsing of Constructive Solid Geometry (CSG) shapes (Sharma et al., 2018) .

Here, we design a probabilistic context-free grammar (PCFG) (Foley et al., 1990) to control the random generation of unlimited shapes.

In a PCFG, a tree is randomly sampled given a set of probabilities. Starting from a root node, the leaf nodes of the tree keeps expanding according to a set of rules. The process is stopped until all leaf nodes cannot expand. Since multiple rules may apply, the parameters in a PCFG define the probability distribution of applying different rules.

In our PCFG, a shape can be constructed by composing two other shapes through union and difference, and this construction can be recursively applied until all leaf nodes are a predefined set of concrete primitive shapes (terminals). The parameters can be the probability of either expanding the node or replacing it with a terminal.

Given our PCFG model with the probability parameters β S , a 3D shape S can be composed:

EQUATION

Rendering training images we use a graphics renderer R to render the composed shape S. The rendering configurations P (e.g. camera poses), are also sampled from a distribution controlled by a set of parameters β R :

EQUATION

Now that we have Eq. 3 and 4, The full function for training data generation can be represented as follows:

EQUATION

where β = (β R , β S ) and r = (r R , r S ).

By choosing different random seeds r, we obtain a set of training images and their 3D ground truth X. The analytical gradient is computed by backpropagating through unrolled training steps (colored in orange). The numerical gradients are computed using finite difference approximation by sampling in a neighborhood of β t (colored in cyan). Then β t is updated using hybrid gradient, and the trained network is used as initialization for the next timestamp t + 1.

Evaluation:



4.2. HYBRID GRADIENT:

After a deep network is trained on synthetic training data X, it is evaluated on a set of validation images X to obtain the generalization loss L.

Recall that to compute the hybrid gradient ∂L ∂β to optimize β, we multiply two gradients: the gradient of network training ∂Lt ∂X and the gradient of image generation ∂X ∂β , as is shown in Fig. 2 .

Analytical gradient from backpropagation:

We assume the network is trained on a a set of previously generated training images X (1) , X (2) , • • • , X (n) . Without loss of generality, we assume mini-batch stochastic gradient descent (SGD) with a batch size of 1 is used for weight update. Let function g denote the SGD step and let l train denote the training loss:

EQUATION

Note that the SGD step g is differentiable with respect to the network weights w (k) as well as the training batch X (k) , if our training loss l train is twice (sub-)differentiable. This requirement is satisfied in most practical cases. To simplify the equation, we assume the training loss l train and the learning rate η do not change during one update step of β, so the variables can be safely discarded in the equation.

Therefore, the gradient of the generalization loss L for each sample X (k) can be computed through backpropagation:

∂L ∂X (k) = ∂L ∂w (k+1) • ∂w (k+1) ∂X (k) = ∂L ∂w (k+1) • g 2 (w (k) , X (k) ) (7) ∂L ∂w (k) = ∂L ∂w (k+1) • ∂w (k+1) ∂w (k) = ∂L ∂w (k+1) • g 1 (w (k) , X (k) ) (8)

with the initial value ∂L ∂w (n+1) computed from the validation loss l eval : ∂L ∂w (n+1) = l eval (w (k+1) , X).

(9)

Aproximate gradient from finite difference For the formulation in Eq. 5, the graphics renderer can be a general black box and non-differentiable. We can approximate the gradient of each rendered image with ground truth X (1) , X (2) , • • • with respect to the generation parameters β, with Basic Random Search, a generalized finite difference method described in (Mania et al., 2018) . First, we sample a set of noise from an uncorrelated multivariate Gaussian distribution (Mania et al., 2018) :

EQUATION

Next, we approximate the Jacobian for each sample (⊗ denotes cross product) (Mania et al., 2018) :

EQUATION

Incremental training Following Yang & Deng (2018), we incrementally update parameters β and network weights w. At timestamp t, we update β t with the hybrid gradient; for network weights, we simply use the latest trained network for initialization in timestamp t + 1:

β t+1 = β t -γ ∂L t ∂β t = β t -γ n i=1 ∂L t ∂X (i) t • ∂X (i) t ∂β t w (1) t+1 = w (n+1) t (12)

5. EXPERIMENTS:

We evaluate our algorithm on three different datasets, and two standard prediction tasks for singleimage 3D. The input is an RGB image and the output is pixel-wise surface normal/depth map.

Specifically, We experiment on the task of surface normal estimation on two real datasets: MIT-Berkeley Intrinsic Images Dataset (MBII) (Barron & Malik, 2015) , which focuses on images of single objects and NYU Depth (Silberman et al., 2012) , which focuses on indoor scenes. For the third dataset, we experiment on the task of depth estimation on the renderings of the scanned human faces in the Basel Face Model dataset (Paysan et al., 2009) .

In all of our experiments, our networks are trained on synthetic images only, and the generalization loss is computed on the validation split of the datasets mentioned above. For MBII, we use pure synthetic shapes (Yang & Deng, 2018) to render training images. We first compare our method with ablation baselines, then show that our algorithm is better than the previous state of the art on MBII.

For NYU Depth, we base our generative model on SUNCG (Song et al., 2017) and augment the original 3D configurations in Zhang et al. (2017) . For Basel Face Model, we sample synthetic faces from a morphable model and evaluate on renderings of scanned faces.

5.1. MIT-BERKELEY INTRINSIC IMAGES:

Following the work of Yang & Deng (2018) , we recover the surface normals of an object from a single image.

Synthetic shape generation:

In Yang & Deng (2018) , a population of primitive shapes such as cylinders, spheres and cubes are evolved and rendered to train deep networks. The evolution operators are defined as transformations of individual shapes, as well as boolean operations of shapes in Constructive Solid Geometry (CSG) (Foley et al., 1990) . In our algorithm, we also use the CSG grammar for our PCFG. We compose our shape in mesh representations, slightly different from the implicit functions in Yang & Deng (2018) . Therefore, we re-implemented their algorithm with mesh representations for fair comparison. For network training and evaluation, we follow Yang & Deng (2018) and train the Stacked Hourglass Network (Newell et al., 2016) on the images, and use the standard split of the MBII dataset for the optimization of β and testing.

We report the performance of surface normal directions with the metrics commonly used in previous works, including mean angle error (MAE), median angle error, mean squared error (MSE), and the proportion of pixels that normals fall in an error range (≤ N • ). See Appendix for detailed definition.

Ablation study We first sample 10 random values of β in advance, then for each β we train a network, with the exact same training and evaluation configurations as in our hybrid gradient. We then report the best, median and worst performance of those 10 networks, and label the corresponding β as β best , β median and β worst . In hybrid gradient, we initialize β 0 from these three values and report the performance on test images also in Table 1 .

From the table we can observe that training with a fixed β can hardly match the performance of our method, even with multiple trials. Instead, our hybrid gradient approach can optimize β to a reasonable performance regardless of different initialization. This simple diagnostic experiment demonstrates that our algorithm is working properly.

Comparison with previous work:

In this experiment, we compare with black-box algorithms including Basic Random Search (Mania et al., 2018) and Shape Evolution (Yang & Deng, 2018) . Because we use mesh implementation instead of implicit computation graph in Yang & Deng (2018) for CSG, we re-implemented Shape Evolution with the same setting for fair comparison. We follow Yang & Deng (2018) for the initialization of β, train the networks and update β for the same number of steps. We then report the test performance of the network which has the best validation performance. The results are shown in Table 2 .

We also run the experiments on the same set of CPUs and GPUs, and plot the test mean angle error with respect to the CPU time, GPU time and total computation time (Fig. 4 ). We see that our algorithm is more efficient than the above baselines. Shapes sampled from our optimized PCFG are shown in Fig. 3 .

Figure 3 : Sampled shapes from our probabilistic context-free grammar, with parameters optimized using hybrid gradient.

Table 2 : Our approach compared to previous work, on the test set of MIT-Berkeley images (Barron & Malik, 2015) . The results show that our approach is better than the state of the art as reported in Yang & Deng (2018) . (Mania et al., 2018) Evolution (Yang & Deng, 2018) Figure 4 : Mean angle error on the test images vs. computation time, compared to two black-box optimization baselines.

5.2. NYU DEPTH:

Scene perturbation We design our scene generation grammar as an augmentation of collected SUNCG scenes (Song et al., 2015) with the cameras from Zhang et al. (2017) :

S => E,P; E => T_shapes * R_shapes * E0; P => T_camera * R_camera * P0; T_shapes => translate(rand_x, rand_y, rand_z); R_shapes => rotate_euler(rand_yaw, rand_pitch, rand_roll);

For each 3D scene S, we perturb the positions and poses of the original cameras (P0) and shapes (E0) using random translations and rotations. The position perturbations follow a mixture of uncorrelated Gaussians, and the perturbations for pose angles (yaw, pitch & roll) follow a mixture of von Mises, i.e. wrapped Gaussians. The vector β consists of the parameters of the above distributions.

Training Setup Our networks are trained on synthetic images only, and evaluated on NYU Depth V2 (Silberman et al., 2012) with the same setup as in Zhang et al. (2017) . For real images in our optimization pipeline, we sample a subset of images from the standard validation images in NYU Depth V2. We initialize our network using the original model in Zhang et al. (2017) and initialize β 0 using a small value. To compare with random β, we construct a dataset of 40k images with a small random β for each image. We then load the same pre-trained network and train for the same number of iterations as in hybrid gradient. We then evaluate the networks on the test set of NYU Depth V2 (Silberman et al., 2012) , following the same protocol. The results are reported in Table 3 . Note that none of these networks has seen a single real image except for validation.

The numbers indicate that our parametrized generation of SUNCG augmentation exceeds the original baseline performance. Note that the network trained with random β is worse than original performance. This means without proper optimization of perturbation parameters, such random augmentation may hurt generalization, demonstrating that good choices of these parameters are crucial for generalization to real images. 

5.3. BASEL FACE MODEL:

Synthetic face generation We exploit an off-the-shelf 3DMM morphable face and expression model (Dai et al., 2017; Zhu et al., 2015; 2016) to generating human 3D models, with face and pose parameters randomly sampled from mixtures of Gaussians or von Mises. Because the number of paramaters for 3DMM is over 400, we only include the first 10 principal dimensions for geometry, texture and expression parameters in the decision vector β, and uniformly sample for the remaining dimensions.

Evaluation We evaluate on the renderings of the scanned human faces (Paysan et al., 2009) . We split the 10 identities into two disjoint sets for validation and test, then use the rendering parameters provided in the dataset to recreate the renderings as well as depth images. For each scan, there are 3 lighting directions and 9 pose angles, creating 135 validation images and 135 test images. For depth evaluation, we use the standard metrics including the relative difference (absolute and squared) and root mean squared error (linear, log and scale-invariant log). The definitions are listed in Eigen et al. (2014) and also detailed in the Appendix. The results in 4 show that our algorithm is able to search for better β so that the network trained on the synthetic faces and generalize better on the scanned faces.

A APPENDIX:

A.1 METRICS Here we detail the metrics that we used in the paper. Assume n i and n * i are the unit normal vector at i-th pixel (of N total) in the prediction and ground truth normal maps, respectively. d i and d * i are depth values of the i-th pixel in the prediction and ground truth depth maps, respectively. • RMSE (linear):

1 N i (d i -d * i ) 2

• RMSE (log):

1 N i (log d i -log d * i ) 2

• RMSE (log, scale-invariant):

1 N i (log d i -log d * i • [ 1 N i (log d i -log d * i )]) 2

A.2 MIT-BERKELEY INTRINSIC IMAGE DATASET Our decision vector β for PCFG is a 29-d vector, with 4 dimensions representing the probabilities of sampling different primitives, 2 for sampling union or difference, 1 for whether to expand the tree node or replace it with a terminal, 6 for translation mean/variance, 6 for scaling log mean/variance, 2 for sphere radius log mean/variance, 2 for box length mean and variance, 4 for cylinder radius and height log mean/variance, and 2 for tetrahedron length log mean/variance.

For optimizing β, we use the mean angle error loss on the validation set as the generalization loss, and use RMSprop (Tieleman & Hinton, 2012) to obtain the gradient for β. Note that some dimensions of β are constrained (such as probability needs to be non-negative), so we simply clip the value of β to valid ranges when sampling near β for finite difference computation and updating β. We present the qualitative results in Fig. 5 . 

:

CONCLUSIONIn this paper, we have proposed hybrid gradient, a novel approach to the problem of automatically optimizing a generation pipeline of synthetic 3D training data. We evaluate our approach on the task of estimating surface normal and depth from a single image. Our experiments show that our algorithm can outperform the prior state of the art on optimizing the generation of 3D training data, particularly in terms of computational efficiency.

https://github.com/YadiraF/face3d

Figures and Tables:

Figure2: The details of using "hybrid gradient" to incrementally update β and train the network. The analytical gradient is computed by backpropagating through unrolled training steps (colored in orange). The numerical gradients are computed using finite difference approximation by sampling in a neighborhood of β t (colored in cyan). Then β t is updated using hybrid gradient, and the trained network is used as initialization for the next timestamp t + 1.

=> E; E => C(E, T(E)) | P; C => union | subtract; P => sphere | cube | truncated_cone | tetrahedron; T => attach * rand_translate * rand_rotate * rand_scale; In this PCFG, the final shape S is generated by recursively composing (C) other shapes E with transformations T, until primitives P are sampled at all E nodes. The parameter vector β consists of three parts: (1) The probability of the different rules; (2) The means and variations of log-normal distributions controlling shape primitives (P), such as the radius of the sphere; (3) The means and variations of log-normal distributions controlling transformation parameters (T), such as scale values. Examples of sampled shapes are shown in Fig. 3. For the generalization loss L, we simply compute the mean angle error of predictions on the training set of the MIT-Berkeley dataset.

Figure 5: The test set of the MIT-Berkeley Intrinsic Images dataset.

Ablation Study: the diagnostic experiment to compare with random but fixed β. We sample 10 values of β in advance, and then train the networks with the same setting as in hybrid gradient. The best, median and worst performance is reported on the test images, and the corresponding values of β are used to initialize β 0 for hybrid gradient for comparison. The results show that our approach is consistently better than the baselines with fixed β.

Table content: <table><tr><td/><td colspan="2">Summary Stats ↑</td><td/><td/><td>Errors ↓</td></tr><tr><td/><td>≤ 11.25 •</td><td>≤ 22.5 •</td><td>≤ 30 •</td><td colspan="2">MAE Median</td><td>MSE</td></tr><tr><td>Fixed β</td><td>19.9% β = β median 20.7% β = β best β = β worst 17.9%</td><td>52.7% 50.9% 46.7%</td><td colspan="2">70.5% 24.0 • 67.5% 24.8 • 64.6% 25.6 •</td><td>21.5 • 22.1 • 23.8 •</td><td>0.2282 0.2461 0.2553</td></tr><tr><td>Hybrid gradient</td><td>22.7% β 0 = β median 24.0% β 0 = β best β 0 = β worst 26.0%</td><td>58.5% 60.1% 58.6%</td><td colspan="2">73.9% 22.5 • 75.7% 21.8 • 73.9% 22.0 •</td><td>19.3 • 18.8 • 19.1 •</td><td>0.2065 0.1938 0.1998</td></tr></table>

The performance of the finetuned networks on the test set of NYU Depth V2Silberman et al. (2012), compared to the original network inZhang et al. (2017). The networks are only trained on the synthetic images. Without optimizing the parameters (random β), the augmentation hurts the generalization performance. With proper search of β using hybrid gradient, we are able to achieve better performance than the original model.

Table content: <table><tr><td/><td colspan="2">Summary Stats ↑</td><td/><td colspan="2">Errors ↓</td></tr><tr><td/><td colspan="3">≤ 11.25 • ≤ 22.5 • ≤ 30 •</td><td colspan="2">Mean Median</td></tr><tr><td>Original (Zhang et al., 2017)</td><td>24.1%</td><td>49.7%</td><td colspan="2">61.5% 28.8 •</td><td>22.7 •</td></tr><tr><td>Augmentation with random β</td><td>23.0%</td><td>48.8%</td><td colspan="2">61.3% 29.2 •</td><td>23.2 •</td></tr><tr><td>Agumentation with Hybrid gradient</td><td>27.3%</td><td colspan="3">52.5% 63.8% 28.1 •</td><td>21.1 •</td></tr></table>

The results on the scanned faces of the Basel Face Model. Our method is able to search for the synthetic face parameters such that the trained network can generalize better.

Table content: <table><tr><td/><td colspan="2">Relative Difference</td><td/><td colspan="2">RMSE</td></tr><tr><td/><td>Absolute</td><td>Squared</td><td>Linear</td><td>Log</td><td>Log (scale inv)</td></tr><tr><td>Training with random β</td><td>0.03718</td><td>9.701 × 10 -3</td><td>0.1395</td><td>0.1014</td><td>0.09717</td></tr><tr><td colspan="2">Basic Random Search (Mania et al., 2018) 0.02330</td><td>1.728 × 10 -3</td><td>0.0581</td><td>0.0299</td><td>0.02700</td></tr><tr><td>Hybrid gradient</td><td colspan="4">0.02256 1.649 × 10 -1 0.0570 0.0293</td><td>0.02603</td></tr></table>

Percentage of n i such that arccos(n i • n * i ) ≤ δ • Mean Squared Error (MSE): 1

Table content: <table><tr><td>• Mean Angle Error (MAE): 1 N</td><td>i arccos(n i • n  *  i )</td></tr><tr><td colspan="2">• Median Angle Error (MAE): median i [arccos(n i • n  *  i )]</td></tr><tr><td>• Threshold δ: N</td><td>i [arccos(n i • n  *  i )] 2</td></tr><tr><td>• Absolute Relative Difference: 1 N</td><td>i |d i -d  *  i |/d  *  i</td></tr><tr><td>• Squared Relative Difference: 1 N</td><td>i (d i -d  *  i ) 2 /d  *  i</td></tr></table>

Peer review (iclr2020):
This work proposes an approach to automatically adapt the generation of synthetic 3D training data such that a downstream neural network performs well on real data. The idea is to adaptively modify the data generation during training such that the network performs optimally on a validation set. The authors propose a gradient-based optimization pipeline, where they unroll the intermingled data generation and training process and compute approximate gradients for the non-differentiable rendering module using random search. Similar approaches have been proposed before, the main idea that sets it apart is random sampling in black-box modules in order to get an estimate for the gradient and to combine this with analytic gradients wherever they are available. I*m a bit surprised that this *hybrid gradient* hasn*t been used before, but a quick literature search seems to confirm this claim. My main concern with the approach is limited scalability. Random search gets harder for higher dimensions. The scalability issue seems to surface already in this work: The face model experiment would feature 400 rendering parameters, but the authors chose to do dimensionality reduction (I assume in order to make the problem tractable with random search). One would argue that any simulation that can transfer to real data reasonably well will likely need orders of magnitude more parameters than what is handled here. The observation is also in line with the other experiments: The grammar for NYUv2 features camera pose and the poses of a handful of shapes and the CSG experiment is very simplistic. The claim that none of the networks has seen a single training image (except for validation) in the NYUv2 experiment is misleading. Clearly, the validation loss provided an error signal throughout the (meta-)training of the approach (c.f Figure 2). In general, the validation set will leak information into the training in this approach. It is thus not surprising that some of the baselines which can*t leverage this information (e.g. random _x0008_eta) will perform worse. Finally, it doesn*t seem to be a good idea to only train on synthetic data (when doing computer vision). I*m missing more realistic experiments and comparisons. How would the approach fare against a modern monocular depth estimation model (for example MegaDepth)? Why only evaluate the normals and not also evaluate the depth maps? Most importantly: Can the model ensure that synthetic data provides value beyond, or in combination with, real data? Some parts of the paper are unclear: - For how many steps is the training loop unrolled? Can you discuss any limitations that the unrolling imposes in practice. E.g. would limited memory be an issue with larger networks and more unrolling steps? - It is not clear to me what the *Basic Random Search* baseline in Table 2 and Table 4 is exactly. - NYUv2: What does it mean to initialize the network using the original model in Zhang et al. 2017? Is that the pre-trained network, or the network architecture initialized with random weights? - Section A.2. mentions using RMSProp for the intrinsic image dataset. Was RMSProp only used on this dataset? If so, why? Problems with citations: - Citing Mania et. al. 2019 for the random search procedure seems like a misattribution. Random search as a DFO method is very old [1], and getting an estimate of the gradient via random sampling has been explored in multiple other work [2,3]. - Missing citation: [4] is close to the presented work and should be cited and adequately discussed. Typos: Page 3, 2nd paragraph *the generated the* Summary: The paper proposes a simple, (to the best of my knowledge) novel technical trick, but it is not clear if the approach would provide value in practical settings. [1] J. Matyas. Random optimization. Automation and Remote control, 1965 [2] A. D. Flaxman, A.T. Kalai, and B.H. Mcmahan, Online convex optimization in the bandit setting: gradient descent without a gradient, ACM-SIAM Symposium on Discrete Algorithms, 2005 [3] Y. Nesterov and V. Spokoiny. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 2017. [4] N. Ruiz, S. Schulter, and M. Chandraker. Learning To Simulate, ICLR 2019