AlgoNet: $C^\infty$ Smooth Algorithmic Neural Networks
Authors: Felix Petersen
Christian Borgelt
Oliver Deussen
Venue:  (2020)
Keywords: Algorithms
Smoothness
Differentiable
Inverse Problems
Adversarial Training
Neural Networks
Deep Learning
Differentiable Renderer
3D Mesh
Turing-completeness
Library

Abstract: Artificial neural networks have revolutionized many areas of computer science in recent years, providing solutions to a number of previously unsolved problems.
On the other hand, for many problems, classic algorithms exist, which typically exceed the accuracy and stability of neural networks.
To combine these two concepts, we present a new kind of neural networks—algorithmic neural networks (AlgoNets).
These networks integrate smooth versions of classic algorithms into the topology of neural networks.
A forward AlgoNet includes algorithmic layers into existing architectures to enhance performance and explainability while a backward AlgoNet enables solving inverse problems without or with only weak supervision.
In addition, we present the algonet package, a PyTorch based library that includes, inter alia, a smoothly evaluated programming language, a smooth 3D mesh renderer, and smooth sorting algorithms.

Under review as a conference paper at ICLR 2020
ALGONET: NETWORKS
Oo SMOOTH ALGORITHMIC NEURAL
Anonymous authors Paper under double-blind review
ABSTRACT
Artificial neural networks have revolutionized many of computer science in recent years, providing solutions to a number of previously unsolved problems. On the other hand, for many problems, classic algorithms exist, which typically exceed the accuracy and stability of neural networks. To combine these two concepts, we present a new kind of neural networks algorrthmi neural networks (AlgoNets). These networks integrate smooth versions of classic algorithms into the topology neural networks. forward AlgoNet includes algorithmic layers into existing architectures to enhance performance and explainability while backward AlgoNet enables solving inverse problems without or with only weak supervision. addition, present the aigonet package, PyTorch based library that includes, inter alia, a smoothly evaluated programming language, a smooth 3D mesh renderer, and smooth sorting algorithms.
INTRODUCTION
Artificial Neural Networks are employed to solve numerous problems, not only in computer science but also in all other natural sciences. Yet, the reasoning for the topologies of neural networks seldom reaches beyond empirically-bei based decisions.
In this work, we present a novel approach to designing neural networks algorithmic neural networks (short: AlgoNet). Such networks integrate algorithms algorithmic layers into the topology of neural networks. However, propagating gradients through such algorithms is problematic, because crisp decisions (conditions, maximum, etc.) introduce discontinuities into the loss function. If one passes from one side of crisp decision to the other, the loss function may change in non-smooth fashion-ii may "jump." That is, the loss function suddenly improves (or worsens, depending on the direction) without these changes being locally noticeable anywhere but exactly at these "jumps." " Hence, gradient descent based training, regardless the concrete optimizer, cannot approach these "jumps" in systematic fashion, since neither the loss function nor the gradient provides any information about these "jumps" in any place other than exactly the location at which they occur. Therefore, smoothing is necessary, such that information about the direction of improvement becomes exploitable by gradient descent also in the area surrounding the "jump." That is, by smoothing, e.g., an if, one can smoothly, by gradient descent, undergo a transition between the two crisp cases using only local gradient information.
Generally, for end-to-end trainable neural network systems, all components should least be C smooth, i.e., continuous, to avoid "jumps." However, having C* smooth, i.e., & times differentiable and then still continuous components with allows for higher-order derivatives and thus prevents unexpected behavior of the gradients. Hence, we designed smooth approximations to basic algorithms where the functions representing the algorithms ideally O smooth. That is, we designed pre- pre-programmed neural networks (restricted to smooth components) with the structure given algorithms.
is favorable. This property of higher smoothness
Algorithmic layers can solve sub-problems of the given problem, act custom algorithmic loss, assist in finding an appropriate solution for (ill- fill-posed) inverse problems. Such algorithmic losses can impose constraints on predicted solutions through optimization with respect the algorithmic loss. Ill-posed problems are a natural application for algorithmic losses and algorithmic layers. For that, introduce the Reconstructive Adversarial Network (RAN) in Sec. 3.3

---

Under review as a conference paper at ICLR 2020
In this work, we describe the basic concept of algorithmic layers and present several applications. In Sec.B.1.1 we start by proving that any algorithm, which can be emulated by Turing machine, can be approximated by a C smooth function. In Sec. 4 present some algorithmic layers that designed to solve underlying problems. In the appendix. we present a case study on 3D geometry reconstruction that demonstrates the applicability of RANs. All described algorithmic layers and models are provided in the algonet package, a PyTorch (Paszke et al. (2017) based library for AlgoNets.
2 RELATED WORK
Related work (Mart in Abadi et al. (2055): Che et al. 2018); Henderson & Ferrari (2018) in neural networks focused on dealing with crisp decisions by passing through gradients for the alternatives of the decisions. There is no smooth transition between the alternatives, which introduces discontinuities in the loss function that hinder learning, which of the alternatives should be chosen. TensorFlow contains a sorting layer (t.... sort) well as while loop construct (t f while while_loop). Since the sorting layer only performs a crisp relocation of the gradients and the while loop has a crisp exit condition, there is gradient with respect the conditions in these layers. Concurrently, we present a smooth sorting layer in Sec. 4.1 and a smooth while loop in Sec. 3.1.1
Theoretical work by DeMillo er al. (DeMillo & Lipton (1993) proved that any program could be modeled by smooth function. Consecutive works (Nesterov (2005); Chaudhuri & Solar- (2011; Yang Barnes (2017)) provided approaches for smoothing programs using, i.a., Gaussian smoothing (Chaudhuri & Solar-Lezama (201T: Yang & Barnes (201)))
ALGONET
To introduce algorithmic layers, we prove that smooth approximations for any Turing computable algorithm exist and explain two flavors of AlgoNets: forward and backward AlgoNets.
3.1 SMOOTH ALGORITTMS
To design smooth algorithm, all discrete cases (e.g., conditions of if statements or loops) have to be replaced by continuous or smooth functions. The essential property is that the implementation is differentiable with respect to all internal choices and does not-as in previous work only carry the gradients through the algorithm. For example, an i statement can be replaced by a sigmoid-weighter sum of both cases. By using smooth sigmoid function, the statement is smoothly interpreted. Hence, the gradient descent method can influence the condition to hold the content of the then case reduces the loss and influence the condition to fail if the loss is lower when the else case is executed. Thus, the partial derivative with respect to a neuron is computed because the neuron is used in the if statement. In contrast, when propagating back the gradient of the then the else case depending on the value of the condition, there is discontinuity at the points where the value of the condition changes and the partial derivative of the neuron the condition equals zero.
82(x) = {
i <0
$1 (I, 5) 1te
with s = 1 (1)
(2)
else
Here, the logistic sigmoid function (Eq. I is a C smooth replacement for the Heaviside sigmoid function (Eq. which is equivalent to the f statement. Alternatively, one could use other sigmoid functions, e.g., the C'l smooth step function x² 2 for x E [0. 1J. and 0 and for all values before and after the given range, respectively.
Another example is the max max-operator, which, in neural networks, is commonly replaced by the exp(x.)
SoftMax operator (SoftMax (x); Σ, exp(=,)
After designing algorithmic layer, use it to enhance a neural network and solve for its inverse by using the reconstructive adversarial neural network (RAN) as shall be described in
Sec.
3.2 and 3.3
2

---

Under review as a conference paper at ICLR 2020
3.1.1 SMOOTH WHILE-PROGRES
In this section, we prove that for any algorithm, C smooth approximations exist. For that, we present smooth and differentiable approximations to an elementary programming language based on the WHILE--lnguaag by Uwe Schöning (Schöning (208).) We do not want imply that such a direct translation of any algorithm into the "HHlE--languaee to make it smooth is the best or even the only option. It is rather meant as fallback solution no better solution can be found for sub-problem. We will present several more efficient and canonical approximations in Sec. 4
The WHiLLEllnggugge is Tomine-complete, and for variables (var) its grammar can be defined as follows:
prog = WHILE var != 0 DO prog END
prog prog
var var
// left and right var unequal
var :- var + 1 var :- var 1
// left and right var equal
// left and right var equal
Here, var E {xn n E No} and while x0 is the output, xn where No are variables initialized to 0 if not set to an imput value. Thus, {xn n E ]+} are inputs and/or local variables used in the computations. Although this interpretation allows only for single scalar output value (x0), an arbitrary number of output values can be reached, either by interpreting additional variables as output or by using multiple WHILE-programm (one for each output value). Provided Church's thesis holds, this language covers all effectively calculable functions. The WHILL-language is equivalent register machines which have no WHILE, but instead IF and GOTO statements.
We generate the approximation to this language by executing all statements only to the extent of their probability. That is, we keep track of a probability p. indicating whether the current statement is still executed. Initially, p 1. In the body of while loop, the
probability for an execution is Pk (z) = =p(odd) dx(x))
(new)
-2
-1
0
1
2
x
where k defines the smoothness of the probability function for exiting the loop Φk. To obtain C smooth
Figure 1: Φu (magenta) and Φo (cyan).
(c"=-1)² =1 - sech For C* smoothness, max(0, |z|) min |x|).. For xo initialized as integers, using Φu, the result for the C WHILLE-pogrra always equals the result for the discrete WHILE-program since the probability is always either 1 or 0. For all & E No. du exists. Fig. [ shows how these exit probability functions behave. Because of the symmetry of Φ, w.l.o.g., we can assume that Vx E IRz-- If the loop (in the discrete version) increases by 1, x will diverge. If x decreases by 1, p converges to O. Since (x(x) 2 @oo (x), it suffices to show that p converges to zero for do. Since p (=new)(x)
WHIL-E-rrograms, we used 0 IR + [0,1]
we used the shouldered fuzzy set du : R - [0, [0,1]:==7
p(old) (x(x) < (x < |x], |x| = 0. Thus, and Φo(x) monotonically decrease and x 2 O. Eventually, (x) < 1. Thus, p(x) < (do(x))) 2-000 O.
x == x x- p(uew) (x) 2
Here, x is the value of the current variable, & the steepness, and p is the probability of the execution. To apply the probabilities on the assignment, increment and decrement operators, we redefine them as:
x0 == x1
x0
x1 + x1
p) x0
(3)
x0 = x0 +1
+ x0 = x0 +
(4)
x0: x0
x0 x0 p
(5)
Since the hyperbolic secant is C smooth, our version of the WHILE-language is C smooth. While the probability converges to zero, it (in most cases) never reaches zero, and the loop would never exit. Thus, we introduce > and exit the loop if p S e or a maximum number of iterations is reached. Although this introduces discontinuities, by choosing an of numerically negligible size, the discontinuities also become numerically negligible.
3

---

Under review as a conference paper at ICLR 2020
(a) x2
(b) C WHILBE(11 x2)
(c) C WHILEE(11, x2)
Figure Multiplication of x1 and x2 implemented as general multiplication, C smooth WHILLE-progran and C smooth WHILE-program. The height indicates the result of the function. Contrary the common notion, the color indicates not the values but the analytic gradient of the function.
As an experiment, we implemented the multiplication on positive integers, as shown on the left:
WHILE x2 != 0 DO
WITH := 1; 11 = x(x2) DO
x3 = x1
x3 := px-
- -pi) x3
WHILE x3 1= 0 DO
WITH p2 :- P/2 : -P- (x33) DO
x0 : x0
x0 := x0 P2
x3 == x3
x3 x3 P2
END
WHILE P2 2 E
x2
x2
x2 x2 P1
END
WHILE p1 2 c
Contrary to the discrete implementation, the smooth interpretation (as on the right) can interpolate the result for arbitrary values x1, Rt.
Since the WHILE WHILE-language is Turine-complete, any high-level program could, in principle, using an appropriate compiler, be translated into an equivalent program WHILE-language. To this WHILE-progrand automatic smoothing could be applied using the rules that are illustrated here. Of course, there are better ways of smoothing: manual smoothing using domain-specific knowledge and smoothing using higher-level language outperform the low-level automatic smoothing. For example, in higher-level language, the multiplication would be implemented since is smooth itself. knowledge, algorithms could be reformulated in such a way that smoothing possible in more canonical way. To translate a WHILE- WHooEaproggam into a neural network, the WHILE loops are considered as recurrent sub-networks.
Using
3.2 FORWARD ALGONET
The AlgoNet can be classified into two flavors, the forward and the backward AlgoNet. To create a forward AlgoNet, we use algorithmic layers and insert them into a neural network. By doing so, the neural network may or may not find better local minimum by additionally employing the given algorithm. We do by using one of the following options for each algorithmic layer:
Insert between two consecutive layers (Fig. 3a
Insert between two consecutive layers and also skip the algorithmic layer (Fig 3b
Add a residual connection and apply the algorithmic layer on the residual part FFig
AlgoNet
AlgoNet
AlgoNet
(a) fully applicd AlgoNet layer
shortcut AlgoNet layer
(c) residual AlgoNet layer
Figure 3: Different styles of the forward AlgoNet.
4

---

Under review as a conference paper at ICLR 2020
B'B
C'EA
a E
a2h
620
D
0
120
BEB
Reconstruction
1
B
Figure 4: RAN System overview. The reconstructor receives an object from the input domain A and predicts the corresponding reconstruction. The reconstruction, then, is validated through our smooth inverse. The latter produces objects in a different domain, B. which are translated back to the input domain A for training purposes (b2a). Unlike in traditional GAN systems, the purpose of our discriminator D is mainly to indicate whether the two inputs match in content, not in style. Our novel training scheme trains the whole network via five different data paths, including two which require another domain translator, a2b.
Generally, algorithmic layers do not have trainable weights. Regarding the accuracy, the output of C smooth WHILLE-roggrams differs from the discrete WHILLE-programe by small factor and offset. The output of C smooth WHILE-programg equals the output of discrete WHILE-prooe -programss for integer inputs (discrete WHILE- p-rggaams fail for non-integer inputs). One could counter this factor and offset for C smooth WHIL-E-rooramss by adding an additional weight and a bias to each assignment in the WHILLE-program For that, one should regularize these weights and biases to be close to one and zero, respectively. These parameters could be trained on data set of integer input/output pairs the respective discrete WHILE-program to fit the algorithmic layer. Moreover, the algorithmic layer could also be trained to fit the surrounding layers better.
3.3 BACKWARD ALGONET: RECONSTRUCTIV ADVERSARIAL NETWORKS (RAN)
While forward AlgoNets can use arbitrary smooth algorithms-ol of course, an algorithm directly related the problem might perform better backward AlgoNets use an algorithm that solves the inverse of the given problem. For example, a smooth renderer for 3D-reconstruction. a smooth iterated function system (IFS) for solving the inverse-problem of IFS, and a smooth text-to- toseshh synthesizer for speech recognition. While backward AlgoNets could be used in supervised settings, they are designed for unsupervised or weakly supervised solving of inverse-problems. Their concept is the following:
Input (E A) Reconst
Goal smooth inverse - Smooth version of input (E B)
This structure similar to auto-encoders and the encoder-renderer architecture presented by Che et al. (Che et al. (2018)) Such an architecture, however, cannot directly be trained since there is a domain shift between the input domain A and the smooth output domain B. Thus, we introduce domain translators (a2b and (2n) translate between these two domains. Since training is extremely hard with three consecutive components, of which the middle one is highly restrictive, we introduce a novel training schema for these components: the reconstructive adversarial network (RAN). For that, we also include a discriminator to allow for adversarial training of the components a2b and 62a. Of our five components four are trainable (the reconstructor R, the domain translators a2b and b2a, and the discriminator D), and one is non- non-trainable (the smooth inverse Inv).
Since, initially, neither the reconstructor nor the domain translators are trained, we are confronted with causality dilemma. A typical approach for solving such causality dilemmas is to solve the two components coevolutionarily by iteratively applying various influences towards a common solution. Fig. 4 depicts the structure of the RAN, which allows for such coevolutionary training scheme.
The discriminator receives two inputs, one from space A and one from space B. One of these inputs (either A or B) receives two values, a real and fake value; the task of the discriminator is to distinguish between these two, given the other input. For training, the discriminator is trained to distinguish between the different path combinations for the generation of inputs. Consecutively, the generator modules are trained to fool the discriminator. This adversarial game allows training the RAN.
5

---

Under review as a conference paper at ICLR 2020
In the following, we will present this process, well as its involved losses, in detail. Our optimization of R, a2b, 62a, and D involves adversarial losses, cycle-consistency losses, and regularization losses. Specifically, we solve the following optimization:
min min min max L R a2b
or in greater detail
min min min max a2b
Σ (oc Li) + freg-
02a D
R
620
D
where or is weight in [0, 1] and L, and L. shall be defined below. Lrog denotes the (optional) regularization losses imposed on the reconstruction output.
We define 8' D" B and d', a" E A in dependency of A according Fig. 4 as
6' a2b(a)
=" Inv R(a)
b2a(b')
a" b2a(b").
With that, our losses are (without
weights)
L1 = A [log D(a, b²]] + Ea-A (log(1 b(), + Eo- 1116" - - 1111]]
L2 = E [log (a, 8²)] + E [lo((1 D(a", + E [||a" all1]
L3 EarA olog (a, b')] + E [og(1 - bi)", + Ban Il|a' a/11] + Ea- [||b" 8 [|]
L1 A log (a, 8*)] + Eu- [log(1 D(a', + Earallaa a|l1]] + E B Allb" - 6' [1]] (()),
C5 A [log (a, b'] + A
d'))]+ + Eu~Allla' a|l1].
We alternately train the different sections of our network in the following order:
1. The discriminator D
2. The translation from B to A (b2a)
3. The components that perform translation from A to B (R+Inv, a2b)
For each of these sections, separately train the five losses L1, L2, L3, L4, and Lz. In experiments, we used one Adam optimizer (Kingma & Ba 2014)) for each trainable component (R, a2b, 62a, and D).
4 APPLICATIONS
In this section, we present specific AlgoNet-layers. Specifically, we present smooth sorting algorithm, a smooth median, finite differences layer, weighted SoftMax, smooth iterated function systems, and smooth 3D mesh renderer. We will present the smooth 3D mesh renderer in greater detail the appendix, where we will also present the respective 3D mesh reconstruction applying the RAN.
4.1 SOFTSORT
The SoftSort layer smooth sorting algorithm that is based on a parallelized version of bubble sort (Knuth/ (1998)) (see especially Section 5.3.4: Networks for Sorting), which sorts a tensor along an array of scalars by repeatedly exchanging adjacent elements necessary. Fig. 3jshows of the SoftSort algorithm. Contrasting our approach, the sorting layer in TensorFlow (Mart in Abadi ctt al. (2015)) is not smooth and does not consider gradients with respect to the ordering induced by the sorting.
structure
4.2 FINITE DIFFERENCES
The finite differences method, which was introduced by Lewy et al. (Lewy H. (1928)), is an essential tool for finding numerical solutions of partial differential equations. In analogy, the finite differences layer uses finite differences to compute the spatial derivative for one or multiple given dimensions of tensor. For that, subtract the tensor from itself shifted by one in the given dimension. Optionally, we normalize the result by shifting the mean to zero and/or add padding to output an equally sized tensor. Thus, is possible integrate spatially- or temporally-derivatnn layer into neural networks.

---

Under review as a conference paper at ICLR 2020
unsorted scalars
1st exchange
2d exchange
sorted scalars
(1)
do
bo
Q1
(1)
11
b1
(1
22
Q2
(1)
Q2
(2)
b2
On
(1 e
(1)
di
(2)
bn
Mi
M2
M3
M
& :=:00 a [0++1
(1-1) (1-1))
where 0 x
(6)
1te
n-1
M M2
b=a- M =
Figure 5: The structure of SoftSort. Here, the exchanges of adjacent elements are represented by matrices M, By multiplying these matrices with tensor a, we obtain b, the sorted version of a. By instead multiplying with tensor we obtain t': t sorted with respect to Using that, we can also sort a tensor with respect to a learned metric. For sorting n values, we need n - steps for an even n and steps for odd n: to get probabilistic coarse sorting, even fewer steps may suffice. denotes the steepness of the sorting such that for - oo we obtain a non-smooth sorting and for infinitely many sorting operations, all resulting values equal the mean of the input tensor, In the displayed graph, the two recurrent layers are unrolled in time.
4.3 WEIGHTEDDSOFTMAX
The weighted SoftMax (short: woSoftaax) allows a list that is fed to the SoftMax operator to be smoothly sliced by weights indicating which elements are the list. That is, there are two inputs, the actual values (x) and weights (w) from (0; indicating which values of should be considered for the SoftMax. Thus, moSoftMax can be used when the maximum value of values, for which an additional condition also holds, is searched by indicating whether the additional condition holds with weights wy (0; 1]. We define the weighted SoftMax as:
exp(x,, wy
exp(x, log w.)
(x
(8)
exp(x;) we
Σ E
2 exp(x; log w))
SoftMax, (x; + log w;)
Accordingly, we define the weighted SoftMin (analogue SoftMax//SftMMin) as wSSfftMinxx w) wSSftMax((Xx w). By that, we enable a smooth selection to apply the SoftMax/SoftMinn function only to relevant values.
4.4 MODTMEDIAN
The mean is a commonly used measure for reducing tensors, e.g., for normalizing a tensor. While the median is robust against outliers, the mean sensitive to all data points. This has two effects: firstly, the mean is not the most representative value because it is influenced by outliers; secondly, the derivative of normalization substantially depends on the positions outliers. That is, outliers, which might have accelerated gradients in the first place, can inffuence all values during a normalization like I' : x Σ. While one would generally avoid these potentially malicious gradients by cutting the gradients of this is not adequate if changes in expected. To reduce the influence of outliers in a smooth way, we propose the SoftMedian, which comes in two styles, a precise and slower as well significantly faster version that only discards fixed number of outliers.
7

---

Under review as a conference paper at ICLR 2020
The precise version sorts the tensor with SoftSort and takes the middle value(s). For that, it is not necessary to carry out the entire SoftSort; instead, only those computations that influence the middle value need to be taken into account.
The faster variant to compute the SoftMedian (of degree j) is by its recursive definition in which influence of the minimum and maximum values is reduced as follows: SoftMed(() MinoftMin (mnSoftMin(xx (of + moSof Moft MofM where SoftMed(x) = |
4.5 SMOOTH ITERATED FUNCTION SYSTEMS
Iterated function systems (IFS) allow the construction of various fractals using only set of parameters. For example, pictures of plants like Barnsley's fern can be generated using only 4 x 6 = 24 parameters. Numerous different plants and objects can be represented using IFS. Since IFS are parametric representations, they can be stored in very small space and be adjusted. This can be used, e.g., in a computer game to avoid unnatural uniformity when rendering vegetation by changing the parameters slightly, so that each plant looks slightly different. Finally, there are very fast algorithms to generate images from IFS. While IFS provide many advantages, solving the inverse problem of IFS, i... finding IFS representation for any given image, is very hard and still unsolved. Towards solving this inverse problem, we developed a t smooth approximation to IFS.
Given two-dimensional IFS with n bi-linear functions (f. fi v) == (æ + 01,, + 02 43, y + 04,i + 05 + d)), we repeatedly randomly select one function ʃ E (Ji)ie(1..)) and apply to an initial position or the proceeding result. We do that process arbitrarily often and plot every intermediate step. Since it is not meaningful to interpolate multiple functions, because IFS rely on randomized choices, and to provide consistency, we perform these random choices in advance.
The difficulty in this process is the rasterization since crisp decision correlating pixels to points can be made. Thus, we correlate each pixel to each point with a probability E [0; 1] where p = 1 is full correlation and p 0 means correlation at all. By applying Gaussian smoothing on the locations of the points, for each point, the probabilities p E (0; 1) for each pixel define the correlation. Concluding, for each pixel, the probabilities for all points lie in the area of that pixel known. By aggregating these probabilities, we achieve a smooth rasterization.
We tested the smooth IFS by optimizing its parameters to fit image. For that, by setting the standard deviation, different levels of details can be optimized.
4.6 SMOOTH RENDERER
Lastly, include C smooth 3D mesh renderer to the AlgoNet library, which projects triangular mesh onto an image while considering physical properties like perspective and shading. Compared to previous differentiable renderers, this renderer fully and not only locally differentiable. Moreover, the continuity of the gradient allows for seamless integration into neural networks by avoiding unexpected behavior altogether. By taking the decision which triangles cover a pixel, in analogy to the smooth rasterization in Sec. 4.5) the silhouette of the mesh can be obtained. Consecutively. by computing which of these triangles is the closest to the camera smoothly, our renderer's r's depth buffer is smooth. That allows for color handling and shading.
The smooth renderer is presented in greater detail in Appendix A Consecutively, in Appendix B we the smooth renderer as smooth inverse for the RAN to solve the inverse problem of 3D reconstruction, A selection of results is presented in Fig. more results are presented in Appendix C
5 DISCUSSION AND CONCLUSION
We presented AlgoNets as a new kind of layers for neural networks, a C Turing complete interpreter, and RANs as novel technique for solving ill-posed inverse problems. Moreover, in the appendix, we demonstrate with a case study on 3D reconstruction that the RAN works even in complex settings. We have implemented the presented layers on top of Py Torch and will publish our AlgoNet library upon publication of this work. Concurrent with their benefits, AlgoNets, such as the aforementioned rendering layer, can get computationally very expensive. On the other hand, the rendering layer is
8

---

Under review as a conference paper at ICLR 2020
Input
Prediction
Alternative View
Input
Prediction
Alt. view
Figure 6: Results for the reconstruction of the ShapeNet (Chang et al. 2015))
classes airplane, and car. Lett:
input. Middle: prediction from the same angles. Right: predictions from alternative viewpoints. Bottom: single-view reconstruction result from the UT Zappos50K dataset (Yu & Grauman (2014) (camera-captured images). Although the object has strong textures, it is adequately reconstructed.
very powerful since it allows training a 3D reconstruction without 3D supervision using the RAN. Since the RAN has a very complex architecture that requires particular training paradigm, it can also take relatively long to train it. To accommodate this issue, we found that by increasing some loss weights and introducing probability of whether computation executed, the training time be reduced by a factor of two or more. Unfortunately. due to length restrictions, we have only been able present subset of all implemented algorithmic layers.
(2000) have been very successful in practice for efficient puck-propagation. However, in contrast to such network where the rectification allows to activate or deactivate neurons, we use the gradients produced by the differentiable algorithm not just for adapting parameters but employ the gradient of the smooth algorithm to train other parts of the network. For doing so, the gradient of the algorithm should not be discontinuous (as it would be with a ReLUU); ; so having least a C*l smoothness is clearly beneficial to allow for effective training.
Rectified Linear Units (ReLUs) (Nair& & Hinton
The AlgoNet could also be used in the realm of explainable artificial intelligence GGilpin al. (2018) by adding residual algorithmic layers into neural networks and then analyzing the neurons of the trained AlgoNet. For that, network activation and/or network sensitivity can indicate the relevance of the residual algorithmic layer. To compute the network sensitivity of an algorithmic layer, the gradient with respect additional weights (constant equal to one) in the algorithmic layer could be computed. By that, similarities between classic algorithms and the behavior of neural networks could be inferred. An alternative approach would be to gradually replace parts of trained neural networks with algorithmic layers and analyzing the effect on the new model accuracy.
In the future, we will develop high-level smooth programming language to improve smooth representations of higher-level programming concepts. Adding trainable weights to the algorithmic layers to improve the accuracy of smooth algorithms and/or allow the rest of the network to influence the behavior of the algorithmic layer is subject to future research. The similarities of our smooth WHILE- -roggaams analog computing as well quantum computing shall be explored in future work. Another future objective is the exploration of neural networks not with a fixed but instead smooth topology. Finally, loss ranking would an additional application for the SoftSort algorithm.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang. Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Rich 3D Model Repository. 12 2015. URL /ttps:////aaxxi
abs/1512. 03012
Swarat Chaudhuri and Armando Solar-Lezama. Smoothing Program Soundly and Robustly. In Ganesh Gopalakrishnan and Shaz Qadeer (eds.), Computer Aided Verification, pp. 277-292, Berlin, Heidelberg, 2011. Springer Berlin Heidelherg. ISBN 978-3-642-221 10-1.
Chengqian Che, Fujun Luan, Shuang Zhao, Kavita Bala, and loannis Gkioulekas. Inverse Transport Networks. 9 2018. URI.
org/ abs 809. .0009..00 10800
M de Berg. Ray Shooting, Depth Orders and Hidden Surface Removal. Lecture Notes in Computer Science. Springer, 1993. ISBN 9783540570202. URL httph::tthhhhhttooe ottp.p////////oooooo.o.o.ooo...
bbooks?idbb1 INPTC3W QC
Richard A. DeMillo and Richard J. Lipton. Comments on "Defining Software by Continuous Smooth Functions. IEEE Transactions on Software Engineering, 19(3):30-339, 1993. ISSN 00985589. doi: 10.1109/32.2211400
Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Explaining Explanations: An Overview of Interpretability of Machine Learning. 2018. URL
/
https::/aaxiv. http:///aaaavvvooaa/////e/e/////
www.oogg///b... 1106. 00069
Paul Henderson and Vittorio Ferrari Learning to Generate and Reconstruct 3D Meshes with only 2D Supervision. 7 2018. URL https:////aa arwiv. org/ abs/ / 1807 09259
Diederik P Kingma and Jimmy Ba. Adam: (A) Method for Stochastic Optimization. CoRR, abs/1412.6, 2014.
org 0ese/142.6600 6980
D E Knuth. The Art of Computer Programming: Volume 3: Sorting and Searching. Pearson Education, 1998. ISBN 9780321635785. URL httph::ttt/ooott got.lks.oooolle...e..... debboksseeee/ooskkeaaaa
CYOLBAAAQBAJ
Friedrichs K Courant R Lewy H. Über die partiellen Differenzengleichunnen der mathematischen
Physik. Mathematische Annalen, 100:32-74, 1928.
httpt::::/eemmll.ooooooooo 119998
Thomas Luft, Oliver Deussen, Thomas Luft, and Oliver Deussen. Real-time Watercolor Illustrations of Plants Using Blurred Depth Test. Proceedings of the 4th International Symposium cho-ppootreaalistic Animation and Rendering, 21 (2006):159-166 2006. doi: 10.1145/1124725 1124732.
Mart'in Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp. Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur. Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems, 2015. URL http::ttt////tt //ww::::::/w..../.. www.wwwwwwww..oooooooooooooooo.. http://hhww.oe//ttttttttttttt 7/www. htt.a//c......oooo.ooo..o//////.
Vinod. Nair and Geoffrey E. Hinton. Rectified Linear Units Improve Restricted Boltzmann Machines. ICML, 27, 2010.
Yu Nesterov. Smooth minimization of non-smooth functions. Mathematical Programming. 103(1): 127-152, 2005. ISSN 00255610. doi: 10.1007/110107-004-5553
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS-W. W, 2017.
10

---

Under review as a conference paper at ICLR 2020
Uwe Schöning. Theoretische Informatik kurz gefasst. Spektrum Akademischer Verlag, 2008. ISBN 9783827418241.
Yuting Yang and Connelly Barnes. Approximate Program Smoothing Using Mean- Varianee Statistics, with Application to Procedural Shader Bandlimiting. 6 2017. URL https::/arxiv...rrr
abs/ 01208
A Yu and K Grauman. Fine-Grained Visual Comparisons with Local Learning. In Computer Vision and Pattern Recognition (CVPR), 2014.

---

Under review as a conference paper at ICLR 2020
ADDITIONAL MATERIAL: A CASE STUDY ABOUT 3D RECONSTRUCTION
Here, we describe how the AlgoNet to solve complex problem in computer graphics without supervision: the reconstruction of 3D geometry from single real-world images. This problem requires complex system that can only be implemented by including algorithmic concepts provided by the AlgoNet and the RAN.
In Appendix A, we present our C smooth differentiable 3D mesh renderer in greater detail. Based on that, we present unsupervised 3D geometry reconstruction as an application of the RAN in Appendix B. We present and discuss our 3D geometry reconstruction results in Appendix C. Finally, in Appendix D, we present additional implementation details.
A
C SMOOTH RENDERER
In this section, we present our C Smooth Renderer that avoids any discontinuities at occlusions or dis-occlusions. Having this property, the renderer's back-propagated gradients can be properly used to modify the 3D model. This is critical for integrating the renderer into neural network. The typical discontinuity problem occurs during triangle rasterization, where the visibility of a triangle, due to occlusion or dis-occlusion, causes an abrupt change in the image. For example, if during the optimization process, the backside of a predicted object self-intersects its front, traditional differentiable renderers not able to provide reasonable gradient towards reversing such an erroncous self-intersection since they cannot differentiate with respect to occlusion. To overcome this problem, our approach offers a soft blending scheme, that is continuous even through such intersections.
As in the general rendering approach, first, we apply view transformations all triangles to bring them from object space into perspective projection space coordinates. This process is generally already fully differentiable.
Consecutively, one needs rasterization to correlate triangles to pixels. General rasterization consists of two steps, for each pixel one needs to collect all the triangles that cover that pixel, and then employ a z-buffer to determine which of them is visible in the pixel.
Instead of collecting all triangles that fit the xy-coordinates of a given pixel, we determine a probability value of whether a triangle fits pixel for each triangle and pixel. This constitutes the visibility tensor V as shall be described in section A.2
Our key idea is to use a visibility test that enables reasoning beyond occlusion, using only smooth functions to avoid abrupt changes.
(a) Discrete case
(b) Smooth case
Smoott e-uullr 5
Rather than taking a discrete decision of which
Visibilty
triangle is the closest and thus visible, we softly blend their visibility, which goes along with an idea from stabilizing non potttooreeaaiitcc rendering results (Luft et al.) (2066). By using a SoftMin-based function, we determine the closest and thus most visible face. But using the simple SoftMin of the z-positions in camera space would result in only the single closest triangle being most visible. Thus, we need to incorporate the visibility tensor V that tells us which triangles cover a given pixel. Instead, we weight the SoftMin with the visibility tensor V by introducing the weighted SoftMin (noSoftMin). Taking the roSoftMin of the z-positions in camera space, constitutes smooth z- bufffer as shall be described in greater detail in section A.
Dist. camera cyan (c) Visibility implied by the discrete z-buffer (for Tab
Dist. camera to cyan face
(d) Visibility implied by
the smooth z-buffer 3
(for 7
Figure 7: Visualization of the smooth depth buffer and occlusion: 77 shows three triangles rendered in a standard way, in 76 the same triangles are rendered smoothly. Whit While in the discrete case small change in depth can result in sudden change of color (7c). our smooth depth-oriented rendering (7d) avoids that and therefore is differentiable everywhere.
12

---

Under review as a conference paper at ICLR 2020
(a) check whether point is triangle
(b) sigmoid edge a
wrt.
(c) odge b
sigmoid
wrt.
(d)
sigmoid edge
wrt.
(e)
product of
sigmoids
Figure 8: Visualization of the smooth rasterization. While the magenta point lies outside the triangle, the cyan point lies inside the triangle; this can be determined by measuring on which side of the edges point lies. In subfigure 8d is smoothly determined which parts of the image lie inside and which parts lie outside the triangle with respect to the edges a-c. This is combined by multiplication (visibility tensor V) in subfigure 8e
This smooth z-buffer leads to our C Smooth Renderer, where the z-positions of triangles is differentiable with respect occlusions. In previous differentiable renderers, only the xy-coordinates were locally differentiable with respect to occlusion.
Let us assume to have three triangles (see Fig. 7ap, where bottom left pixel (marked with #) with respect to the z- e-position of the cyan face. During the process of optimizing the geometry. triangles might change their order with regard the depth and abrupt color changes might appear. As shown in Figure 7c] the color value of the pixel (implied by the rasterization of the triangles) is constant except for one single point. this point of intersection, the rasterization is discontinuous; at all other points, the derivative with respect to the z-position is O. Employing the smooth rasterization and smooth z-buffer as Figure 7 the visibility pixel never absolute, but rather soft blend. Thus, it is differentiable, and optimizations can be solved with simple gradient descent.
examine the behavior of the
Finally, need compute the color values of the triangles. For that, composed of Jlinn-Phong diffuse, and ambient shading. We restrict the color to grayscale since we do not reconstruct the color in the RAN. Since the function of color is already differentiable directly use
lightning model
Figure 9 shows comparison between smooth renderings and Blender rendering of the Stanford bunny.
A.1 SMOOTHI Z-BUFFER 3
Our rasterization step similar to the bufuffer algorithm, but instead of displaying the single closet triangle and its z-distance in each pixel, we display blend of triangles that project to the pixel.
We define the Smooth z-buffer for pixel (Pe,Py), triangle T. and opacity o as follows:
3ppz Pa. T) roSoft.Minoo- z-dist(camera, V (pz. Pg.
We define the weighted SoftMin (analogue to SoftMin/SoftMaxx) as: woSoftMinxx, w) = roSoftMax x,w) w) where the weighted SoftMax defined
exp(x:)
(x,,
w exp(x:) SoftMax; + log w:)
Thus, for pixel, the closest triangle is represented with high visibility, while triangles further away have weaker visibility. The visibility tensor V. as shall be defined in Section A.2 contains the extent to which a given triangle covers a given pixel. We use as a weight for the toSoftMin, which allows considering only the relevant triangles in the SoftMin operator.
The opacity o is how it affects the results.
setting accelerating the strength of the SoftMin. See Figure 9 to
Similar to the painter's algorithm dee Berg (1993) we do not explicitly handle special cases of cyclical overlapping polygons that cause depth ordering errors. Our smooth renderer is not
13

---

Under review as a conference paper at ICLR 2020
(a) low low 0
(b) high low o
(c) low 8, high
(d) high 8, high
(c) Blender result
Figure 9: Stanford bunny rendered by the smooth renderer using different edge smoothnesses (s) and opacities (o): In 9a and [9b] the low opacity causes, e.g., one of the ears to be still
visible through the head of the bunny (for usage of o see section A. 1). Da 99 the low edge steepness causes smoother edges (for usage of s see section A.2p On the right: The Stanford bunny rendered by Blender (e).
sensitive these cases. When polygons have similar distance the camera, their opacity will also be very similar and thus not only the front polygon but also the one behind is visible.
A.2 VISIBILITY TENSOR V
In the general rendering approach, the discrete choice, whether a triangle covers a pixel is just trivial check. In the smooth approach, shown in Figure we determine the pixels that correspond to triangle by checking for each pixel whether the directed distances from the pixel to each edge are all positive. This yields the visibility tensor V for triangle T = (e1, (21, e2) with e, (V1, v2) and 0 = (0 Piyy, and steepness s as follows:
((aa Py. := =
II
Ur.2 Uz,1
Un,1 Ps S
Ug21
Uy,1 Py
(T
with m Soft
The sign of the directed distances the three edges indicates which side of the edge a pixel is. By applying a sigmoid function (a) on that directed distances, we get a value close to if the pixel lies inside and value close to 0 if the pixel lies outside the triangle with respect to given edge. By taking the product of the values for all three edges, the result (E [0, 1] smoothly indicates whether a pixel lies in or outside triangle. Since this draws the triangles only from one direction, we add the same term with the negative directed distances to make the visibility tensor triangle orientation invariant:
Vorient.ánv. (pax Py; T)
Σ
II
U2.2 - Ux.1
Up,1 Ps
$ m
Uy.2 - Uy.1
0y.1
Py
GE (=(
For Figure the visibility tensor V looks as follows:
V(pa/PesPy Py/
example
Distanon
The steepness s is hyper-parameter setting the steepness of the sigmoid function. See Figure see how affects the results.
14

---

Under review as a conference paper at ICLR 2020
B 3D GEOMETRY RECONSTRUCTIVE ADVERSARIAL NETWORK
Using our RAN framework, we train reconstructor (pix2vex) without 3D supervision, i.e., without the need for the actual 3D models corresponding to the input images during training. The architecture of the reconstructor (pi (2ix2vex) is pix2pix network followed by two fully connected layers predicting the coordinates of the points of a 3D mesh. The pix2vex shall be described in greater detail in Appendix D.2] The aforementioned causality dilemma is the following: To train the reconstruction, we need to compare the smoothly rendered images of the predicted shape to the input, which requires style transfer. On the other hand, to train the style translation, we need know what properly smoothly rendered image (corresponding to the input image) looks like. As discussed before, solve the two components coevolutionarily by iteratively applying various influences towards common solution.
The key idea is train an adversarial discriminator D discriminate between the different ways to obtain pairs of images from A (identity, p2v-S/--2aa, a2b-b2a) and B (p2v-SR, a2b). This allows the three components p2v, a2b and 62a to trained to fool D. In designing such strategy. we exploited the following insights:
Since pix2pix networks are lazy and their capabilities are restricted, the discriminator can be implicitly trained in a way that the content between pairs of images (A and B) will be similar. The rationale behind this is the following: for the pix2pix x to hold the cycle-consistency of p2v-SR-b2 it is much easier for the image translator to only do style-transfer from content-wise similar image than to reconstruct the input from a smooth rendering of different object.
To let the discriminator know what a general smoothly rendered image should look alike, we train it by rendering randomly guessed 3D models. After doing so, the discriminator can be used to train a2b to output images from B.
Training a conventional GAN is a relatively straightforward task, since only a single binary decision (real vs. fake) has to be taken. Training the RAN, shown in Fig. 4 is much more convoluted, since instead of only a single binary decision two decisions have to be made: one between three choices (A-input real fake generated by p2v-SR-b22 vs. a25- (2b) and one between two choices (B-input only fake generated by p2v-SR vs. a2b). These paths represent all possibilities to obtain an image of A respectively B-i.e., the diseriminator has differentiate between all possible ways to generate its input and thus fooling the discriminator leads to common solution for all these paths.
Since it is easier to train binary discriminator, our solution is break the RAN into five sub-RANs. which all have to take only a single binary decision (real vs. fake), as depicted in Figure II] These sub-RANs are alternately trained like conventional GANs by training their discriminator to differentiate between the "real" and "fake" input and training the other modules to fool the
A-
ST
pdr
D
(a) Lab-RAN 1
(b) sub- ROD--RAN2 2
(c) Bub-RANN3 3
Figure 11: The five sub-RANs constituting the RAN. Sub-RAN 1 (a) has discriminator switch at differentiating between p2v+SR and a26. The other four routes differentiate between two images from space A, while the "real" input for the discriminator is always the input image. The "fake" input is the result of a round trip that either involves P2v+5R and b2a (b, c) or a2h and 52a (d. e). This round trip is required for cycle-consistency loss that diminishes mode collapse. Meanwhile, the B input is the result of either P2v+SR (b, d) or abb (c, e). By that, p2v and a2h can be trained mutually.
C.C.A
62s
D
a U.C.A.
D
(d) Sub-RAN 4
(e) Sub-RANS 5
input B,
15

---

Under review as a conference paper at ICLR 2020
Input
Prediction
Altemative View
Inpor (only 2 of
Prediction (only of 4) Alternative View
Inpur (only 2 of
Prediction (only 2 Altemative View
Figure 12: Four-view reconstruction of the ShapeNet (Chang et al. (2015) classes airplane, car, and sofa. Left: input. Middle: prediction from the same angles. Right: predictions from alternative viewpoints.
discriminator. If there two modules to be trained at once, the training is split into two steps: the module next to the discriminator trained first and the one after the input is trained second (e.g.. 82a, which is close to the discriminator, is trained first, and a2b is trained second). This helps to avoid mode collapse. Since the relevance of these sub-RANs differs, their influence is weighted. For example, training a path with the pix2vex module (sub-RAN (a) in Figure [1] carries more weight than training the cycle of the two image translators (sub-RAN (c)). For the discriminator D, we use the binary cross-entropy loss. For training, an L loss between any two images of the same image space is applied as defined in Section 3.3
This constitutes the RAN an unsupervised way to find an appropriate internal representation which in turn requires the pix2pix x networks to perform a minimum of content-wise changes.
C RESULTS AND CONCLUSION
We evaluate our reconstruction results on synthetic as well camera-captured images. While using synthetic images allows highly controlled experiments, the training and evaluation based on camera-captured images demonstrates that our approach can be applied to real-world scenarios.
For creating synthetic images, used the ShapeNet dataset (Chang al. (2015) of categorized 3D meshes that has also been used for many other 3D reconstruction tasks. We rendered the 3D meshes using Blender with resolution of typically 128 x 128 pixels and from multiple directions by using lighting hyperparameters different from the lighting hyperparameters that we used in the SR the RAN. This avoids unintended implicit supervision of the process.
16

---

Under review as a conference paper at ICLR 2020
Input
Pred.
Alt. view
Input
Prod.
Alt. view
Figure 13: Single-view reconstruction results from the UT Zappos50K dataset (camera-captured images).
In the general case we used sets of images from four azimuths (Aazimuth = 90%) for our training. The results for this setting are shown in Fig. [2] In this case, the viewpoints are from altitudes which are not contained in the training data, e.g., the training images of cars were always from an altitude of thus, a perfect reconstruction of diagonal view of the models is harder. Since our smooth renderer does not consider shadows, reconstructing the sofa in line 5 especially hard.
In addition, we conducted studies on modified settings as presented and described in Fig. 14 In (a-c), we performed view trainings with the following modifications to the training data: In (a), we randomized the azimuth of the images with standard deviation of 5 In subfigure (b), randomly assigned the position of the light source for each set of images. In subfigures (cl) and (c2), we trained on the and airplane classes simultaneously.
In (d) and (e), we randomized, but supervised, the difference between azimuths for the four images. L.e. if multiple images have the same azimuth, the input data is effectively three fewer images. In (dI) and (d2), we predicted the images from only two input images; in (el-e4), we show single-view reconstructions. Since these reconstructions are trained on single resp. dual view only, the quality of entirely unseen parts of the reconstruction is lower.
Training was performed on GTX Titan Xp GPUs on the basis of Float32. In our experiments, used a uniform sphere with 162 vertices and 320 faces as base model. The networks for the presented results have been trained for between one and three weeks on a single GTX Titan Xp.
When processing low resolution images (64 x 64 pixels) combined with high mesh resolution (642 vertices, 1280 faces) faces have sub-pixel size. Thus, it occurs that single vertices dissociate themselves from the mesh since they are not visible any more. The problem of dissociating vertices is even worse if, instead of considering the directed distance to edges the faces only the distance to the faces is considered in the smooth visibility test.
For training on camera-captured images, we used single-view images of shoes (Yu & Grauman (2014)). Since these images are all typically taken from more or less the same direction, we use mirrored versions of the images and pretend this would be the view from the other side. We employed this small trick since many objects such as shoes commonly roughly symmetric. Moreover, the back of the shoe could not be reconstructed without even having any training sample from the back side. Since this problem is highly ill-posed, results could still be improved nevertheless,, they
17

---

Under review as a conference paper at ICLR 2020
the first of their kind. Figure 13 presents the results of this single-view 3D mesh reconstruction which was trained on camera-capuuue camera-aptuurd images from single direction.
In the proceeding experiments, we have demonstrated a robust way to reconstruct 3D geometry from only 2D data, eliminating the need for ground truth 3D models, or prior knowledge regarding materials and lighting conditions. In addition, we have demonstrated how a globally differentiable renderer is crucial the learning process-even if designing one induces differences the appearance of the produced renderings. We alleviate this difference through the use of image domain translation. The success of the reconstruction is driven by restriction of the information flow and by the laziness of pix2pix ix networks, which easily perform image-style exchanges but struggle in changing the content an image-a property that we exploit. Thus, our approach not informed by data but instead by an understanding of the real world.
The presented experiments indicate that our RAN architecture is suitable more than just its current application of 3D reconstruction, but instead to variety of inverse problems.
In addition, given the right training dataset, we believe the performance of the RAN for camera-captured single-view image 3D reconstructions could be significantly improved.
D IMPLEMENTATION DETAILS
D.1 REGULARIZATION LOSSES
The regularization losses (Greg) on the reconstructed meshes with descending relevance are:
The angle of normals of adjacent faces should be as similar as possible (loss uses the L norm).
The lengths of edges should be as similar as possible (loss uses the L norm).
The distance the mean vertex of adjacent vertices should be as small possible to imply regular mesh and also reduce the curvature of the mesh (loss uses the L norm).
D.2 NETWORK ARCHITECTURES
Here, we describe the topologies of the components p2v, a2b, 62a, and D. our experiments we typically used an image resolution of n x 128 x 128 and number of vertices v 162 we will base the following details on that assumption.
Let Ck denote Convolution Seavolution-LeakyRlllee layer with filters of size 4 x 4 and stride of 2. Let the negative slope of the LeakyReLU be 0.2. From the fifth convolutional layer on, we apply a 30% dropout.
The pix2pix network is symmetric residual network with the following blocks defining the first 2-
half: 21111222---0000112-511--22-----1
pix2vex is based on the pix2pix network. It is followed by two fully connected layers (n2 v)) and the sigmoid function.
243 and
+3
a2b and b2a are pix2 x22pix networks with the first two or more residual layers. This is followed by the sigmoid function.
The discriminator D is defined C64 CC128 C256 001--551--00 followed by the sigmoid function.
18

---

Under review as a conference paper at ICLR 2020
Iaput (only 2 of
Pred. fonly 4)
Alt. view
(el)
(c2)
Pred.
Alt. view
(dl)
(d2)
Input
Pred.
Alt, view
Input
Pred.
Alt. view
(+1)
(o2)
(c3)
(o4)
Figure 14: Experiments showing the robustness of our approach. (a-c): four-view training with the following modifications to the training data: (a) randomized azimuth of the images; (b) randomly assigned position of the light source; (c) simultaneous training of car and airplane classes; (d) predicting images from only two input images; (e) single-view reconstructions.
19


Reviews:
[{"review_id": "rylxfnmRFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper conceptualizes a neural network architecture that integrates smoothed versions of traditional algorithms into the network topology. The AlgoNet concept employs smoothed versions of algorithms implemented in the WHILE language, with options for different levels of differentiability. The paper outlines a forward version of AlgoNet based on traditional, skip, and residual connections, and a backward version for solving inverse problems in a reconstructive, autoencoder-like fashion. As smoothing introduces a form of domain shift, the paper the reconstructive adversarial network that that employs \"domain translators\" as well as a discriminator that are trained in an adversarial fashion. The paper concludes by describing versions of AlgoNet for various different algorithms.\n\nThe general idea of being able to better control the behavior of neural networks by better leveraging known structure (e.g., algorithms) is appealing. However, the paper does not go beyond conceptualizing how this might be done in a hand wavy manner. It is difficult to see what if anything can be learned from the paper, let alone what practical utility it has, which is important given that the paper claims to propose a new neural network architecture.\n\nThe paper would benefit from a discussion of empirical results in the main text, with baseline comparisons. With the exception of a single figure, the details are relegated to the appendices.\n\nThe related work discussion is surprisingly short, given the attention that has been paid to designing/optimizing different network topologies. The work that is discussed is very narrow in scope (see below).\n\nThe  paper devotes too much discussion to the challenges introduced by non-differentiable layers and the advantages of increasing degrees of differentiability (this is about half of the intro and most of the related work).\n\nDespite the lack of experimental demonstrations, the paper is one page over the suggested 8 page limit.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rylUQbwIiB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes \"AlgoNets\", which are differentiable implementations of classical algorithms. Several AlgoNets are described, including multiplication algorithm implemented in the WHILE programming language, smooth sorting, a smooth while loop, smooth finite differences and a softmedian.\n\nThe paper additionally presents RANs (similar to GANs but with an AlgoNet embedded) and Forward AlgoNets (where the the AlgoNet is embedded in a feedforward net). \n\nThe smooth implementations normally amount to replacing hard functions with soft equivalents, for example \"if\" conditions are replaced by logistic sigmoids.\n\nThe research direction in this paper is very interesting and could lead to important advancements, however a strong argument needs to be presented to the readers about why this way of making algorithms smooth is better than other published or obvious techniques.\n\nThe argument could be theoretical, proving for example faster convergence under certain assumptions, or it could be empirical, showing that the method achieves better results than other techniques on some benchmarks. I could not see however any such arguments in this paper.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BylqVJ3pKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a few different results surrounding smooth relaxations of classical algorithms. While the ideas (particularly the smooth rendering system) are interesting, the limited nature of the experiments and the lack of comparisons with baselines or prior work makes it difficult for me to support acceptance.\n\nSome feedback:\n\n- Re: \"the output of C∞ smooth WHILE-programs differs from the discrete WHILE-programs by a small factor.\" Can you talk about the math here? Maybe mention bump functions (some of your relaxations have infinite support, and others have finite support, i.e. they use bump functions--this is likely to be an important distinction in practice).\n\n- I'd want to see examples of softsort in action; especially examples that help me understand that the gradients are meaningful.\n\n- The smooth renderer, and in particular the advantages it has over existing differentiable renderers, seem like the most important contributions of the paper (although they're relegated to the appendix). Can you expand on what you mean by \"fully\" vs \"locally\" differentiable? Can you provide empirical comparisons with other differentiable renderers?\n\n- Can you optimize a scene with an unknown or variable number of triangles?\n\n- I'm skeptical that all of the complexity of the RAN architecture and training setup is necessary. Is it possible to compare with other options, including baselines that leave out one or more of the components or training losses?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper does not provide theory or experiment to justify the various proposed relaxations. In its current form, it has very limited scope.
Decision: Reject