Title: DENSE MORPHOLOGICAL NETWORK: AN UNIVERSAL FUNCTION APPROXIMATOR

Abstract: Artificial neural networks are built on the basic operation of linear combination and non-linear activation function. Theoretically this structure can approximate any continuous function with three layer architecture. But in practice learning the parameters of such network can be hard. Also the choice of activation function can greatly impact the performance of the network. In this paper we are proposing to replace the basic linear combination operation with non-linear operations that do away with the need of additional non-linear activation function. To this end we are proposing the use of elementary morphological operations (dilation and erosion) as the basic operation in neurons. We show that these networks (Denoted as DenMo-Net) with morphological operations can approximate any smooth function requiring less number of parameters than what is necessary for normal neural networks. The results show that our network perform favorably when compared with similar structured network.

1. INTRODUCTION:

In artificial neural networks, the basic building block is an artificial neuron or perceptron that simply computes the linear combination of the input (Rosenblatt, 1958) . It is usually followed by a non-linear activation function to model the non-linearity of the output. Although the neurons are simple in nature, when connected together they can approximate any continuous function of the input (Hornik, 1991) . This has been successfully utilized in solving different real world problems like image classification (Krizhevsky et al., 2012) , semantic segmentation (Long et al., 2015) and image generation (Isola et al., 2017) . While these models are quite powerful in nature, their efficient training can be hard in general (LeCun et al., 2012) and they need support of specials techniques, such as batch normalization (Ioffe & Szegedy, 2015) and dropout (Srivastava et al., 2014) , in order to achieve better generalization capabilities. Their training time also depends on the choice of activation function (Mishkin et al., 2017) .

In this paper we are proposing new building blocks for building networks similar to neural network. Here, instead of the linear combination operation of the artificial neurons, we use a non-linear operation that eliminates the need of additional activation function while requiring a small number of neurons to attain same performance or better. More specifically, We use morphological operations (i.e. dilation and erosion) as the elementary operation of the neurons in the network. Our contribution in this paper is building a network with these operations that has the following properties.

1. Networks built with with dilation-erosion neurons followed by linear combination can approximate any continuous function given enough dilation/erosion neurons. 2. As dilation and erosion operation are non-linear by themselves, requirement of separate non-linear activation function is eliminated. 3. The use of dilation-erosion operation greatly increases number of possible decision boundaries. As a result, complex decision boundaries can be learned using small number of parameters.

The rest of the paper is organized as follows. Section 2 describes the prior work on morphological neural network. In Section 3, we introduce our proposed network and prove its capabilities theoretically. We further demonstrate its capabilities empirically on a few benchmark datasets in Section 4. Lastly Section 6 concludes the paper.

2. RELATED WORK:

Morphological neuron was first introduced by Davidson & Hummer (1993) in their effort to learn the structuring element of dilation operation in images. Similar effort has been made to learn the structuring elements in a more recent work by Masci et al. (2013) . Use of morphological neurons in a more general setting was first proposed by Ritter & Sussner (1996) . They restricted the network to a single layer architecture and focused only on binary classification task. To classify the data, these networks use two axis parallel hyperplanes as the decision boundary. This single layer architecture of Ritter & Sussner (1996) has been extended to two layer architecture by Sussner (1998) . This two layer architecture is able to learn multiple axis parallel hyperplanes, and therefore is able to solve arbitrary binary classification task. But, in general the decision boundaries may not be axis parallel, as a result this two layer network may need to learn a large number of hyperplanes to achieve good results. So, one natural extension is to incorporate the option to rotate the hyperplanes. Taking a cue from this idea, Barmpoutis & Ritter (2006) proposed to learn a rotational matrix that rotates the input before trying to classify the data using axis parallel hyperplanes. In a separate work by Ritter et al. (2014) the use of L 1 and L ∞ norm has been proposed as a replacement of the max/min operation of dilation and erosion in order to smooth the decision boundaries.

Ritter & Urcid (2003) first introduced the dendritic structure of biological neurons to the morphological neurons. This new structure creates hyperbox based decision boundaries instead of hyperplanes.

The authors have proved that with hyperboxes any compact region can be estimated, therefore any two class classification problems can be solved. A generalization of this structure to the multiclass case has also been done by (Ritter & Urcid, 2007) . Sussner & Esmi (2011) had proposed a new type of structure called morphological perceptrons with competitive neurons, where the output is computed in winner-take-all strategy. This is modelled using the argmax operator and this allows the network to learn more complex decision boundaries. Later Sossa & Guevara (2014) proposed a new training strategy to train this model with competitive neurons.

The non-differentiability of the max-min operations has forced the researchers to propose specialized training procedures for their models. So, a separate line of research has attempted to modify these networks so that gradient descent based optimizer can be used for training. Pessoa & Maragos (2000) have combined the classical perceptron with the morphological perceptron. The output of each node is taken as the convex combination of the classical and the morphological perceptron. Although max/min operation is not differentiable, they have proposed methodology to circumvent this problem. They have shown that this network can perform complex classification tasks. Morphological neurons have also been employed for regression task. de A. Arajo (2012) has utilized network architecture similar to morphological perceptrons with competitive learning to forecast stock markets. The argmax operator is replaced with a linear function so that the network is able to regress forecasts. The use of linear activation function enables the use of gradient descent for training which is not possible with the argmax operator. For morphological neurons with dendritic structure Zamora & Sossa (2017) had proposed to replace the argmax operator with a softmax function. This overcomes the problem of gradient computation and therefore gradient descent is employed to train the network. So, this retains the hyperbox based boundaries of the dendritic networks, but facilitates easy training with gradient descent.

3. DENSE MORPHOLOGICAL NETWORK:

In this section we introduce the basic components and structure of our network and establish its approximation power.

3.1. DILATION AND EROSION NEURONS:

Dilation and Erosion are two basic operations of our proposed network. Given an input x ∈ R d and some structuring element s ∈ R d+1 , dilation (⊕) and erosion ( ) neurons computes the following two functions respectively Where x = [x, 0] and x k denotes the k th component of vector x . The 0 is appended to the input x to take care of the 'bias'. Here we try to learn the structuring element (s). Note that erosion operation can also be written in the following form.

EQUATION

x s = min k (x k -s k ). (2) ⊕ x s z + L ∂ ∂z + ∂s L ∂ ∂s L ∂ ∂z + ∂z + =

x s = -max k (s k -x k )

(3)

3.2. GRADIENT OF DILATION AND EROSION:

Artificial neural networks are trained using back-propagation. To be able to use the dilation/erosion neurons as a drop-in replacement of the artificial neurons, we must be able to compute the gradient of this operation. Here we show the gradient of the dilation operation. For erosion the gradient computation will be similar. Figure 1 shows the computational graph model of dilation operation and its gradient flow. Lets assume L is the loss, we are trying to optimize. We need to compute ∂L ∂s to be able to update the structuring element. This can be computed using the chain rule as follows.

EQUATION

Here ∂L ∂z + will be the input gradient to this node which will be available from the node following this node. We need to compute ∂z + ∂s . Now as s ∈ R d+1 we can write the following.

EQUATION

Now for each ∂z + ∂si the gradient is computed as follows,

EQUATION

)

3.3. NETWORK STRUCTURE:

The Dense Morphological Net or 'DenMo-Net', in short, that we propose here is a simple feed forward network with some dilation and erosion neurons followed by classical artificial neurons (Figure 2 ). We call the layer of dilation and erosion neurons as the dilation-erosion layer and the following layer as the linear combination layer. Let's assume the dilation-erosion layer contains n dilation neurons and m erosion neurons, followed by c neurons in the linear combination layer. Let x ∈ R d is the input to the network. Let z + i and z - j be the output of i th dilation neuron and j th erosion node, respectively. Then we can write,

EQUATION

- j = x s - j (8)

where, s + i and s - j are the structuring elements of the i th dilation neuron and j th erosion neuron respectively. Note that i ∈ {1, 2, . . . , n} and j ∈ {1, 2, . . . , m}. The final output from a node of the linear combination layer is computed in the following way.

EQUATION

where ω + i and ω - j are the weights of the artificial neuron in the linear combination layer. In following subsection we show that g(x) can approximate any continuous function f : R d → R. Here we show that with the linear combination of dilation and erosion, any function can be approximated, and the approximation error decreases with increase in the number of neurons in the dilation-erosion layer. Before that we need to describe some concepts.

g 1 (x) g 2 (x) g c (x) z 1 z n z 1 z m Dilation Neurons ⊕ ⊕ ⊖ ⊖ Erosion Neurons Dilation-Erosion Layer x 1 x 2 x d x d+1

Definition 1 (k-order Hinge Function (Wang & Sun, 2005 )) A k-order hinge function consists of (k + 1) hyperplanes continuously joined together. it is defined by the following equation,

EQUATION

)

Definition 2 (d-order hinging hyperplanes (d-HH) (Wang & Sun, 2005 )) A d-order hinging hyperplanes (d-HH) is defined as the sum of multi-order hinge function as follows,

EQUATION

with α i ∈ {-1, 1}, k i ≤ d.

From Wang & Sun (2005) the following can be said about hinging hyperplanes.

Proposition 1 For any given positive integer d and arbitrary continuous piece-wise linear function f : R d → R, there exists finite, say N , positive integers η(k) ≤ d+1, 1 ≤ k < N and corresponding

EQUATION

)

This says that any continuous piece-wise linear function of d variables can be written as an d-HH, i.e. the sum of multi-order hinge functions. Now to show that our network can approximate any continuous functions, we show the following.

Lemma 1 g(x) is sum of multi-order hinge functions.

The proof of this lemma is given in Appendix A. Basically we show that g(x) can written as the sum of l hinge functions in the following form.

EQUATION

where l = m + n (number of neurons in the dilation-erosion layer), α i ∈ {1, -1} and φ i (x)'s are d-order hinge function.

Proposition 2 (Stone-Weierstrass approximation theorem) Let C be a compact domain (C ⊂ R d ) and f : C → R a continuous function. Then there exists a continuous piece wise linear function g such that for all x ∈ C, |f (x) -g(x)| < for some > 0.

Theorem 1 (Universal approximation) Only a single dilation-erosion layer followed by a linear combination layer can approximate any continuous smooth function provided there are enough nodes in dilation erosion-layer.

Sketch of Proof:

From lemma 1 we know that our DenMo-Net with of n dilation and m erosion neurons followed by a linear combination layer computes g(x), which is a sum of multi-order hinge functions. Now from proposition 1 we get that any continuous piecewise linear function can be written by a finite sum of multi-order hinge function. Now from Proposition 2 we can say that any continuous function can be well approximated by a piecewise linear function. In general if l → ∞ then → 0. If we increase the number of neurons in the dilation-erosion layer the approximation error decreases. Therefore, we can say that a DenMo-Net with enough dilation and erosion neurons can approximate any continuous function.

3.5. LEARNED DECISION BOUNDARY:

The DenMo-Net we have defined above learns the following function,

EQUATION

Where each φ(x) is collection of multiple hyperplanes joined together. Therefore the number of hyperplanes learned by the network with l neurons in the dilation-erosion layer is much more than l. Each morphological neuron allows only one of the inputs to pass through because of max / min operation after addition with the structuring element. So, effectively each neuron in the dilation-erosion layer chooses one component of the d-dimensional input vector. Depending on which component is being chosen, the final linear combination layer computes the hyperplane by taking either all the components of the input or only some of them (when a subset of input components is chosen more than once in the dilation-erosion layer). Note that this choice depends on the input and the structuring element together. For a network with d dimensional input data and l neurons (l ≥ d) in the dilation-erosion layer, theoretically (d + 1) l -1 hyperplanes can be formed in d dimension. Out of the all possible planes only l P d × (d + 1) l-d planes can span anywhere in the d dimensional space. Therefore, increasing the number of neurons in the dilation-erosion layer exponentially increases the possible number of hyperplanes, i.e., the decision boundaries. This implies that, using only a small number of neurons, complex decision boundaries can be learned.

4. RESULTS:

Here we empirically validate the power of our DenMo-Net and demonstrate its advantages in comparison with other networks like artificial neural networks with different activation functions i.e. tanh (NN-tanh) and ReLU (NN-ReLU) and Maxout network (Goodfellow et al., 2013) . As our network is defined with all possible connections between two consecutive layers, we have compared with only similar structured networks. We have chosen the maxout network for comparison, because it uses the max function as a replacement of the activation function but with added nodes to compute the maximum. The experiments have been carried out on a toy dataset with two concentric circles for visualizing the decision boundaries and also on benchmark datasets like MNIST (LeCun et al., 1998) , Fashion-MNIST (Xiao et al., 2017) , CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton, 2009) . For all the tasks we have used categorical cross entropy as the loss and in the last layer softmax function is used. In the training phase, all the networks have been optimized using Adam (learning rate= 0.001, β 1 = 0.9, β 2 = 0.999) optimizer (Kingma & Ba, 2014) with mini batches of size 32. In all the experiments, we have used same number of dilation and erosion neurons in dilation-erosion layer unless otherwise stated.

Class 2 Class 1 Legend:

-1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 -1.5 as it introduces extra parameters with max function to achieve non-linearity. In the maxout layer we have taken maximum among h = 2 features. As we see in the figure 3b the network learns (2 * h =) 4 straight lines when trying to classify these data. For the same data and two neurons in dilation-erosion layer, our DenMo-Net has learned 6 lines to form the decision boundary (figure 3c ).

Although from equation 14 we can say that we get at most 8 lines, only two of them can be placed anywhere in the 2D space while others are parallel to the axes. For this reason, we are getting two slanted lines and the remaining lines are parallel to the axes. The classification accuracy achieved by the networks along with their number of parameters is reported in table 1. The difference in the accuracy clearly shows the power of DenMo-Net.

4.2. MNIST DATASET:

MNIST dataset (LeCun et al., 1998) contains gray scale images of hand written numbers (0-9) of size 28 × 28. It has 60,000 training images and 10,000 test images. Since our network does not support two dimensional input, we have converted each image to a column vector (in row major order) before giving it as input. The network we use follows the structure we have previously defined: input layer, dilation-erosion layer and linear combination layer computing the output. As in this dataset we had to distinguish between 10 classes of images, 10 neurons are taken in the output layer. In table 2 we have shown the accuracy on test data after training the network for 150 epochs with different number of nodes (l) in the dilation-erosion layer. The change of test accuracy over the epochs is shown in figure 4a . It is seen that increasing number of nodes in the dilation-erosion layer helps to increase non-linearity, and thus it results in better accuracy on test data. We get test average accuracy of 98.43% after training 3 times with the DenMo-Net of 200 dilation and 200 erosion neurons (Table 3 ) up to 400 epochs. We have also experimented when only dilation, only erosion and both type of neurons were used in the dilation-erosion layer (Figure 4b ). We see that using both erosion only and both dilation and erosion neurons giving better accuracy. The Fashion-MNIST dataset (Xiao et al., 2017) has been proposed with the aim of replacing the popular MNIST dataset. Similar to the MNIST dataset this also contains 28 × 28 images of 10 classes and 60,000 training and 10,000 testing samples. While MNIST is still a popular choice for benchmarking classifiers, the authors' claim that MNIST is too easy and does not represent the modern computer vision tasks. This dataset aims to provide the accessibility of the MNIST dataset while posing a more challenging classification task.

For the experiment, we have converted the images to a column vector similar to what we have done for the MNIST dataset. We have taken 400 dilation and 400 erosion nodes in the dilation-erosion layer for this experiment. We have trained the network separately 3 times up to 300 epochs. The reported test accuracy (Table 3 ) is the average of the 3 runs. We see that our method gives better results.

4.4 CIFAR-10 DATASET CIFAR-10 ( Krizhevsky & Hinton, 2009 ) is natural image dataset with 10 classes. It has 50,000 training and 10,000 test images. Each of them is a color image of size 32 × 32. The images are converted to column vector before they are fed to the DenMo-Net. For all the networks we compare with, the experiments have been conducted with keeping the number of neurons same in the hidden layer. For maxout network each hidden neuron have two extra nodes over which the maximum is computed. In table 4 we have reported the average test accuracy obtained over three run of 150 epochs. The change of accuracy over epochs is also shown in figure 5a when number of hidden neurons is 600. As it can be seen from both the table and the figure that DenMo-Net achieves the best accuracy in all the cases. Maxout network lags behind even with more number of parameters. This happens because our network is able to learn more hyperplanes with number of parameters similar to normal artificial neural networks. When using only a single type of neurons in our network, we see a different result for this dataset (Figure 5b ). The network takes time to learn with only erosion neurons. The situation improves a little when using only dilation neurons. When using both type of morphological neurons, the network is able to perform better by leveraging the power of both the operations.

4.5 CIFAR-100 DATASET CIFAR-100 (Krizhevsky & Hinton, 2009) 6b ). Using only erosion neurons, the networks lags behind in terms of accuracy. But interestingly, using erosion only the network is able to perform better than using both type of morphological neurons.

5. DISCUSSIONS:



5.1. GRADIENT PROPAGATION:

In our network we are learning the structuring element and the weights of the linear combination layer using gradient descent method. While learning the structuring elements we may encounter two kinds of problem. Dilation/erosion operation involves max/min operation. As shown in Section 3.2, the gradient of the loss with respect to the structuring element contains a single non-zero element. Therefore, only a single element of the structuring element is updated. So, the learning can be slow.

On the other hand, some values of the structuring element may not get updated at all.

5.2. STACKING MULTIPLE LAYERS:

We have defined the network and have shown its properties when only three layers are employed in the network. Although it may seem stacking multiple of these layers would translate to better results, the results are showing the opposite. Straight-forward stacking of the layers we have used in our network can give rise to two kinds of network.

Type-I Multiple dilation-erosion layer, followed by a single linear combination layer at the end.

Type-II Dilation-Erosion layer followed by a linear combination layer. Then another dilationerosion layer followed by one more linear combination layer and so on.

For the network of Type-I, it can be argued that the network is performing some combination of opening and closing operation, and their linear combination. As there are dilation-erosion (DE) layers back to back, the problem of gradient propagation is amplified. As a result it takes much more time to train than single layer architecture (Table 6 ).

Similar explanation doesn't work for Type-II networks. From Figure 7 we see that the network has tendency to overfit. We believe its understanding requires further exploration.

6. CONCLUSION:

In this paper we have proposed a new class of networks that uses both normal and morphological neurons. These network consists of three layers only: input layer, dilation-erosion layer with dilation and erosion neurons followed by linear combination layer giving the output of the network with normal artificial neurons. We have done our analysis using this three layer network only, but its deeper version can also be explored. We have shown that this three layer architecture can approximate any APPENDIX A PROOF OF LEMMA 1

From equation 9 we have

EQUATION

Now this equation can be rewritten as follows

EQUATION

where s + ik and s - ik denote the k th component of the i th structuring element of dilation and erosion neurons, respectively. The above equation can be further expressed in the following form,

EQUATION

Where β + i , β - i , d + ik and d - ik are define in the following way

β + i = ω + i if ω + i ≥ 0 -ω + i if ω + i < 0 β - i = -ω - i if ω - i ≥ 0 ω - i if ω - i < 0 d + ik = s + ik ω + i if ω + i ≥ 0 -s + ik ω + i if ω + i < 0 d - ik = s - ik ω - i if ω - i ≥ 0 -s - ik ω - i if ω - i < 0 α + i = 1 if ω + i ≥ 0 -1 if ω + i < 0 α - i = -1 if ω - i ≥ 0 1 if ω - i < 0

Now, without any loss of generality we can write equation 17 as follows

EQUATION

where

β i = β + i if i ≤ n β - i-n if n < i ≤ m + n d ik = d + ik if i ≤ n d - (i-n)k if n < i ≤ m + n α i = α + i if i ≤ n α - (i-n) if n < i ≤ m + n

Finally, we can rewrite equation 18 as

EQUATION

where l = m + n, α i ∈ {1, -1} and φ i (x)'s are of the following form

EQUATION

with

EQUATION

In equation 20, v T ik x + d ik is affine and α i φ i (x) is a d-order hinge function. Hence l i=1 α i φ i (x) i.e., g(x) represents sum of multi-oder hinge function.

:



Figures and Tables:

Figure 1: Forward and backward pass through a node computing dilation

Figure 2: Single Layer DenMo-Net with n dilation and m erosion neuron and c output neurons

Figure 3: Decision boundaries of different networks

Figure 4: Test accuracy achieved over epochs in the MNIST dataset

Figure 6: Test accuracy over epochs on CIFAR-100 dataset

Training accuracy achieved on the circle dataset by different networks

Table content: <table><tr><td>Methods</td><td colspan="3">Hidden nodes Parameters Training accuracy</td></tr><tr><td>NN-ReLU</td><td>2</td><td>12</td><td>68.87</td></tr><tr><td>NN-tanh</td><td>2</td><td>12</td><td>69.10</td></tr><tr><td>Maxout Network (h=2)</td><td>2</td><td>18</td><td>87.17</td></tr><tr><td>DenMo-Net</td><td>2</td><td>12</td><td>91.6</td></tr><tr><td colspan="2">4.1 VISUALIZATION WITH A TOY DATASET</td><td/><td/></tr></table>

Accuracy on MNIST dataset with different architectures

Table content: <table><tr><td>Neurons in dilation-erosion layer (l)</td><td>10</td><td>50</td><td>100</td><td>200</td></tr><tr><td>Test Accuracy</td><td colspan="4">76.35 93.38 95.51 96.85</td></tr><tr><td>4.3 FASHION-MNIST DATASET</td><td/><td/><td/><td/></tr></table>

Achieved accuracy in the test set

Table content: <table><tr><td>Dataset</td><td>DenMo-Net</td><td>Test accuracy State of the art</td></tr><tr><td>MNIST</td><td colspan="2">98.43 (l = 400) 99.79 (Wan et al., 2013)</td></tr><tr><td colspan="3">Fashion-MNIST 89.87 (l = 800) 89.70 (Xiao et al., 2017)</td></tr></table>

Test accuracy achieved on CIFAR-10 dataset by different networks

Table content: <table><tr><td>Architecture</td><td colspan="6">l=200 parameters accuracy parameters accuracy parameters accuracy l=400 l=600</td></tr><tr><td>NN-tanh</td><td>616,610</td><td>48.88</td><td>1,233,210</td><td>49.39</td><td>1,849,810</td><td>51.24</td></tr><tr><td>NN-ReLU</td><td>616,610</td><td>49.28</td><td>1,233,210</td><td>50.43</td><td>1,849,810</td><td>52.25</td></tr><tr><td colspan="2">Maxout-Network 1,231,210</td><td>49.51</td><td>2,462,410</td><td>50.10</td><td>3,693,610</td><td>51.51</td></tr><tr><td>DenMo-Net</td><td>616,610</td><td>51.84</td><td>1,233,210</td><td>53.41</td><td>1,849,810</td><td>54.49</td></tr></table>

is a image dataset similar to CIFAR-10 but with 100 classes with 600 images in each. There are 500 training and 100 testing images for each class. The training has been done similar to what is done for CIFAR-10. Network has been trained with batch size 100. We have reported the average test accuracy of 3 run with 100 epochs each in table 5. The change of test accuracy over the epochs is plotted in figure6a. The results show trend similar to what is observed in other dataset. DenMo-Net is giving better result with comparable number of trainable parameters and trains much faster. When the type of neurons is restricted in our network, the results are similar to what we have seen for CIFAR-10 dataset (Figure

Table content: <table/>

Comparison with Baseline CIFAR100

Table content: <table><tr><td>Architecture</td><td colspan="6">l=200 parameters accuracy parameters accuracy parameters accuracy l=400 l=600</td></tr><tr><td>NN-tanh</td><td>634,700</td><td>19.50</td><td>1,269,300</td><td>19.62</td><td>1,903,900</td><td>20.46</td></tr><tr><td>NN-ReLU</td><td>634,700</td><td>17.83</td><td>1,269,300</td><td>19.63</td><td>1,903,900</td><td>20.77</td></tr><tr><td colspan="2">Maxout-Network 1,249,300</td><td>21.58</td><td>2,498,500</td><td>21.49</td><td>3,747,700</td><td>21.69</td></tr><tr><td>DenMo-Net</td><td>634,700</td><td>23.65</td><td>1,269,300</td><td>25.89</td><td>1,903,900</td><td>26.93</td></tr></table>

Peer review (iclr2019):
 * Update: Thanks for you answer and clarification. While the Morph-net appears novel, the authors only report result for image classification task and don*t achieve as good performance as standard convolutional baselines. Given the current empirical evaluation, I find hard to assess how significant is the contribution. I would encourage the authors to either compare on a task where dense networks achieve state-of-art performances or extend their approach to 2D inputs. * Review This paper introduces Morph-Net, a new architecture that intertwine morphological operator such as dilation/erosion with linear layer. Authors first show than Morph-Net are universal approximator. Morph-Net can be expressed as a sum of multi-order hinge functions which can approximate any continuous function. They then validate empirically the Morph-Net on MNIST, FashionMNIST, CIFAR10 and CIFAR100 datasets. In particular, authors investigate a 3 layers fully-connected Morph-Net and shows that it can outperform its Tanh/Relu/Maxout counterparts. The paper is a nice read also some specific point could be clarify. For instance it is not clear how the structuring elements of the dilation/erosion are learned? Are the learned simply through backpropagation? Also, it is not clear to me how Morph-Net differs from the previously proposed morphological neurons? Empirical evaluation of Morph-Net could be improved as well. In particular, authors focus on image classification task. While they show that Morph-Net can outperform other fully connected architecture, the results on CIFAR10/100 seems low compared to convolutional network. It raises the question of the advantages of Morph-Net over convolutional neural networks ? Authors also limit their exploration to 3-layer networks. Why don’t you explore deeper network for both baseline and Morph-Net? Finally, if I am not mistaken, authors use the same set of hyperparameters for the baselines/Morph-Net? It is not clear to me if the hyperparameters are optimal for all the approach? They might give an unfair advantage to one of the baseline or Morph-Net? Overall, this paper present a nice idea. Showing the Morph-Net is an universal approximator is a nice result. However, the empirical evaluation could be improved. It is not clear to me at this point if Morph-Net brings a benefit compare to convolutional net for image classification task.