Title: GEOMETRIC OPERATOR CONVOLUTIONAL NEURAL NETWORK

Abstract: The Convolutional Neural Network (CNN) has been successfully applied in many fields during recent decades; however it lacks the ability to utilize prior domain knowledge when dealing with many realistic problems. We present a framework called Geometric Operator Convolutional Neural Network (GO-CNN) that uses domain knowledge, wherein the kernel of the first convolutional layer is replaced with a kernel generated by a geometric operator function. This framework integrates many conventional geometric operators, which allows it to adapt to a diverse range of problems. Under certain conditions, we theoretically analyze the convergence and the bound of the generalization errors between GO-CNNs and common CNNs. Although the geometric operator convolution kernels have fewer trainable parameters than common convolution kernels, the experimental results indicate that GO-CNN performs more accurately than common CNN on CIFAR-10/100. Furthermore, GO-CNN reduces dependence on the amount of training examples and enhances adversarial stability.

1. INTRODUCTION:

Convolutional Neural Networks have been successfully applied in many fields during recent decades, but the theoretical understanding of the deep neural network is still in the preliminary stages. Although CNNs have strong expressive abilities, they have two clear deficiencies. First, as complex functional mappings, CNNs, like black boxes, cannot take full advantage of domain knowledge and prior information. Second, when little data is available for a certain task, CNNs' generalization ability weakens. This is due to overfitting, which may occur due to the large number of parameters and the large model size. Stemming from these two defects, a great deal of research has been done to modify CNNs (Dai et al., 2017; Wang et al., 2018; Sarwar et al., 2017) . Before CNNs were applied, traditional geometric operators had developed quite well. Each geometric operator represents the precipitation of domain knowledge and prior information. For example, the Sobel operator (Works) is a discrete difference operator, which can extract image edge information for edge detection. The Schmid operator (Schmid, 2001 ) is an isotropic circular operator, which extracts texture information from images for face recognition. The Histogram of Oriented Gradients (HOG) (Dalal & Triggs, 2005 ) is a statistic operator of gradient direction, which extracts edge direction distributions from images for pedestrian detection and other uses.

Many computer vision tasks require domain knowledge and prior information. For example, in Cao et al. (2015) , the texture information from the image is used for an auxiliary diagnosis of a fracture. Geometric operators can make use of domain knowledge and prior information, but cannot automatically change parameter values by learning from data. Convolutional Neural Networks have strong data expression abilities and learning abilities, but they struggle to make use of domain knowledge. For better data learning, we have combined the two. It is natural to directly use geometric operators for pre-processing, and then classify the data through a Convolutional Neural Network (Yao et al., 2016) . However, this method uses human experience to select geometric operator parameter values, and then carries out the Convolutional Neural Network learning separately. This method is a kind of two-stage technique, and without reducing parameter redundancy in a Convolutional Neural Network, it is difficult to achieve global optimization. The method proposed in this paper directly constructs geometric operator convolution and then integrates geometric operator convolution into a Convolutional Neural Network to form a new framework -the Geometric Operator Convolutional Neural Network. This method achieves global optimizations and utilizes the properties of geometric operators.

In summary, the contributions of this work are as follows:

• This framework can integrates many conventional geometric operators, which reveals its broad customization capabilities when handling diverse problems.

• In theory, the same approximation accuracy and generalization error bounds are achieved when geometric operators meet certain conditions.

• The Geometric Operator Convolutional Neural Network not only reduces the redundancy of the parameters, but also reduces the dependence on the amount of the training samples.

• The Geometric Operator Convolutional Neural Network enhances adversarial stability.

2. RELATED WORK:

In recent years, Convolutional Neural Networks have been widely used in various classification and recognition applications (Krizhevsky et al., 2012; Hu et al., 2014) . Convolutional Neural Networks have achieved advanced success in various problems. All CNNs adopt an end-to-end approach to learning; however, each unique task is associated with its own distinctive domain knowledge and prior information. Thus, to improve classification accuracy, researchers use priori information that is tailored to each specific task and each specific Convolutional Neural Network. One way to do this is to use the traditional image processing algorithm as a preprocessing step. Another way is to use the traditional image processing algorithm to initialize convolution kernels.

Classification accuracy is a primary concern for researchers in the machine-learning community. Different pre-processing models, such as filters or feature detectors, have been employed to improve the accuracy of CNNs. One example of this is the Gabor filter with CNN (Daugman, 1988) . The Gabor filter is a feature extractor based on human vision. Besides the Gabor filter, some people also use Fisher vectors (Cimpoi et al., 2014) , sparse filter Banks (Pfister & Bresler, 2015) , and the HOG algorithm (Lu et al., 2018) combined with a CNN to improve accuracy. Based on the human visual system, these filters are found to be remarkably well-suited for texture representation and discrimination. In the works by Kwolek (2005) and Mounika et al. (2012) , the Gabor filter is used to extract features from the input image in a pre-processing step. However, these methods require a kind of two-stage procedure that may not reach the optimal global solution.

In addition, some scholars use traditional image processing algorithms to initialize convolutional kernels, such as building a Feature Pyramid Network with an image pyramid for multi-scale feature extraction (Lin et al., 2017) . Geometric operators are widely used in traditional image processing algorithms. Many researchers use the Gabor filter to fix the first convolution layer, while other layers, which are common convolution layers, can be trained to improve their accuracy (Yao et al., 2016; Sarwar et al., 2017) . John et al. simultaneously adopted the weight of the first layer convolution with the Gershgorin circle theorem and the Gabor filter constraint to improve the classification accuracy when Convolutional Neural Networks propagated backward. In Calderón et al. (2003) and Chang & Morgan (2014) , the authors have attempted to get rid of the pre-processing overhead by introducing Gabor filters in the first convolutional layer of a CNN. In addition, some researchers use filters to initialize multiple convolutional kernels. Lu et al. (2018) only used the Gabor function to create kernels in four directions to initialize the convolutional kernels from a Convolutional Neural Network. These methods change the initialization weight and use domain knowledge, but they do not reduce the redundancy of model parameters, and they do not enhance the transformation ability of the model.

In this paper, a new network, the Geometric Operator Convolutional Neural Network, is proposed. This method integrates geometric operators, namely the filters, into a convolutional neural network. This network can not only make use of domain knowledge and prior information, but also reduce the redundancy of network parameters and enhance the ability of model transformation. This network's construction is described in detail in the following section. , 1999) , the Roberts operator (Rosenfeld, 1981) , the Laplace operator (van Vliet et al., 1989) , the Gabor operator (Han & Ma, 2007) , and so on. 

3.2.1. GABOR OPERATOR:

In order to study the frequency characteristics of local range signals, Gabor (1946) proposed the famous "Window" Fourier transform (also called the short-time Fourier transform, STFT) in the paper "Theory of communication" in 1946. This is now known as the Gabor operator; when combined with images, it is referred to as the Gabor filter. Until now, the Gabor filter has undergone many developments, and its primary characteristics are listed below. First, the Gabor filter has the advantages of both spatial and frequency signal processing. As shown in Eqn. 1.0, the Gabor operator is essentially a Fourier transform with a gaussian window. For an image, the window function determines its locality in the spatial domain, so the spatial domain information from different positions can be obtained by moving the center of the window. In addition, since the gaussian function remains the same after the Fourier transform, the Gabor filter can extract local information in the frequency domain. Second, the Gabor filter's response to biological visual cells may be an optimal feature extraction method. In 1985 , Daugman (1985) extended the Gabor function to a 2-dimensional form and constructed a 2D Gabor filter on this basis. It was surprising to find that the 2D Gabor filter was also able to maintain consistency with the mammalian model of retinal nerve cell reception. Third, the Gabor kernels are similar to the convolution kernels from the first convolutional layer in the CNN. From the visualization of the first convolutional layer in AlexNet, which was proposed by Krizhevsky et al. (2012) . Some convolution kernels present geometric properties, as in the kernel function from the Gabor filter. From this feature, it can also be explained that there are parameter redundancies in the Convolutional Neural Network, and the Gabor operator can be convoluted and integrated into CNN. Lastly, the Gabor filter can extract directional correlation texture features from an image.

g θ,φ,γ,σ,λ (x, y) = exp - x 2 + y 2 2σ 2 cos 2πx λ + φ x = xcosθ + ysinθ y = -xsinθ + ycosθ (1.0)

Since the Gabor operator combines with the CNN in the image, better feature expressions can be obtained. There are two main binding methods. First, the image is preprocessed by the Gabor operator, and then its features are extracted by the CNN. Next, the Gabor operator is convoluted to form a convolution layer, and then we integrate this convolution into the common Convolutional Neural Network. The second approach is used in this article. As shown in Eqn. 1.0, the Gabor kernel function has 5 parameters, which are obtained by learning and then regenerated into an m×m kernel. We replace the common convolution kernels with these Gabor kernels to form a convolutional layer. However, for the common convolutional layer, an m × m convolution kernel is generated by an identity mapping, which requires m 2 parameters. So, our method reduces the number of trainable parameters in the convolutional layer.

3.2.2. SCHMID OPERATOR:

In 2001, Schmid (2001) proposed a Gabor-like image filter, namely the Schmid operator. As shown in Eqn. 2.0, its composition is similar to the kernel function of the Gabor operator, so it retains the properties of the Gabor operator. In addition, the Schmid operator has rotation invariance. So, the Schmid operator is convoluted, and we integrate this convolution into common Convolutional Neural Network. This network improves the model's adversarial stability to rotation and improve the image feature extraction effect. Similar to the convolution of the Gabor operator, as shown in Eqn. 2.0, the Schmid kernel function has two parameters, which are obtained by learning and then generated by the Schmid kernel. Finally, we replace common convolution kernels with Schmid kernels to form a convolutional layer.

F σ,τ (r) = exp - r 2 2σ 2 cos 2πτ r σ r = x 2 + y 2 (2.0)

In this paper, only two geometric operator convolutions are explained. Similarly, for other geometric operators, operator kernels are generated by operator kernel functions, which replace common convolution kernels to form a convolutional layer. Due to the diversity of geometric operators, different geometric operators can be replaced with geometric operator convolutions, so the geometric operator convolution is customizable. There is a kind of geometric operator to form any kind of geometric operator convolution. Consequently, a question that must be addressed is how we combine multiple geometric operators with common CNNs to form GO-CNNs.

3.3. GEOMETRIC OPERATOR CONVOLUTIONAL NEURAL NETWORK:

Since the visualization of the first layer of convolution kernels maintains some geometric characteristics, we replace the convolution kernel in the first convolutional layer by kernel generated from geometric operators, and denote this kind of CNN as Geometric Operator Convolutional Neural Network (GO-CNN). In GO-CNN, kernels from the first convolutional layer are calculated by parameters of various geometric operator functions, and we call these operator functions as generator functions. Then, we concatenate all the calculated convolutional kernels in the last dimension to obtain a complete convolutional kernel. The full procedure is illustrated in Fig. 1 . The generated convolution kernel is used as the weight of the first convolution layer in the Geometric Operator Convolutional Neural Network, and then the common convolution layer and output layer are connected. In this way, we have defined the forward propagation of the whole Geometric Operator Convolutional Neural Network. So, in backward propagation, the gradient of loss is transferred to the convolution kernel; this process is different from the usual convolution. Here, the convolution kernel generated by the geometric operator needs to further use the chain derivative rule (i. e., Eqn. 3.0, where L is the loss function, w is each convolution kernel, and p i is the parameter to generate each convolution kernel) to transfer the gradient to the parameters of each convolution kernel.

Then, trainable parameters are updated by gradient descent algorithms, and the whole GO-CNN is complete.

∂L ∂p i = ∂L ∂w ∂w ∂p i (3.0)

4. THEORETICAL ANALYSES:

The whole framework of the Geometric Operator Convolutional Neural Network has been introduced above. Next, we describe how to theoretically analyze the GO-CNN. It is theoretically proved that although the number of trainable parameters in the GO-CNN decreases, the effectiveness for computer vision tasks does not decrease. All detailed proofs are expanded in Appendix B.

4.1. DEFINITION OF DATA AND LOSS FUNCTION:

• We denote the input by S = {I i } N i=1 , the corresponding label is {y i |y i = 0 or 1} N i=1 .

• The loss function is Mean Square Error.

• The output of the neural network is ỹi for each input I i , and the empirical loss function is defined as follows:

ÊS [h] = 1 N N i=1 (ỹ i -y i ) 2 (4.0)

Definition 1 (Parametric Convolutional Kernel Space). Let f be a function that maps vector from R n to matrix in R m×m , n, m ∈ N + , and we call this function as convolution kernel generator function. Then we define Parametric Convolutional Kernel Space K f as:

K f = {[f (p 1 ), f (p 2 ), • • • ,f (p od )], p i ∈ D ⊂ R n , i = 1, 2, • • • , od}.

(5.0)

We call n the parameter number, m the kernel size, od (short for output dimension) the output dimension. Since a convolutional kernel in a parametric convolutional kernel spaces is generated by function f , we call f as the generator function, and f i,j (p) = f (p)[i, j] as the pixel generator function.

Since kernel can be generated from a generator function by fewer parameters than ordinary kernel, the amount of trainable parameters of GO-CNN can be much smaller. However, reduction in parameters often causes loss of performance as the hypothesis space is smaller. In the simplest situation, we replace the ordinary kernel in the first convolutional layer by the parameter kernel generated from a parametric convolutional kernel space, and study on it. Definition 2 (GO-CNN). Assume that K f is a parametric convolutional kernel space. If the kernel in the first convolutional layer of a convolutional neural network is generated from K f , we call this network GO-CNN. We denote the set of GO-CNN by G f . GO-CNN is almost exactly the same as common CNN, except for the kernel in the first convolutional layer. We treat the first convolutional layer as a function from images to outputs, which then act as input of the following layer. If this function is not an injective function, meaning that different inputs can be mapped to identical outputs, then the network takes these identical outputs as the input of the following layers, meaning that the final outputs are still the same. However, the image inputs of the first convolutional layer are different, and corresponding labels can also be different. Thus, when the final outputs are the same, errors must occur.

Therefore, we need to choose kernel carefully to make the function be an injective function. Since the convolution operator is a linear operator, we have the following proposition. Proposition 1. If the kernel of a convolutional layer, denoted by w, satisfies the following: I * w = 0 ⇔ I = 0, ∀I (6.0) where I is the layer input and * is the convolution operation, then this convolutional layer is an injective function.

We find a necessary and sufficient condition for a convolutional layer to be an injective function. But which kernel satisfies this condition? In the proposition below, we show that 3 × 3 kernel generated by Gabor kernel function satisfies this condition. Proposition 2. Let f be the Gabor kernel function, that is

f x,y (θ, σ, γ, λ, ψ) = exp(-x 2 +γ 2 y 2 2σ 2

)cos(2π x λ + ψ), where x = xcosθ + ysinθ, y = -xsinθ + ycosθ. Let K f be the corresponding parametric convolutional kernel space with kernel size m equal to 3 and sufficient output dimension od. Then, there exists kernel in K f satisfies the condition (6.0).

As the kernel generated from K f could not meet the (6.0), we have the following definition: Definition 3 (Well-Defined GO-CNN). Let G ∈ G f , if there is a kernel generated by K f that satisfies (6.0), we call G a well-defined GO-CNN. We denote the set of all well-defined GO-CNNs as G * f . Corollary 1. If the generator function f is Gabor kernel function, the GO-CNN is well-defined. Now, let us consider a Convolutional Neural Network with one convolutional layer and two fully connected layers, and we will study the convergency of common CNN and GO-CNN. The detailed mathematical expression is expanded in Appendix B. Theorem 1. For any F ∈ F, where F is the set of common CNN, if the first fully connected layer is wide enough, the empirical loss of a well-defined GO-CNN can be that of common CNN controls. That is, for an arbitrary > 0, there exists d * ∈ N + and G ∈ G * f , such that when d 1 ≥ d * , the following inequality holds:

| ÊS [G] -ÊS [F ]| ≤ (7.0) Theorem 2. For any F ∈ F, where F is the set of common CNN, if the first fully connected layer is wide enough, the generalization error of a well-defined GO-CNN can be that of common Convolutional Neural Network controlled. That is, for an arbitrary > 0, there exists d * ∈ N + and G ∈ G * f , such that when d 1 ≥ d * , the following inequality holds:

ÊD [G] ≤ ÊS [F ] + 2 Ra S (F) + log(1/δ) 2N + (8.0)

In Theorem. 2, we know that well defined GO-CNNs have almost the same generalization error as common CNNs. Therefore, we need to find which GO-CNNs are well defined.

As GO-CNN with Gabor kernel function as the generator function is well defined, we have the following corollary.

Corollary 2. Let f be Gabor kernel function, for any F ∈ F, if the first fully connected layer is wide enough, the generalization error GO-CNN G, which applies f as the generator function can be that of F controlled. That is, for an arbitrary > 0, there exists d * ∈ N + and G ∈ G * f , such that when d 1 ≥ d * , the following inequality holds:

ÊD [G] ≤ ÊS [F ] + 2 Ra S (F) + log(1/δ) 2N + (9.0)

More generally, if there are many generator functions in the first convolutional layer of a GO-CNN, when the number of kernels generated by Gabor kernel function is sufficient enough, this GO-CNN is also well defined. Therefore, we have the following corollary. 

Corollary 3. Let {f 1 , f 2 , • • • , f T }

5. EXPERIMENTS:

All experiments are performed on a single machine with CPU Intel Core i7-7700 CPU @ 3.60GHz × 8, GPU TITAN X (Pascal), and RAM 32G. Experimental details and more experiments are given in Appendix C, respectively.

5.1. APPROXIMATION ACCURACY AND GENERALIZATION ERROR:

Theoretical analyses ensures that the GO-CNN has the same approximation accuracy and the same upper bound for generalization error as the common CNN. We verify this using two kinds of experiments on CIFAR-10/100 datasets. For the GO-CNN, the convolution kernels from the first layer are half trainable Schmid kernels and half trainable Gabor kernels. The basic network frameworks used in these experiments are ResNet18, ResNet34, and ResNet50 (He et al., 2016) .

According to the cross-entropy curve of CIFAR-10/100 train sets, GO-CNN's value initially fell faster than the common CNN's, eventually almost reaching the same value. It is verified that GO-CNN achieves the same approximation accuracy as the common Convolutional Neural Network.

According to the error rate curve of CIFAR-10/100 verification sets, the value of GO-CNN is lower than that of the common CNN. In addition, as shown in Table 1 , the GO-CNN on the CIFAR-10 test set was 0.4% more accurate than the common CNN. On the CIFAR-100 test set, the GO-CNN was 0.5% more accurate than the common CNN. It is verified that the GO-CNN achieves the same generalization error bound as the common CNN.

5.2. GENERALIZATION:

In many practical applications, such as the military, medical care, and so on, annotated data are often insufficient. Thus, a model's generalization ability for small data sets is of great importance.

For CIFAR-10/100 and MNIST datasets, their train sets are large and their test sets are small. So, in these numerical experiments, the test set is directly used to train the model, and the train set is used to evaluate the model. For numerical experiments with CIFAR-10/100 datasets, the techniques and models used are the same as in Sec. 5.1. For numerical experiments with MNIST dataset, the basic network structure used in the experiment is LeNet (LeCun et al., 1998) . Similarly, in the GO-CNN, the first convolutional layer is replaced by the operator convolutional layer. The convolution kernels from the first layer are composed of trainable Gabor kernels and Schmid kernels. The other convolutional layers are the common convolutional layers.

As shown in Table 2 , from the perspective of the accuracy of MNIST and CIFAR-10/100, after the train set drops to one-fifth of the original train set, the accuracy of the common CNN falls faster than the GO-CNN. Moreover, the GO-CNN more accurate than the common CNN on the original train set. That is to say, the GO-CNN is better at predicting unknown data than the common CNN. The GO-CNN not only reduces the redundancy of the parameters, but also reduces the dependence on the amount of training samples.

5.3. ADVERSARIAL STABILITY:

The current machine learning model, including the neural network and other models, is vulnerable to attacks from adversarial samples. In addition, CNNs show instability under attacks against adversarial samples (Goodfellow et al., 2014) . So, it is very important to study the stability of adversarial It can be seen from Table 3 that when the test set is randomly rotated within 90 degrees, the difference of the GO-CNN is 1.21% lower than that of the common CNN. This verifies that the GO-CNN enhances the adversarial stability of rotated samples. When the small Gaussian disturbance (the mean is 0, the standard deviation is 0.3) is applied to the test set, the difference of the GO-CNN is 0.6% lower than that of the common CNN. This indicates that the GO-CNN enhances the adversarial stability of Gaussian disturbance adversarial samples. In sum, the GO-CNN enhances the adversarial stability of certain adversarial samples.

In the above experiments, the Geometric Operator Convolutional Neural Network uses a priori knowledge from the field of medicine and provides a better recognition effect. Experiments about intelligent medical diagnoses of bone fractures is given in Appendix C. Although the trainable parameters decrease, GO-CNN still reaches the same approximation accuracy and a slightly lower generalization error upper bound when compared with the common CNN. And the GO-CNN reduces the dependence on training samples and enhances the adversarial stability of certain adversarial samples.

6. CONCLUSION AND FUTURE RESEARCH:

In this paper, we present a novel framework named the Geometric Operator Convolution Neural Network, where the kernel in the first convolutional layer is replaced with kernels generated by geometric operator functions. This new network boasts several contributions. Firstly, the GO-CNN is customizable for diverse situations. Secondly, there is a theoretical guarantee in the learning framework of the GO-CNN. Thirdly, the GO-CNN reduces the dependence on training samples. Lastly, the GO-CNN enhances adversarial stability. In the future, we can explore a more appropriate geometric operator convolution block.

A APPENDIX:

A supplementary explanation about the Gabor and Schmid operators.

The Gabor kernels are similar to the convolution kernels from the first convolutional layer in the CNN. An illustration of this similarity is shown in Figure 2 . In addition, the Gabor filter can extract directional correlation texture features from an image. As shown in Figure 3 , there are 40 Gabor 

F ⊆ G ⇒ Ra S (F) ≤ Ra S (G) Ra S (F + G) ≤ Ra S (F) + Ra S (G)

(11.0) Lemma 3. (Mohri et al., 2012) Let z be a random variable of support Z and distribution D.

Let S = {z 1 , z 2 , • • • , z N } be a data set of N i.i.d. samples drawn from D. Let F be a hypothesis class satisfying F ⊆ {f | f : Z → [a, a + 1]}. Fix δ ∈ (0, 1)

. With probability at least 1 -δ over the choice of S, the following holds for all h ∈ F:

E D [h] ≤ ÊS [h] + 2 Ra S (F) + log(1/δ) 2N (12.0) Proof of Proposition1.

Assume that the proposition is not true, then there exist I 1 = I 2 , such that I 1 * w = I 2 * w. Thus, if we set I = I 1 -I 2 , we have I * w = (I 1 -I 2 ) * w = 0, since * is a linear operator, which means that I = 0 according to the condition. Therefore, the assumption is not true, and the conclusion is proved.

Proof of Proposition2.:

Assume that there exists I ∈ R 3×3 , I = 0, such that I * k = 0 holds for ∀k ∈ K f .

We write I in the following matrix way:

I =

a 00 a 01 a 02 a 10 a 11 a 12 a 20 a 21 a 22 (13.1)

We define the pixel generator function f ij to be f x-1,y-1 (θ, σ, γ, λ, ψ). Then, we have the following equivalence:

I * k = 0 ⇐⇒ 2 i=0 2 j=0 a ij f ij = 0. (13.2)

We will choose a variety of different parameters to discuss.

I. θ = 0, λ = 1, ψ = 0.

Since θ = 0, we have x = x, y = y, and the following:

           f 00 = f 02 = f 20 = f 22 = exp -1+γ 2 2σ 2 f 01 = f 21 = exp -1 2σ 2 f 10 = f 12 = exp -γ 2 2σ 2 f 11 = 1 (13.3)

We make the following shorthands for conveniency:

h 1 = exp - 1 + γ 2 2σ 2 h 2 = exp - 1 2σ 2 h 3 = exp - γ 2 2σ 2

(13.4) From Eqn.13.2, we can get:

(a 00 + a 02 + a 20 + a 22 )h 1 + (a 01 + a 21 )h 2 + (a 10 + a 12 )h 3 + a 11 = 0.

(13.5)

The equation above means that,

∃ b 0 , b 1 , b 2 , b 3 ∈ R, such that b 0 + b 1 h 1 + b 2 h 2 + b 3 h 3 = 0. (13.6)

Differentiate on both sides of parameter γ and get:

- γ σ 2 b 1 h 1 - γ σ 2 b 3 h 3 = 0 =⇒b 1 + exp γ 2 σ 2 b 3 = 0 (13.7)

Since Eqn.13.7 holds for ∀γ, σ, which indicates that:

b 1 = b 3 = 0 (13.8)

In the same way, we can get the following equation from Eqn.13.8:

b 2 h 2 + b 0 = 0 b 0 = b 2 = 0 (13.9)

Therefore, we have the following equations:

      

a 00 + a 02 + a 20 + a 22 = 0 a 01 + a 21 = 0 a 10 + a 12 = 0 a 11 = 0 (13.10)

II. θ = 0, λ = 3, ψ = π/3.

In the same way, we have the following equations:

             f 20 = f 21 = f 22 = 0 f 00 = f 02 = 1 2 h 1 f 10 = f 12 = 1 2 h 3 f 01 = 1 2 h 2 f 11 = 1 2 (13.11)

And we can get:

(a 00 + a 02 )h 1 + a 01 h 2 + (a 10 + a 12 )h 3 + a 11 = 0, (13.12) which indicates that:

a 00 + a 02 = 0 a 01 = 0 (13.13) From Eqn.13.10, we can get:

a 00 + a 02 = 0 a 01 = a 21 = 0 (13.14)

III. θ = 0, λ = 3, ψ = -π/3.

We can get the following equations in the way just the same as discussed in situation II:

a 20 + a 22 = 0 (13.15)

IV. θ = π/2, λ = 3, ψ = ±π/3.

We have x = y, y = x this time, and we can get the following equations as the way discussed in situation II III, :

   a 00 + a 20 = 0 a 02 + a 22 = 0 a 10 = a 12 = 0 (13.16)

Combine equations 13.14 13.15 13.16, we can get:

a 00 = -a 02 = a 22 = -a 20 = v a 01 = a 10 = a 12 = a 21 = 0 (13.17) V. θ = π/4, λ = 2, ψ = √ 2π. We have x = √ 2 2 (x + y), y = √ 2

2 (y -x) and the following:

     f 00 = exp -1 σ 2 f 02 = f 20 = exp -γ 2 σ 2 f 22 = exp -1 σ 2 cos(2 √ 2π) (13.18)

Therefore, we have

(1 + cos(2 √ π)) exp(- 1 σ 2 )v + 2 cos(2 √ 2π) exp(- γ 2 σ 2 )v = 0 =⇒ v = 0 =⇒ a 00 = a 02 = a 20 = a 22 = 0 (13.19)

Combine equations 13.10 13.17 13.19, we can find that a ij = 0, f ori, j = 0, 1, 2, which means that I = 0. Therefore, the assumption that I = 0 is not true.

For an arbitrary sized input I, we can focus on the 3 × 3 sized submatrix that will do inner product with the convolution kernel and get the same conclusion.

Proof of Corollary1.:

From Prop.2, the conclusion is obvious.

For the common CNN, denoted by F , we define the convolution kernel as k F . The weights of the rest of fully connected layers are {a F,1 , a F,2 }, and the biases of three layers are {b F,0 , b F,1 , b F,2 }. Let σ stand for sigmoid activation function, then the convolutional layer C F and the fully connected layer F C F can be defined as follows:

EQUATION

Then, the last two fully connected layers can be defined as:

D F (x) = F C F,2 • σ • F C F,1 (x) (14.2)

Therefore, the output before activation, denoted by F (x), and after activation, denoted by F (x), are defined as:

F (x) = D F • σ • C F (x) F (x) = σ • F (x) (14.3)

We denote the set of common CNN as F, that is, F = {F }, and the output before activation and after activation of input I i as F i , Fi .

For a GO-CNN G, we similarly define the convolutional kernel to be k g , and the weights and biases are {a

G,1 , a G,2 , b G,0 , b G,1 , b G,2 }.

Then, we have the following shorthand when the input is x:

C G (x) = x * k g + b G,0 F C G,k = a G,k x + b G,k , k = 1, 2 D G (x) = F C G,2 • σ • F C G,1 (x) G(x) = D G • σ • C G (x) G(x) = σ • G(x) (14.4)

We denote the output before activation and after activation of input I i as G i , Gi as well.

We maintain the same neuron number for each corresponding layer in common CNN and GO-CNN, that is to say, dim(b F,k ) = dim(b G,k ), k = 0, 1, 2, since the approximation ability is different when the neuron number is different. We define the width of each layer as

d k = dim(b F,k ) = dim(b G,k ), k = 0, 1, 2.

For the common CNN, denoted by F , we define the convolution kernel as k F . The weights of the rest of fully connected layers are {a F,1 , a F,2 }, and the biases of three layers are {b F,0 , b F,1 , b F,2 }. Let σ stand for sigmoid activation function, then the convolutional layer C F and the fully connected layer F C F can be defined as follows:

C F (x) = x * k F + b F,0 F C F,k (x) = a F,k x + b F,k , k = 1, 2 (15.1)

Then, the last two fully connected layers can be defined as:

D F (x) = F C F,2 • σ • F C F,1 (x) (15.2)

Therefore, the output before activation, denoted by F (x), and after activation, denoted by F (x), are defined as:

F (x) = D F • σ • C F (x) F (x) = σ • F (x) (15.3)

We denote the set of common CNN as F, that is, F = {F }, and the output before activation and after activation of input I i as F i , Fi .

For a GO-CNN G, we similarly define the convolutional kernel to be k g , and the weights and biases are {a

G,1 , a G,2 , b G,0 , b G,1 , b G,2 }.

Then, we have the following shorthand when the input is x:

C G (x) = x * k g + b G,0 F C G,k = a G,k x + b G,k , k = 1, 2 D G (x) = F C G,2 • σ • F C G,1 (x) G(x) = D G • σ • C G (x) G(x) = σ • G(x) (15.4)

We denote the output before activation and after activation of input I i as G i , Gi as well.

We maintain the same neuron number for each corresponding layer in common CNN and GO-CNN, that is to say, dim(b F,k ) = dim(b G,k ), k = 0, 1, 2, since the approximation ability is different when the neuron number is different. We define the width of each layer as

d k = dim(b F,k ) = dim(b G,k ), k = 0, 1, 2.

Proof of Theorem1. Notice that

N ( ÊS [G] -ÊS [F ]) = i G2 i -F 2 i -2y i ( Gi -Fi ) = i ( Gi -Fi )( Gi + Fi -2y i ) (16.1)

Apply absolute value on both sides

N | ÊS [G] -ÊS [F ]| ≤ i | Gi -Fi || Gi + Fi -2y i | ≤ 4 i | Gi -Fi | (16.2)

The last inequality holds as

| Fi |, | Gi |, |y i | ≤ 1.

We can fix parameters of ordinary CNN, so that there is a mapping between input I i and output Fi , and the mapping function is F as we have defined.

We can also fix parameters of C G , and choose the convolution kernel of C G that satisfies (??) since G is a well-defined GO-CNN, so that C G is an injective function, which means that C -1 G exists. In the same time, D G = F C G,2 • σ • F C G,1 can be treated as a one hidden layer neural network.

Define a new hypothesis:

h = σ • F • C -1 G • σ -1 ranges in [0, 1], according to Lemma.1, we can find parameters {a G,k , b G,k }, k = 1, 2, such that |σ • D G (x) -h(x)| ≤ /4, ∀x. (16.3) Replace x by σ • C G (I i ) we can get | Gi -Fi | = | G(I i ) -F (I i )| = |σ • D G • σ • C G (I i ) -σ • F (I i )| = |σ • D G (σ • C G (I i )) -σ • F • C -1 G • σ -1 (σ • C G (I i ))| ≤ /4 (16.4)

Combine with (16.2), we can get

| ÊS [G] -ÊS [F ]| ≤ 4 N i | Gi -Fi | ≤ 4 N N 4 = (16.5)

Proof of Theorem2.

From Theorem.1, we know that G satisfies the following inequality:

| ÊS [G] -ÊS [F ]| ≤ (17.1)

From Lemma.3, we know that

ÊD [G] ≤ ÊS [G] + 2 Ra S (G * f ) + log(1/δ) 2N ≤ ÊS [F ] + 2 Ra S (G * f ) + log(1/δ) 2N + (17.2)

Since G * f ⊂ F, we have the following inequality from Lemma.2:

Ra S (G * f ) ≤ Ra S (F) (17.3)

Combined with (17.2), we have

ÊD [G] ≤ ÊS [F ] + 2 Ra S (F) + log(1/δ) 2N + (17.4)

The conclusion is proved! Proof of Corollary2.

From Theorem.2 and Corollary.1, this conclusion is obvious.

Proof of Corollary3.:

Let K be the set of k i such that the generator function of k i is f ti = f t * and denote the concatenation of all these k i as k.

Suppose that there exists an input x, satisfies that x * k i = 0, i = 1, 2, • • • , od, then x * k = 0, ∀ k ∈ K. Therefore, x * k holds for any parameters. However, it is conflict with Prop.2.

Therefore, the conclusion is proved!

C APPENDIX:

Approximation accuracy and generalization error Recognizing objects in an actual scene is not dependent on corresponding domain knowledge but on humans' prior information. For object recognition tasks, the Geometric Operator Convolutional Neural Network's recognition effect is worth exploring. The commonly used public data sets for common object recognition are CIFAR-10 (ten categories) and CIFAR-100 (100 categories). They are all three-channel color images with a resolution of 32×32. The train set contains 50,000 images and the test set contains 10,000 images. The basic network frameworks used in these experiments are ResNet18, ResNet34, and ResNet50 (He et al., 2016) , which mainly consist of a new residual structure unit. In these experiments, four paddings are added on the four edges. Then, a random 32×32 cropping is performed, and a data enhancement method is carried out, which involves turning the image up and down. For both testing and training, the images' pixels are normalized to a 0-1 distribution. The Stochastic gradient descent optimization algorithm with 0.9 the momentum (Loshchilov & Hutter, 2016) is used during the training process. The batch size is 100, the initial learning rate is 0.1, and the weight decay is 0.0005. The learning rate is reduced by one fifth per 60, 120, and 160 epochs. We report the performance of our algorithm on a test set after 200 epochs based on the average over five runs.

As shown in Figure 5 , according to the cross-entropy curve of the CIFAR-10 and CIFAR-100 train sets, GO-CNN's value initially fell faster than the common CNN's, eventually almost reaching the same value. It is verified that Geometric Operator Convolutional Neural Network achieves the same approximation accuracy as the common Convolutional Neural Network. According to the error rate curve of the CIFAR-10 and CIFAR-100 verification set (Figure 6 ), the value of Geometric Operator Convolutional Neural Network is lower than that of the common Convolutional Neural Network. Feature visualization One way to evaluate a model is through visualizing the features that the model extracts; this is called feature visualization. T-SNE (Maaten & Hinton, 2008) or PCA (Jolliffe, 2011) are generally used for visualization. The T-SNE visualization maps data points to a two-dimensional or three-dimensional probability distribution through affinitie transformation. Then, the data points are displayed with a two-dimensional or three-dimensional plane.

In this paper, a two-dimensional T-SNE visualization is adopted to display the CIFAR-10 features extracted by the model. As shown in Figure 7 , the CIFAR-10 features extracted by the Geometric Operator Convolutional Neural Network are evenly separated from each other in the two-dimensional visualization of T-SNE, while the features extracted from the common Convolutional Neural Network are mixed. It is apparent that the features extracted by the GO-CNN are more separable; in other words, the features learned by the GO-CNN are more distinguishable and easy to classify with the last fully connected layer.

Generalization MNIST is a public, handwritten recognition dataset with a total of ten classes. This dataset is a channel image with 28×28 resolution and a clean background. The train set contains 50,000 images and the test set contains 10,000 images. For numerical experiments with the MNIST data set, the adaptive moment estimation (Kingma & Ba, 2014) optimization algorithm is used. In addition, as an image enhancement strategy, the image padding is increased to 32×32 during the training process. The batch size is 100, the initial learning rate is 0.001, and the weight decay is 0.0005. The learning rate stays the same until reaching 20,000 iterations. Consequently, we complete 20,000 iterations on one test set and average the performance over five runs in order to report the final performance evaluation of our algorithm.

Application Medical images in China are developing rapidly, but specialist doctors are short of resources, and they are mainly concentrated in big cities and big hospitals. Many small and mediumsized cities do not have sufficient diagnostic imaging capacities, so many patients have to go to big cities in order to access better medical resources and obtain better treatment. Similarly, there are few According to Table 4 , the Geometric Operator Convolutional Neural Network is the most accurate. Moreover, the fracture recall of the two-stage method is 0.77% higher than that of the Convolutional Neural Network, indicating that domain knowledge from the field of medicine is important for intelligent diagnosis. The fracture recall of the Geometric Operator Convolutional Neural Network is 2.21% higher than that of the two-stage method, which indicates that the Geometric Operator Convolutional Neural Network does make use of medical knowledge for fracture diagnosis. The integration of geometric operator into the deep neural network indeed achieve global optimization. 

:



Figures and Tables:

Figure 1: For each out channel, parameters (the first row) are fed into a generator function (the second row), and generate a kernel (the third row). Then, kernels are concatenated to get the convolution kernel in the first convolutional layer. In the second row, G(•) is Gabor kernel function, S(•) is Schmid kernel function, and I(•) is identity mapping.

Figure 2: The similarities between the CNN's first convolutional kernels and Gabor kernels.

Figure 3: The results of the Gabor operator on an image

Figure 4: The results of the Schmid operator on an image

Figure 5: Log of cross entropy curve during training in common ResNet 18-34-50 and GO-ResNet 18-34-50.

Figure 8: Bone images

Each operator has different characteristics. Therefore, different geometric operators are used in different application scenarios, according to the characteristics of each unique problem. For example, SIFT looks for feature points in different scale spaces for pattern recognition and image matching. The Roberts operator uses local differences to find edges for edge detection, and the Laplace operator uses isotropic differentials to retain details for image enhancement.

Table content: <table><tr><td>Geometric operators represent the precipitation of domain knowledge and prior knowledge. The</td></tr><tr><td>GO-CNN is proposed in this paper, which uses the characteristics of geometric operators. The first</td></tr><tr><td>step in this framework is to convolve geometric operators. In this paper, the Gabor operator and</td></tr><tr><td>the Schmid operator are mainly used as examples to illustrate how to carry out convolutions and</td></tr><tr><td>integrate these convolutions into CNNs. Other geometric operators in subsequent studies employ</td></tr><tr><td>similar concepts.</td></tr><tr><td>3.2 CONVOLUTION OF GEOMETRIC OPERATORS</td></tr></table>

The model's accuracy rates averaged over five experiments on the test set

Table content: <table><tr><td>-</td><td colspan="2">CIFAR-10 CIFAR-100</td></tr><tr><td>common ResNet18</td><td>94.79%</td><td>77.06%</td></tr><tr><td>GO-ResNet18</td><td>95.17%</td><td>77.59%</td></tr><tr><td>common ResNet34</td><td>95.27%</td><td>78.26%</td></tr><tr><td>GO-ResNet34</td><td>95.77%</td><td>78.72%</td></tr><tr><td>common ResNet50</td><td>94.44%</td><td>78.45%</td></tr><tr><td>GO-ResNet50</td><td>94.72%</td><td>79.50%</td></tr></table>

The accuracy of the test set for the small train set and the large train set (in brackets)

Table content: <table><tr><td>-</td><td>CIFAR-10</td><td>CIFAR-100</td><td>MNIST</td></tr><tr><td colspan="2">common ResNet18 84.96%(94.79%)</td><td>44.97%(77.06%)</td><td>-</td></tr><tr><td>GO-ResNet18</td><td colspan="2">86.21%(95.17%) 47.03%(77.59%)</td><td>-</td></tr><tr><td colspan="2">common ResNet34 82.33%(95.27%)</td><td>44.74%(78.26%)</td><td>-</td></tr><tr><td>GO-ResNet34</td><td colspan="2">86.36%(95.77%) 49.00%(78.72%)</td><td>-</td></tr><tr><td colspan="2">common ResNet50 83.86%(94.44%)</td><td>45.93%(78.45%)</td><td>-</td></tr><tr><td>GO-ResNet50</td><td colspan="2">85.64%(94.72%) 47.09%(79.50%)</td><td>-</td></tr><tr><td>common LeNet</td><td>-</td><td>-</td><td>97.75%(99.22%)</td></tr><tr><td>GO-LeNet</td><td>-</td><td>-</td><td>97.97%(99.24%)</td></tr></table>

The adversarial stability of randomly rotated samples and Gaussian disturbance samples. The loss compared to the original accuracy is in brackets, the smaller value is marked in green.

Table content: <table><tr><td>-</td><td colspan="3">original randomly rotated Gaussian disturbance</td></tr><tr><td colspan="2">common LeNet 99.22%</td><td>58.97%(-40.25%)</td><td>95.69%(-3.53%)</td></tr><tr><td>GO-LeNet</td><td>99.24%</td><td>60.20%(-39.04%)</td><td>96.31%(-2.93%)</td></tr></table>

Experimental results of intelligent diagnosis

Table content: <table><tr><td>TestSet</td><td>CNN</td><td colspan="2">two-stage GO-CNN</td></tr><tr><td>accuracy</td><td>92.38%</td><td>93.05%</td><td>93.98%</td></tr><tr><td>fracture recall</td><td>87.97%</td><td>88.74%</td><td>90.95%</td></tr><tr><td colspan="2">non-fracture recall 96.57%</td><td>96.17%</td><td>96.87%</td></tr></table>

Peer review (iclr2019):
 I appreciate the responses authors provided to my comments. Many of my main concerns have been addressed there. Having said this, I do not consider the contribution of this paper substantial given the fact the a more involved framework (CKN) has been already known. I honestly believe that the findings of the authors are interesting but doubt that we use Gabor features in the first layer of CNNs from now on. If the paper is accepted, I kindly encourage the authors to reflect their answers to my comments in their work. For the sake of completeness, I have incorporated my comments and authors responses below. Good luck ------------------------------------------------------------------------------------------------------------- I would like to get my head around a few points and appreciate clarifications on - To me the title is a bit misleading, especially the word geometric seems a bit far-stretched. can you explain why geometric is essential here? - In experiments, given that the Gabor filters have less tunable parameters, did the authors keep the number of channels similar to their CNN counterparts or increased the number of channels (Gabor filters) to match the number of tunable parameters in CNNs? - have authors evaluated their solution on large scale problems, say image classification on image-net? - what will happen if more than 1 layer of Gabor filters is considered? I am curious to know whether the performance decreases or not - how is your proposal different from previous studies where learning the parameters of a kernel function (e.g., CKN by Mairal et al.) has been discussed? I am not very much convinced that the proposed approach is brand-new. - can you explain how you initialize the Gabor filters? Can you also report [min,max] accuracies over say 10runs if random initialization is used? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Re: Some clarifications ICLR 2019 Conference Paper273 Authors 25 Oct 2018ICLR 2019 Conference Paper273 Official CommentReaders: Everyone Comment: Q1 feedback: The title of our paper is originated from geometric operators in traditional image process algorithms. Geometric properties in geometric operators, such as symmetry, account for the word formation. The kernels in the first convolutional layer in AlexNet .cite{krizhevsky2012imagenet} show similarity with Gabor operator kernels in vision, and there is redundancy of parameters in CNNs, which inspire us to combine geometric operators in the traditional image process algorithms into CNNs. Q2 feedback: In our experiments, the output channel number of the first convolutional layer keeps the same between common CNN and the corresponding GO-CNN. Therefore, the number of trainable parameters of common CNN is larger than that of the corresponding GO-CNN. There is theoretical guarantee in our article that, under less parameter degrees of freedom, the approximation ability of the Geometric Operator Convolutional Network is not worse than that of common CNN, and the parameter redundancy of common CNN is further verified. Q3 feedback: The main purpose of this article is to validate and analysis the potential of the proposed architecture in it, and experiments on large scale dataset are not considered in this article. We are going to study applications by the proposed architecture on large scale datasets in the future. Q4 feedback: One of the motivations in this paper is to observe that the visualization of the first layer of the AlexNet convolution kernel has certain geometric characteristics, such as symmetry and volatility, and parameter redundancy. The visualization of other layer convolution kernels does not show obvious geometric characteristics (.cite{krizhevsky2012imagenet} ). In addition, in the MNIST experiment, the experimental effect of replacing the two-layer ordinary convolution kernel with the geometric operator kernel is the same as that of replacing the first-layer convolution kernel. At the same time, in order to reduce the computational complexity and the convenience of theoretical analyses, only the results of replacing one layer are considered in this paper, and the research of replacing multiple layers will be carried out in the future. Q5 feedback: There are many related studies that have similarities with the method presented in this article, which are discussed in the part of related work. As for CKN mentioned, there are at least three differences with the GO-CNN proposed in this article: 1. The number of trainable parameters about convolution kernels is unchanged in CKN comparing to common CNN, however that of the corresponding GO-CNN is smaller. 2. CKN calculates the convolution kernel by approximating the Gaussian kernel with linear functions, and GO-CNN directly learn proper parameters of the geometric operator functions from training samples to get the convolution kernel. 3. CKN uses an energy function as loss for better performance in approximation, and GO-CNN uses directly cross entropy as loss for better performance in classification. Therefore, GO-CNN proposed in this article is very different from CKN, and other similar studies discussed in the related work section. Q6 feedback: In this article, we use random uniform initializer to initialize the five parameters in Gabor kernel function, wherein .phi_{init} ~ U(-.pi, .pi), .lambda_{init} ~ U(2, 10), .theta_{init} ~ U(0, 2.pi), .sigma_{init} ~ U(0, 2.pi), and .gamma_{init} ~ U(0, 1). The model’s accuracy rates over ten experiments on the Cifar-10/100 test set if random initialization is used are listed as following: 1. GO-ResNet18 achieves accuracy of 95.17%±0.13% in Cifar-10 and 77.59%±0.04% in Cifar-100. 2. GO-ResNet34 achieves accuracy of 95.77%±0.14% in Cifar-10 and 78.26%±0.03% in Cifar-100. 3. GO-ResNet50 achieves accuracy of 94.72%±0.08% in Cifar-10 and 79.50%±0.06% in Cifar-100.