# A Simple Convolutional Neural Network with Rule Extraction
## 3 Proposed Model
### The DIMLP Model  
DIMLP differs from a standard MLP in the number of connections between the input layer and the first hidden layer. Specifically, any hidden neuron receives only a connection from an input neuron and the bias neuron, while all other layers are fully connected [7]. The activation function above the first hidden layer of a typical DIMLP is a sigmoid function given as:  
\[\sigma(x)=\frac{1}{1+\exp(-x)}. \tag{6}\]  
For the first hidden layer a step function or its generalization corresponding to a staircase activation function is used. For simplicity, we first give the step function \(\tau(x)\), which is a particular case of the staircase function with only one step:  
\[\tau(x)=\left\{\begin{array}{ll}1&\text{if }x>0;\\ 0&\text{otherwise.}\end{array}\right. \tag{7}\]  
The key idea behind rule extraction from DIMLPs is the precise localization of axis-parallel discriminative hyperplanes. In other words, the input space is split into hyper-rectangles representing propositional rules. Specifically, the first hidden layer creates for each input variable a number of axis-parallel hyperplanes that are effective or not, depending on the weight values of the neurons above the first hidden layer. As an example, Figure 5 illustrates an elementary DIMLP network with a hidden neuron, a weight \(w\) between the input neuron and the hidden neuron and a bias \(b\) between the bias neuron and the hidden neuron. Because of the step function as the activation of the hidden neuron, a potential hyperplane discriminator lies in \(-b/w\). It will depend on the layers above the first hidden layer, whether the hyperplane discriminator will be effective or not. Generally, these hyperplanes are parallel to the axis of the input neurons. Hence they represent possible rule antecedents.  
The starting point of the rule extraction algorithm is the list of all potential hyperplane discriminators. The number of these hyperplanes depends on the number of stairs in the staircase activation function. Then, a decision tree is built and rules are generated from each tree path. Typically, at this stage the number of rules and antecedents is too large. Hence, a greedy algorithm progressively removes antecedents and rules. More details on the rule extraction algorithm can be found in [8].  
Since the CNN defined in the previous Section is trained with a Softmax function in the output layer (cf. Equation (4)), we replace the sigmoid by it. As stated previously, the activation function  
Figure 5: A Discretized Interpretable Multi Layer Perceptron (DIMLP) network that potentially creates a discriminative hyperplane in \(-b/w\). The activation function of the hidden neuron is a step function, while for the output neuron it is a sigmoid.  
in the first hidden layer of DIMLPs is a staircase function \(S(x)\), with \(\Theta\) stairs that approximate the Identity function (\(I(x)\)) on a compact interval:  
\[I(x)=x; \tag{8}\]  
\[S(x)=R_{min},\ \ \ \text{if}\ x\leq R_{min}; \tag{9}\]  
\(R_{min}\) represents the abscissa of the first stair. By default \(R_{min}=-1\).  
\[S(x)=R_{max},\ \ \ \text{if}\ x\geq R_{max}; \tag{10}\]  
\(R_{max}\) represents the abscissa of the last stair. By default \(R_{max}=1\). Between \(R_{min}\) and \(R_{max}\), \(S(x)\) is:  
\[S(x)=I(R_{min}+\left[\theta\cdot\frac{x-R_{min}}{R_{max}-R_{min}}\right]( \frac{R_{max}-R_{min}}{\theta})). \tag{11}\]  
Square brackets indicate the integer part function, with \(\theta=1,\ldots\Theta\). The approximation of the Identity function by a staircase function depends on the number of stairs \(\Theta\). The larger the number of stairs the better the approximation.