# Quotient Network-A Network Similar to ResNet but Learning Quotients
## 3 Quotient Network
### The Rules of Designing
#### 3.3.1 Choice of Activation Function for Quotients  
**The value range should not be too large or too small and should avoid negative numbers.** If it is too large, it will lead to difficulty in learning the desired features, which are computed in the quotient network by a large number of point-to-point scalar multiplications, \(y=h_{n}\times h_{n-1}\ldots h_{2}\times h_{1}\times x\), so the direct exponential growth makes an explosion of computational results more likely compared with traditional networks, and a gradient explosion more likely during backpropagation. If there is no restriction on the range of \(h\) for each computation, this will easily lead to unsuccessful training of the network, and in our experiments the results behave as white noise. If it is too small, the range of features that can be updated each time will be too small, and more layers will be needed to achieve the same effect, thus reducing the network's performance. Moreover, the value range of the function should remain positive because, for general activation functions (e.g., ReLU, Sigmoid), the representation of the learned useful features tends to be positive, and, for the useless ones, close to or equal to 0. If multiplied by a negative number, this structure is destroyed,  
Figure 1: The residual module (**left**) and the quotient module (**right**).  
especially when used with ReLUs in the same network. If wanting to enlarge or reduce the features, multiplying by a positive number is more straightforward and intuitive.  
**The function should pass through the (0, 1) point.** Like ResNet, we want to make it easier for the network to learn the identity mapping. When the weight parameters of the network are small, the weighted sum tends to be 0. At this time, ensuring that the value after activation is close to 1 helps to keep the previous features unchanged when multiplied by the previous features.  
**The function should be globally differentiable.** The value range of the activation function is bounded, but zero gradients cannot be used like ReLU in areas where the value range is close to the upper and lower bounds. Half of the domain in ReLU is in the unsaturated area. However, most domains of this activation function are the regions where the function values are close to the upper and lower bounds. Therefore, if the gradient in the region close to the upper and lower limits is completely zero, a large number of neurons will not be updated, affecting the performance and efficiency of the quotient network's learning.  
Equation (1), as the activation function, can perfectly meet the above requirements. It can ensure that the function is positive, passes through the (0, 1) point, is globally differentiable, and can control the value range through \(\alpha\). In the Appendix A, we will compare the experimental results of different activation functions to illustrate the effectiveness of the design principles and the fact that the network using the activation function of Equation (1) can indeed show good performance.  
\[activate(x)=sigmoid(x-\ln(\alpha-1))\times\alpha \tag{1}\]