## 3 Adacomp Method
### Idea 2: Approximate Unknown Terms
* _When \(f(w_{t-1})-f(w_{t})<0\), we should decrease \(\gamma\) to restrict the progress along the wrong direction._
* _When \(f(w_{t-1})-f(w_{t})>0\), we should increase \(\gamma\) to make aggressive progress._
* _When \(\gamma\) is too small, we should increase \(\gamma\) to prevent Adacomp from becoming trapped in a local minimum._
* _When \(\gamma\) is too large, we should decrease \(\gamma\) to stabilize the training process._  
These requirements motivate us to design Adacomp based on the _arctan_ function, which is flexible to small values but robust and bounded to large values. The main principle is as follows:  
_Decompose \(\gamma\) into three parts, \(\Pi_{1}\), \(\Pi_{21}\), and \(\Pi_{22}\). \(\Pi_{1}\) is used to compensate the learning rate to accelerate training when the loss function decreases. However, the compensation should be inverse to learning rate and bounded. \(\Pi_{21}\) and \(\Pi_{22}\) are used to penalize the learning rate to stabilize training when the loss function increases, but with a bounded amplitude when the learning rate is too large or small._  
Based the above principle, we reformulate Equation (8) as:  
\[\gamma\leftarrow\frac{\gamma}{2}+\gamma\mathbb{I}_{\Delta_{t}> \epsilon}\,\Pi_{1}+\gamma\mathbb{I}_{\Delta_{t}<-\epsilon}(\Pi_{21}+\Pi_{22}), \tag{9}\]  
where \(\mathbb{I}\) is the indicator function of \(\Delta_{t}\overset{\triangle}{=}f(w_{t-1})-f(w_{t})\) and \(\epsilon\) is a small positive constant. Many functions satisfy the above discipline. To reduce the difficulty of designing \(\Pi_{1},\Pi_{21},\Pi_{22}\), we define the following expressions, where only one parameter \(\beta\) needs to be tuned. Furthermore, as experimental results show (refer to Figure 11), Adacomp is not so sensitive to \(\beta\). \[\Pi_{1}=\frac{1}{2}+\frac{\arctan(\Delta_{t}+1/\gamma)}{5\pi},\Pi_{21}=\frac{1 }{4}-\frac{\arctan(\gamma)}{\beta\pi},\;\Pi_{22}=\frac{1}{4}-\frac{\arctan(- \Delta_{t}+1/\gamma)}{5.5\pi},\]  
where \(1/2<\beta\leq 5\) is a parameter used to control the adjustment amplitude of the learning rate. We first explain the meaning of \(\epsilon\) and then \(\Pi_{1},\Pi_{21}\), and \(\Pi_{22}\). The meaning of \(\epsilon\). We replace the hard threshold \(\Delta_{t}<0,>0\) with soft threshold \(\Delta_{t}<-\epsilon,>\epsilon\) on two aspect considerations. _First_, this can alleviate the impacts of \(\Delta_{t}\)'s randomness on learning rate. _Second_, when \(\Delta_{t}\in[-\epsilon,\epsilon]\) for small positive values such as \(\epsilon=10^{-5}\), we halve \(\gamma\). Actually, in such a case the training has converged and halving \(\gamma\) can make training more stable. Now, we explain how to set expressions of \(\Pi_{1},\Pi_{21}\), and \(\Pi_{22}\). * Expression of \(\Pi_{1}\). Note that \(\Pi_{1}\) works only if \(\Delta_{t}>0\). In this case, we should increase \(\gamma\) to speed up training. However, the increment should be bounded and inverse to current \(\gamma\) to avoid divergence. Based on these considerations, we define \(\Pi_{1}\) as \(\Pi_{1}=\frac{1}{2}+\frac{\arctan(\Delta_{t}+1/\gamma)}{5\pi}\), where \(1/2\) is used to keep \(\gamma\) unchanged and \(\arctan(\cdot)/(5\pi)\) is used to ensure that the increment amplitude is at most \(\gamma/10\). * Expressions of \(\Pi_{21}\) and \(\Pi_{22}\). Note that \(\Pi_{21},\Pi_{22}\) work only if \(\Delta_{t}<0\), where the twice \(1/4\) is used to keep \(\gamma\) unchanged and the remaining terms are used to control the decrement amplitude. In this case, we should decrease \(\gamma\) to prevent too much movement along the incorrect direction. However, the decrement should be bounded to satisfy the two following properties. * When \(\gamma\) is small, the decrement of \(\Pi_{21}+\Pi_{22}\) should be less than the increment in \(\Pi_{1}\) given the same \(|\Delta_{t}|\) unless \(\gamma\) is forced to zero, which potentially leads to training stopping too early. Therefore, we define \(\frac{\arctan(\cdot)}{5.5\pi}\) in \(\Pi_{21}\) to satisfy \(\frac{\arctan(\cdot)}{5.5\pi}<\frac{\arctan(\cdot)}{5\pi}\).