# Gradient Agreement Hinders the Memorization of Noisy Labels
## 3 Proposed Method
### Gradient Agreement Updating  
Algorithm 1 briefly describes Gradient Agreement Updating. The training procedure of \(\theta_{A}\) and \(\theta_{B}\) can be divided into two phases. Due to the large random volatility of the gradient directions at the early stage of training process, gradient agreement updating is not applied in the first phase. Meanwhile, previous works [2,3] have observed that DNNs tend to fit training samples with clean labels before memorizing noisy labels. Thus, in the first phase, we warm up the two models using the standard cross-entropy loss and gradient descent for a certain number of epochs. At this stage, parameters of the two identical nets \(\theta_{A}\) and \(\theta_{B}\) are initialized with the same values. The data fed into \(\theta_{A}\) and \(\theta_{B}\) is also identical in every batch to avoid unnecessary gradient divergence. In the next phase, the two nets will gradually over-fit to noisy labels if following the training method in the first phase. Therefore, to hinder the memorization of noisy labels, gradient agreement updating is applied. \(\theta_{A}\) and \(\theta_{B}\) each consists of \(N\) parameters. We can group the corresponding parameters of the two networks to obtain \(N\) parameter pairs \(\rho_{i}=\{\mathcal{P}_{i}^{A},\mathcal{P}_{i}^{B}\};i\in(1,\ldots,N)\). In every iteration, using cross entropy loss and the annotation labels, a pair of gradients \(\
abla\mathcal{P}_{i}^{A}\) and \(\
abla\mathcal{P}_{i}^{B}\) for \(\rho_{i}\) is obtained after each backward  
Figure 2: Overview of the proposed Gradient Agreement Learning (GAL) framework. propagation. In contrast to standard gradient descent methods which directly add gradients to \(\rho_{i}\), an intermediate parameter pair \(\hat{\rho}_{i}\) is first calculated following Equation (2):  
\[\hat{\mathcal{P}}_{i}^{A} =\mathcal{P}_{i}^{A}+\eta\
abla\mathcal{P}_{i}^{A} \tag{2}\] \[\hat{\mathcal{P}}_{i}^{B} =\mathcal{P}_{i}^{B}+\eta\
abla\mathcal{P}_{i}^{B}\]  
where \(\eta\) is the learning rate. We then reshape \(\hat{\mathcal{P}}_{i}^{A}\) and \(\hat{\mathcal{P}}_{i}^{B}\) to \(D\)-dimension vectors \(\mathbf{v}_{i}^{A}\) and \(\mathbf{v}_{i}^{B}\), respectively. \[\mathbf{v}_{i}^{A} =\mathit{Reshape}(\hat{\mathcal{P}}_{i}^{A}),\mathbf{v}_{i}^{A} \in(1,D) \tag{3}\] \[\mathbf{v}_{i}^{B} =\mathit{Reshape}(\hat{\mathcal{P}}_{i}^{B}),\mathbf{v}_{i}^{B} \in(1,D)\]  
To measure whether the gradients are clean or noisy, we propose a gradient agreement coefficient \(g_{i}\) which is defined as follows:  
\[g_{i}=\Phi(\mathbf{v}_{i}^{A},\mathbf{v}_{i}^{B})=\frac{(\mathbf{v}_{i}^{A}) \mathbf{v}_{i}^{B}}{\sqrt{\Sigma_{j=1}^{\text{D}}(\mathbf{v}_{ij}^{A})^{2}} \times\sqrt{\Sigma_{j=1}^{\text{D}}(\mathbf{v}_{ij}^{B})^{2}}} \tag{4}\]  
After acquiring \(g_{i}\), we then apply parameter updating as follows:  
\[\widetilde{\mathcal{P}}_{i}^{A} =\mathcal{P}_{i}^{A}+\mathbb{1}_{g_{i}\geq\tau}*\eta\
abla \mathcal{P}_{i}^{A} \tag{5}\] \[\widetilde{\mathcal{P}}_{i}^{B} =\mathcal{P}_{i}^{B}+\mathbb{1}_{g_{i}\geq\tau}*\eta\
abla \mathcal{P}_{i}^{B}\]  
where \(\tau\) is the threshold for determining whether the gradients of a pair of parameters reach an agreement.