# Leveraging Expert Knowledge for Label Noise Mitigation in Machine Learning
## 2 Materials & Methods
### Rule-Weight Algorithm  
The label noise processing part of the Rule-Weight algorithm is shown in Figure 2. First, the detected label noise samples are weighted to reduce the bad influence of the data to the model training. Then the model is retrained with weighted data. As such, the loss function for model training is less affected by label noise data samples. Therefore, the model is more accurate. Algorithm 1 shows the procedure for updating the weight, and training the model in Rule-Weight algorithm. To speed up the training process, the training set is split into smaller mini-batches and the loss function is calculated for each mini-batch. The update process is executed for each mini-batch data sample and stops after a predefined number of loops \(T\). First of all, a \(roughModel\) is trained from an initial model \(model\) using the training algorithm \(trainAlgo()\) with whole training set \(TrainingSet\). Let \(J(Y,Label)\) be the distance between the label \(Label\) of all samples in \(TrainingSet\) and its predicted output \(Y\) and \(\epsilon\) is the weight vector for each data sample in \(TrainingSet\), the loss function is defined as \(\bar{L}(e\odot J(Y,Label))\), where \(\odot\) is the element-wise dot product. At the beginning, all elements of \(\epsilon\) are 1, i.e., all the data samples are treated equally. The model is trained according to weighted data samples at every step. Therefore, a temporary variable \(Model_{t}\) is declared to store the temporary model at every step \(t\) and at initialization step \(Model_{0}\) is assigned to \(model\). At loop \(t\), a mini-batch data sample \(X_{t}\) and the corresponding label \(Label_{t}\) is randomly drawn from training set by the function \(SampleMiniBatch()\). After that, the \(roughModel\) is used to predict the output \(Y_{t}\) of the mini-batch data sample \(X_{t}\). The logical inference machine \(RuleCheck()\) checks if the predicted output \(Y_{t}\) of the input \(X_{t}\) violates the given rules and differs from the label \(Label_{t}\). In other words, a data sample is considered as a correct label if its predicted output is equal to its label or the relationships between that data sample  
Figure 2: The diagram of label noise detection part and label noise processing part (Rule-Weight). and its predicted output satisfy all given rules. The violating data samples are put in the label noise set \(D_{t}\). Most of the label noise are in \(D_{t}\) and should be weighted less than the non-violating data to reduce their effect. The optimal weight can be found by applying the gradient descent algorithm for the mini-batch data. This optimal weight is used to train the temporary model \(Model_{t}\) later. To do that, the loss of model \(Model_{t}\) for the \(j^{th}\) data sample of \(D_{t}\) is calculated using \(Loss_{j}()\). The gradient \(g_{t}\) is calculated over all data samples in the set \(D_{t}\) using the calculated loss. The weight \(\epsilon_{t+1}\) for each data sample in \(X_{t}\) is updated based on the gradient \(g_{t}\) with learning rate \(\alpha\). Initially, all elements of \(\epsilon_{0}\) are 1. After obtaining the proper weight, a weighted mini-batch data \(WeightedMB_{t}\) is formed by updating these weights for \(X_{t}\) at \(Weight()\) function. \(WeightedMB=[\epsilon_{t+1}X_{t},Label_{t}]\). \(WeightedMB\) is then used to train \(Model_{t}\) which should be more accurate than the previous one. The loss function in the training algorithm \(trainAlgo()\) is \(\bar{L}_{t}(\epsilon_{t+1}\odot J(Y_{t},Label_{t}))\). The \(FinalModel\) is the \(Model_{t}\) after \(T\) times training with the weighted mini-batch data sample.