# Machine Unlearning by Reversing the Continual Learning
## 4 Machine Unlearning by Reversing the Continual Learning
### Unlearning with DMM  
We first consider continual learning with IMM, given task A (dataset \(D_{r}\)) and task B (dataset \(D_{f}\)), and a model \(M_{r}\) that has learnt task A (trained with \(D_{r}\)). The objective of continual learning is to build on \(M_{r}\) to learn task B while retaining knowledge of task A. In short, one wants to obtain the model \(M\) that has learned task C (dataset \(D=D_{r}+D_{f}\)). In IMM, the moments of posterior distributions are matched in an incremental way.  
We denote the true posterior parameter distribution of \(M\) as \(p\), the true posterior parameter distribution of \(M_{f}\) as \(p_{f}\), and the true posterior parameter distribution of \(M_{r}\) as \(p_{r}\). Assuming that the posterior parameters of all models follow Gaussian distribution, then the true distributions \(p,p_{f}\) and \(p_{r}\) can be approximated by \(h,h_{f}\) and \(h_{r}\), respectively:  
\[p\approx h(w;\mu,\Sigma)=\frac{1}{\sqrt{2\pi^{n}|\Sigma|}}\exp\biggl{[}-\frac {1}{2}(w-\mu)^{T}\Sigma^{-1}(w-\mu)\biggr{]} \tag{5}\]  
\[p_{f}\approx h_{f}\Big{(}w_{f};\mu_{f},\Sigma_{f}\Big{)}=\frac{1}{\sqrt{2\pi^{ n}|\Sigma_{f}|}}\exp\biggl{[}-\frac{1}{2}\Big{(}w_{f}-\mu_{f}\Big{)}^{T} \Sigma_{f}^{-1}\Big{(}w_{f}-\mu_{f}\Big{)}\biggr{]} \tag{6}\]  
\[p_{r}\approx h_{r}(w_{r};\mu_{r},\Sigma_{r})=\frac{1}{\sqrt{2\pi^{n}|\Sigma_{ r}|}}\exp\biggl{[}-\frac{1}{2}(w_{r}-\mu_{r})^{T}\Sigma_{r}^{-1}(w_{r}-\mu_{r}) \biggr{]} \tag{7}\]  
where \(\mu\) is the mean of the distribution and \(\Sigma\) is the covariance matrix in the distribution.  
The dimension of \(\mu,\mu_{f},\mu_{r}\) is \(d\) and the dimension of \(\Sigma,\Sigma_{f},\Sigma_{r}\) is \(d\times d\), where \(d\) is the dimension of \(w\). Since \(D=D_{f}+D_{r}\), the model \(M\) can be calculated from \(M_{f}\) and \(M_{r}\)The specific idea is to minimize the weighted sum of the KL-divergence between the two sub-distributions \(h_{r},h_{f}\) and the overall distribution \(h\):  
\[\mu^{*},\Sigma^{*}=\underset{\mu,\Sigma}{\operatorname{argmin}}\Big{[}(1-\alpha) KL\Big{(}p\|p_{f}\Big{)}+\alpha KL(p\|p_{r})\Big{]} \tag{8}\]  
where \(1>\alpha>0\). Equation (8) has a closed solution:  
\[\mu^{*}=(1-\alpha)\mu_{f}+\alpha\mu_{r} \tag{9}\]  
\[\Sigma^{*}=(1-\alpha)\bigg{[}\Sigma_{f}+\Big{(}u_{f}-\mu^{*}\Big{)}\Big{(}\mu _{f}-\mu^{*}\Big{)}^{T}\bigg{]}+\alpha\Big{[}\Sigma_{r}+(\mu_{r}-\mu^{*})(\mu_{ r}-\mu^{*})^{T}\Big{]} \tag{10}\]  
where \(\mu^{*}\) and \(\Sigma^{*}\) are solutions of Equation (8).  
Recall our problem: delete the data \(D_{f}\) from \(M\) to obtain \(M_{r}\), and make sure that \(M_{r}\) has no residual information about \(D_{f}\). We propose the DMM, in which the moments of posterior distributions are matched in a decreasing way. We can approximate \(M\) as the solution of Equations (9) and (10); then, we can calculate \(\mu_{r}\) and \(\Sigma_{r}\) according to the following equation:  
\[\mu_{r}=\frac{1}{1-\alpha}\mu^{*}-\frac{\alpha}{1-\alpha}\mu_{f} \tag{11}\]  
\[\Sigma_{r}=\frac{1}{1-\alpha}\Sigma^{*}-\frac{\alpha}{1-\alpha}\Big{[}\Sigma_ {f}+\big{(}\mu_{r}-\mu^{*}\big{)}(\mu_{r}-\mu^{*})^{T}\Big{]}-\Big{(}\mu_{f}- \mu^{*}\Big{)}\Big{(}\mu_{f}-\mu^{*}\Big{)}^{T} \tag{12}\]  
where \(1>\alpha>0\).  
Since \(M_{r}\) is the optimal \(M^{\prime}\), the model owner can complete the deletion request according to Equations (11) and (12). Specifically, after receiving a deletion request from a user, the model owner trains the model \(M_{f}\) on the dataset to be removed, calculates \(\mu_{r}\) and \(\Sigma_{r}\) according to Equations (11) and (12), and finally uses \(\mu_{r}\) and \(\Sigma_{r}\) to generate the parameters of \(M_{r}\). Note that our method does not require covariance information, since the calculation of \(\mu_{r}\) does not require \(\Sigma\).