# Pruning Adapters with Lottery Ticket
## 2 Materials and Methods
### Pruning Schemes  
Here, we systematically propose three pruning strategies for weights, neurons, and the adapter layer, respectively, which are formalized in the following Algorithms 1-3.  
**Weight Pruning:** We prune a weight solely by its magnitude. At iteration \(t\), we calculate the absolute value of weights \(|\theta_{t}|\) in all adapters. We save the non-zero value and its corresponding position in \(\theta_{t}\) in a list \(R\). We sort \(R\) in the descending order by the absolute value. The last \(p\%\) of \(R\) with the lowest magnitude are pruned, and their corresponding positions in mask and \(\theta_{t}\) are both set to 0. The remaining parameters are set back to the initial values, and the training restarts without the pruned weights.  
```
Result: a list of tuple containing values of importance and positions in weight matrix \(R\) is an empty list;  Weights of adapter at iteration \(t\) as \(\theta_{t}\); forlayer \(l\) in \(\theta_{t}\)do forweight \(a\) in layer \(l\)do if\(a\) is not zerothen  Value of importance \(Imp_{a}=|a|\) ;  Position of weight \(a\) in \(\theta_{t}=Pos_{a}\) ;  Append tuple \((Imp_{a},Pos_{a})\) to list \(R\) ;  end if  end for Sort list \(R\) with \(Imp\)  end for
```  
**Algorithm 1**Identify the importance of weights  
**Neuron Pruning:** If we need to prune neurons, this can be done by the \(L_{2}\) norm of their associated weights. An adapter module consists of a down-project weight matrix and an up-project weight matrix with residual connection. To prune the neuron inside an adapter, we will simplify the up-project weight matrix because the column in the weight matrix corresponds to the connection to the neuron. Let \(\theta_{t,l,up}\) be the up-project weight matrix of the adapter layer \(l\) at iteration \(t\) and \(a_{i,j}\) denote the weight parameters in \(\theta_{t,l,up}\). Therefore, the importance of neuron \(i\) in the adapter of size \(N\) is \(\sum_{j=0}^{N}a_{i,j}^{2}\). Likewise, neurons are sorted in the descending order by their importance, and the last \(p\%\) of the neurons are pruned and their mask is set to 0. Then, the remaining parameters are put back to their initial values.  
```
Result: a list of tuple containing values of importance and positions in weight matrix \(R\) is an empty list; The size of adapter is \(N\) ; Weights of adapter at iteration \(t\) as \(\theta_{t}\); forlayer \(l\) in \(\theta_{t}\)do  up-project weight matrix \(l_{u}\) in layer \(l\) ; forcolumn \(i\) in \(l_{u}\)do if\(\sum_{j=0}^{N}a_{i,j}\) is not zerothen  Value of importance \(Imp_{c}=\sum_{j=0}^{N}a_{i,j}^{2}\) ;  Position of column \(a\) in \(\theta_{t}=Pos_{c}\) ;  Append tuple \((Imp_{c},Pos_{c})\) to list \(R\) ;  end for  end for Sort list \(R\) with \(Imp\)  end for
```  
**Algorithm 2**Identifify the importance of neurons  
**Adapter Pruning:** Finally, we can even prune adapters based on their sum of weights magnitude. Let \(\theta_{t,l}\) be the weights of the adapter at layer \(l\) at iteration \(t\) and \(a_{i,j}\) denote the parameters in \(\theta_{t,l}\). The importance of an adapter of size \(N\) with input size of \(H\) is \(\sum_{i,j}^{N,H}|a_{i,j}|\). Adapters are sorted by the sum of weights in the descending order as well, and the \(p\%\) smallest adapters in list \(R\) are removed from the model. The remaining adapters step back to their initial weights for the re-training.  
```
Result: a list of tuple containing values of importance and the number of layers \(R\) is an empty list; The size of adapter is \(N\) ; Input size of adapter is \(H\) ; Weights of adapter at iteration \(t\) as \(\theta_{t}\); forlayer \(l\) in \(\theta_{t}\)do iflayer \(l\) not prunedthen  Value of importance \(Imp_{l}=\sum_{i,j=0}^{N,H}|a_{i,j}|\) ;  Append tuple \((Imp_{l},l)\) to list \(R\) ;  end for Sort list \(R\) with \(Imp\) end
```  
**Algorithm 3**Identifify the importance of adapter layers