# IoT-Portrait: Automatically Identifying IoT Devices via Transformer with Incremental Learning
## 4 System Design
### Class Incremental Learning Method  
When a new device joins the network, we need to use all device samples to retrain the classification model. This will consume much time to retrain and requires much space to store all samples. At the same time, in real task scenarios, the past samples cannot be fully preserved due to issues such as privacy, data storage, and computing power limitations. To ensure the accuracy of device identification and less training time, we use a little storage space to save exemplars of old classes and use the Fine Tuning with Distillation Loss (FTDL) method to retrain the old classifier.  
The FTDL method adds the distillation loss into the original cross-entropy loss function. When training the new model, the distillation loss will calculate the discrepancy in the output logits between the old model and the new model, which will make the predictions of the new model as close as possible to that of the old model. Compared with the Fine Tuning (FT) method which preserves the knowledge learned from the old data by fixing the parameters of the old model and learns new knowledge by retraining the dense layer, the FTDL method uses distillation loss [28; 29] to retrain the classifier and can restore the prediction results of the old model to the greatest extent while learning new data. Compared with the Multiple Binary Classifiers (MBCs) [30], which train multiple binary classifiers that each class \(i\) has a corresponding binary classifier \(C_{i}\), the FTDL method only trains one multi-classifier and retrains it when necessary. Although the MBC method does not need to retrain existing classifiers and just train a new binary classifier for the new device, the cost of the memory and prediction time of the FTDL method is much less than the MBC method.  
Suppose that we store a total of \(N_{stored}\) samples, the classifier can currently classify \(c_{1}\) categories, and \(c_{2}\) new devices join the network. We use distillation loss to retain old knowledge, and the loss function is shown in Equation (3). Suppose that the output of the old model can be computed by \(f(x)\) and that of the new model corresponds to \(g(x)\), and \(f_{y}(x_{i})\) represents the value corresponding to class \(y\) in the output logits of the old model of \(x_{i}\). The value of \(\delta\) is 1 when the true label of sample \(i\) is y and 0 when not. For a new class sample, the corresponding label is a vector like \(\{0,0,0,1,0,...,0\}\), in which the class number of this sample corresponds to 1 and others correspond to 0. For exemplars of old classes, we predict these data through the old model, expand the dimension of output logits \(\{f_{0}(x_{i}),f_{1}(x_{i}),...,f_{c_{1}}(x_{i})\}\) to \(c_{1}+c_{2}\), and regard the result \(\{f_{0}(x_{i}),...,f_{c_{1}}(x_{i}),0,...,0\}\) as labels instead of the class number. In this way, we can restore the prediction results of the old model to the greatest extent while learning new data.  
\[Loss=-\sum_{i=1}^{n}[\sum_{y=1}^{c_{1}}(f_{y}(x_{i})\log g_{y}(x_{i})+(1-f_{y} (x_{i}))\log(1-g_{y}(x_{i})))+\sum_{y=c_{1}+1}^{c_{2}}(\delta_{y=y_{i}}\log g_ {y}(x_{i})+\delta_{y\
eq y_{i}}\log(1-g_{y}(x_{i})))] \tag{3}\]  
After training the new model, we will randomly select \(\frac{N_{stored}}{c_{1}+c_{2}}\) exemplars for each class and store them in memory.