# IoT-Portrait: Automatically Identifying IoT Devices via Transformer with Incremental Learning
## 4 System Design
### Multi-Classifier Training
#### 4.3.4 Model Training  
We use a hybrid architecture of the encoders of the transformer and a simple classifier based on softmax to train the model. There are two main reasons to use the transformer architecture. First, the traffic of the device is regular in time series, and both the transformer network and Recurrent Neural Network (RNN) can well mine the regularity of the features presented in the time series. Second, compared with RNN, which must be calculated in chronological order, the parallel processing mechanism of the transformer can bring higher computational efficiency and significantly speed up the training. In addition, RNN is prone to gradient explosion and gradient disappearance during training, which will affect the model training, while the transformer does not have such problems. The encoders of transformer contain six encoder layers, each of which consists of a self-attention layer and a fully connected layer. Through the transformer, we hope to find the characteristics of the adjacent packet groups. Furthermore, the output of the transformer layer is passed to the fully connected layer, and the softmax function is used to generate the corresponding probability that the sample corresponds to this label.  
The purpose of each layer and the specific parameter settings are depicted as follows:  
(1) Input layer: The size of the input layer is \(\mathit{packet\_num}\ \times\ (\mathit{packet\_len}+1)\). For image data processing, the input usually has three dimensions. The first two dimensions contain image information, and the last one contains the number of channels, usually 3, indicating R, G, and B. However, our processed data are a two-dimensional matrix in which data packets are spliced into time series. Hence, the channel is 1, and we omit it.  
(2) Transformer layer: In the input, we use a \((\mathit{packet\_len}+1)\)-dimensional vector to represent a packet. For the input sequence, the transformer uses positional encoding to record the packet order and self-attention mechanism to mine the relationship between different packet vectors. Since the attention computation involves a parameter calculated by \(\sqrt{\mathit{model\_dim}/\mathit{heads\_num}}\), and \(\mathit{heads\_num}\) is 8, \(\mathit{model\_dim}\) is the vector dimension of embedding, which is \(\mathit{packet\_len}+1\). So we set the dimension of the packet vector as 32.  
(3) Flattening: The flattening operation reduces the data dimension. The flattened one-dimensional data are then passed to the next layer.  
(4) Dense layer: After the transformer layer, the next result is obtained through a fully connected layer. The size of units corresponds to the total class number.  
(5) Softmax layer: The ultimate goal of the neural network is to classify. Hence, the classification result of the upper layer is optimized by softmax regression. The final result is a probability distribution for the classification.  
After training the classifier, a few exemplars of samples per device will be stored in memory. When new devices join the network, data packet collection and traffic pre-processing should be performed again. Then, the multi-classifier will be retrained with stored samples and new device samples using the class incremental method. The detailed retraining process and class incremental method we use will be introduced in Section 4.5.