# MT-FBERT: Malicious Traffic Detection Based on Efficient Federated Learning of BERT
## 2 Related Works
### Federated Learning  
FL was first introduced by Google in 2016 [38]. It is an emerging deep learning paradigm that facilitates data sharing without requiring data to be uploaded to a central server [39], thereby addressing data silos and enhancing data privacy [40].  
In recent years, FL has been applied to the task of network traffic classification. [PERSON] et al. [41] proposed FLIC, a federated-learning traffic classification protocol that can distinguish new applications in real-time without compromising privacy. [PERSON] et al. [42] proposed an Edge Device Identification (EDI) method based on FL. This method has a faster training speed and shorter training time than the centralized learning method. [PERSON] et al. [43] presented a cross-silo horizontal FL scheme to classify the traffic based on supervised feature-based deep learning and proved that the federated model is also comparable to the centralized training approach. [PERSON] et al. [44] proposed a federated approach for privacy-preserving network traffic classification in heterogeneous environments. This approach incorporates a client selection algorithm designed to proactively and intelligently identify clients with low skewness, enabling their participation in model aggregation within the federated learning framework. The aforementioned methods, which combine FL with supervised learning, depend on labeled samples for network traffic classification. In addition, these FL training methods focus on the smaller datasets and the use of relatively small network models for training.  
Recently, the technology integrating FL with pre-trained models has been proposed [45]. On one hand, this technology can ensure user privacy and effectively utilize more training data. On the other hand, pre-trained learning can alleviate the dependence on labeled data. Yet, current FL training methods typically require substantial communication and computational resources [46; 47]. This renders training complex models within the FL framework exceptionally challenging, particularly for pre-trained models that have achieved remarkable progress in natural language processing tasks, such as GPT [32] and BERT [31].  
To enable more complex models to adapt to the limited hardware resources and communication bandwidth of federated learning clients, FedSplitBERT [46] proposes a novel framework that reduces communication costs by partitioning BERT encoder layers into local and global components. Parameters of the local part are trained exclusively by local clients, while global part parameters are optimized through gradient aggregation across multiple clients. FedPETuning [47] investigates parameter-efficient tuning methods to alleviate the communication overhead of pre-trained models in federated learning, which keep most pre-trained model parameters frozen and only fine-tune additional lightweight parameters or a subset of parameters for downstream tasks. These methods enhance efficiency by designing the parameters for model updates in federated learning. However, these approaches primarily focus on downstream task fine-tuning and show limited adaptability to pre-training tasks [48]. Additionally, their effectiveness in the malicious network traffic detection task remains unapplied and uninvestigated. It is critical to incorporate network packet data analysis components for extracting features from network traffic in malicious network traffic detection.  
Given the uniqueness of network packet data analysis and the requirement for high accuracy via pre-training and fine-tuning, sharing actual network traffic data for training purposes may raise privacy concerns. We propose MT-FBERT for malicious network traffic detection, an efficient, privacy-preserving architecture specifically designed to address these considerations.