D-Band 4.6 km 2 \(\times\) 2 MIMO Photonic-Assisted Terahertz Wireless Communication Utilizing Iterative Pruning Deep Neural Network-Based Nonlinear Equalization  
[PERSON]  
S.-Y. bands demonstrate the potential of each band in enhancing RoF system transmission performance [8; 9; 10]. Notably, [PERSON] et al. achieved 4.6 km wireless transmission in the D-band in 2022, setting a record for the longest transmission distance [9]. These studies provide valuable experience and data support for the application of RoF systems across various bands. However, RoF systems may suffer from various nonlinear effects, which affect system performance, especially at high bit rates and high spectral efficiencies. Previously, researchers mostly used traditional equalization algorithms such as Decision Feedback Equalizer (DFE), Least Mean Square (LMS), and Volterra to handle nonlinearity [11; 12; 13; 14]. In recent years, researchers have improved these algorithms to suit different systems. Specifically, using the MIMO structure Volterra nonlinear equalization (VNE) algorithm, [PERSON] et al. successfully transmitted 25-Gbaud 16-QAM signals over a 4.6 km 2 \(\times\) 2 MIMO wireless system at the 125 GHz D-band in 2023, achieving a minimum BER of 2.05 \(\times\) 10\({}^{-2}\)[15]. However, these improved traditional equalization algorithms have limited capabilities. To address these limitations, the use of machine learning (ML) technology in RoF systems has been proposed and has become an active research area [16]. Unlike traditional methods, ML methods learn system impairments from training data. Compared with traditional signal processing methods, ML-based schemes consider and handle all impairments simultaneously, as well as the interactions between different types of impairments. Researchers have experimented with and applied various types of ML methods. In the non-neural network domain, researchers achieved 6 GHz RoF transmission using the k-means algorithm in 2010 [17]. Subsequently, researchers improved the performance of the k-means algorithm in 6 GHz RoF systems using the Fuzzy c-means Gustafson-Kessel (FCM-GK) algorithm [18]. Researchers also verified the effectiveness of the k-nearest neighbors (KNN) algorithm and support vector machine (SVM) in reducing BER in RoF systems under single-channel and single-polarization conditions [19; 20]. In 2017, SVM was applied to a single-carrier 2.4 Gbps 56.2 GHz RoF system [21]. In 2020, researchers proposed and validated the effectiveness of a deep reinforcement learning (DRL) method based on the proximal policy optimization (PPO) algorithm, which can be effectively extended to MIMO systems [22]. Non-neural network algorithms have only been validated in simulations and low-speed, short-distance RoF systems. Neural networks have been more maturely applied in high-speed, long-distance RoF systems. Researchers have verified that neural network algorithms significantly outperform traditional algorithms in handling nonlinear effects in RoF systems [23]. In 2017, researchers demonstrated the advantage of DNN over VNLE in a 5 Gbps 60 GHz RoF system [24]. In 2020, researchers began applying convolutional neural networks (CNNs) and Binary Convolutional Neural Networks (BCNNs) to a 5 Gbps 60 GHz RoF system [25]. In recent years, researchers have improved DNN and CNN algorithms, introducing complex-valued neural networks (CVNNs) and 2D-CNN to achieve equalization of complex signals in RoF communication systems. These algorithms further reduced the BER of DNNs and CNNs [26; 27]. Subsequently, researchers applied complex neural networks to dual-polarization systems, achieving transmission below the hard decision forward error correction (HD-FEC) BER threshold in a 30 GBaud 320 GHz THz system [28]. In the latest research, Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) have also been introduced for signal equalization in RoF systems [29; 30].