## 1 Introduction  
The fast progression of worldwide information is instigating significant modifications in mobile communication technology. With the advent of the 6G communication era, burgeoning applications like the metaverse, autonomous driving, and holographic communication necessitate wireless networks characterized by ultra-high data speeds (about 100 Gbps), ultra-low latency (sub-millisecond), and ultra-large connection density [1]. The conventional microwave frequency ranges are becoming progressively insufficient for these demands. In this context, the D-band (110-170 GHz) has emerged as a highly attractive contender for 6G wireless communication. The enormous bandwidth of the D-band facilitates substantial data transmission, offering the capability for high-rate communications [2]. Photonics-aided millimeter-wave generation uses optical heterodyne beating and optical frequency comb generation to overcome bandwidth, transmission loss, and phase stability issues in conventional electronic devices at the D-band (110-170 GHz) [3]. This method uses photonic techniques to achieve high spectral purity, low phase noise, and wide bandwidth scalability, enabling the reliable generation of high-frequency millimeter-wave signals in the demanding D-band frequency range [4]. RoF technology integrates the high-bandwidth, low-loss attributes of optical communication with the adaptability of wireless communication [5, 6, 7]. In a RoF system, radio-frequency (RF) signals are modulated onto optical carriers, transmitted over optical fibers, and subsequently converted back to RF signals at the receiver. This method capitalizes on the low-loss characteristics of optical fibers to efficiently increase signal transmission distance and employs the high-bandwidth benefits of optical communication to improve system transmission capacity [8, 9, 10, 11]. The nonlinear properties of optoelectronic devices, such as modulators and photodetectors, generate supplementary harmonic components during signal modulation and detection, disrupting the original signal. The Kerr effect in fiber transmission generates nonlinear phase noise by rendering the optical signal's phase contingent upon its intensity. These effects collectively result in amplitude distortion and phase noise, significantly impairing system performance. The restricted linewidth of lasers presents another challenge: while laser output is not a perfect monochromatic signal but encompasses a specific frequency range (i.e., linewidth), it results in random phase drift. As transmission distance and signal rate escalate, the accumulation of phase drift causes rotation and diffusion of the constellation diagram of received signals, degrading signal quality and elevating the bit error rate [12]. In D-band RoF systems, addressing nonlinear distortion and phase noise compensation is a significant research problem. Classical nonlinear approaches like the Volterra series are based on linear time-invariant (LTI) system theory. These methods estimate nonlinear distortion using polynomial models to demonstrate how power amplifiers (PAs) link input and output as polynomial terms and delayed input signals. Limitations include the fast growth of model complexity as nonlinear order rises, which requires more processing power and hardware. In real-world application, high-order model parameters are hard to estimate, fit too tightly to training data, and perform poorly in varied conditions, resulting in inaccurate compensation performance. Due to their robust nonlinear fitting characteristics, deep neural networks (DNNs) have seen extensive use in nonlinear compensation in recent years. They can autonomously acquire complex signal feature patterns to mitigate nonlinear distortion [13, 14]. Nevertheless, DNNs have significant constraints in phase noise compensation. Phase noise is a random process that changes over time, making it hard for traditional statistical learning methods to effectively understand its changing characteristics.