## 3 Experimental Setup  
The C-band 80 Gb/s coherent optical PAM4 transmission system setup and DSP block diagram are shown in Figure 3a. At the transmitter, a pseudo-random sequence of the length 211-1 was generated using MATLAB (R2022b) and mapped into PAM4 symbols. PAM4 symbols were upsampled by a factor of 4 and pulse-shaped using a root-raised cosine filter with a roll-off factor of 0.01. Then, the resulting PAM4 signal was loaded into an arbitrary waveform generator (AWG) to generate a 40 GBaud PAM4 electrical signal. After that, the 40 GBaud PAM4 electrical signal from AWG was amplified by a linear electrical amplifier (EA) and then fed into an optical modulator for electrical-optical conversion. Due to the insufficient bandwidth of the electro-absorption modulated laser (EML) in our laboratory for this experiment, we used a Mach-Zehnder modulator (MZM) with a 3 dB bandwidth of 28 GHz as the optical modulator. The laser source was an external cavity laser (ECL) with a center wavelength of 1550 nm and linewidth of 500 Hz. Semiconductor optical amplifiers (SOAs) are considered a promising solution for increasing output power and are commonly integrated with distributed feedback (DFB) lasers and electro-absorption modulators (EAMs) to form EML, thus compensating for modulator losses [22]. Consequently, the SOA is used to amplify the signal power before the PAM4 signal enters the standard single-mode fiber (SSMF). Figure 3b shows the experimentally measured SOA input-output curves, and it can be seen that excessive input power provides higher output power but also leads to SOA gain saturation. At the receiver, the received optical power (ROP) of the signal was adjusted using a variable optical attenuator (VOA), and the polarization state was controlled by a polarization controller (PC). The tunable semiconductor laser (TSL) with an output power of 12 dBm and a linewidth of 100 kHz was used as the local oscillator. Subsequently, the signal was intrudyne-detected by a 25 GHz integrated coherent receiver (ICR), generating two IQ signals. Next, the detected signal was digitized at a rate of 160 GSa/s using a real-time digital signal analyzer (DSA). We performed offline DSP on the received data using MATLAB. Initially, the PAM4 complex signal formed by the combination of the two IQ signals was subjected to matched filtering and then downsampled by a factor of 4. Next, we determined the starting point of the corresponding training sequence by calculating the correlation between the received signal and the training sequence. Then, the training sequence was used to optimize and obtain the best tap coefficients for the PA-DFE and MA-DFE, thereby equalizing the PAM4 signal. Finally, PAM4 signals are hard-determined and demapped, and then the BER is calculated. The experimentally measured MZM transmission curve is depicted in Figure 4a. When employing it for intensity modulation, the DC bias voltage setting affects the extinction ratio of the PAM4 signal. Typically, setting the DC bias point at the quadrature point ensures linear output from the MZM. However, this configuration typically results in a lower extinction ratio of the PAM4 signal, thereby reducing its resistance to interference. If the DC bias voltage is adjusted to achieve a high extinction ratio, the offset of the bias point will cause a part of the MZM output to enter the nonlinear region, resulting in signal distortion. Similarly, this issue arises when using EAM for PAM4 modulation. Figure 3: (a) C-band 80 Gb/s coherent optical PAM4 transmission system diagram and DSP block diagram (The black arrows represent the electrical circuit, and the red arrows represent the optical path.). (b) Experimentally measured input–output curve of the SOA. It is worth emphasizing that our proposed PA-DFE and MA-DFE can mitigate these nonlinear impairments to some extent, so that the optimal bias point may not be at the quadrature point.