# Compensation of Limited Bandwidth and Nonlinearity for Coherent Transponder
## 1 Introduction  
Today's telecommunications infrastructure relies on optical fiber communications systems where the coherent in-phase quadrature (IQ) optical transceiver is an essential component. To deliver a large amount of information over a long distance, the high-order quadrature amplitude modulation (QAM) running at a high baud-rate has been adopted in the long-haul transmission system [1; 2]. The information is carried on two orthogonal domains. One domain is the in-phase (I) and quadrature (Q), and the other domain is the two orthogonal polarizations, which are X and Y. Thus, four tributary channels are formed, namely XI, XQ, YI, and YQ. Within each tributary, there are impairments due to the limited bandwidth and nonlinearity. Among these tributaries, there are degrading effects due to the time skew and power imbalance. In this work, we mainly focus on the impairments within each tributary. The limited bandwidth ultimately determines the highest achievable baud rate of the coherent IQ transponder. It is critical to compensate for the limited bandwidth, particularly for the high baud rate coherent IQ transponder. For example, the state-of-art system runs at the all-electronically multiplexed symbol rate of 180 GBd (giga baud) [3]. The nonlinearity causes uneven distribution of the constellation points in the complex plane and eventually limits the signal-to-noise ratio (SNR). Thus, it is equally important to mitigate the nonlinearity for the coherent IQ transponder using the high order QAM. As an example, the next generation coherent DWDM system uses the probability shaping constellation (PSC) QAM [4], and the state-of-the-art demonstration utilizes 4096-PSC-QAM [5]. The limited bandwidth is usually mitigated by the finite impulse response (FIR) filter through the pre-emphasis process in the digital domain. For example, a 61 GBd coherent system can still be demonstrated when the overall electrical bandwidth of the coherent IQ transceiver is less than 15 GHz [6]. To set the FIR filter, one needs to measure the system's bandwidth accurately. A traditional method is to apply a stimulus, for example a sinusoidal signal, to the Mach-Zehnder modulator (MZM). By scanning the frequency of the stimulus and measuring the optical power from the output of MZM, one can determine its bandwidth. To limit the influence of the nonlinearity, the sinusoidal signal has a small amplitude. However, the actual analog signal generated by real data is a large signal applied to the MZM to utilize the dynamic range of the coherent IQ transmitter fully. On the other hand, the bandwidth measured with a large signal is influenced by the nonlinearity, making the measurement result inaccurate. Multiple methods have been demonstrated to mitigate the nonlinearity. In [7], a lightwave component analyzer is used to determine the limited bandwidth by performing a small-signal measurement. The limited bandwidth is compensated by the first-order kernel of the Volterra filter. Next, the constellation diagram is recovered by the coherent receiver through the digital signal processing (DSP). The second-order kernel and the third-order kernel of the Volterra filters are adaptively updated through the indirect learning algorithm. In such a routine, the nonlinearity is mitigated by minimizing the error function. In [8], the indirect learning algorithm is further expanded so that the limited bandwidth, the time skew between the in-phase tributary and the quadrature tributary, and the nonlinearity are simultaneously compensated by the Volterra filter. In [9], the limited bandwidth, the time skew, and the nonlinearity are simultaneously detected by a high-speed photodiode and a sampling oscilloscope. The high-order Volterra filter is used to compensate those impairments. In [10], a look-up table (LUT) mitigates the nonlinearity and pattern-dependent distortion. The 7-symbol LUT is trained by determining the difference between the training symbol and the actual sample value for the signal.