# Advanced DSP for Coherent Optical Fiber Communication
## 1 Introduction  
With the development of Erbium doped fiber amplifier (EDFA), wavelength division multiplexing (WDM), dispersion management and optical fiber nonlinearity compensation technologies, optical fiber communication capacity has been rapidly improved over the past few decades. In addition, the research on high-order modulation formats has also increased the transmission capacity and spectral efficiency (SE) of optical fibers. Due to a smaller bandwidth occupied (for the same bit rate), systems with higher SE are usually more tolerant to chromatic dispersion (CD) and polarization mode dispersion (PMD). The fault tolerance to CD and PMD are particularly crucial in high bit-ratetransmission systems. In order to achieve a high SE, early works used direct detection (incoherent detection) and amplitude-based modulation (1-dimension). Actually, higher SE can be obtained using coherent detection and 2-dimension modulation formats. Coherent detection attracted intensive research in the 1980s due to its high receiver sensitivity. With the invention of EDFA and the development of WDM systems, research in coherent optical communication ceased in early 1990s, because of the difficulty and complexity of system implementation, especially the complicated realization of optical phase locked loop. Phase and polarization management turned out to be the major obstacles for the practical implementation of conventional coherent receivers. Fortunately, both phase and polarization management can be realized in the electrical domain using DSP. With the increasing demand on transmission capacity, coherent optical communication has attracted widespread attention again in recent years, and has become an unprecedented and promising approach for realizing high-capacity long-haul optical communication systems [1; 2; 3; 4; 5]. From the beginning of this century, coherent detection combined with large-bandwidth analog-to-digital converter (ADC), digital-to-analog converter (DAC) and DSP has increased the achievable capacity of optical fiber communication systems [6; 7; 8; 9; 10]. In coherent optical communication, information is encoded onto optical carrier waves in the complex domain, and the optical signal carrying all information is transmitted to the front end, after undergoing a series of linear and nonlinear impairments. In order to recover the transmitted signal and obtain the original information, it is necessary to measure the full complex electric field of the light wave, which means that the phase and the magnitude of optical carrier field both need to be detected. Using coherent detection, the complex field of the received signal can be completely acquired, and the linear transmission impairments, such as CD and PMD, can be fully compensated by using static and adaptive DSP. One of the main benefits of coherent optical communication is the possibility of compensation of transmission impairments using DSP. Compensation of linear impairments such as CD and PMD are now implemented using DSP. Compensation of nonlinear impairments brings about an increase in the fundamental capacity limit for fiber transmission, which is an important research area for coherent optical communications. Advanced DSP has also been developed in the frequency domain to reduce the computational complexity and maintain all of the advantages of time domain equalization (TDE) based adaptive digital filters. Machine learning (ML) techniques have become one of the most promising disciplines and is beneficial to optical fiber communication applications such as nonlinearity mitigation, optical performance monitoring (OPM), carrier recovery, in-band optical signal-to-noise ratio (OSNR) estimation and modulation format classification, and especially, advanced DSP. In this paper, the latest development and progress of DSP approaches for coherent optical communications are reviewed. The paper is organized as follows: Section 2 describes the principle and schematic of the coherent detection as well as linear impairments equalization.