## 2 The Principles of Polybinary Transformation
Figure 4b,c show the PDF diagrams after duobinary transformation and tribinary transformation, respectively. Compared with Figure 4a,b, we find that after duobinary transformation, the serious overlap of the constellation points makes them difficult to distinguish. In this case, the decoding algorithm cannot work effectively. However, if we utilize tribinary transformation, the situation is improved. As shown in Figure 4c, the overlapping phenomenon seems greatly reduced. After using tribinary transformation instead of duobinary transformation,  
Figure 4: The PDF of n-level polybinary transformation for QPSK signal under the time-packing factor of 0.5: (**a**) original QPSK signal; (**b**) after duobinary transformation; (**c**) after tribinary transformation. Figure 3: The PDF of n-level polybinary transformation for 16 QAM signal: (**a**) 1-level polybinary transformation of original signal; (**b**) duobinary transformation; (**c**) tribinary transformation. as in Figure 4c, we can see that the 16 clusters are clearly distinguishable and the subsequent decoding algorithms will complete the decoding work. Based on the above discussions, we can find that an appropriate n-level polybinary transformation can somewhat solve the ISI problems in the FTN system. However, when using the incorrect transformation, we cannot achieve the elimination of ISI in the FTN signal, instead aggravating the ISI problem. Therefore, we need to explore how to use the appropriate n-level polybinary transformation and achieve the best performance of the decoding algorithm. We build a 28 GBaud PDM-FTN-QPSK system as a simulation platform, as shown in Figure 5, to test the influence of n-level polybinary transformation on the performance of MLSE under the different time-packing factors. The roll-off factor is 0.15 for the model of pulse shaping. We assume that there exist no frequency offset (FO) or PN for the transmitter laser and local laser. We also assume that the channel impairments, such as chromatic dispersion (CD), polarization mode dispersion (PMD) and rotation of state of polarization (RSOP), have been equalized by corresponding DSP algorithms. (We have conducted work to jointly equalize PMD and RSOP in the FTN system under a small time-packing factor [17]. In this paper, we only focus our attention on coping with the problem of ISI induced in the FTN system.) Therefore, the remaining impairments that we considered only included ISI and ASE noise in the FTN system. As we have discussed, the appropriate n-level polybinary transformation is a prerequisite for the MLSE algorithm to solve the signal's ISI. Next, we will present the comparisons. We let the signal pass through three combinations of n-level polybinary transformation and subsequent MLSE. In other words, we allow the signal to pass through module 1 in Figure 5. The first combination is 1-level polybinary transformation + MLSE. This means that only MLSE is used to implement ISI elimination. The second combination is 2-level polybinary transformation + MLSE (the equivalent name is duobinary transformation + MLSE). The third combination is 3-level polybinary transformation + MLSE (the equivalent name is tribinary  
Figure 5: Simulation platform for 28 GBaud PDM-FTN-QPSK system. transformation + MLSE). The comparison of the aforementioned three combination algorithms' performance for the elimination of ISI is shown in Figure 6. Compared with the other two combinations, we can find that MLSE (0.95-1.00 time-packing factor), duobinary transformation + MLSE (0.55-0.90 time-packing factor) and tribinary transformation + MLSE (0.45-0.50 time-packing factor) have better performance, as shown in Figure 6a-c. Why does the phenomenon in which the combination of n-level polybinary transformation and MLSE leads to different decoding performance under different time-packing factors occur? We will explain this phenomenon as follows to ensure that we choose the appropriate n-level polybinary transformation for eliminating ISI under a certain time-packing factor.