## 1 Introduction
### Previous Work and Background  
Multiple approaches have been analyzed to date to improve the effectiveness of LDOTNs. These are addressed as follows. The authors explore nonlinearities in 16 QAM and 64 QAM coherent optical communication by using the k-nearest neighbor algorithm in [17]. Experimental results were successfully performed for 800 km path cover. In [18], the dynamic equalization approach was proposed to handle digital-to-analog conversion resolution issues. Nonlinear deterministic operations were considered for this proposed process. Furthermore, the Viterbi algorithm was developed to maintain dynamic quantization issues. In [19], the authors proposed a joint processing algorithm for carrier-less amplitude and phase modulation to improve the system fidelity for the short and metro network. The penalties due to nonlinearities are minimized using Volterra filters and multiple input and output Volterra filters. In [20], the authors concluded the future capacity increase and discussed the lower bounds as a function of fiber bandwidth, modulation formats, and nonlinearity compensation. In [21], the researchers used the 32 QAM and 64 QAM modulation formats over standard single-mode fiber to offset nonlinear issues. In [22], the authors investigated the nonlinear impairments generated due to the high capacity transmitted signal, implementing the 64 QAM modulation scheme. Then, the achieved results were compared with 16 QAM and 32 QAM modulation techniques. In [23], the authors used the technique of artificial neural networks to minimize the nonlinear impairments in LDOTNs. In [24], geometric and probabilistic methods were presented in 64 Qam modulation format for analyzing nonlinear impacts. In [25], the authors applied techniques such as Kalman smoother and Turbo equalization to resolve inter-channel nonlinearity issues. The 64 QAM advanced modulation format was also installed in this work. The 64 QAM modulation system with the support vector machine was used by the researchers in [26] to determine the optimum solution to signal distortion factors. In [27], the authors utilized optical backpropagation and dispersion compensation fiber methodologies with a 64 QAM modulation scheme for compensating nonlinear factors. Similarly, in [28; 29], the authors launched techniques including nonlinear frequency division multiplexing with a 32 QAM modulation format and orthogonal frequency division multiplexing based on Trellis-coded modulation, respectively, for balancing the nonlinear effects inside LDOTNs. However, these discussed methods were found to be effective and not tailored to the specific scenario. Moreover, the practical implementation of these recent approaches is complex, and no real-time implementation is presented. The modification of the soft decoding method at the receiver side is proposed in this paper, aiming to minimize the NLs' impact, which degrades the channel's statistics. In [30], various modulation formats were studied for enhancing the capacity of optical transmission systems. The effects of modulation formats were evaluated in terms of fiber Bragg gratings, the optical fiber amplifier and dispersion compensation fiber techniques. The nonlinear issues were reduced in [31] for an optical communication system using neural network nonlinear equalizers in 16 QAM modulation format. Nonlinearities are investigated in this paper for the multichannel system.