Optical fiber communication has been the most versatile and revolutionary medium of communication that has helped connect millions of people across the globe. These traits have seen tremendous development in the sector of optical fiber communica- tion. There are now many desired qualities expected out of an optical fiber communication such as large transmission capacity and a smaller amount of loss. Communication systems operating with higher transmission rates have found an essential role of non-linear fiber optics. The behavior of light in non-linear media is described by non-linear optics. Cross-phase modulation (XPM) and self-phase modulation (SPM) are a few non-linearities occurring in the optical fiber. Non-linear effects reduce the performance of optical fibers. In this paper, a comparative study of non-linear effects in erbium-ytterbium co-doped fiber is reviewed using simulation. The results are compared with optical systems utilizing normal fiber. The benefits of using Er-Yb fiber have been demonstrated by analyzing effective systems comprising the Er-Yb co-doped fiber and their ability to mitigate the non-linearities. Keywords Er-Yb · SPM · XPM · Q-factor Introduction The rapid development of erbium (Er)-doped optical fiber amplifiers (EDFA) in the 1550-nm band has greatly improved different Er- and ytterbium (Yb)-doped fiber amplifiers (EYDFAs) and 1550-nm EDFAs.1 Specifically, for high-energy applications, EYDFAs are vital, counting long-distance optical communications.2 As of late, EYDFAs have been utilized in modern specialized areas, such as ultra- high-repetition-rate beat sources.3 EYDFs co-doped with Yb attain several advantages over erbium-doped fibers (EDFs) with benefits such as higher Er concentration and, signifi- cantly, a bandwidth and a greater absorption intensity around 980 nm. High-power cladding-pumped optical fiber ampli- fiers utilizing multimode laser diodes use the properties of EYDFs.4 Various stages of energy levels associated with the Er-Yb system are shown through a model in Fig. 1. Yb ions in an EYDF are excited to the 2F5/2 state by pumping light in 800–1000 nm. Because of the energy transfer from the Yb ions, while they regress to the 2F7/2 state, the 4I11/2 state is achieved by Er ions due to excitation. To transition to 4I13/2, a non-radiative process is utilized by the Er ions that are excited to 4I11/2, forming a population inversion between the 4I13/2 and 4I15/2 states. Stimulated emission causes the ampli- fication of the incident optical signal. Yb ions and Er ions cluster together with low solubility towards a silica host and have the same ionic radius. In this manner, the separation between Yb ions and Er ions decreases, and energy transfer continues in an incredibly efficient way.5 There have been various works done in investigating and mitigating non-linear effects. The authors in Refs. 6 , 7 and 8 have been able to reduce the non-linearities using multiple methods. The methodologies adopted by the authors in Ref. 6 use a regenerative system based on SPM with a bandwidth of 10 Gb/s; the authors in Ref. 7 have developed a regenera- tive system based on highly non-linear fibers (HNLF); the authors in Ref. 8 utilized a multimode non-linear fiber with a numerical approach. The work done by these authors has led us to study the benefits of using a system comprising Er-Yb fiber to overcome the issues developed in the above positions. We have investigated the properties of Er-Yb fiber by comparing it with other similar systems consisting of * Aditya Kumar aditya18rocks@gmail.com Gausia Qazi gausia.qazi@nitsri.net 1 Department of Electronics and Communication Engineering, National Institute of Technology, Srinagar, Jammu, and Kashmir, India A. Kumar, G. Qazi 1 3 only a normal fiber for transmission. The issues were poor quality factor (Q-factor) in Ref. 6 and a limited data rate.6–8 To study the benefits of using the Er-Yb fiber, we have performed a comparative study, analyzing the quality factor and data rates of an optical communication system contain- ing Er-Yb fiber to the optical systems proposed by authors in Refs. 9 and 10. The findings observed during the comparison will help overcome the shortcomings of the above works, thus providing a simple yet more efficient system to mitigate non-linearities using Er-Yb co-doped fiber. The paper discusses the origin of non-linearity and dis- cusses self-phase modulation (SPM) in Sect. 2 and cross- phase modulation (XPM) in Sect. 3. The simulation setup and its properties are discussed in Sect. 4. The results and analysis are presented in Sect. 5, and the conclusion is given in Sect. 6. Non-linearity in its preliminary stage originated due to the impact of an applied field on the anharmonic motion of bounded electrons. The total polarization (P ) is affected by this anharmonic motion; electric dipoles induce P, and its value is not linear but sums up additionally as a general relation given by where χ(l) (l = 1, 2, … ) is lth order susceptibility and per - mittivity of vacuum is given by ε 0. The linear susceptibil- ity χ(1) provides the dominant contribution to P . For sum- frequency generation and second harmonic generation, the second-order susceptibility χ(2) is responsible. χ(2) vanishes for a symmetrical molecule such as silica. Therefore, the second-order non-linear refractive effects are not exhibited by optical fibers. For non-linear effects in fibers of the low - est order, the third-order susceptibility χ(3) is responsible.11 Er-Yb fiber has been selected for analysis due to its high efficiency, low noise,12 and cost-effectiveness.13 Values of non-linearities must be kept as low as possible for effective communication.14(1) P=/u1D7000/u1D712(1)E1+/u1D7000/u1D712(2)E2+/u1D7000/u1D712(3)E3+⋯Self‑Phase Modulation The intensity-dependent phase transition of the fiber is trig- gered by the intensity-dependent refractive index.15 Kerr non-linearities for the propagating light pulse through the fiber form a separate phase transmission for the pulse height relative to the leading and trailing ends of the light pulse. Such an effect that causes modifications to the pulse spec- trum is known as self-phase modulation (SPM). A time- varying frequency is created by the time-varying phase due to the instantaneous frequency of a wave, which is the time derivative of its phase. Therefore, SPM is responsible for the alteration and broadening of the frequency spectrum of the pulse. Dispersion-like effects are produced due to the spec- tral broadening caused by SPM, limiting the transmission rates in some of the optical communication systems of the long-haul category, depending on its chromatic dispersion and type of fiber utilized for transmission.16 A time-varying refractive index is caused due to the increase in intensity variation. It falls as the pulse goes past at any single point in the fiber, while the optical pulse propagates through the fiber. A non-linear phase shift, which varies with time, is caused due to the temporally varying index change. The optical field phase changes are given by Ref. 17 The linear refractive index is denoted by n , and n2 indi- cates the non-linear refractive index of the fiber. (|E |2 is the optical signal intensity). The free-space wave number is denoted by k0 and contains the value of 2/u1D70B∕/u1D706 , λ is the optical signal wavelength, and L denotes the fiber length.17 Intensity dependence of phase fluctuations causes different phase shifts in different parts of the pulse leading to fre- quency chirping. A frequency shift in the upper side is pro- duced in the leading edge of the optical pulse, whereas the frequency shift in the lower side is produced in the trailing edge. Therefore, a significant effect of SPM is to broaden the pulse spectrum, with the temporal shape untouched. In high-transmitted power systems, SPM effects are more pro- nounced because the effect of chirping is proportional to signal power transmitted.18 An induced phase shift in SPM is proportional to the intensity, occurring due to the non-linear refractive index value.19 The formula between refractive index and intensity can be described as: where the non-linear index is denoted by n2 and optical intensity is denoted by I(t). The pulse phase is given by:(2) /u1D719=/parenleft.s1n+n2/uni007C.varE/uni007C.var2/parenright.s1k02L (3) Δn=n2I(t) Fig. 1 Energy levels in the Er-Yb system. A Comparative Study of SPM and XPM in Er -Yb Co -Doped Fiber 1 3 here L is the propagated distance by the pulse, and the car - rier frequency is given by ω0 is the and λ0 denotes the wave- length of the pulse.19 Cross‑Phase Modulation Effects of XPM and SPM are related. Distinguishable and overlapping pulses with different wavelengths or polari- zations are involved in this case. Phase modulation of the overlapping pulse(s) is caused by the intensity variations of one pulse that modulates the fiber's refractive index. This modulation of phase in SPM converts into a modula- tion in frequency that broadens the spectrum of the pulse. XPM is, therefore, displayed as a mechanism of crosstalk between channels. This occurs when the dispersive optical fiber undergoes intensity modulation during transmission, or when phase encoding is used with the increase in the number of channels, the strength of XPM increases. Decreasing the channel spacing also increases the strength of XPM. Nevertheless, there is no exchange of energy among chan- nels, differentiating the effect from other crosstalk processes in which a signal power increase in one channel arises only through a power reduction in another. It can be inferred that the overall intensity is the square of the number of two electric field amplitudes, which makes the total strength of XPM twice that of SPM. This influence is diminished as pulses with different wavelengths or polarizations are typi- cally not balanced by group velocity, and thus overlap is not (4) /u1D719(t)=/u1D7140t−kx=/u1D7140t−2/u1D70B /u1D7060n(I)Lpreserved.16 The shift for the ith channel in an N-channel transmission system can be given as,18 The above equation contains a factor 2, which has its ori- gin in the form of non-linear susceptibility and indicates the same amount of power; the XPM is twice as effective as SPM. The first term and the second term in the above equa- tion represent the contribution of SPM and XPM, respec - tively. It can be inferred that XPM is effective only when the interacting signals are superimposed in time.17 Total non-linear phase shift in a fiber of length L:19 Simulation Setup of the System OptiSystem 0920 simulation software platform was used to study the various parameters responsible for surging SPM and XPM effects.9 The authors in Ref. 10 have also utilized OptiSystem to study non-linearities in fiber. A system to study SPM analysis has been created and is shown in Fig. 2. The system operates at a wavelength of 1550 nm. The power provided to the input is varied from 10 dBm to 20 dBm. The length of fiber is assumed to be 100 km. The transmission rate is fixed at 10 Gb/s. The Er-Yb fiber length is set at 1 m, and a pump laser is used to amplify the signal and improve efficiency. SPM effect is introduced in the fiber. The analysis to find the eye-diagram is done using a bit error rate (BER) (5) /u1D719i nl=/parenleft.s4 Pi+2N/uni2211.s1 n≠ipi/parenright.s4 knlLeﬀ (6) /u1D719NL=/parenleft.s22/u1D70B /u1D706/parenright.s2 n2/parenleft.s1/uni007C.varI1/uni007C.var2+b/uni007C.varI2/uni007C.var2/parenright.s1 Fig. 2 SPM simulation setup. A. Kumar, G. Qazi 1 3 analyzer to devise the Q-factor that determines the distortion in the signal. A photodetector PIN, low-pass Bessel filter (LPBF) with a cut-off frequency set at 0.65 * bitrate, and a BER analyzer constitute the receiver section (Table I). A wavelength divison multiplexing (WDM) system consisting of two channels has been utilized to analyze the effects of XPM, and it is shown in Fig. 3. To generate sig- nals in the system, two optical Gaussian pulse generators and two user-defined bit sequence generators are used to provide input signals. Multiplexing of two signals over a single channel is done using an ideal multiplexer. The signal after the process of multiplexing is further passed to the fiber and passed through EYDF. The length of the EYDF is set at 1 m. The length of optical fiber is assumed to be 50 km. A BER analyzer is used to obtain the Q-Factor for the assumed signal. A photodetector PIN, LPBF with a cut-off frequency set at 0.70 * (bitrate), BER analyzer, and 3R regenerator constitute the receiver section (Table II).Results and Analysis In this section, the analysis of the effects of SPM and XPM on the fiber is shown. Input power and transmission rates are some parameters that have been changed over time to provide conclusive evidence of the effects on the fiber. Eye‑Diagram Analysis of Self‑Phase Modulation Marvin and Pratheesh9 demonstrated a simulation based on optical fiber to analyze the SPM effect. They obtained various values of the Q-factor by varying the input power of the source. They varied the power from 10 to 20 dBm and found that the Q-factor completely vanished at 20 dBm power input.9 We analyzed the Q-Factor by varying the power from 10 dBm to 20 dBm and obtained the following results. Initially, the input power is set at 10 dBm and 15 dBm. The output optical signals associated are shown in Fig. 4a and b, respectively. Now, we vary the input power from 17 dBm to 20 dBm. The output optical signals are shown in Fig. 5a and 5b, respectively. We observed that the Q-factor at 10 dBm power input was 5.06, and at 15 dBm, the power input was 8.38. When Table I Important parameters and their values pertaining to the SPM system Parameter Value Bitrate 10 × 109 Bits/s Length of optical fiber 100 km Length of Er-Yb Co-doped fiber 1 m Core-radius of Er-Yb Co-doped fiber 2 µm Doping-radius of Er-Yb Co-doped fiber 2 µm Er-ion density of Er-Yb Co-doped fiber 51.4 × 1024 m-3 Yb-ion density of Er-Yb Co-doped fiber 620 × 1024 m-3 Er metastable lifetime 10 ms Yb metastable lifetime 1.5 ms Low-pass Bessel filter cut-off-frequency 0.65 × bitrate Fig. 3 XPM simulation setup.Table II Important parameters and their values in the XPM system Parameter Value Length of optical fiber 50 km Length of Er-Yb Co-doped fiber 1 m Core-radius of Er-Yb Co-doped fiber 2 µm Doping-radius of Er-Yb Co-doped fiber 2 µm Er-ion density of Er-Yb Co-doped fiber 51.4 × 1024 m-3 Yb-ion density of Er-Yb Co-doped fiber 620 × 1024 m-3 Er metastable lifetime 10 ms Yb metastable lifetime 1.5 ms Low-pass Bessel filter cut-off-frequency 0.70 × bitrate A Comparative Study of SPM and XPM in Er -Yb Co -Doped Fiber 1 3 Fig. 4 (a) Eye-diagram for SPM at 10 dBm input, (b) Eye-diagram for SPM at 5 dBm input. Fig. 5 (a) Eye-diagram for SPM at 17 dBm input, (b) Eye-diagram for SPM at 20 dBm input. Table III Eye-diagram parametric values (SPM).Parameter 10 dBm 15 dBm 17 dBm 20 dBm Max. Q-factor 5.06 8.38 10.79 4.32 Min. BER 2.02815 × 10-72.4637 × 10-171.64342 × 10-277.45267 × 10-6 Eye height 1.79711 × 10-5107 × 10-6214 × 10-6196 × 10-6 Threshold 3.08781 × 10-59.16885 ×10-5118×10-6291×10-6 Decision inst. 0.53 0.55 0.55 0.53 A. Kumar, G. Qazi 1 3 the power input was increased to 17 dBm, the Q-factor increased to 10.79, and at 20 dBm, the Q-factor decreased to 4.31. The Er-Yb fiber is able to compensate SPM at lower power inputs. Table III gives the values of param- eters related to the eye-diagram. At higher power input rates, the eye-opening decreases, thus SPM grows and depletes the signal. The Er-Yb fiber gave a Q-Factor of 4.31 at 20 dBm input, whereas the Q-factor in the normal fiber van - ished.9 The comparative results have been demonstrated in Fig. 6 indicating possible transmission quality in the case of Er-Yb fiber. The Er-Yb system generates a better Q-Factor for higher input power conditions, whereas in Ref. 9, the Q-Factor deteriorates and vanishes. As a point of benefit, the system with Er-Yb at 20 dBm input power manages to pro- vide transmission quality, but the other system completely fails. Thus, Er-Yb fiber has better operating conditions. XPM Analysis by Eye‑Diagram Based on Varying Transmission Rates In this section for the simulation, the rate of transmission is altered from 10 Gb/s to 40 Gb/s to analyze the effect of XPM on varying transmission rates. The Q-factor is considered to determine the strength of the signal. Initially, the input transmission rate is set at 10 Gb/s and 20 Gb/s. The corresponding output eye-diagrams are por - trayed in Fig. 7a and b, respectively. Now, the input transmission rate to the fiber is varied from 30 Gb/s to 40 Gb/s. The corresponding eye-diagrams are portrayed in Fig. 8a and b, respectively. The parametric values of the eye-diagram are shown in Table IV. The Q-factor obtained at 10, 20, 30, and 40 Gb/s are 18.78, 5.16, 3.7, and 5.4. These data are shown in Fig. 9, indicating a decrease in Q-Factor. Still, the rate of decrease is slower in this fiber, increasing slightly at the end, evidently showing that the Er-Yb fiber supports higher transmission rates. Thus, Er-Yb fiber can give good quality factors at vari- ous transmission rates and has a better-compensating ability than normal fiber.10 Fig. 6 Comparative study of SPM. Fig. 7 (a) Eye-diagram for XPM at 10 Gb/s input, (b) Eye-diagram for XPM at 20 Gb/s input. A Comparative Study of SPM and XPM in Er -Yb Co -Doped Fiber 1 3 Based on Varying Input Power Bhusari, Deshmukh, and Jagdale10 demonstrated a simula- tion based on optic fiber to analyze XPM effects and have obtained various values of Q-factor by varying the input power of the source from 0 dBm to 12 dBm. They found that the Q-factor decreased from 4.7 to 2.7, causing degradation of the input signal.10 We analyzed the Q-Factor by varying powers from 0 dBm to 20 dBm and obtained the following results. Initially, the input power is set at 0 dBm and 5 dBm. The output optical signals associated are shown in Fig. 10a and b, respectively. Now, the power given to the input of the fiber is varied from 10 dBm to 20 dBm. The corresponding eye-diagrams are shown in Fig. 11a and b, respectively. Here, despite varying power inputs, the Q-factor con- stantly stays around 5 and decreases slightly at the end. Fig- ure 12 indicates the comparative plot between the considered Fig. 8 (a) Eye-diagram for XPM at 30 Gb/s input, (b) Eye-diagram for XPM at 40 Gb/s input. Table IV Eye-diagram parametric values (XPM).Parameter 10 Gb/s 20 Gb/s 30 Gb/s 40 Gb/s Max. Q-factor 18.79 5.17 3.18 5.4 Min. BER 4.94205 × 10-791.22285 × 10-17752×10-62.95298 × 10-8 Eye height 216×10-6139×10-61.59369 × 10-5160×10-6 Threshold 125×10-6411×10-5402×10-6400×10-6 Decision inst. 0.53 0.55 0.68 0.66 Fig. 9 Study of XPM by transmission rates. A. Kumar, G. Qazi 1 3 systems. The Er-Yb fiber generates higher quality signals with better Q-Factor than the system in Ref. 10. Thus, Er-Yb fiber can give better results as compared to normal fiber uti- lized in Ref. 10. The parametric values of the eye-diagram are shown in Table IV (Table V).Conclusion Mitigation of non-linearities has been effectively shown to be achieved with optical communication systems compris- ing Er-Yb fiber, generating good Q-Factor, and providing a Fig. 10 (a) Eye-diagram for XPM at 0 dBm input, (b) Eye-diagram for XPM at 5 dBm input. Fig. 11 (a) Eye-diagram for XPM at 10 dBm input, (b) Eye-diagram for XPM at 20 dBm input. A Comparative Study of SPM and XPM in Er -Yb Co -Doped Fiber 1 3 higher signal quality. Non-linear effects in Er-Yb co-doped fiber are relatively lower than the normal fiber, and it can mitigate the non-linearities to a better extent. Systems capac- ity and signal quality are reduced by SPM and XPM affect. SPM grows and drains the signal at a comparatively lower rate when input power is increased. XPM also performs a similar slower degradation of signal. Thus, Er-Yb co-doped fiber may be adopted if the signal quality required is of emi- nent importance. The study can be further extended by ana - lyzing the effects of different input signal waveforms on the fiber and studying the nature of four-wave mixing.Conflict of interest The authors declare that they have no known com- peting financial interests or personal relationships that could have ap- peared to influence the work reported in this paper.