## 1 Introduction  
In recent years, cavity optomechanics, exploring the interaction between optical and mechanical modes via radiation pressure [1], has received considerable attention owing to its possible applications in ultrasensitive force measurement [2] and quantum information processing [3]. It is an outstanding challenge in cooling a macroscopic mechanical oscillator of an optomechanical system to its quantum ground state. With the rapid development of the experimental technologies, it is now possible to reduce the average phonon occupancy number of the macroscopic mechanical oscillator to below unity [4, 5]. This enables us to observe numerous quantum optical phenomena in the optomechanical system, such as the mechanical squeezed state below zeropoint fluctuations [6, 7, 8], entanglement between optical and mechanical resonators [9, 10], non-classical correlations between single photons and phonons [11], normal mode splitting [12, 13], and electromagnetically induced transparency (EIT) [14, 15, 16, 17]. The EIT has been theoretically studied in the Fabry-Perot-type optomechanical system [14], and has been displayed experimentally in different optomechanical systems, including a toroid microresonator [15], an optomechanical crystal nanocavities [16], and a silica microsphere [17]. The EIT plays an important role in realizing slow light and fast light [18], storage and retrieval of an optical pulse [19], and light switching [20]. The review of the EIT in the optomechanical systems can be seen in [21, 22]. Up to date, almost all of the EIT studies in the optomechanical systems have been done with linear mechanical oscillators since the intrinsic nonlinearity of a mechanical oscillator is generally weak, and it can be ignored. However, the recent works have shown that the strong nonlinearity of the mechanical oscillator can be achieved by geometrical and material effects of the mechanical oscillators [23], electrostatic coupling mechanism [23], strong driving [24], and coupling a linear mechanical oscillator to an auxiliary system [25]. Hence, a nonlinear mechanical oscillator has been recently introduced into the optomechanical system. It has been analyzed that a quartic nonlinear mechanical oscillator which includes a quartic term in its potential energy in the optomechanical system can be prepared in a Fockstate [26] and a squeezed state [27]. Moreover, a cubic nonlinear mechanical oscillator which includes a cubic term in its potential energy in the optomechanical system can also be prepared in a squeezed state, which is shown in Appendix B in Ref. [27]. On the other hand, when an intense coherent coupling field with frequency \(\omega_{c}\) and a weak coherent probe field with frequency \(\omega_{p}\) are combined in an optomechanical system, many sidebands at frequencies \(\omega_{c}\pm n\delta\) (\(n\) is an integer representing the sideband order and \(\delta=\omega_{p}-\omega_{c}\)) are created in the output field due to the nonlinear optomechanical coupling. It has been shown that the second-order upper sideband generation in the output field is much weaker than the input probe field [28]. The enhancement of the second-order upper sideband generation has potential applications in precise measurements of electrical charges [29] and the effective mass of the mechanical resonator [30]. It has been shown that the enhancement of the second-order upper sideband generation in the optomechanical system can be achieved by using Coulomb-interaction [29], a degenerate parametric amplifier [30], a nonlinear Kerr medium [31], and two-level atoms [32]. As mentioned above, the nonlinearity of a mechanical oscillator can be used to prepare nonclassical mechanical states [26; 27]. In order to better assess its performance, different methods have been proposed to measure the nonlinearity strength of the mechanical oscillator [33; 34].