## 1 Introduction  
Stimulated Brillouin scattering (SBS) is a nonlinear effect that initiates from the nonlinear interaction between the optical field and acoustic field. In recent decades, SBS has drawn extensive attention due to its wide applications and significant effects on various optical fiber-based systems [1, 2]. Among the various parameters that describe SBS, threshold is the most important. When the input power exceeds the SBS threshold, large amounts of forward transmission power convert to the backscattering Stokes light [3]. Simultaneously, the systems suffer from serious noise increases [4]. Consequently, SBS threshold is the maximum input power allowed in many optical fiber-based systems. Many researchers studied the SBS threshold in various optical fiber systems [4, 5, 6, 7]. For continuous single-frequency light, the SBS threshold can be calculated simply by the famous equation \(P_{th}=21A_{eff}/(g_{B}L_{eff})\), in which \(A_{eff}\) is the effective mode area, \(g_{B}\) is the Brillouin gain factor and \(L_{eff}\) is the effective length [8]. However, optical pulse, rather than the continuous light, is more widely used in practical systems. To determine the SBS threshold in pulsed systems, a concept of effective interaction length was proposed to determine the SBS threshold for optical pulses [9, 10]. The effective interaction length indicates that the SBS threshold is mainly determined by the average threshold and keeps almost constant whatever the repetition rate is. However, the situation changes when other nonlinear effects occur simultaneously with SBS. Modulation instability (MI) is another important nonlinear effect in optical fiber, which results from the combined interaction of Kerr effect and group velocity dispersion. When MI and SBS occur simultaneously, the effect of MI on Brillouin scattering must be taken into account. Some researchers have studied the effect of MI on Brillouin scattering in Brillouin-based optical fiber systems. [PERSON] et al. found that the dominance of MI suppresses SBS gain in SBS-based broadband slow light systems in an experiment, but no detailed theoretical analysis was presented [11]. A few groups studied the effect of MI on Brillouin-based distributed fiber sensors, and the results show MI severely limitsthe sensing range and degrades performance [12, 13, 14, 15]. However, most of the work was conducted experimentally; only a few simulation results were reported. In a typical remote interferometric fiber sensing system, the fiber length in the system can be tens or hundreds of kilometers. In addition, the typical optical pulse has a duration of hundreds of nanoseconds and a repetition rate of tens to hundreds of kilohertz [16]. The remote interferometric fiber sensing system offers a specific platform in which various nonlinear effects occur simultaneously. For SBS, the threshold in the remote interferometric fiber sensing system cannot be simply determined by the method proposed in references [9, 10] for the simultaneous occurrence of other nonlinear effects. In our previous work [17], a complicated competition among SBS, modulation instability (MI) and stimulated Raman scattering (SRS) was observed in an experiment. The results indicate that rich physical mechanisms exist behind the complicated competition, which deserves careful research. In reference [17], an important phenomenon is that the SBS threshold increases for the occurrence of MI. However, as far as we know, no detailed theoretical study has been demonstrated to reveal the physical mechanisms. In this paper, we study the effect of MI on SBS threshold in long optical fiber both in theory and experiment. The Brillouin process is analyzed based on the coupled-wave equation. The results show that MI converts the Brillouin pump to symmetric sidebands, which leads to the Brillouin gain decreasing and SBS threshold increasing significantly. The effects of the pulse repetition rate and the ASE noise level on the SBS threshold are analyzed via numerical simulation.