## 4 Stability of the Nonlinear Modes: Spectral Analysis  
The natural next step is to identify the stability of the solutions. This is monitored by using the linearization ansatz:  
\[\Psi=e^{i\eta z}\left(\psi+\delta\left[a(x,y)e^{\
u z}+b^{*}(x,y)e^{\
u z} \right]\right) \tag{7}\]  
which yields the order \(\delta\) linear system  
\[\left[\begin{array}{cc}M_{1}&M_{2}\\ -M_{2}^{*}&-M_{1}^{*}\end{array}\right]\left[\begin{array}{c}a\\ b\end{array}\right]=-i\
u\left[\begin{array}{c}a\\ b\end{array}\right] \tag{8}\]  
where \(M_{1}=\
abla^{2}+U-\mu+2\sigma|\psi|^{2}\), \(M_{2}=\sigma\psi^{2}\). Thus, \(\max(\mathrm{Re}(\
u))>0\) corresponds to instability and \(\max(\mathrm{Re}(\
u))=0\) corresponds to (neutral) stability.  
In the bottom left panel of Figure 3, the power curve is drawn with stability and instability as determined by \(\
u\) noted by the solid or dashed curve, respectively. The other three plots in Figure 3 show the maximum real part of eigenvalues \(\
u\) for each of the three branches: the red dashed curve is the max real part of real eigenvalue pairs, and the blue dotted curve is the max real part of eigenvalue quartets with a nonzero imaginary part. The former corresponds to exponential instabilities associated with pure growth, while the latter indicate so-called oscillatory instabilities, where growth is present concurrently with oscillations. In Figure 4, we plot some example eigenvalues in the complex plane for some sample unstable solutions. The dominant unstable eigenvalues within these can be seen to be consonant with the growth rates reported for the respective branches (and for these parameter values) in Figure 3.  
The overarching conclusions from this stability analysis are as follows. The lowest \(\mu\) branch, being the ground state in the defocusing case, is always stable in the presence of the self-defocusing nonlinearity. For the parameters considered, generic stability is also prescribed for the third branch under self-defocusing nonlinearity. The middle branch has a narrow interval of stability and then  
Figure 4: Eigenvalues are plotted in the complex plane \((Re(\
u),Im(\
u))\) for a few representative solutions. One can compare the maximal real part with Figure 3. For example, the top left complex plane plot here shows that for branch 1 at \(\mu\approx 0.71\), the eigenvalues with the maximum real part are complex; this agrees with the top left plot of Figure 3 where at \(\mu\approx 0.71\), the blue dotted curve representing complex eigenvalues is bigger. Similarly, one can check whether the other three eigenvalue plots here also agree with what is shown in Figure 3, the top right for branch 1, the bottom left for branch 2 and the bottom right for branch 3.  
becomes unstable, initially (as shown in the top right of Figure 3) via an oscillatory instability and then through an exponential one. In the focusing case (that was also focused on in [19] for the second and third branch), all three branches appear to be stable immediately upon their bifurcation from the linear limit, yet all three of them subsequently become unstable. Branch 1 (that was not analyzed previously) features a combination of oscillatory and exponential instabilities. Branch 2 features an oscillatory instability which, however, only arises for a finite interval of frequencies \(\mu\), and the branch restabilizes. On the other hand, for branch 3, when it becomes unstable, it is through a real pair. Branches 2 and 3 terminate in a saddle-center bifurcation near \(\mu=1.9\). The eigenvalue panels of Figure 4 confirm that the top panels of branch 1 may possess one or two concurrent types of instability (in the focusing case), branch 2 (bottom left) can only be oscillatorily unstable in the focusing case (yet as is shown in Figure 3, it can feature either type of instability in the defocusing case), while for branch 3, when it is unstable in the focusing case, it is via a real eigenvalue pair.