Strain-induced two-dimensional topological crystalline insulator in bilayer SnTe

Domain: physics
Published: 2026-01-21
Editorial outcome: major_revision

Abstract
Topological crystalline insulators host topological phases of matter protected by crystal symmetries. Despite the prediction of a two-dimensional topological crystalline insulator, materials challenges have, to the best of our knowledge, thus far prevented its experimental realization. Here we report the growth and characterization of bilayer SnTe on the 2H-NbSe2 substrate by molecular beam epitaxy and scanning tunneling microscopy. We experimentally observe bilayer SnTe experiences compressive strain and exhibits two anticorrelated, periodically modulated pairs of conducting edge states within a large band gap exceeding 0.2 eV. We identify these states as topological edge states through first-principles calculations. Finally, we experimentally probe the coupling of adjacent topological edge states and demonstrate the resulting energy shift driven by a combination of electrostatic interactions and tunneling coupling. Our work opens the door to investigations of tunable two-dimensional topological states, of potential impact for spintronics and nanoelectronics applications at room temperature.

Introduction
Topological insulators (TIs) feature an energy gap in the bulk and nontrivial surface or edge states that connect the valence and conduction bands1–3. The discovery of quantum spin Hall (QSH) effect in HgTe/CdTe quantum wells4 has sparked intensive research on novel topological materials extending from three-dimensional (3D) to two-dimensional (2D) systems5–10. Meanwhile, crystal symmetries are discovered to play an analogous role as time-reversal symmetry of QSH insulators11–13, which can protect the boundary states from backscattering and thus result in dissipationless carrier transport14. The topological phase is protected by the simultaneous combination of time-reversal and mirror symmetries, which in the following we refer to as time-reversal-mirror symmetry. Unlike QSH described by Z2 invariant, the topology in topological crystalline insulators (TCIs) is classified by the finite Chern number in each symmetry sector, having a net zero total Chern number. 3D TCIs have been realized in semiconductor tin telluride (SnTe)15,16, Pb1−xSnxSe17,18, and Pb1−xSnxTe19,20, in which topological surface states are highly dependent on the surface orientations depending on whether time-reversal-mirror symmetry is broken or preserved. If the time-reversal-mirror symmetry with respect to the (110) bulk mirror plane is broken, the Dirac cones would acquire a band gap and Dirac fermions would obtain mass17,21. The (001) SnTe films are predicted to be 2D TCIs both in the single atomic layer (AL) limit22 and above 5 ALs23. They should host two pairs of edge states protected by the time-reversal-mirror symmetry z→−z (z is normal to the film) through a nonzero time-reversal-mirror Chern number.

The hallmark of a 2D TCI is the presence of metallic edge states along the sample boundary. Higher-order one-dimensional topological hinge states24,25 and surface step edge states26 have been theoretically predicted and experimentally observed, but it remains a challenge to realize a strong 2D TCI with a wide band gap due to the complicated fabrication of thin films and strong substrate-induced hybridization effects. Particularly, only the odd number of ALs are expected to be 2D TCIs because they keep the symmorphic crystal symmetry with nonzero time-reversal-mirror Chern number15. However, each monolayer of (001) SnTe ultrathin films in the rock-salt structure contains 2 ALs and thus we always get an even number of ALs in experiments27, which prevents the band inversion necessary for a topological phase transition. Besides, a strong ferroelectric phase also emerges in the ultrathin limit28,29. Realization of a 2D TCI could be achieved with strain, which allows tailoring the electronic band gap and electronic phase transitions of few-AL SnTe films30.

In this work, we utilize epitaxial growth to fabricate bilayer SnTe (4 ALs) islands on the bulk 2H-NbSe2 and investigate atomic-scale structure and electronic properties using low-temperature scanning tunneling microscopy (STM) and spectroscopy (STS). We experimentally observe strong edge modes that can be rationalized through density functional theory (DFT) calculations showing the top 3 ALs of SnTe are decoupled from the first 1 AL due to strong interfacial interactions. The concomitant compressive strain drives our effective 3-AL system to undergo a crossover from trivial ferroelectricity to a TCI phase. Finally, we probe experimentally the coupling of edge states from adjacent bilayer SnTe islands and reveal significant energy shifts. Our study highlights a pathway for creating strain-induced emergent quantum states of matter, which could significantly impact energy-efficient electronics.

Results
Experimental realization of strained 3 atomic layers
We prepared ultrathin SnTe films on a 2H-NbSe2 substrate using the molecular beam epitaxy (MBE, see the Methods section for details on sample preparation). Since each monolayer consists of 2 ALs in the rock-salt SnTe structure, the growing island thickness is always an integer multiple of monolayer in ultrathin films. By carefully controlling the growth conditions, we achieved monolayer-by-monolayer growth. As shown in Fig. 1a, the monolayer forms large irregular islands where the second layer subsequently nucleates. The height profile (see Supplementary Information Fig. S6) confirms that both layers are one monolayer thick. The stripes visible in Fig. 1b extend from the monolayer to the bilayer and they arise from the moiré patterns resulting from a large mismatch between the square SnTe lattice (Te-Te distance of 4.5 Å) and the hexagonal 2H-NbSe2 lattice (Se-Se distance of 3.4 Å). Monolayer SnTe islands exhibit a buckled and dislocated structure with varying stripe orientations, serving as the seed layer for the bilayer. In addition, the monolayer stripe patterns reflect a periodically varying strain field in the bilayer (see Supplementary Information Fig. S7). The strain field predominantly distributes along strained moiré patterns, with one unit cell of strain field being shown by orange color in Fig. 1c. By fast Fourier transform (FFT) (in the inset) of atom-resolved bilayer SnTe topography in Fig. 1c, we find the lattice constants along the [100] and [010] directions are approximately 4.17 Å and 4.33 Å, respectively, which indicates bilayer SnTe is biaxially compressed. This scenario is quite different from the strain-free ultrathin SnTe films grown on graphene and graphite28,29.

To gain further insight into the structural and electronic properties of this strained system, particularly for the 4-AL SnTe/2H-NbSe2 heterostructure, we performed DFT calculations based on the experimentally extracted lattice parameters. Figure 1d presents the side view of our heterostructure model after full relaxation (see also Supplementary Information Fig. S8). The calculations reveal that the metallic NbSe2 substrate provides a high density of states near the Fermi level, facilitating charge transfer and inducing p-type doping into the SnTe due to the alignment of the Fermi level. The charge transfer induces strong hybridization between the orbitals of the SnTe bottom layer and the underlying 1H-NbSe2, causing the Sn atoms in the bottommost AL to bond directly with the substrate. Due to the substantial lattice mismatch, the upper 3 ALs become structurally decoupled from the wetting layer, exhibiting noticeable distortion. The computed interlayer distances are consistent with the experimental height profile (Supplementary Information Fig. S9). The projected density of states (PDOS) for the topmost AL, shown in Fig. 1e, aligns well with the experimental tunneling spectra (Fig. 1f), where the bulk band gap edges are marked by black dashed lines. Further, the layer-wise PDOS (shown in Supplementary Information Fig. S9d) suggests that the PDOS in the bottom layer bonded to the substrate is quite different from the remaining 3 ALs. In addition to charge transfer from the substrate, Sn vacancies estimated at a surface density of 1013 cm−2 further promote p-type doping, shifting the Fermi level below the valence band maximum31. The bilayer bulk gap, determined from negative differential resistance measurements, ranges from 0.2 to 0.3 eV (see Supplementary Information Fig. S15). Together, these findings indicate that the experimental system effectively realizes a strained 3-AL SnTe structure on top of a 1-AL wetting layer, stabilized on the 2H-NbSe2 substrate. This interpretation, supported by both structural and spectroscopic evidence, provides a solid foundation for theoretical investigation into the topological properties of this system.

Lifshitz transition driven by biaxial strain
Building on this understanding, we now focus on the theoretical analysis of the topological characteristics of the strained 3-AL SnTe structure, using parameters fitted from the DFT-calculated configuration where the bottom 1-AL SnTe is bonded to the substrate. To ensure the robustness of our theoretical model and its relevance to the experimentally realized structure, we also analyze the general behavior of few-layer SnTe systems (see Supplementary Information Figs. S1 and S2). As shown in Fig. 2a, a 3-AL SnTe structural model has a cubic rock-salt structure with a Sn-Sn spacing of 4.5 Å and preserves the reflection symmetry z→−z with respect to the atomic layer in the middle plane (pink shadow). This system hosts two equivalent high-symmetry X points in the 2D Brillouin zone. On the other hand, ultrathin SnTe films are known to spontaneously distort along the {110} direction into a parallelogram structure and form a ferroelectric phase27. This in-plane 2D ferroelectricity does not break the vertical mirror symmetry. This allows for competition between ferroelectricity and a TCI phase. Ab initio calculations enable us to establish a phase diagram of the 3-AL SnTe system as a function of biaxial strain.

Figure 2b shows ferroelectric displacement δFE of this system evolves with the lattice parameter. Above the critical value of a ≃ 4.38 Å, the phonon instability associated with the ferroelectric distortion appears (see Supplementary Information Fig. S5) as the lattice parameters expand. Meanwhile, this system opens a band gap at the X point. These findings indicate ferroelectric order occurs in the xy plane of biaxially tensile 3-AL system. When the lattice parameters reduce to the critical value, the band gap closes. Below the critical value, the band gap reopens, as shown in Fig. 2d. This change in Fermi surface topology corresponds to a Lifshitz transition driven by the change from the tensile to compressive strain. The gap closing at the X point is the critical point of a phase transition (Fig. 2c), where biaxial compressive strain and spin-orbit coupling can induce a band inversion from Sn and Te orbitals (see Supplementary Information Fig. S3). The electronic structure induces the stabilization of the high-symmetry crystalline phase, quenching the ferroelectric displacement. We further experimentally exclude the possibility of ferroelectricity in our compressive system by probing band bending of conduction band at the edges (shown in Supplementary Information Fig. S22).

The topological invariant of a strained 3-AL SnTe system can be obtained in the gapped phases by extracting a Wannier Hamiltonian from DFT calculations. The results show that the time-reversal-mirror Chern number vanishes CM± = 0 in the tensile regime while the time-reversal-mirror Chern number is CM± = ±2 in the compressive regime (Fig. 2d), signaling the emergence of a TCI phase. This finding is schematically depicted in Fig. 2c. Figure 2e displays the computed time-reversal-mirror Berry curvature for each of the symmetry sectors (±1) of the operator M = M(001) Θ, where M(001) is the (001) mirror and Θ is the time-reversal symmetry operator. In the topological phase, the mirror Berry curvature is well localized around two X points and possesses an opposite sign for each symmetry sector. In the ferroelectric phase, the mirror Berry curvature is not localized and changes the sign with each symmetry sector, resulting in CM± = 0. The bulk-boundary correspondence require that there must be gapless edge states in the biaxially compressive 3-AL system. The time-reversal-mirror Chern number of ±2 allows two pairs of Dirac cones located around X points in the 2D Brillouin zone, and with further projection into the specific edges, we obtain two pairs of edge state subbands along the X-Γ-X direction (Fig. 2f). Co-propagating edge states carry identical mirror eigenvalues. At the band crossings at Γ and X points, edge states with opposite mirror eigenvalues propagate in the opposite direction. The two pairs of edge state subbands exhibit linear dispersion around the crossing points, extending into the bulk band gap. In contrast, there are no gapless edge states connecting valence and conduction bands in the ferroelectric phase, as shown in Fig. 2g. Although we used slightly different lattice parameters compared to the experimental values, the essential conclusions remain unchanged: while the band gap size can be tuned by lattice parameters, the two phases preserve their respective band structures and topological characteristics.

Experimental evidence of topological edge states
We present two direct experimental proofs to verify the existence of a 2D TCI in our effective 3-AL system. The first one is that we observe two strong peaks associated to two pairs of edge states extending to the bulk gap along the specific edges of bilayer SnTe islands. As shown in Fig. 3a, the dI/dV spectrum taken at the bilayer edge exhibits two pronounced peaks located around 0.5 V (peak 1) and 1.55 V (peak 2). We attribute these peaks to edge states (see Supplementary Information Figs. S10 and S11). These experimental results can be compared with calculations by computing the density of states (DOS) in the bulk and edge of the compressive 3-AL system, where we obtain two sharp peaks with one residing in the bulk gap and the other within the conduction band of bulk states. The low-energy (high-energy) peak at the edge is produced by the substantially enhanced DOS induced by the flatter dispersion of the pair of edge states at the X (Γ) crossing point (Fig. 2f). Therefore, each peak corresponds to the crossing of counter-propagating pairs of edge states. The edge state subbands above the peak 1 are fully within the bulk gap (from 0.5 to 0.7 eV) and are also spatially localized (see Supplementary Information Fig. S12). The discrepancy of the energy peak positions between theory and experiment stems from the typical underestimation of the band gap in DFT calculations.

To resolve the spatial modulation of edge states, we first conduct dI/dV spectroscopy across the opposite edges of a bilayer island (top panel in Fig. 3b). At each edge, there are two edge state peaks located at low and high energies, outlined by the dashed oval frames in the middle panel. These peaks are highly localized around the edges. Similar to the observations in Fig. 3a, the peak 1 almost resides the bulk gap (bottom panel) while the peak 2 lies inside the conduction band. It is important to note that the exact energy position of the edge states depends on the strain field, e.g., the strain field can shift the energy of the peak 1 beyond the bulk gap. Series of local DOS (LDOS) oscillations between 0.2 and 1 V as shown in the bottom panel of Fig. 3b are likely to be related to the LDOS oscillations in the SnTe wetting layer (see further details in the Supplementary Information Fig. S14). We also performed dI/dV spectroscopy along the blue line cut indicated in Fig. 3c (top panel). The results show that both peak 1 and peak 2 modulate periodically with the moiré pattern at the island edges (two middle panels). FFT analysis further reveals that the edge-state crossings occur at low energy near the X point and at higher energy near Γ (see Supplementary Information Fig. S13) in agreement with the theoretical prediction shown in Fig. 2f. The detailed dispersions of these edge-state crossings are difficult to extract experimentally due to resolution limitations. Interestingly, the two peaks are anticorrelated, exhibiting a π phase difference. The energy shift of the peak 2 is approximately 2 times as large as the peak 1. To rationalize these observations, we developed a phenomenological model. Considering that strain induces changes in the band gap (Fig. 2d), by renormalizing the energy gap as dictated by the strain modulation, we account for the impact of the strain field on the energy position of each pair of edge states. As shown in the bottom panel, our calculations reproduces the experimental results. Moreover, dI/dV maps at different energies (see Supplementary Information Fig. S18) reveal that the two pairs of edge states span the bulk energy gap along the perimeters of bilayer islands at 0.5 and 0.6 V. They only emerge along the sharp and straight edges with specific atomic terminations of either Sn or Te atoms, a characteristic observed in both regular and irregular bilayer SnTe structures (see Supplementary Information Figs. S4, S19~S21).

Since the edge states in our system are protected by time-reversal-mirror symmetry, the second piece of evidence to confirm a 2D TCI is to observe a band gap in the edge states when time-reversal-mirror symmetry is broken. Time-reversal-mirror symmetry can be disrupted spontaneously or through external perturbations such as perpendicular electric field or an in-plane magnetic field23. In contrast with time-reversal symmetry, disorder can always break time-reversal-mirror symmetry. In our effective 3-AL system, intrinsic atomic defects at the edges locally and randomly break the z→−z mirror symmetry about the middle plane. As a result, the edge states open up a band gap, as shown in Fig. 3d. This is supported by theoretical simulations (see Supplementary Information Fig. S16). The depth of the opened band gap depends on the extent to which mirror symmetry is broken at the edges. Remarkably, we find that the edge states are substantially resilient to the effects of the substrate, the moiré potential, and even a sizable electric field (see Supplementary Information Figs. S16 and S17).

The link between theory and experiment can be further strengthened by investigating the strain-driven transition between the topological and trivial phases. We have achieved precise control over the in-plane lattice constants of the bilayer SnTe by tuning the growth parameters. This tunability allows us to modulate the system across the topological phase boundary. When the lattice constants are smaller than the critical value identified in Fig. 2d-i.e., in the compressively strained regime-the bilayer SnTe islands consistently exhibit robust topological edge states. In contrast, when the lattice constants exceed this critical value, entering the tensile strain regime, the edge states vanish along all observed edges. Details are shown in Supplementary Information Figs. S23–S25. This experimental observation directly supports the strain-driven topological phase transition predicted by theory and further validates the topological nature of the observed edge states in our system.

Coupling of topological edge states
The interaction of edge states in 3D TCIs has been studied26,32. We probe the possibility of observing coupling between the edge states in our 2D TCI by examining two closely adjacent bilayer islands, as illustrated in Fig. 4a. High-resolution topography indicates no significant structural distortions or atomic defects that could disrupt mirror symmetry; therefore, a gap opening in the edge states is not expected. To eliminate the influence of strain, which can shift edge state energy, the tunneling spectra were taken at equivalent points on the same stripe pattern, ensuring a consistent strain field. Our dI/dV spectra reveal that peak 1 shifts to lower energy compared to isolated edges as indicated by the black arrow (Fig. 4b) and its intensity weakens significantly around 0.6 V. The energy shift due to the proximity of adjacent edges exhibits an exponential decay with increasing distance between the two edges, as seen in Fig. 4c, with a decay length of approximately 5 nm. This effect is most evident in the dI/dV mapping over the bilayer edges at different energies (Fig. 4d–g), as outlined by the orange and red rectangular frames.

We propose two possible origins for the observed energy shift, both of which are expected to coexist in our system. First, we experimentally find that the edge states shift to lower energies compared to isolated edges. This behavior can be explained by electrostatic coupling: the presence of adjacent edges leads to interactions that modify the energy position of the edge states. The strength of this coupling depends on both the edge-to-edge separation and the local charge distribution. Importantly, this charge redistribution stems from edge reconstruction, a phenomenon that becomes particularly pronounced in materials that lie close to an electronic phase transition, where small structural changes can strongly affect the electronic configuration. Second, there is a possibility of substrate-mediated tunneling between adjacent edges, allowing for interaction through proximity. To assess the magnitude of this effect, we developed a model where the electronic modes in one bilayer island can tunnel into another island through an effective barrier potential V (details in Supplementary Information Fig. S26). This tunneling coupling leads to a shift of the crossing Dirac point. Comparison with the experimental results is possible by relating V to the distance d between neighboring edges through λd = ln(V), where t is the hopping energy of SnTe orbitals (on the order of eV) and λ is a finite phenomenological parameter that can be adjusted based on experimental data. This model results in an exponential decay of the energy shift as λd increases, consistent with our experimental results. In summary, it is likely that a combination of electrostatic interactions and tunneling coupling is responsible for the energy shift of edge modes for closely spaced SnTe edges.

Methods
Experimental methods
Ultrathin SnTe films were grown on the freshly cleaved bulk crystal 2H-NbSe2 by molecular beam epitaxy under ultrahigh vacuum conditions (UHV, base pressure ≈ 2 × 10−10 mbar). Before the growth, compound material SnTe was fully degassed and 2H-NbSe2 substrate was also outgassed at ~100 °C. During the growth, the SnTe powder of 99.999% purity loaded into a Knudsen cell was evaporated onto the substrate held at room temperature for 3 min. After the growth, the sample was annealed at ~50 °C. Finally, it was inserted into low-temperature STM head housed in the same UHV system. All the subsequent STM and STS measurements were performed at T = 4.7 K. The etched tungsten tip was used for all the STM scan and electrical measurement. STM images were scanned in the constant-current mode and all the dI/dV spectra were recorded by a standard lock-in techniques while sweeping the sample bias in an open feedback loop configuration, with a peak-to-peak bias modulation of 5~20 mV at a frequency of 400-800 Hz.

Theoretical methods
DFT calculations were performed with the standard Perdew-Burke-Ernzerhof (PBE) functional38 as implemented in the QUANTUM-ESPRESSO package39. Ultra-soft pseudopotentials from the pslibrary (https://dalcorso.github.io/pslibrary/) version 1.0.0 were used to describe the interactions between electrons and ions40, while the wave functions were expanded using a plane-wave basis set with a kinetic energy cutoff of 50 Ry. The integration over the Brillouin zone was performed using a 8×8×1 grid. After structural relaxation with the force threshold of 0.01 eV/Å, all electronic properties were obtained considering spin-orbit coupling with the full-relativistic versions of the same pseudopotentials used previously and Brillouin zone sampling of 10×10×1. The Wannier90 package41 was employed to obtain tight-binding Hamiltonians in the basis of maximally localized Wannier functions, including s and p orbitals for both Sn and Te atoms. We fitted the DFT band structures with the tight-binding Hamiltonians considering an energy window of more than 20 eV and a frozen window covering 2 eV.

The same code and pseudopotentials were used to relax and to obtain electronic properties of SnTe/NbSe2 heterostructures. The Brillouin zone was integrated using a smaller 2×2×1 grid, and the vdw-df-ob86 vdW functional was used instead of standard PBE in order to account for weak dispersion interactions that might occur between the layers42. All calculations in this part were performed without taking spin-orbit coupling interactions into account.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "The research focuses on 2D Topological crystalline insulators (TCIs). This is interesting because materials challenges have thus far prevented its experimental realization. The authors systematically report bilayer SnTe on the 2H-NbSe2 substrate by molecular beam epitaxy and scanning tunneling microscopy. Additionally topological edge states are identified through first principles calculations. I particaulary like the realization and explanation of TCI phase by tuning the compressive strain. The results further motivate investigation of intrinsic atomic defects that locally break the time-reversal-mirror symmetry. Another key aspect of the paper involves the interaction of adjacent topological edge states which driven by a combination of electrostatic interactions and tunneling coupling. The paper is well written, detailed and motivates further investigations in the field of 2D TCI.\nI have a few questions which can further improve the paper for the audience of Nature Communications\n1.Can the authors talk about the lattice mismatch between the substrate and SnTe in terms of the exact percentage? This should be useful in quantitative understanding the distorted vs undistorted Sn-Te bonds in the two layers.\n2.SnTe undergoes structural phase transformation and local distortions and Sn off centring. How are the nature of these structural changes getting influenced because of the substrate?\n3.The effective strained 3-AL SnTe system needs further explanation. I miss the point where the transition is shifted to calculations on 3AL system. How is the strained 1AL bonded to the substrate influencing the overall properties.\n4.The discussion on competition between ferroelectricity and a TCI phase is limited. This is a key concept and needs further explanation in the main text correlating the phonon frequencies, band inversions, lattice parameters.\n5.Figure 2 is very impressive, although it is not translated well in the text and the discussion is very weak.\n6.What modifies the strength of the electric field or charge between adjacent edges? For further studies is it the chemical nature of atoms, strain or materials near electronic phase transition?\n7.In Figure S8, what are the bond length of the 1st AL SnTe with the substrate? Wasn't it established that there are bonds between them which are not shown in Figure S8. Figure S8 seems to show vdw like interactions between the substrate and SnTe.\n8.What is the role of charge transfer between the substrate and SnTe? What drives this?\n9.Can the authors comment on the methodology involved in getting the simulated resilience of pristine edge states against moiré potential and point defects at the edge. Although this is an interesting figure lack of any technical details makes it difficult for reproducing results.\n10.Looking at projected band structure oor pdos of the substrate+4AL would be interesting. The different layer wise projection would make it easier to follow.\n11.A brief discussion to Lifshitz transition and how is it linked with the Fermi surface topology would be useful.\n12.A final comment on the abstract. One of the main conclusions of the paper is strain as a powerful control parameter. However this is not motivated in the abstract.", "reviewer_2": "I have read the manuscript entitled “Strain-induced two-dimensional topological crystalline insulator” by Liwei Jing et al. The authors report the realization of a two-dimensional topological crystalline insulator (2D TCI) state in a 2-monolayer (4 atomic layers) SnTe film grown on a cleaved surface of 2H-NbSe2. They propose the formation of an effective 3 atomic-layer (3AL) SnTe, isolated from the interfacial 1 AL SnTe. Based on STM/STS data, in combination with DFT calculations, the authors claim to observe edge states characteristic of a 2D TCI. Two key experimental signatures are presented to support this claim (please see details in latter part). Furthermore, an intriguing coupling between adjacent conducting edges is demonstrated, interpreted as arising from electrostatic interaction and tunneling coupling.\nThe topic is of broad interest and fits well within the scope of Nature Communications. The motivation and context are clearly articulated, and the STM/STS data at 4 K are of high quality. However, I have major concerns that prevent me from recommending publication at this stage, though I acknowledge the potential significance of the work. My detailed comments are as follows:\nIguessthatthemanuscripthas accumulatedtoomuchdetailduringpreviousrevisioncycles. Whilethekeyclaimsareclearlypositionedandunderstandabletome, itisdifficulttoextractthecoreexperimentalfindingsandtheir solidity as an experimentalist. In particular, what is being observed—and how the authors interpret those observations — needs to be more transparent and systematically presented. Since the manuscript is dense, given the tight timeline, I apologize if I have overlooked or misunderstood anything (I’ve tried my best not to). Further details are as follows.\n(1) Validity of Structural Modeling and Interface Justification on (Sec on II-A) The STM/STS experiments alone are not sufficient to establish a 2D TCI state; the conclusions heavily rely on structural modeling and corresponding DFT calculations. I have the following questions/comments regarding this point, particularly related to the interface model in Fig. 1d:\nReferee comment:A The connection between the first and second paragraphs of Section II-A is unclear at least for me. In other words, I feel a logical jump. It is not obvious how the authors justify simplifying the complex topography (Figs. 1a–b and S6) into the idealized model in Fig. 1d. More explanation is needed about how such simplification is allowed, especially in light of the following complications.\nReferee comment:B The fifth line of Sec on II-A states that 2 ML becomes effectively 3 AL. This raises a natural question: what happens in the 1 ML case? Would it then correspond to effective 1 AL? Can this picture further strengthen the authors’ main conclusion? On the other hand, the use of the term \"amorphous\" seems wired for me —if it were truly amorphous, epitaxial growth wouldn’t be feasible. Then, modeling as in Fig. 1d is not justified\nReferee comment:C The role of spatially varying strain should be addressed more clearly. The manuscript seems to suggest that its influence is minor, mostly affecting energy shifts. However, given the apparent correlation between defect density and stripe patterns (at least visually), could this strain have a more significant effect? If not, please explain why it can be safely neglected in drawing the main conclusions.\nReferee comment:D Figs. 1a, S6, and S20 suggest the coexistence of multiple crystal orientations, likely inferred from surface facet angles. How does this structural inhomogeneity affect the main conclusions? Is NbSe2 substrate domain formation expected or observed? What does the exposed NbSe2 surface look like after SnTe growth? Can the authors still justify a uniform interface model under such conditions? Alternatively, could this domain diversity be used positively, e.g., to increase statistics and strengthen the main claim?\n(2) Edge State Evidence (Section on III). Two key pieces of experimental evidence are provided: (i) a sharp peak in Fig. 3a, and (ii) gap opening due to a point defect in Fig. 3d. I have the following concerns:\nReferee comment:A The large energy window mismatch between Fig. 3a and the bottom panel of Fig. 3d is not reader- friendly. Presenting these data on a consistent energy window would significantly improve clarity.\nReferee comment:B The connection between Fig. 3d and Fig. S16 (right panel) is ambiguous. While the authors state that theory supports the experiment, it is unclear whether Fig. 3d supports local edge destruction, while the theory models a more global effect.\nReferee comment:C Additionally, if the edge is locally destroyed in Fig. 3d (right), how is the remaining edge still clearly visible in the topography (left)? Please clarify.\n(3) Clarification on of Fig. 4c (Edge Coupling Analysis).\nIn Fig. 4c, the x-axis is labeled r, presumably denoting the distance between two positions on edge. How are so many data points obtained in this figure? Furthermore, as noted in the main text, energy shifts along the y-axis could be affected by strain. The construction of Fig. 4c requires a more detailed explanation.\n(4) Global Spectral Evolution: Missing x-y Context. Due to the fragmented nature of the data presentation, it is difficult to grasp the overall spectral evolution across space. A global map of key spectral features would help. For example, it would be helpful to unify the energy scale (e.g., 0–2 eV) and plot the peak positions related to Figs. 3a and 3d in real space, focusing on the region containing the two adjacent SnTe islands. This would greatly aid the reader's understanding of the spatial structure of edge states and their interactions, based on high-statistic data. Since energy resolution is not needed in this case, mapping won’t take so much time. Also, if needed, sample growth will be reproducible, not like cleaving an uncleavable bulk crystal. Although it is a bit demanding, I suggest showing such in this case.\nMinor Comments\nReferee comment: (5) In Fig. 1f, how is the gap determined experimentally? If the assumption is that the peak near 1.7 eV in Fig. 1f corresponds to that in Fig. 1e (right), please state this explicitly.\nReferee comment: (6) There is a mismatch in energy scales between the top and bottom panels of Fig. 3a, which does not appear in Fig. 1e–f. Please unify the energy scale for consistency and better readability.", "reviewer_3": "In this work, the authors claim the realization of a two-dimensional topological crystalline insulator (2D TCI) in a bilayer SnTe/2H-NbSe2 system. This behavior is attributed in part to the significant compressive strain induced by the substrate. Density functional theory (DFT) calculations suggest a strain-driven topological phase transition when the lattice constant is reduced below 4.4 Å. Scanning tunneling microscopy (STM) and spectroscopy (STS) measurements reveal two peaks, which are attributed to edge states. If validated, this system could represent a topological insulator with one of the largest band gaps observed, even accounting for the known underestimation in DFT calculations, as shown in Fig. 2d. However, before this work can be considered for publication in Nature Communications or any peer-reviewed journal, the following major concerns must be addressed:\n1. The topological properties were calculated only for a free-standing SnTe layer. Although the manuscript claims that the bottom SnTe layer is \"detached\" from the others, this assertion lacks sufficient evidence. Given that the bottom layer imposes strain and may introduce defects, it is likely to influence the electronic properties of the entire bilayer system. A more realistic model including the substrate is necessary to assess the robustness of the topological phase.\n2. The topological band gap, induced by spin-orbit coupling, is typically small, as illustrated in Fig. 2f. However, Fig. 2d suggests a much larger band gap, and the connection between these figures is unclear. A more thorough explanation of the apparent discrepancy is needed.\n3. The experimental identification of the observed peaks as topological edge states is not sufficiently substantiated. The data in Figs. 3b and 3c are ambiguous, and key features such as topological protection are not demonstrated. The peaks could instead arise from trivial edge states due to missing neighbors. Furthermore, topological edge states should exhibit gapless dispersion to be useful for edge transport, a hallmark that is not evident in the current data.\nIn summary, the manuscript contains significant uncertainties on both theoretical and experimental fronts. Further analysis and clarification are required before it can be considered for publication."}, "author_response": ""}, {"round": 2, "reviewer_comments": {"reviewer_1": "The revised manuscript satisfactorily addresses all my previous comments. The authors have made significant improvements by incorporating additional results and clarifying key aspects of the study. I find the manuscript to be substantially strengthened and recommend it for acceptance in its current form.", "reviewer_2": "In this second round of review, the authors have made a substantial effort to address nearly all of the questions and concerns previously raised by the referees. The revised version now includes additional experimental data as well as complementary analyses that significantly improve the overall quality and reliability of the work. Although the interfacial structure of the system remains rather complicated—indeed, certain aspects of it are still not fully clarified—it is understandable that an exact or fully quantitative modeling may be difficult at this stage. Despite these limitations, the authors have successfully demonstrated the intriguing and rich physical nature of the two-dimensional topological crystalline insulator (2D TCI), particularly highlighting the emergence of its characteristic edge states and the possible interactions among them. The combination of molecular beam epitaxy (MBE) growth, scanning tunneling microscopy (STM), and density functional theory (DFT) calculations presents a well-balanced and convincing multidisciplinary approach. Overall, while the work may still be somewhat marginal, mostlly in terms of complex structure, I find the results interesting and of potential relevance to the broader condensed matter and materials communities. Therefore, I am to support its publication in Nature Communications.", "reviewer_3": "Overall, the authors’ responses to the reviewers are satisfactory. However, I remain unconvinced by their assertion of the topological nature of the gap states. Nevertheless, the publication of this work could serve as a useful stimulus for further investigations into this and related materials systems.\nThere are also issues with the presentation of the results in Fig. 3. The length scales appear inconsistent: the black region in the top panel of Fig. 3b measures approximately 40 Å, whereas the corresponding widths in the two lower panels are less than 15 Å. Moreover, in Fig. 3c it is not specified which edge the results correspond to.\nFinally, on page 4, if two edge states are expected to interact with each other, should one observe two distinct peaks (splitting) for points 1 and 2?"}, "author_response": "REFEREE 1:\nOverall referee comments:The revised manuscript satisfactorily addresses all my previous comments. The authors have made significant improvements by incorporating additional results and clarifying key aspects of the study. I find the manuscript to be substantially strengthened and recommend it for acceptance in its current form.\nAuthor reply: Wewouldliketothanktherefereesomuchfortherecognitionandinsightful suggestions about our work.\nREFEREE 2:\nOverall referee comments: In this second round of review, the authors have made a substantial effort to address nearly all of the questions and concerns previously raised by the referees. The revised version now includes additional experimental data as well as com- plementary analyses that significantly improve the overall quality and reliability of the work. Althoughtheinterfacialstructureofthesystemremainsrathercomplicated—indeed, certain aspects of it are still not fully clarified—it is understandable that an exact or fully quanti- tative modeling may be difficult at this stage. Despite these limitations, the authors have successfully demonstrated the intriguing and rich physical nature of the two-dimensional topological crystalline insulator (2D TCI), particularly highlighting the emergence of its characteristic edge states and the possible interactions among them. The combination of molecular beam epitaxy (MBE) growth, scanning tunneling microscopy (STM), and density functional theory (DFT) calculations presents a well-balanced and convincing multidisci- plinary approach. Overall, while the work may still be somewhat marginal, mostly in terms of complex structure, I find the results interesting and of potential relevance to the broader condensed matter and materials communities. Therefore, I am to support its publication in Nature Communications.\nAuthor reply: We appreciate the referee’s positive comments and significant suggestions in improving our research work very much. We also thank the referee’s interest in the interfacial structure of our system, which would inspire us to study it further.\nOverall referee comments: Overall, the authors’ responses to the reviewers are satis- factory. However, I remain unconvinced by their assertion of the topological nature of the gap states. Nevertheless, the publication of this work could serve as a useful stimulus for further investigations into this and related materials systems. There are also issues with the presentation of the results in Fig. 3. The length scales appear inconsistent: the black region in the top panel of Fig. 3b measures approximately 40 ˚A, whereas the corresponding widths in the two lower panels are less than 15 ˚A. Moreover, in Fig. 3c it is not specified which edge the results correspond to. Finally, on page 4, if two edge states are expected to interact with each other, should one observe two distinct peaks (splitting) for points 1 and 2?\nAuthor reply: We would like to thank the referee for the prudent examination and positive response in our work. In Figure 3b, the color contrast of STM topographic image could be misleading. So we adjusted it and checked the distance and the scale again. As shown in Figure.R1a, The edge states originating from the outmost atomic layer of the bilayer island decay both the bilayer bulk and the monolayer. The distance between two adjacent edge states marked by two vertical dashed lines is 25 ˚A from spectroscopy. Also, we carefully measured the distance between the adjacent edges in STM topography collected by Createc software installed in our machine. As shown in Figure.R1b, the blue and red lines define the two edges and the distance is around 25 ˚A, too. So the length scale look more consistent.\nIn Fig. 3c, the edge corresponds to the left one shown in Figure. R2 a, which we cropped and showed it in the top panel of Fig. 3c in the main Figure. The spatial evolution of edge states along the moir´e is widespread in the bilayer islands. The edge we show here is just one representative.\nWhen two edge states are brought into close proximity, several studies (already cited in the main text as refs. 26 and 32) report that their interaction can lead to an energy splitting due to strong hybridization. Depending on the interaction strength and mechanism, such hybridization can indeed produce distinct split edge-state energies. In our experiments, however, we do not observe a splitting; instead, we find a uniform energy shift. We therefore suggest that this behavior may arise from electrostatic interactions between neighboring edges. Additionally, quantum tunneling between adjacent edge states could also contribute 2 to the observed energy shift.\nFIG. R1. a, Picture copied from Figure 3b of main text. b, Distance measurement between two edges in the bilayer island by raw data collected by the Createc STM software.\n4"}]

Decision letter