# A Rellich-type theorem for the Helmholtz equation in a junction of stratified media

Sarah Al Humaikani, Anne-Sophie Bonnet-Ben Dhia, Sonia Fliss, Christophe Hazard

► To cite this version:

Sarah Al Humaikani, Anne-Sophie Bonnet-Ben Dhia, Sonia Fliss, Christophe Hazard. A Rellich-type theorem for the Helmholtz equation in a junction of stratified media. 2025. hal-05037634

HAL Id: hal-05037634

https://hal.science/hal-05037634v1

Preprint submitted on 23 Apr 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A RELLICH-TYPE THEOREM FOR THE HELMHOLTZ EQUATION IN A JUNCTION OF STRATIFIED MEDIA*

SARAH AL HUMAIKANI†, ANNE-SOPHIE BONNET-BEN DHIA†, SONIA FLISS†, AND CHRISTOPHE HAZARD†

**Abstract.** We prove that there are no non-zero square-integrable solutions to a two-dimensional Helmholtz equation in some unbounded inhomogeneous domains which represent junctions of stratified media. More precisely, we consider domains that are unions of three half-planes, where each half-plane is stratified in the direction orthogonal to its boundary. As for the well-known Rellich uniqueness theorem for a homogeneous exterior domain, our result does not require any boundary condition. Our proof is based on half-plane representations of the solution which are derived through a generalization of the Fourier transform adapted to stratified media. A byproduct of our result is the absence of trapped modes at the junction of open waveguides as soon as the angles between branches are greater than $\pi/2$.

**Key words.** Helmholtz equation, stratified media, open waveguides, generalized Fourier transform, embedded eigenvalues, trapped modes, Rellich theorem, uniqueness theorem

**MSC codes.** 35A05, 35J05, 42A38, 78A50

## 1. Introduction.

**1.1. Motivation and main result.** In the present paper, we study the possible existence of square-integrable solutions to a two-dimensional Helmholtz equation with varying wavenumber in unbounded media that can be interpreted as *junctions of stratified half-planes*.

We first introduce what we call hereafter a *stratified half-plane*, defined in a local coordinate system $(X, Y)$ by

$$ (1.1) \qquad H := \{(X, Y) \in \mathbb{R}^2 \mid Y > 0\} = \mathbb{R} \times \mathbb{R}^+, $$

where we denote $\mathbb{R}^+ := (0, +\infty)$. This half-plane is called stratified in the sense that it is associated with a real-valued wavenumber function $\kappa$ that only depends on the $x$-coordinate, i.e., $\kappa = \kappa(X)$. In other words, the stratification is orthogonal to the boundary of $H$. In this paper, the direction parallel to the $x$-axis is called *transverse*, whereas the perpendicular one (along the $Y$-axis) is called *longitudinal*. Moreover we assume that $\kappa$ is bounded and becomes constant outside a bounded interval, that is, there exist positive numbers $(\kappa_-, \kappa_+)$ and a pair $(x^-, x^+) \in \mathbb{R}^2$ with $x^- \le x^+$ such that

$$ (1.2) \qquad \forall X \in \mathbb{R}, \quad \kappa(X) = \kappa_- \text{ if } X < X^- \quad \text{and} \quad \kappa(X) = \kappa_+ \text{ if } X > X^+. $$

The media we are interested in, illustrated in Figure 1, represent *junctions* of three such stratified half-planes, denoted $H_W$, $H_N$ and $H_E$ (where the indices stand respectively for west, north and east). Each of them is associated with local coordinates $(X_J, Y_J)$ and a wavenumber function $\kappa_J = \kappa_J(X_J)$ for $J \in \{W, N, E\}$, as described above. We choose a global coordinate system $(x, y)$ which coincides with the local system of the northern half-plane, whereas for each of the two other half-planes, the local

<footnote>*Submitted to the editors 04/16/2025.</footnote>

<footnote>†POEMS, CNRS, Inria, ENSTA Paris, Institut Polytechnique de Paris, 91120 Palaiseau, France (sarah.al-humaikani@ensta.fr, anne-sophie.bonnet-bendhia@ensta.fr, sonia.fliss@ensta.fr, christophe.hazard@ensta.fr).</footnote>
<figure><img src="image_2.png" /><figcaption>FIG. 1. The two considered configurations. (a): right-angle, (b): general angle. The light gray represents various constant values of k, whereas the dark gray stands for possible stratifications.</figcaption></figure>

coordinates $(x_J, y_J)$ are obtained from the global ones $(x, y)$ by a rotation of angle $\theta_J$ centered at a given point $(a_J, b_J)$, for $J \in \{\text{W}, \text{E}\}$. More precisely, we have

$$ (1.3) \quad \begin{pmatrix} x_N \\ y_N \end{pmatrix} = \begin{pmatrix} x \\ y \end{pmatrix} \quad \text{and} \quad \begin{pmatrix} x_J \\ y_J \end{pmatrix} = \begin{pmatrix} \cos \theta_J & \sin \theta_J \\ -\sin \theta_J & \cos \theta_J \end{pmatrix} \begin{pmatrix} x - a_J \\ y - b_J \end{pmatrix} \quad \text{for } J \in \{\text{W}, \text{E}\}, $$

where $\theta_W \in (0, +\pi)$ and $\theta_E \in (-\pi, 0)$. We denote by $\Omega$ the union of the three half-planes, i.e.,

$$ \Omega := H_W \cup H_N \cup H_E, $$

and we consider the wavenumber function defined in $\Omega$ by

$$ (1.4) \quad \forall J \in \{\text{W}, \text{N}, \text{E}\}, \quad \forall (x, y) \in H_J, \quad k(x, y) := \kappa_J(x_J(x, y)), $$

where the function $x_J(x, y)$ refers to the mappings (1.3) between global coordinates $(x, y)$ and local ones $(x_J, y_J)$. Of course, for this definition to make sense, it is necessary that for all points $(x, y)$ in the intersection of two half-planes, both possible definitions of $k(x, y)$ coincide. This clearly implies that $k$ must be constant in each intersection, hence that the stratifications must be located outside these intersections. This also implies that $|\theta_J| \ge \pi/2$ for $J \in \{\text{W}, \text{E}\}$, since the stratifications are orthogonal to the boundaries of the $H_J$'s. In other words, we assume that

$$ (1.5) \quad \theta_W \in [+\pi/2, +\pi) \quad \text{and} \quad \theta_E \in (-\pi, -\pi/2]. $$

These conditions lead us to consider the two configurations depicted in Figure 1. The so-called right-angle configuration on the left-hand side corresponds to the limit case where $(\theta_W, \theta_E) = (+\pi/2, -\pi/2)$, for which $\Omega$ is the complement of a semi-infinite strip. In the other case that we call the general angle configuration, we have $(\theta_W, \theta_E) \ne (+\pi/2, -\pi/2)$, which shows that $\Omega$ is now the complement of a triangle.

The aim of the present paper is to prove the following statement.

**THEOREM 1.1.** *Considering $\Omega \subset \mathbb{R}^2$ and a function $k \in L^\infty(\Omega)$ as described above, if $u \in L^2(\Omega)$ satisfies the Helmholtz equation*

$$ (1.6) \quad -\Delta u - k^2 u = 0 \quad \text{in } \Omega $$

in the distributional sense, then $u = 0$ in $\Omega$.
<figure><img src="image_3.png" /><figcaption>FIG. 2. Examples of situations where Theorem 1.1 can be deduced from existing results.</figcaption></figure>

Theorem 1.1 can be deduced from existing results in some particular cases which are illustrated in Figure 2. The simplest situation is the case where $k$ is constant in the whole domain $\Omega$. In the general angle configuration (case (a) of Figure 2), (1.6) implies that $u$ satisfies the Helmholtz equation outside a disk, then the well-known Rellich's uniqueness theorem [14] tells us that $u$ vanishes outside this disk, hence also in $\Omega$ according to the unique continuation principle. In the right-angle configuration (case (b) of Figure 2), Rellich's theorem no longer applies, but one can use instead the uniqueness theorem in [3], which is exactly the statement of Theorem 1.1 when $k$ is constant and $\Omega$ is a conical domain with vertex angle greater than $\pi$, that is, $\Omega := \{(x, y) \in \mathbb{R}^2 \mid y > -|x|\tan\theta\}$ with $\theta \in (0, \pi/2)$. The latter result (together with the unique continuation principle) also yields Theorem 1.1 for some cases of non-constant $k$, as soon as $k$ remains constant in a conical subdomain of $\Omega$ with vertex angle greater than $\pi$. Such a condition is satisfied for instance in the general angle configuration when $k$ is constant in one of the three half-planes (case (c) of Figure 2), or in $H_W \cup H_N$ (or $H_E \cup H_N$) in the right-angle configuration (case (d) of Figure 2).

Let us point out that no boundary condition is required in Theorem 1.1, nor in the uniqueness results [3, 14] mentioned above. Indeed, these results express that in some particular unbounded domains, the $L^2$ framework prohibits the existence of non-trivial solutions to the Helmholtz equation. In other words, whatever the content of $\mathbb{R}^2 \setminus \Omega$ (i.e., the white semi-infinite strip or triangle of Figure 1), Theorem 1.1 applies. Figure 3 shows some possible examples covered by Theorem 1.1: case (a) shows a junction of three stratified media, whereas case (b) represents a more exotic junction of three stratified media together with a periodic medium. From a physical point of view, Theorem 1.1 ensures the absence of *trapped modes*, that is, localized vibrations of an unbounded medium (provided the unique continuation principle in $\mathbb{R}^2 \setminus \Omega$ holds true). From a spectral point of view, if we consider the selfadjoint operator which describes the dynamics of such a junction, the absence of trapped modes amounts to the absence of eigenvalues embedded in the essential spectrum $[0, +\infty)$ of this operator.

The possible existence of trapped modes at the junction of open waveguides initially motivated this paper. By *open waveguide*, we mean a stratified medium which is unbounded in the transverse direction and such that time-harmonic waves can propagate without attenuation in the longitudinal direction while their energy remains localized in the transverse direction, near the *core* of the waveguide (the dark gray areas in Figure 1). It is known that trapped modes do not exist in uniform straight open waveguides, nor for local perturbations of such waveguides [2, 5, 8, 19, 20], nor in straight junctions of such waveguides [4]. Thanks to [3], we also know that such
<figure><img src="image_4.png" /><figcaption>FIG. 3. *Examples of situations where Theorem 1.1 applies.*</figcaption></figure>

modes do not exist in curved open waveguides either, provided that the core of the waveguides and all other possible inhomogeneities can be encased in a cone of opening angle smaller than $\pi$ (as in case (c) of Figure 2). The present paper provides other configurations where this result holds true. In short, Theorem 1.1 implies the absence of trapped modes at the junction of open waveguides, provided the angles between branches are all greater or equal to $\pi/2$.

The above mentioned results, as well as others in the same vein (see e.g. [18]) could suggest a more general result stating the absence of trapped modes in any unbounded domain. This is not true. Such trapped modes are known to occur in *closed waveguides*, i.e., unbounded tubes with bounded transverse section, about which there exists an abundant literature (see for instance the review papers [11, 13]). We also mention the recent result of [10] which proves the existence of an embedded eigenvalue for a so-called quasi-conical domain.

The proof of Theorem 1.1 follows exactly the same lines as in [3]. The key idea, exploited also in [2, 4, 8], dates back to the pioneering work of [19]. There are two main ingredients in the proof.

(i) On the one hand, we use a *half-plane representation* of $u$, whose construction relies on separation of local space coordinates<fnref n="1" /> in each of the three half-planes. In a homogeneous medium, the usual Fourier transform along the transverse x-direction provides such a representation, for it diagonalizes the transverse component of the Helmholtz operator, that is, $-\partial_x^2$. This is no longer possible in a stratified medium, since we cannot express conveniently the usual Fourier transform of $k^2u$ if $k^2$ is not constant. Fortunately we can construct a *generalized Fourier transform* which diagonalizes $-\partial_x^2 - k^2$ for non-constant $k^2$ and then yields a half-plane representation (see [12, 15] for examples of the use of the generalized Fourier transform in open waveguides). From a physical point of view, the latter representation is interpreted as a modal decomposition of $u$ on a continuous family of modes of the stratified medium. There are two categories of modes depending on their behaviour in the longitudinal direction: evanescent or travelling. Since we are concerned with localized (square-integrable) solutions, the modal components of $u$ associated with travelling modes must vanish.

(ii) On the other hand, the second ingredient of the proof is an *analyticity argument*.

<footnote n="1">¹Rellich's uniqueness theorem [14] also relies on separation of variables. But the separation is made with polar (or spherical) coordinates instead of Cartesian coordinates. Let us mention that even in the very particular sub-case of our problem where $k$ only depends on the polar angle (which may happen in the situations considered in section 3), separation of polar coordinates cannot be used to prove Theorem 1.1.</footnote>
ment with respect to the Fourier variable, which ensures that the modal components extend to an analytic function of the Fourier variable. Thanks to the isolated ze-ros principle, the fact that the travelling components of $u$ vanish implies that the evanescent ones also vanish, hence that $u = 0$, which completes the uniqueness proof.

The paper is organized as follows. We start in subsection 1.2 by recalling the sketch of the proof given in [3] for a homogeneous medium in the right-angle configuration. Then, for the sake of clarity, we focus on a particular case of stratified media composed of only two layers, for which more explicit calculations can be made. Section 2 is devoted to the half-plane representation in such a two-layered medium, after the presentation of the generalized Fourier transform in this context. In section 3, we give the proof of Theorem 1.1 for a junction of two-layered media, considering successively the right-angle and the general angle configurations. The case of general stratified media is finally dealt with in section 4.

**1.2. Sketch of the proof for a homogeneous medium.** As our proof of Theorem 1.1 is a generalization of the one presented in [3] in the case of a homogeneous medium, we begin here by recalling the basic ideas of the latter proof. For the sake of simplicity, we restrict ourselves to the right-angle configuration, i.e., $(\theta_W, \theta_E) = (+\pi/2, -\pi/2)$, see Figure 1 (strictly speaking, this case is not considered in [3], but the proof is easily adapted). For greater clarity, we omit most of the technical details concerning the functional framework.

Hence, in this subsection, function $k(x, y)$ is constant, simply denoted $k$.

**First step.** The core ingredient of the proof is a half-plane representation of a function $u \in L^2(H)$ which satisfies

$$-\Delta u - k^2 u = 0 \quad \text{in H},$$

where H is the half-plane defined in (1.1). This representation is obtained by separation of space variables, thanks to the usual Fourier transform in the x-direction, defined by

$$(1.7) \qquad \forall \xi \in \mathbb{R}, \quad Ff(\xi) := \int_{\mathbb{R}} f(x) e^{-i\xi x} dx.$$

Let $\hat{u}$ denote the partial Fourier transform of $u$ in the x-direction, that is, $\hat{u}(\xi, y) := F[u(\cdot, y)](\xi)$. Applying $F$ to the Helmholtz equation shows that for (almost) every $\xi \in \mathbb{R}$,

$$-\partial_y^2 \hat{u}(\xi, \cdot) - (k^2 - \xi^2) \hat{u}(\xi, \cdot) = 0 \quad \text{in } (0, +\infty).$$

Thus there exist some functions $A(\xi)$ and $B(\xi)$ such that

$$\hat{u}(\xi, y) = A(\xi) e^{-\sqrt{\xi^2 - k^2} y} + B(\xi) e^{\sqrt{\xi^2 - k^2} y},$$

where we use for instance the convention $\sqrt{z} := i\sqrt{-z}$ if $z < 0$. As $F$ is unitary from $L^2(\mathbb{R}_x)$ to $L^2(\mathbb{R}_\xi)$, we infer from the initial assumption $u \in L^2(H)$ that $\hat{u} \in L^2(\mathbb{R}_\xi \times (0, +\infty))$, which implies that for (almost) every $\xi \in \mathbb{R}$, $\hat{u}(\xi, \cdot)$ belongs to $L^2(0, +\infty)$. As a consequence, we have $A(\xi) = 0$ for $|\xi| < k$ and $B(\xi) = 0$ for $\xi \in \mathbb{R}$, which yields

$$(1.8) \qquad \forall (\xi, y) \in \mathbb{R} \times [0, +\infty), \quad \hat{u}(\xi, y) = \begin{cases} 0 & \text{if } |\xi| < k, \\ \hat{u}(\xi, 0) e^{-\sqrt{\xi^2 - k^2} y} & \text{if } |\xi| > k. \end{cases}$$
Using the inverse Fourier transform, we finally obtain the so-called half-plane representation of $u$, which provides an expression of $u$ at any point of H from the Fourier transform of its trace on the boundary of H:

$$ (1.9) \qquad \forall (x, y) \in H, \quad u(x, y) = \frac{1}{2\pi} \int_{|\xi|>k} e^{i\xi x} e^{-\sqrt{\xi^2-k^2} y} \hat{u}(\xi, 0) \, d\xi. $$

**Second step.** Let us show now how to use this representation to prove Theorem 1.1 in the right-angle configuration, i.e., when the relations between the local and global coordinate systems (1.3) simplify as

$$ (1.10) \qquad \begin{pmatrix} x_N \\ y_N \end{pmatrix} = \begin{pmatrix} x \\ y \end{pmatrix}, \quad \begin{pmatrix} x_W \\ y_W \end{pmatrix} = \begin{pmatrix} y - b_W \\ -x + a_W \end{pmatrix} \quad \text{and} \quad \begin{pmatrix} x_E \\ y_E \end{pmatrix} = \begin{pmatrix} -y + b_E \\ x - a_E \end{pmatrix}. $$

Suppose that $u \in L^2(\Omega)$ satisfies the Helmholtz equation in $\Omega := H_W \cup H_N \cup H_E$. Hence, we can use the half-plane representation (1.9) for the restriction of $u$ in each of the three half-planes. For $J \in \{W, N, E\}$, we denote by $\varphi_J$ the trace of $u$ on $\partial H_J$, considered as a function of the local coordinate $x_J$, and by $\hat{\varphi}_J$ its Fourier transform with respect to $x_J$. We know from (1.8) for $Y=0$ that $\hat{\varphi}_J(\xi) = 0$ if $|\xi| < k$.

In order to prove that $u=0$, the key argument consists in proving that $\hat{\varphi}_N(\xi)$ extends to an analytic function of $\xi$ in a complex vicinity of the real axis. As $\hat{\varphi}_N(\xi)$ vanishes on the interval $(-k, +k)$, analyticity implies that it vanishes on the whole real axis. The half-plane representation in $H_N$ then tells us that $u$ vanishes in $H_N$, so finally also in the whole domain $\Omega$ by virtue of the unique continuation principle.

It remains to prove the analyticity of $\hat{\varphi}_N(\xi)$. To do so, we start from the definition of $\hat{\varphi}_N$ (recall that $x_N = x$),

$$ \hat{\varphi}_N(\xi) := \int_{\mathbb{R}} u(x, 0) e^{-i\xi x} \, dx $$

and split the integral in three parts by setting

$$ \begin{aligned} \hat{\varphi}_N(\xi) &= \underbrace{\int_{-\infty}^{a_W} u(x, 0) e^{-i\xi x} \, dx}_{=: \hat{\varphi}_{N,W}(\xi)} + \underbrace{\int_{a_W}^{a_E} u(x, 0) e^{-i\xi x} \, dx}_{=: \hat{\varphi}_{N,0}(\xi)} + \underbrace{\int_{a_E}^{+\infty} u(x, 0) e^{-i\xi x} \, dx}_{=: \hat{\varphi}_{N,E}(\xi)} \\ &= \hat{\varphi}_{N,W}(\xi) \end{aligned} $$

Thanks to Morera's theorem<fnref n="2" />, we see that the second integral $\hat{\varphi}_{N,0}(\xi)$ extends to an entire function of $\xi$, since $x$ lies in a bounded interval (so that $e^{-i\xi x}$ remains bounded for $\xi$ in any bounded complex domain). To deal with $\hat{\varphi}_{N,W}(\xi)$ and $\hat{\varphi}_{N,E}(\xi)$, we use the half-plane representation in $H_W$ and $H_E$ respectively. As both cases are similar, we focus on $\hat{\varphi}_{N,W}(\xi)$. Using (1.10), the half-plane representation (1.9) becomes

$$ \forall (x, y) \in H_W, \quad u(x, y) = \frac{1}{2\pi} \int_{|\eta|>k} e^{i\eta(y-b_W)} e^{-\sqrt{\eta^2-k^2}(-x+a_W)} \hat{\varphi}_W(\eta) \, d\eta. $$

Taking $y=0$, we obtain

$$ \hat{\varphi}_{N,W}(\xi) = \frac{1}{2\pi} \int_{-\infty}^{a_W} \left( \int_{|\eta|>k} e^{-i\eta b_W} e^{-\sqrt{\eta^2-k^2}(-x+a_W)} \hat{\varphi}_W(\eta) \, d\eta \right) e^{-i\xi x} \, dx. $$

<footnote n="2">²Morera's theorem tells us that if $f$ is a continuous complex-valued function defined in a given open disk $D$ of the complex plane, such that $\int_T f(\xi) d\xi = 0$ for every triangular path $T \subset D$, then $f$ is analytic in $D$. To apply it to $\hat{\varphi}_{N,0}(\xi)$, we simply use Fubini's theorem.</footnote>
Fubini's theorem then yields

$$ 
\begin{aligned}
\hat{\varphi}_{N,W}(\xi) &= \frac{e^{-i\xi a_w}}{2\pi} \int_{|\eta|>k} \left( \int_{-\infty}^{a_w} e^{(\sqrt{\eta^2 - k^2} - i\xi)(x-a_w)} dx \right) e^{-i\eta b_w} \hat{\varphi}_w(\eta) d\eta \\
&= \frac{e^{-i\xi a_w}}{2\pi} \int_{|\eta|>k} \frac{e^{-i\eta b_w} \hat{\varphi}_w(\eta)}{\sqrt{\eta^2 - k^2 - i\xi}} d\eta.
\end{aligned}
$$

Using again Morera's theorem, we conclude that the latter integral extends to an analytic function of $\xi$ in $C \setminus i\mathbb{R}$ (since the denominator does not vanish). Proceeding in a similar way on the east side, we have finally proved that $\hat{\varphi}_N(\xi)$ extends to an analytic function of $\xi$ in $C \setminus i\mathbb{R}$, which completes the proof.

**Remark 1.2.** To ensure that the above approach is rigorous, we need in particular to justify that all the integrals we have used are well defined in the Lebesgue sense. Among other things, we must have $\hat{\varphi}_J \in L^1(\mathbb{R})$ for each $J \in \{W, N, E\}$. This is not true in general, since the hypotheses of Theorem 1.1 do not provide any regularity assumption on the $\varphi_J$'s. This explains why in the sequel, we will use, for the half-plane representations, traces on lines which are contained in the interior of $\Omega$ instead of traces on the boundaries of the half-planes (see subsection 2.3).

**2. Fourier half-plane representation for a two-layered medium.** In this section and the next one, we are concerned with the simplest case of layered media, composed of only two layers. We therefore consider a wavenumber function $\kappa$ given by

$$ (2.1) \qquad \kappa(x) := \begin{cases} \kappa_- & \text{if } x < 0, \\ \kappa_+ & \text{if } x > 0, \end{cases} $$

where $\kappa_-$ and $\kappa_+$ are positive constants. The next section provides the proof of Theorem 1.1 for a junction of such media. The present section focuses on the core ingredient of the proof, that is, the extension of the half-plane representation (1.9) to a two-layered medium. This requires us to introduce a generalized Fourier transform which diagonalizes the $x$-dependent part of the Helmholtz operator, more precisely the unbounded selfadjoint operator $A : D(A) \subset L^2(\mathbb{R}) \to L^2(\mathbb{R})$ defined by

$$ (2.2) \qquad \forall \Psi \in D(A) := H^2(\mathbb{R}), \quad A\Psi := -\partial_x^2\Psi - \kappa^2\Psi. $$

We will see that this transform can be interpreted as an operator of “decomposition” on a family of generalized eigenfunctions of $A$ which are introduced below.

**2.1. Generalized eigenfunctions.** As $A$ is selfadjoint, its spectrum is necessarily real. Its eigenelements are thus pairs $(\lambda, \Psi) \in \mathbb{R} \times H^2(\mathbb{R})$ such that $A\Psi = \lambda\Psi$, that is,

$$ (2.3) \qquad -\partial_x^2\Psi - \kappa^2\Psi = \lambda\Psi \quad \text{in } \mathbb{R}. $$

This equation can also be derived from the Helmholtz equation by separation of variables, i.e., by searching for solutions to $-\Delta u - \kappa^2 u = 0$ in the form $u(x, y) = \Psi(x) e^{\pm \sqrt{\lambda}y}$. The latter behavior in the $y$-direction will be implicitly understood below in the physical interpretation of the generalized eigenfunctions.

It is easily seen that with the assumption $\Psi \in H^2(\mathbb{R})$, the only possible solution to (2.3) is $\Psi = 0$ for any $\lambda \in \mathbb{R}$, which shows that the point spectrum of $A$ is empty. However, if we forget this assumption and search for bounded solutions, the dimension
of the space of bounded solutions is 0 if $\lambda < -\max(\kappa_2^-, \kappa_+^2)$, 1 if $-\max(\kappa_2^-, \kappa_+^2) \le \lambda < -\min(\kappa_2^-, \kappa_+^2)$ and 2 if $\lambda \ge -\min(\kappa_2^-, \kappa_+^2)$. We introduce a particular basis of this space. Denoting

$$ (2.4) \qquad \Lambda^{\pm} := (-\kappa_{\pm}^{2}, +\infty), $$

we consider two functions $\Psi^{\pm} : \Lambda^{\pm} \times \mathbb{R} \to \mathbb{C}$ defined for all $(\lambda, x) \in \Lambda^{\pm} \times \mathbb{R}$ by

$$ (2.5) \qquad \Psi^{\pm}(\lambda, x) := \begin{cases} e^{\mp i \beta^{\pm}(\lambda)x} + R^{\pm}(\lambda) e^{\pm i \beta^{\pm}(\lambda)x} & \text{if } \pm x > 0, \\ T^{\pm}(\lambda) e^{\mp i \beta^{\mp}(\lambda)x} & \text{if } \mp x > 0, \end{cases} $$

where

$$ (2.6) \qquad \beta^{\pm}(\lambda) := \begin{cases} \sqrt{\lambda + \kappa_{\pm}^2} & \text{if } \lambda \ge -\kappa_{\pm}^2, \\ i\sqrt{-\lambda - \kappa_{\pm}^2} & \text{if } \lambda < -\kappa_{\pm}^2, \end{cases} $$

$$ (2.7) \qquad R^{\pm}(\lambda) := \frac{\beta^{\pm}(\lambda) - \beta^{\mp}(\lambda)}{\beta^{+}(\lambda) + \beta^{-}(\lambda)} \quad \text{and} \quad T^{\pm}(\lambda) := \frac{2\beta^{\pm}(\lambda)}{\beta^{+}(\lambda) + \beta^{-}(\lambda)}. $$

For a given $\lambda$, $\Psi^{\pm}(\lambda, \cdot)$ is called a generalized eigenfunction of A, because it is a solution to the eigenvalue equation (2.3), but it does not belong to $D(A)$ (nor to $L^2(\mathbb{R}))$, which justifies the word generalized. This function can be interpreted as the response of the two-layered medium to an incident wave with unit amplitude and wavenumber $\beta^{\pm}(\lambda)$. In the notation $\Psi^{\pm}$, the sign indicates the side where the incident wave lies, i.e., $x > 0$ for + and $x < 0$ for -. The behaviour of $\Psi^{\pm}(\lambda, \cdot)$ depends on whether $\beta^{\pm}(\lambda)$ is real or imaginary. To fix ideas, let us assume for instance that $\kappa_- > \kappa_+$, i.e., $-\kappa_-^2 < -\kappa_+^2$.

First consider $\Psi^{+}(\lambda, \cdot)$ for a given $\lambda \in \Lambda^{+} := (-\kappa_{+}^2, +\infty)$. Noticing that both $\beta^{-}(\lambda)$ and $\beta^{+}(\lambda)$ are real, we see that function $\Psi^{+}(\lambda, \cdot)$ can be written more explicitly as

$$ \Psi^{+}(\lambda, x) = \begin{cases} e^{-i\sqrt{\lambda+\kappa_{+}^2}x} + R^{+}(\lambda)e^{i\sqrt{\lambda+\kappa_{+}^2}x} & \text{if } x > 0, \\ T^{+}(\lambda)e^{-i\sqrt{\lambda+\kappa_{-}^2}x} & \text{if } x < 0. \end{cases} $$

Hence, on the + side, it represents the superposition of an incident wave which propagates from $+\infty$ and a reflected wave which propagates back towards $+\infty$ and whose amplitude is given by the reflection coefficient $R^{+}(\lambda)$, which is real. On the - side, it becomes a transmitted wave which propagates towards $-\infty$ and whose amplitude is given by the transmission coefficient $T^{+}(\lambda)$, which is also real.

Consider now $\Psi^{-}(\lambda, \cdot)$ for a given $\lambda \in \Lambda^{-} := (-\kappa_{-}^2, +\infty)$. Two situations occur. On the one hand, if $\lambda > -\kappa_{+}^2$, as both $\beta^{-}(\lambda)$ and $\beta^{+}(\lambda)$ are real, the interpretation of $\Psi^{-}(\lambda, \cdot)$ is the same as above by swapping the roles of both sides. On the other hand, if $\lambda \in (-\kappa_{-}^2, -\kappa_{+}^2)$, then $\beta^{-}(\lambda)$ is still real, but $\beta^{+}(\lambda)$ becomes imaginary, so that

$$ \Psi^{-}(\lambda, x) = \begin{cases} e^{i\sqrt{\lambda+\kappa_{-}^2}x} + R^{-}(\lambda)e^{-i\sqrt{\lambda+\kappa_{-}^2}x} & \text{if } x < 0, \\ T^{-}(\lambda)e^{-i\sqrt{-\lambda-\kappa_{+}^2}x} & \text{if } x > 0. \end{cases} $$

Hence, on the - side, it represents the superposition of an incident wave which propagates from $-\infty$ and a reflected wave with a reflection coefficient $R^{-}(\lambda)$ which is now
<figure><img src="image_5.png" /><figcaption>FIG. 4. Behavior of the generalized eigenfunctions in the case where $K_{-} > K_{+}$.</figcaption></figure>

a complex number such that $|R^{-}(\lambda)| = 1$. On the + side, the transmitted wave is now evanescent. This case corresponds to the well-known phenomenon of total reflection.

These comments are summarized in Figure 4. Of course, similar interpretations hold in the case where $K_{-} < K_{+}$, swapping the roles of both sides.

The following proposition gathers some properties which are needed in the sequel.

**PROPOSITION 2.1.** (i) The generalized eigenfunctions are bounded, in the sense that

$$ (2.8) \qquad \forall (\lambda, x) \in \Lambda^{\pm} \times \mathbb{R}, \quad |\Psi^{\pm}(\lambda, x)| \le 2. $$

(ii) Consider

$$ (2.9) \qquad \mathbb{D} := \mathbb{C} \setminus (-\infty, -\min(\kappa_{-}^2, \kappa_{+}^2)]. $$

Each of the following functions has an analytic continuation in $\mathbb{D}$: $\beta^{\pm}(\lambda)$, $R^{\pm}(\lambda)$, $T^{\pm}(\lambda)$, as well as $\Psi^{\pm}(\lambda, x)$ for any fixed $x \in \mathbb{R}$. Moreover the analytic continuations of $\Psi^{\pm}(\lambda, x)$ are bounded in any compact set of $\mathbb{D} \times \mathbb{R}$.

*Proof.* (i) As $\beta^{\pm}(\lambda) \in \mathbb{R}^{+} \cup i\mathbb{R}^{+}$ for $\lambda \in \Lambda^{\pm}$, we have $|R^{\pm}(\lambda)| \le 1$ and $|T^{\pm}(\lambda)| \le 2$ on $\Lambda^{\pm}$, which yields (2.8) since the exponentials involved in the expression (2.5) are either oscillating or evanescent.

(ii) Consider the definition (2.6) of $\beta^{\pm}(\lambda)$ restricted to $\lambda \in (-\kappa_{\pm}^2, +\infty)$. Using the principal determination of the complex square root, i.e., $\sqrt{z} := |z|^{1/2}e^{i(\arg z)/2}$ with $\arg z \in (-\pi, +\pi)$, we see that $\beta^{\pm}(\lambda)$ has an analytic continuation in $\mathbb{C} \setminus (-\infty, -\kappa_{\pm}^2]$. Hence, as both functions $\beta^{-}(\lambda)$ and $\beta^{+}(\lambda)$ are involved in the definition (2.7) of $R^{\pm}(\lambda)$ and $T^{\pm}(\lambda)$, the latter functions extend to analytic functions in $\mathbb{D}$, which is the common domain of analyticity of $\beta^{-}(\lambda)$ and $\beta^{+}(\lambda)$ (note that the denominator $\beta^{+}(\lambda) + \beta^{-}(\lambda)$ cannot vanish in $\mathbb{D}$). Therefore the same holds for $\Psi^{\pm}(\lambda, x)$ for any fixed $x \in \mathbb{R}$. Moreover, as $e^{\pm i\beta^{\pm}(\lambda)x}$ is bounded in any compact set of $\mathbb{D} \times \mathbb{R}$, so is $\Psi^{\pm}(\lambda, x). \quad \square$

**Remark 2.2.** In this section we interpreted complex exponentials as being ingoing or outgoing based on an implicit convention, a time-dependence in $e^{-i\omega t}$. For a time-dependence in $e^{i\omega t}$, the $\Psi^{\pm}$'s would then be replaced by their conjugates and everything that follows would hold true with this alternative choice.
**2.2. Generalized Fourier transform.** The usual Fourier transform $F$ defined in (1.7) can be interpreted as an operator of “decomposition” on the family of functions $\Psi_{\xi}(x) := e^{i\xi x}$ for $\xi \in \mathbb{R}$, which appear as generalized eigenfunctions of the operator $-i\partial_x$ in the sense that they satisfy $-i\partial_x \Psi_{\xi} = \xi \Psi_{\xi}$. Moreover $F$ diagonalizes $-i\partial_x$, in that it transforms the action of this operator into a multiplication by $\xi$, which can be written in a synthetic form $-i\partial_x = F^{-1}\xi F$. In this relation, $\xi$ plays the role of a spectral variable that lies in the spectrum of $-i\partial_x$, which covers the whole real line $\mathbb{R}$.

Similar properties hold for every selfadjoint operator in a Hilbert space (see, e.g. [1]). Theorem 2.3 below tells us that that the projection on the family of generalized eigenfunctions defined in (2.5) diagonalizes the operator $A$ introduced in (2.2). In order to make precise the functional framework in which this diagonalization takes place, we need to introduce a few notations to complement (2.4)-(2.7).

Considering the functions $\rho^{\pm} : \Lambda^{\pm} \to \mathbb{R}$ given by

$$ (2.10) \qquad \forall \lambda \in \Lambda^{\pm}, \quad \rho^{\pm}(\lambda) := \frac{1}{4\pi \beta^{\pm}(\lambda)} = \frac{1}{4\pi \sqrt{\lambda + \kappa_{\pm}^2}}, $$

we denote $L^2(\Lambda^{\pm}, \rho^{\pm})$ the weighted space of square integrable functions on $\Lambda^{\pm}$ for the measure $\rho^{\pm}d\lambda$, i.e., the space composed of measurable functions $\hat{\varphi}^{\pm} : \Lambda^{\pm} \to \mathbb{C}$ such that

$$ \|\hat{\varphi}^{\pm}\|_{L^2(\Lambda^{\pm}, \rho^{\pm})}^2 := \int_{\Lambda^{\pm}} |\hat{\varphi}^{\pm}(\lambda)|^2 \rho^{\pm}(\lambda) d\lambda < +\infty. $$

We will also use the space $L^1(\Lambda^{\pm}, \rho^{\pm})$ composed of measurable functions $\hat{\varphi}^{\pm} : \Lambda^{\pm} \to \mathbb{C}$ such that

$$ \int_{\Lambda^{\pm}} |\hat{\varphi}^{\pm}(\lambda)| \rho^{\pm}(\lambda) d\lambda < +\infty. $$

Then we define the Hilbert space

$$ \hat{\mathcal{H}} := L^2(\Lambda^{-}, \rho^{-}) \times L^2(\Lambda^{+}, \rho^{+}), $$

which acts as the spectral space in the diagonalization theorem below. The proof of this theorem is based on the spectral theorem, following the same lines as in the pioneering book of Titchmarsh [17]. An explicit construction can be found for instance in [12].

**THEOREM 2.3.** *The generalized Fourier transform $\mathcal{F}$ defined for all $\varphi \in L^1(\mathbb{R}) \cap L^2(\mathbb{R})$ by*

$$ (2.11) \qquad \mathcal{F}\varphi := (\mathcal{F}^{-}\varphi, \mathcal{F}^{+}\varphi) \quad \text{with} \quad \mathcal{F}^{\pm}\varphi(\lambda) := \int_{\mathbb{R}} \varphi(x) \overline{\Psi^{\pm}(\lambda, x)} \,dx \quad \text{for } \lambda \in \Lambda^{\pm}, $$

extends by density to a unitary operator from $L^2(\mathbb{R})$ to $\hat{\mathcal{H}}$. It diagonalizes $A$ in the sense that

$$ (2.12) \qquad \forall \varphi \in H^2(\mathbb{R}), \quad A\varphi = \mathcal{F}^{-1}\lambda \mathcal{F}\varphi, $$

where the notation $\lambda$ stands for the operator of multiplication by $\lambda$ in $\hat{\mathcal{H}}$. Moreover, the inverse transform $\mathcal{F}^{-1}$ writes explicitly as follows, for all $(\hat{\varphi}^{-}, \hat{\varphi}^{+}) \in \hat{\mathcal{H}}$ such that
$\hat{\varphi}^{\pm} \in L^{1}(\Lambda^{\pm}, \rho^{\pm}):$

$$
(2.13) \quad \forall x \in \mathbb{R}, \quad \mathcal{F}^{-1}(\hat{\varphi}^{-}, \hat{\varphi}^{+})(x) = \sum_{\pm} \int_{\Lambda^{\pm}} \hat{\varphi}^{\pm}(\lambda) \Psi^{\pm}(\lambda, x) \rho^{\pm}(\lambda) \, d\lambda.
$$

Remark 2.4. Note that formula (2.11) makes sense for all functions $\varphi \in L^1(\mathbb{R})$, since (2.8) tells us that $\Psi^{\pm}(\lambda, x)$ remains bounded. But for $\varphi \in L^2(\mathbb{R}) \setminus L^1(\mathbb{R})$, it becomes an improper integral whose definition follows from a density argument as in the definition of the usual Fourier transform in $L^2(\mathbb{R})$. Again thanks to (2.8), the same holds true for formula (2.13), which makes sense for all $\hat{\varphi}^{\pm} \in L^1(\Lambda^{\pm}, \rho^{\pm})$.

Theorem 2.3 provides us a spectral decomposition of $L^2(\mathbb{R})$ associated with operator $A$. On the one hand, $\mathcal{F}$ acts as an operator of “decomposition” on the family composed of the $\Psi^{\pm}$’s, in the sense that it furnishes the spectral components $\mathcal{F}\varphi$ of $\varphi$. These components are functions of the variable $\lambda$ that lies in the spectrum of $A$, which is purely continuous and given by $\bar{\Lambda}^{-} \cup \Lambda^{+} = [-\max(\kappa_{-}^{2}, \kappa_{+}^{2}), +\infty)$. On the other hand, $\mathcal{F}^{-1}$ appears as an operator of “recomposition” which allows us to reconstruct a function $\varphi$ from its spectral components.

It is instructive to understand how this generalized Fourier transform relates to the usual one in the particular case of a homogeneous medium, that is, when $\kappa_{-} = \kappa_{+}$. Simply denoting $\kappa$ this common value, we have in this case $\Lambda^{-} = \Lambda^{+} = (-\kappa^{2}, +\infty)$ and

$$
\forall \lambda \in (-\kappa^2, +\infty), \quad \beta^{-}(\lambda) = \beta^{-}(\lambda) = \sqrt{\lambda + \kappa^2}, \quad R^{\pm}(\lambda) = 0 \quad \text{and} \quad T^{\pm}(\lambda) = 1.
$$

Therefore,

$$
\forall (\lambda, x) \in (-\kappa^2, +\infty) \times \mathbb{R}, \quad \Psi^{\pm}(\lambda, x) = e^{\mp i \sqrt{\lambda + \kappa^2} x}.
$$

This shows that the definition (2.11) of the generalized Fourier transform simplifies
as

$$
\mathcal{F}\varphi := (\mathcal{F}^{-}\varphi, \mathcal{F}^{+}\varphi) \quad \text{with} \quad \mathcal{F}^{\pm}\varphi(\lambda) := \int_{\mathbb{R}} \varphi(x) e^{\pm i \sqrt{\lambda+\kappa^2} x} \, dx \quad \text{for } \lambda \in (-\kappa^2, +\infty).
$$

We observe that we return to the definition (1.7) of the usual Fourier transform simply
by setting $\xi = \mp\sqrt{\lambda + \kappa^2}$. More precisely, introducing the operator $\mathcal{C}$ associated to
this change of variable, defined for a function $g$ of the $\xi$-variable by

$$
\mathcal{C}g := (\mathcal{C}^{-}g, \mathcal{C}^{+}g) \quad \text{where} \quad \mathcal{C}^{\pm}g(\lambda) := g(\mp\sqrt{\lambda+\kappa^2}) \quad \text{for } \lambda \in (-\kappa^2, +\infty),
$$

we have

$\mathcal{F} = \mathcal{C} \mathcal{F}.$

Note that this formula allows us to derive the unitary nature of $\mathcal{F}$ asserted in Theorem 2.3 from that of the usual Fourier transform. Indeed we know on the one hand that $\mathcal{F}$ is unitary from $L^2(\mathbb{R}_x)$ to $L^2(\mathbb{R}_\xi, (2\pi)^{-1})$. On the other hand, noticing that

$$
\frac{1}{2\pi} \int_{\mathbb{R}} |g(\xi)|^2 d\xi = \sum_{\pm} \int_{-\kappa^2}^{+\infty} |g(\mp\sqrt{\lambda+\kappa^2})|^2 \frac{1}{4\pi\sqrt{\lambda+\kappa^2}} d\lambda,
$$
we infer that $\mathcal{C}$ is unitary from $L^2(\mathbb{R}_\xi, (2\pi)^{-1})$ to $\hat{\mathcal{H}}$. Thus their product is unitary from $L^2(\mathbb{R}_x)$ to $\hat{\mathcal{H}}$. The above formula explains the presence of the weight functions $\rho^\pm(\lambda)$ in the spectral space of the generalized Fourier transform.

**2.3. Deriving the half-plane representation.** We are now able to generalize the half-plane representation (1.9) to a two-layered medium. The technical difficulty mentioned in Remark 1.2 leads us to modify our definition (1.1) of H, without loss of generality. From now on, H denotes the half-plane defined in a local coordinate system $(X, Y)$ by

$$ (2.14) \qquad H := \{(X, Y) \in \mathbb{R}^2 \mid Y > -\varepsilon\}, $$

where $\varepsilon > 0$ is a fixed parameter whose role is simply that the x-axis $\Sigma := \mathbb{R} \times \{0\}$ is now contained in H. Our aim is to derive a half-plane representation of a solution $u \in L^2(H)$ to the Helmholtz equation in H, i.e., to express $u$ by means of its trace on $\Sigma$. As $\Sigma \subset H$, interior regularity of solutions to the Helmholtz equation (2.15) shows this trace is well defined as a continuous function (since $u \in H_{\text{loc}}^2(H)$).

**PROPOSITION 2.5.** *Suppose that $u \in L^2(H)$ satisfies*

$$ (2.15) \qquad -\Delta u - \kappa^2 u = 0 \quad \text{in H,} $$

in the distributional sense, where $\kappa = \kappa(X)$ is given in (2.1). Then,

$$ (2.16) \quad \forall (x, y) \in \mathbb{R} \times [0, +\infty), \quad u(x, y) = \sum_{\pm} \int_{\mathbb{R}^{+}} \hat{\varphi}^{\pm}(\lambda) \Psi^{\pm}(\lambda, x) e^{-\sqrt{\lambda}y} \rho^{\pm}(\lambda) \, d\lambda, $$

where $\Psi^\pm$ and $\rho^\pm$ are defined respectively in (2.5) and (2.10), and $\hat{\varphi}^\pm := \mathcal{F}^\pm u(\cdot, 0)$ has the following properties:

$$ (2.17) \qquad \hat{\varphi}^{\pm}(\lambda) = 0 \quad \text{if } \lambda \in (-\kappa_{\pm}^2, 0) \quad \text{and} \quad \lambda^s \hat{\varphi}^{\pm}(\lambda) \in L^1(\mathbb{R}^+, \rho^{\pm}) \quad \text{for } s > -\frac{3}{4}. $$

**Remark 2.6.** Note that the integral in (2.16) is well defined (in the Lebesgue sense). Indeed $\Psi^\pm(\lambda, x)$ is bounded (see (2.8)), as well as $e^{-\sqrt{\lambda}y}$ for $y \in [0, +\infty)$, and $\hat{\varphi}^\pm$ belongs to $L^1(\Lambda^\pm, \rho^\pm)$ (according to (2.17) with $s = 0$).

**Proof.** As $u \in L^2(H)$, we know that for almost every $y \in (-\varepsilon, +\infty)$, we have $u(\cdot, y) \in L^2(\mathbb{R})$, so that we can define its partial generalized Fourier transform in the x-direction by $\hat{u}(\cdot, y) := (\hat{u}^-(\cdot, y), \hat{u}^+(\cdot, y))$ where $\hat{u}^\pm(\cdot, y) := \mathcal{F}^\pm u(\cdot, y)$. As $\mathcal{F}$ is unitary from $L^2(\mathbb{R})$ to $\hat{\mathcal{H}}$, we have

$$ \|u(\cdot, y)\|_{L^2(\mathbb{R})}^2 = \sum_{\pm} \|\hat{u}^{\pm}(\cdot, y)\|_{L^2(\Lambda^{\pm}, \rho^{\pm})}^2. $$

Integrating on $(-\varepsilon, +\infty)$ with respect to $y$ then yields

$$ (2.18) \qquad \begin{aligned} \|u\|_{L^2(H)}^2 &= \sum_{\pm} \int_{-\varepsilon}^{+\infty} \int_{\Lambda^{\pm}} |\hat{u}^{\pm}(\lambda, y)|^2 \rho^{\pm}(\lambda) \, d\lambda \, dY \\ &= \sum_{\pm} \int_{\Lambda^{\pm}} \|\hat{u}^{\pm}(\lambda, \cdot)\|_{L^2(-\varepsilon, +\infty)}^2 \rho^{\pm}(\lambda) \, d\lambda, \end{aligned} $$
where the second equality follows from Fubini's theorem. This shows in particular that

$$ (2.19) \qquad \text{for almost every } \lambda \in \Lambda^{\pm}, \quad \hat{u}^{\pm}(\lambda, \cdot) \in L^2(-\varepsilon, +\infty). $$

We prove now that applying the partial generalized Fourier transform to the Helmholtz equation (2.15) yields

$$ (2.20) \qquad -\partial_{\gamma}^2 \hat{u}^{\pm}(\lambda, \cdot) + \lambda \hat{u}^{\pm}(\lambda, \cdot) = 0 \quad \text{in } (-\varepsilon, +\infty) $$

in the distributional sense, for almost every $\lambda \in \Lambda^{\pm}$. The idea is to use test functions with separated variables, i.e., in the form $\psi(x)\chi(y)$. Indeed, the Helmholtz equation (2.15) amounts to saying that for all $\psi \in D(\mathbb{R})$ and $\chi \in D(-\varepsilon, +\infty)$ (usual spaces of compactly supported indefinitely differentiable functions), we have

$$ \langle\langle -\partial_x^2 u - \partial_y^2 u - \kappa^2 u, \psi \chi \rangle\rangle_x_y = 0, $$

where we distinguish the anti-duality products in the X and Y variables, which can be interchanged. From the definition of derivatives in the distributional sense, we deduce that

$$ \langle\langle u, -(\partial_x^2 \psi) \chi - \psi (\partial_y^2 \chi) \rangle\rangle_x_y - \langle\langle \kappa^2 u, \psi \chi \rangle\rangle_x_y = 0. $$

or equivalently

$$ -\langle\langle u, \chi \rangle\rangle_y, \partial_x^2 \psi \rangle\rangle_x - \langle\langle \kappa^2 \langle u, \chi \rangle_y, \psi \rangle\rangle_x - \langle\langle u, \partial_y^2 \chi \rangle\rangle_y, \psi \rangle\rangle_x = 0. $$

As $u \in L^2(H)$, the anti-duality products in this expression can be replaced by integrals, which allows us to rewrite it as

$$ \int_{\mathbb{R}} \langle u(x, \cdot), \chi \rangle_{\gamma} \overline{(-\partial_x^2 \psi(x) - \kappa(x)^2 \psi(x))} dx - \int_{\mathbb{R}} \langle u(x, \cdot), \partial_y^2 \chi \rangle_{\gamma} \overline{\psi(x)} dx = 0. $$

Interpreting both integrals as inner products in $L^2(\mathbb{R})$, we can use the generalized Plancherel's identity following from the unitary nature of $\mathcal{F}$ from $L^2(\mathbb{R})$ to $\hat{\mathcal{H}}$. Denoting $(\hat{\psi}^{-}, \hat{\psi}^{+}) := \mathcal{F}\psi$, we infer that the first integral is equal to

$$ \sum_{\pm} \int_{\Lambda^{\pm}} \langle \hat{u}^{\pm}(\lambda, \cdot), \chi \rangle_{\gamma} \overline{\lambda \hat{\psi}^{\pm}(\lambda)} \rho^{\pm}(\lambda) d\lambda. $$

Indeed, on the one hand, we have $\mathcal{F}(-\partial_x^2 \psi - \kappa^2 \psi) = \mathcal{F}(A\psi) = \lambda \mathcal{F}\psi$ thanks to (2.12). On the other hand, the generalized Fourier transform of $x \to \langle u(x, \cdot), \chi \rangle_y$ is given by $\langle \hat{u}^{\pm}(\lambda, \cdot), \chi \rangle_y$, which amounts to interchanging $\mathcal{F}$ with the Y-anti-duality products. This permutation is obvious when $\mathcal{F}$ applies to functions with separated variables; as these functions span $L^2(H)$, the general case follows from a density argument. Computing the second integral yields finally

$$ \sum_{\pm} \int_{\Lambda^{\pm}} \left( \lambda \langle \hat{u}^{\pm}(\lambda, \cdot), \chi \rangle_{\gamma} - \langle \hat{u}^{\pm}(\lambda, \cdot), \partial_{\gamma}^2 \chi \rangle_{\gamma} \right) \overline{\hat{\psi}^{\pm}(\lambda)} \rho^{\pm}(\lambda) d\lambda = 0, $$

As this equality holds true for all $\psi \in D(\mathbb{R})$, we infer that the quantity between parentheses must vanish for almost every $\lambda \in \Lambda^{\pm}$ and every $\chi \in D(-\varepsilon, +\infty)$, which is exactly equation (2.20).
From this equation, we deduce that there exist functions $A^{\pm}(\lambda)$ and $B^{\pm}(\lambda)$ such that

$$ \hat{u}^{\pm}(\lambda, Y) = A^{\pm}(\lambda) e^{-\sqrt{\lambda}(Y+\epsilon)} + B^{\pm}(\lambda) e^{+\sqrt{\lambda}(Y+\epsilon)}, $$

where we keep the convention $\sqrt{\lambda} := i\sqrt{-\lambda}$ if $\lambda < 0$. Thanks to (2.19), we infer that $A^{\pm}(\lambda) = 0$ if $\lambda \in (-\kappa_{\pm}^2, 0)$ and $B^{\pm}(\lambda) = 0$ for all $\lambda \in \Lambda^{\pm}$. Therefore,

$$ \hat{u}^{\pm}(\lambda, Y) = A^{\pm}(\lambda) e^{-\sqrt{\lambda}(Y+\epsilon)} \quad \text{where} \quad A^{\pm}(\lambda) = 0 \text{ if } \lambda \in (-\kappa_{\pm}^2, 0). $$

According to the definition $\hat{\varphi}^{\pm} := \mathcal{F}^{\pm}u(\cdot, 0)$, this can be written equivalently

$$ \hat{u}^{\pm}(\lambda, Y) = \hat{\varphi}^{\pm}(\lambda) e^{-\sqrt{\lambda}Y} \quad \text{where} \quad \hat{\varphi}^{\pm}(\lambda) = 0 \text{ if } \lambda \in (-\kappa_{\pm}^2, 0). $$

Using the expression (2.13) of $\mathcal{F}^{-1}$, we obtain the half-plane representation (2.16), where the existence of the integral will be justified, once (2.17) is proven (see Remark 2.6).

To do so, notice that for almost every $\lambda \in \Lambda^{\pm}$,

$$ \|\hat{u}^{\pm}(\lambda, \cdot)\|_{L^2(-\varepsilon, +\infty)}^2 = |\hat{\varphi}^{\pm}(\lambda)|^2 \int_{-\varepsilon}^{+\infty} e^{-2\sqrt{\lambda}Y} dY = \frac{|\hat{\varphi}^{\pm}(\lambda)|^2 e^{2\sqrt{\lambda}\varepsilon}}{2\sqrt{\lambda}}. $$

Hence we deduce from (2.18) that

$$ \|u\|_{L^2(\mathbb{H})}^2 = \sum_{\pm} \int_{\mathbb{R}^{+}} \frac{|\hat{\varphi}^{\pm}(\lambda)|^2 e^{2\sqrt{\lambda}\varepsilon}}{2\sqrt{\lambda}} \rho^{\pm}(\lambda) d\lambda, $$

which shows that $\lambda^{-1/4} e^{\sqrt{\lambda}\varepsilon} \hat{\varphi}^{\pm}(\lambda) \in L^2(\mathbb{R}^{+}, \rho^{\pm})$. Therefore Cauchy-Schwartz inequality yields

$$ \int_{\mathbb{R}^{+}} \lambda^{s} |\hat{\varphi}^{\pm}(\lambda)| \rho^{\pm}(\lambda) d\lambda \le \left\| \lambda^{-1/4} e^{\sqrt{\lambda}\varepsilon} \hat{\varphi}^{\pm}(\lambda) \right\|_{L^{2}(\mathbb{R}^{+}, \rho^{\pm})} \left\| \lambda^{s+1/4} e^{-\sqrt{\lambda}\varepsilon} \right\|_{L^{2}(\mathbb{R}^{+}, \rho^{\pm})}, $$

where the last norm is bounded provided $s > -3/4$, which proves (2.17). □

<figure><img src="image_6.png" /></figure>

The following corollary of Proposition 2.5 tells us about the behaviour of $u$ in any non-transverse direction.

**COROLLARY 2.7.** Let $u$ defined as in Proposition 2.5. Then for any half-line $\Gamma_{\alpha} := \{(t \cos \alpha, t \sin \alpha) \in \mathbb{R}^2 \mid t > 0\} \subset \mathbb{H}$ where $\alpha \in (0, \pi)$, we have $u|_{\Gamma_{\alpha}} \in L^1(\Gamma_{\alpha})$.

*Proof.* Using the half-plane representation (2.16), then (2.8) and Fubini's theorem, we obtain

$$ \begin{align*} \int_{0}^{+\infty} |u(t \cos \alpha, t \sin \alpha)| dt & = \int_{0}^{+\infty} \left| \sum_{\pm} \int_{\mathbb{R}^{+}} \hat{\varphi}^{\pm}(\lambda) \Psi^{\pm}(\lambda, t \cos \alpha) e^{-\sqrt{\lambda}t \sin \alpha} \rho^{\pm}(\lambda) d\lambda \right| dt \\ & \le 2 \sum_{\pm} \int_{0}^{+\infty} \int_{\mathbb{R}^{+}} |\hat{\varphi}^{\pm}(\lambda)| e^{-\sqrt{\lambda}t \sin \alpha} \rho^{\pm}(\lambda) d\lambda dt \\ & \le 2 \sum_{\pm} \int_{\mathbb{R}^{+}} \frac{|\hat{\varphi}^{\pm}(\lambda)|}{\sqrt{\lambda} \sin \alpha} \rho^{\pm}(\lambda) d\lambda, \end{align*} $$
<figure><img src="image_7.png" /><figcaption>FIG. 5. Junctions of two-layered media.</figcaption></figure>

where the last integral is bounded, according to (2.17) with $s = -1/2$.

<figure><img src="image_8.png" /></figure>

**3. Junction of two-layered media.** In this section, we come back to the junction of stratified media as defined in subsection 1.1, but we restrict ourselves to the particular case of two-layered media, as considered in section 2. Hence each of the three half-planes $H_W, H_N$ and $H_E$ is characterized by a two-valued wavenumber function $K_J$ for $J \in \{W, N, E\}$, where $K_J(X) = K_{J,\pm}$ if $\pm X_J > 0$. By construction, we have

$$ (3.1) \quad K_{W,+} = K_{N,-} \quad \text{and} \quad K_{E,-} = K_{N,+}, $$

as well as $K_{W,-} = K_{E,+}$ if $(\theta_W, \theta_E) \neq (+\pi/2, -\pi/2)$.

The various objects introduced in section 2 for a single half-plane will simply be indexed by $J \in \{W, N, E\}$: the transverse wavenumbers $\beta_J^{\pm}(\lambda)$, the reflection and transmission coefficients $R_J^{\pm}(\lambda)$ and $T_J^{\pm}(\lambda)$, the generalized eigenfunctions $\Psi_J^{\pm}(\lambda, x_J)$, the weight functions $\rho_J^{\pm}(\lambda)$ and the generalized Fourier transforms $\mathcal{F}_J$ (see (2.5)-(2.7), (2.10) and Theorem 2.3).

Recall that we have replaced the initial definition (1.1) of $H$ by (2.14) so that the x-axis $\Sigma := \mathbb{R} \times \{0\}$ is now contained in $H$. The same goes for each half-plane $H_J$ for $J \in \{W, N, E\}$, whose local coordinates are still related to the global ones $(x, y)$ by (1.3). We denote by $\Sigma_J$ the $x_J$-axis of $H_J$, which is contained in $H_J$. Note that the three points $(a_W, b_W)$, $(0, 0)$ and $(a_E, b_E)$ are located respectively at the intersection of $\Sigma_J$ with the interface of the two-layered medium filling $H_J$.

We prove now Theorem 1.1 in the context described above. For the sake of clarity, we begin by the case of the right-angle configuration (see Figure 5, left).

**3.1. Right-angle configuration.** We follow the same approach as in the second step of subsection 1.2. Recall that in the right-angle configuration, the relations between the local and global coordinate systems are given by (1.10).

Suppose that $u \in L^2(\Omega) = H_W \cup H_N \cup H_E$. Hence, we can use the half-plane representation of Proposition 2.5 for the restriction of $u$ in each of the three half-planes. For $J \in \{W, N, E\}$, we denote by $\varphi_J$ the trace of $u$ on $\Sigma_J$, considered as a function of the local coordinate $x_J$, and by $(\tilde{\varphi}_J^-, \tilde{\varphi}_J^+) := \mathcal{F}_J \varphi_J$ its generalized Fourier transform with respect to $x_J$.

In order to prove that $u = 0$, the key argument consists in proving that $\tilde{\varphi}_N^{\pm}(\lambda)$ extends to an analytic function of $\lambda$ in the domain

$$ (3.2) \quad \mathbb{D}_N := \mathbb{C} \setminus (-\infty, -\min(K_{N,-}^2, K_{N,+}^2)] $$
introduced in Proposition 2.1. As (2.17) tells us that $\hat{\varphi}_N^\pm(\lambda)$ vanishes on the interval $(-K_{N,\pm}^2, 0)$, analyticity implies that it vanishes also on $(0, +\infty)$, hence $\hat{\varphi}_N^\pm = 0$. The half-plane representation (2.16) in $H_N$ then tells us that $u$ vanishes in $H_N$, so finally also in the whole domain $\Omega$ by virtue of the unique continuation principle.

It remains to prove that $\hat{\varphi}_N^\pm(\lambda)$ has an analytic continuation in $D_N$. We focus on the case of $\hat{\varphi}_N^+(\lambda)$. The case of $\hat{\varphi}_N^-(\lambda)$ can be dealt with in the same way. We start from the definition (2.11) of $\hat{\varphi}_N^+(\lambda)$ (recall that $x_N = x$):

$$ \forall \lambda \in (-K_{N,+}^2, +\infty), \quad \hat{\varphi}_N^+(\lambda) := \int_{\mathbb{R}} \varphi_N(x) \overline{\Psi_N^+(\lambda, x)} dx. $$

According to Remark 2.4, this integral is well defined (in the Lebesgue sense) if $\varphi_N \in L^1(\mathbb{R})$, which is not obvious. This actually follows Corollary 2.7 applied for both half-planes $H_W$ and $H_E$, which ensures that $u|_{\Sigma_N \cap H_W}$ and $u|_{\Sigma_N \cap H_E}$ are integrable, that is, $\varphi_N|_{(-\infty, a_W)} \in L^1(-\infty, a_W)$ and $\varphi_N|_{(a_E, +\infty)} \in L^1(a_E, +\infty)$. We can then split the above integral in three parts as follows:

$$ \begin{aligned} (3.3) \quad \hat{\varphi}_N^+(\lambda) = & \underbrace{\int_{-\infty}^{a_W} \varphi_N(x) \overline{\Psi_N^+(\lambda, x)} dx}_{=} \underbrace{\int_{a_W}^{a_E} \varphi_N(x) \overline{\Psi_N^+(\lambda, x)} dx}_{=} \underbrace{\int_{a_E}^{+\infty} \varphi_N(x) \overline{\Psi_N^+(\lambda, x)} dx}_{=} \underbrace{\hat{\varphi}_{N,E}^+(\lambda)}_{=: \hat{\varphi}_{N,0}^+(\lambda)} \end{aligned} $$

First consider the second integral $\hat{\varphi}_{N,0}^+(\lambda)$. Recall that Proposition 2.1 tells us that for any $x \in \mathbb{R}$, the function $(-K_{N,+}^2, +\infty) \ni \lambda \to \Psi_N^+(\lambda, x)$ has an analytic continuation in $D_N$ and $(\lambda, x) \to \Psi_N^+(\lambda, x)$ is bounded in any compact set of $D_N \times \mathbb{R}$. Then, the function $(-K_{N,+}^2, +\infty) \ni \lambda \to \Psi_N^+(\lambda, x)$ also has an analytic continuation in $D_N$, simply defined by $\overline{\Psi_N^+(\lambda, x)}$, and $(\lambda, x) \to \overline{\Psi_N^+(\lambda, x)}$ is bounded in any compact set of $D_N \times \mathbb{R}$. Using Morera's theorem, we infer that $\hat{\varphi}_{N,0}^+(\lambda)$ has an analytic continuation in $D_N$.

These arguments cannot apply for $\hat{\varphi}_{N,W}^+(\lambda)$ nor $\hat{\varphi}_{N,E}^+(\lambda)$, since $\overline{\Psi_N^+(\lambda, x)}$ is exponentially increasing as $x \to +\infty$ or $x \to -\infty$ for non-real $\lambda$. The idea is to use the half-plane representation in $H_W$ and $H_E$ respectively. Consider the case of $\hat{\varphi}_{N,W}^+(\lambda)$. Denoting $u_W$ the restriction of $u$ to $H_W$, considered as a function of the local coordinates $(x_W, Y_W)$, the half-plane representation (2.16) writes as

$$ (3.4) \quad u_W(x_W, Y_W) = \sum_{\pm} \int_{\mathbb{R}^+} \hat{\varphi}_W^{\pm}(\mu) \Psi_W^{\pm}(\mu, x_W) e^{-\sqrt{\mu} Y_W} \rho_W^{\pm}(\mu) d\mu, $$

for all $(x_W, Y_W) \in \mathbb{R} \times [0, +\infty)$. According to (1.10), for all $x \in (-\infty, a_W)$, we have $\varphi_N(x) = u(x, 0) = u_W(-b_W, -x + a_W)$. Hence, for $\lambda \in (-K_{N,+}^2, +\infty)$,

$$ \hat{\varphi}_{N,W}^{+}(\lambda) = \int_{-\infty}^{a_W} \left( \sum_{\pm} \int_{\mathbb{R}^{+}} \hat{\varphi}_{W}^{\pm}(\mu) \Psi_{W}^{\pm}(\mu, -b_{W}) e^{-\sqrt{\mu}(-x+a_{W})} \rho_{W}^{\pm}(\mu) d\mu \right) \overline{\Psi_{N}^{+}(\lambda, x)} dx. $$
In order to use Fubini's theorem, note that thanks to (2.8), we have

$$ 
\begin{aligned}
& \int_{-\infty}^{a_w} \int_{\mathbb{R}^+} \left| \hat{\varphi}_W^{\pm}(\mu) \Psi_W^{\pm}(\mu, -b_W) e^{-\sqrt{\mu}(-x+a_W)} \overline{\Psi_N^+(\lambda, x)} \right| \rho_W^{\pm}(\mu) \, d\mu \, dx \\
& \le 4 \int_{\mathbb{R}^+} \left( \int_{-\infty}^{a_w} e^{-\sqrt{\mu}(-x+a_W)} \, dx \right) \left| \hat{\varphi}_W^{\pm}(\mu) \right| \rho_W^{\pm}(\mu) \, d\mu \\
& = 4 \int_{\mathbb{R}^+} \frac{|\hat{\varphi}_W^{\pm}(\mu)|}{\sqrt{\mu}} \rho_W^{\pm}(\mu) \, d\mu,
\end{aligned}
$$

which is bounded, according to (2.17) with $s = -1/2$. Switching the integrals in the above expression of $\hat{\varphi}_{N,W}^{\pm}(\lambda)$ and using the expression (2.5) of $\Psi_N^+(\lambda, x)$ then yields

$$ \hat{\varphi}_{N,W}^{+}(\lambda) = \sum_{\pm} \int_{\mathbb{R}^{+}} T_{N}^{+}(\lambda) \left( \int_{-\infty}^{a_{W}} e^{-\sqrt{\mu}(-x+a_{W})+i\beta_{N}^{-}(\lambda)x} dx \right) \hat{\varphi}_{W}^{\pm}(\mu) \Psi_{W}^{\pm}(\mu, -b_{W}) \rho_{W}^{\pm}(\mu) d\mu, $$

since $T_N^+(\lambda)$ and $\beta_N^-(\lambda)$ are real on $(-K_{N,+}^2, +\infty)$. Computing the integral between parentheses, we finally obtain

$$ (3.5) \qquad \hat{\varphi}_{N,W}^{+}(\lambda) = T_{N}^{+}(\lambda) e^{i\beta_{N}^{-}(\lambda)} a_{W} \sum_{\pm} \int_{\mathbb{R}^{+}} \frac{\hat{\varphi}_{W}^{\pm}(\mu) \Psi_{W}^{\pm}(\mu, -b_{W})}{\sqrt{\mu} + i\beta_{N}^{-}(\lambda)} \rho_{W}^{\pm}(\mu) \, d\mu. $$

Now, we can use again Morera's theorem to conclude that $\hat{\varphi}_{N,W}^{+}(\lambda)$ has an analytic continuation in $\mathbb{D}_N$, since $\sqrt{\mu} + i\beta_N^-(\lambda)$ cannot vanish for $\mu > 0$ and $\lambda \in \mathbb{D}_N$.

We can proceed in a similar way for $\hat{\varphi}_{N,E}^{+}(\lambda)$. Using the half-plane representation in $H_E$, we obtain instead of (3.5)

$$ \hat{\varphi}_{N,E}^{+}(\lambda) = \sum_{\pm} \int_{\mathbb{R}^{+}} \left( \frac{e^{i\beta_{N}^{+}(\lambda)} a_{E}}{\sqrt{\mu} - i\beta_{N}^{+}(\lambda)} + R_{N}^{+}(\lambda) \frac{e^{-i\beta_{N}^{+}(\lambda)} a_{E}}{\sqrt{\mu} + i\beta_{N}^{+}(\lambda)} \right) \hat{\varphi}_{E}^{\pm}(\mu) \Psi_{E}^{\pm}(\mu, b_{E}) \rho_{E}^{\pm}(\mu) \, d\mu, $$

which also extends to an analytic function of $\lambda$ in $\mathbb{D}_N$ thanks to Proposition 2.1. This completes the proof.

**3.2. General angle configuration.** We only detail here the items of the proof which differ from the right-angle configuration presented above. Indeed, the ideas are exactly the same, but the calculations are more complicated, since the relations between the local and global coordinate systems are given now by the general form (1.3) with $(\theta_W, \theta_E) \neq (+\pi/2, -\pi/2)$ (see Figure 5, right). These relations show in particular that the intersection points of $\Sigma_W \cap \Sigma_N$ and $\Sigma_E \cap \Sigma_N$ are located respectively at $(a_{NW}, 0)$ and $(a_{NE}, 0)$ where

$$ a_{NW} := a_W - b_W \cot \theta_W \quad \text{and} \quad a_{NE} := a_E - b_E \cot \theta_E. $$

Therefore the splitting (3.3) of $\hat{\varphi}_N^+(\lambda)$ becomes here

$$ \hat{\varphi}_N^+(\lambda) = \underbrace{\int_{-\infty}^{a_{NW}} \varphi_N(x) \overline{\Psi_N^+(\lambda, x)} dx}_{=: \hat{\varphi}_{N,W}^+(\lambda)} + \underbrace{\int_{a_{NW}}^{a_{NE}} \varphi_N(x) \overline{\Psi_N^+(\lambda, x)} dx}_{=: \hat{\varphi}_{N,0}^+(\lambda)} + \underbrace{\int_{a_{NE}}^{+\infty} \varphi_N(x) \overline{\Psi_N^+(\lambda, x)} dx}_{=: \hat{\varphi}_{N,E}^+(\lambda)} $$
As previously, the second integral $\hat{\varphi}_{N,0}^{+}(\lambda)$ has an analytic continuation in $\mathbb{D}_N$. But as we will see, this is no longer true for the two other integrals. We focus on $\hat{\varphi}_{N,W}^{+}(\lambda)$. The half-plane representation (3.4) is still valid, but (1.3) shows now that

$$ \forall x \in (-\infty, a_{NW}), \quad \varphi_N(x) = u(x, 0) = u_W(x, Y_W(x)) \quad \text{where} \\ X_W(x) := (x - a_W) \cos \theta_W - b_W \sin \theta_W \quad \text{and} \quad Y_W(x) := (a_{NW} - x) \sin \theta_W. $$

Therefore (3.4) yields

$$ \hat{\varphi}_{N,W}^{+}(\lambda) = \int_{-\infty}^{a_{NW}} \left( \sum_{\pm} \int_{\mathbb{R}^{+}} \hat{\varphi}_{W}^{\pm}(\mu) \Psi_{W}^{\pm}(\mu, x_{W}(x)) e^{-\sqrt{\mu} Y_{W}(x)} \rho_{W}^{\pm}(\mu) d\mu \right) \overline{\Psi_{N}^{+}(\lambda, x)} dx. $$

From Fubini's theorem, we obtain

$$ (3.6) \qquad \hat{\varphi}_{N,W}^{+}(\lambda) = \sum_{\pm} \int_{\mathbb{R}^{+}} A^{\pm}(\lambda, \mu) \hat{\varphi}_{W}^{\pm}(\mu) \rho_{W}^{\pm}(\mu) d\mu \quad \text{where} \\ A^{\pm}(\lambda, \mu) := \int_{-\infty}^{a_{NW}} e^{-\sqrt{\mu} Y_{W}(x)} \Psi_{W}^{\pm}(\mu, x_{W}(x)) \overline{\Psi_{N}^{+}(\lambda, x)} dx. $$

Using the expression (2.5) of $\Psi_W^\pm$ and $\Psi_N^+$ (note that $x_W(x) > 0$ for all $x \in (-\infty, a_{NW})$), as well as the fact that $\beta_W^+(\mu) = \beta_N^-(\mu)$ (see (3.1)), the calculation of the latter integral yields

$$ \begin{aligned} A^{-}(\lambda, \mu) &= \frac{T_{W}^{-}(\mu) e^{+i\beta_{N}^{-}(\mu) x_{W}(a_{NW})}}{D^{+}(\lambda, \mu)} T_{N}^{+}(\lambda) e^{+i\beta_{N}^{-}(\lambda) a_{NW}} \\ A^{+}(\lambda, \mu) &= \left( \frac{e^{-i\beta_{N}^{-}(\mu) x_{W}(a_{NW})}}{D^{-}(\lambda, \mu)} + \frac{R_{W}^{+}(\mu) e^{+i\beta_{N}^{-}(\mu) x_{W}(a_{NW})}}{D^{+}(\lambda, \mu)} \right) T_{N}^{+}(\lambda) e^{+i\beta_{N}^{-}(\lambda) a_{NW}} \end{aligned} $$

where we have denoted

$$ D^{\pm}(\lambda, \mu) := \sqrt{\mu} \sin \theta_W \pm i \beta_N^-(\mu) \cos \theta_W + i \beta_N^-(\lambda). $$

For a fixed $\mu > 0$, the function $A^\pm(\lambda, \mu)$ has an analytic continuation in any complex subdomain of $\mathbb{D}_N$ (see (3.2)) in which the denominators $D^-(\lambda, \mu)$ and $D^+(\lambda, \mu)$ do not vanish. On the one hand, $D^-(\lambda, \mu)$ cannot vanish, for

$$ \forall \lambda \in \mathbb{D}_N, \forall \mu > 0, \quad \mathrm{Im}(D^-(\lambda, \mu)) = -\beta_N^-(\mu) \cos \theta_W + \mathrm{Re}(\beta_N^-(\lambda)) > 0, $$

since $\cos \theta_W \le 0$ and $\mathrm{Re}(\beta_N^-(\lambda)) > 0$ for all $\lambda \in \mathbb{D}_N$ (according to our choice of the complex square root, see the proof of Proposition 2.1). On the other hand,

$$ \begin{aligned} D^{+}(\lambda, \mu) = 0 &\iff \sqrt{\lambda + K_{N,-}^2} = -\sqrt{\mu + K_{N,-}^2} \cos \theta_W + i\sqrt{\mu} \sin \theta_W \\ &\iff \lambda = (\mu \cos(2\theta_W) - K_{N,-}^2 \sin^2 \theta_W) - i \left(\sqrt{\mu} \sqrt{\mu + K_{N,-}^2} \sin(2\theta_W)\right). \end{aligned} $$

The latter equation defines a curve $\Lambda_{N,W}$ parameterized by $\mu$ in the upper complex half-plane $\mathrm{Im}\,\lambda > 0$ (since $\sin(2\theta_W) < 0$, see (1.5)), which reaches the real axis at $\lambda_{N,W} := -K_{N,-}^2 \sin^2 \theta_W$ when $\mu = 0$. As a consequence, going back to (3.6), we deduce from Morera's theorem that $\hat{\varphi}_{N,W}^{+}(\lambda)$ has an analytic continuation in $\mathbb{D}_N \setminus \Lambda_{N,W}$.
<figure><img src="image_9.png" /><figcaption>FIG. 6. Relevant domain of analyticity of $\hat{\varphi}_N^+$ (gray area) for $K_{N,+} = 10$, $K_{N,-} = 9$, $\theta_E = -5\pi/6$, $\theta_W = 2\pi/3$.</figcaption></figure>

We proceed in the same way to deal with $\hat{\varphi}_{N,E}^+(\lambda)$. The only differences in the computations is that we use now the expression of $\Psi_N^+(\lambda, x)$ for $x > 0$ (see (2.5)) and $\Psi_E^\pm$ instead of $\Psi_W^\pm$. We are then faced with the possible cancellation of four denominators, which leads us to solve the following four equations:

$$ \sqrt{\mu} \sin \theta_E \pm i \beta_N^+(\mu) \cos \theta_E \pm i \beta_N^+(\lambda) = 0, $$

where both $\pm$ are independent. As above, only two of them have a solution, which yields

$$ \lambda = (\mu \cos(2\theta_E) - K_{N,+}^2 \sin^2 \theta_E) \pm i \left( \sqrt{\mu} \sqrt{\mu + K_{N,+}^2} \sin(2\theta_E) \right). $$

Instead of one curve, we then obtain two curves $\Lambda_{N,E}^\pm$ which are complex-conjugate to each other and reach the real axis at $\lambda_{N,E} := -K_{N,+}^2 \sin^2 \theta_E$. We conclude again that $\hat{\varphi}_{N,E}^+(\lambda)$ has an analytic continuation in $\mathbb{D}_N \setminus (\Lambda_{N,E}^- \cup \Lambda_{N,E}^+)$.

To sum up, $\hat{\varphi}_N^+(\lambda)$ has an analytic continuation in $\mathbb{D}_N \setminus (\Lambda_{N,W} \cup \Lambda_{N,E}^- \cup \Lambda_{N,E}^+)$. The curves are represented in Figure 6. We are interested in the analyticity of $\hat{\varphi}_N^+(\lambda)$ in the gray area: as $\hat{\varphi}_N^+(\lambda)$ vanishes on the interval $(\max(\lambda_{N,W}, \lambda_{N,E}), 0)$, it implies that $\hat{\varphi}_N^+(\lambda)$ also vanishes on $(0, +\infty)$.

Finally, the same arguments apply to $\hat{\varphi}_N^-(\lambda)$, so that Theorem 1.1 is now proved for a junction of two-layered media in the general angle configuration.

**Remark 3.1.** Notice that when $(\theta_W, \theta_E)$ tends to $(+\pi/2, -\pi/2)$, the union of the curves represented in Figure 6 formally tends to the half-line $(-\infty, -\min(K_{N,-}^2, K_{N,+}^2)]$ on the real axis. We then recover the domain of analyticity $\mathbb{D}_N$ of the right-angle configuration (see (3.2)).

**4. Extension to the general stratification case.** We consider now the junction of general stratified media as described in subsection 1.1. We follow the same steps as previously, first focusing on a half-plane, explaining how the generalized Fourier transform is written in that case, detailing the properties of the half-plane representation and then using them to derive Theorem 1.1.

**4.1. Generalized Fourier transform.** We consider again the unbounded self-adjoint operator A defined by (2.2) where K now satisfies the general assumptions of subsection 1.1: it is a bounded function which is constant outside a bounded interval,
more precisely $\kappa(X) = \kappa_{\pm}$ if $\pm X > \pm X^{\pm}$. A main difference with the two-layered medium is that, in addition to the generalized eigenfunctions appearing previously, the operator may now have $L^2$ eigenfunctions. This means that there may exist pairs $(\lambda, \Psi) \in \mathbb{R} \times H^2(\mathbb{R}) \setminus \{0\}$ such that

$$ (4.1) \qquad -\partial_x^2 \Psi - \kappa^2 \Psi = \lambda \Psi \quad \text{in } \mathbb{R}. $$

Using that $\Psi$ has to be decreasing where $\kappa$ is constant, we infer that there can exist such eigenpairs only for $\lambda < -\max(\kappa_{-}^2, \kappa_{+}^2)$ otherwise non-trivial solutions to (4.1) cannot be $L^2$. As this equation is one dimensional, all eigenvalues are of multiplicity one. Writing

$$ \kappa_M^2 := \ess\sup_{x \in (x^-, x^+)} (\kappa^2(x)), $$

and denoting $\Lambda^P$ the possibly empty point spectrum of $A$, multiplying (4.1) by $\overline{\Psi}$ and integrating by parts, it follows that

$$ (4.2) \qquad \Lambda^P \subset [-\kappa_M^2, -\max(\kappa_-^2, \kappa_+^2)] \subset \mathbb{R}^- $$

Then, an obvious necessary condition for the existence of eigenvalues is

$$ \kappa_M^2 > \max(\kappa_-^2, \kappa_+^2). $$

We can exhibit sufficient conditions ensuring the existence of at least one eigenpair. For instance, if $\kappa_-$, $\kappa_+$, a sufficient condition [16, Corollary 4.5.2] is that $\kappa$ satisfies

$$ \int_{\mathbb{R}} (\kappa^2(x) - \kappa_+^2) dx > 0. $$

Using the min-max principle, it is also possible to derive conditions on $\kappa$ ensuring the existence of any given number of eigenvalues. For instance, if $\kappa = \kappa_M$ on an interval of size $l$ and there exists a non-zero integer $n$ such that

$$ \kappa_M^2 - \max(\kappa_-^2, \kappa_+^2) > \frac{n^2 \pi^2}{l^2}, $$

then there exist at least $n$ eigenvalues.

Finally, it is proven in [16, Theorem 4.7.1] that there is at most a finite number of eigenvalues.

In what follows, for the sake of conciseness, we consider the situation where the point spectrum is not empty. The empty case then follows easily since we will see in Proposition 4.3 that the existence of eigenfunctions does not impact the core of the proof. We denote by $N \ge 1$ the number of eigenvalues $(\lambda^n)_{n=1}^N$ and for all $n \in [1, N]$, by $\Psi^n: \mathbb{R} \to \mathbb{C}$ an associated eigenfunction normalized in $L^2(\mathbb{R})$ associated to $\lambda^n$.

As previously, we also consider generalized eigenfunctions (which are bounded but not $L^2$ solutions to (4.1)) for $\lambda > -\max(\kappa_-^2, \kappa_+^2)$. The dimension of the space of bounded solutions is once again 1 if $-\max(\kappa_-^2, \kappa_+^2) \le \lambda < -\min(\kappa_-^2, \kappa_+^2)$ and 2 if $\lambda \ge -\min(\kappa_-^2, \kappa_+^2)$. The generalized eigenfunctions $\Psi^{\pm}: \Lambda^{\pm} \times \mathbb{R} \to \mathbb{C}$ are now more complicated to describe. They are uniquely defined (see Appendix A) as solutions to (4.1) that have the following form outside of the interval $(x^-, x^+): \forall (\lambda, x) \in \Lambda^{\pm} \times \mathbb{R}$,

$$ (4.3) \qquad \Psi^{\pm}(\lambda, x) := \begin{cases} e^{\mp i \beta^{\pm}(\lambda) x} + R^{\pm}(\lambda) e^{\pm i \beta^{\pm}(\lambda) x} & \text{if } \pm x > \pm x^{\pm}, \\ T^{\pm}(\lambda) e^{\mp i \beta^{\mp}(\lambda) x} & \text{if } \mp x > \mp x^{\mp}, \end{cases} $$
for some complex coefficients $T^{\pm}(\lambda)$, $R^{\pm}(\lambda)$. If $\kappa$ is piecewise constant, we can still retrieve explicit expressions for the generalized eigenfunctions in $(x^{-}, x^{+})$ and for the associated coefficients $T^{\pm}(\lambda)$ and $R^{\pm}(\lambda)$ [12, Section 2.3], but in general, it is no longer possible.

We are now ready to give the diagonalization theorem for the general stratification case. A proof can be found in [12, Section 2]. The spectral space adapted to this situation writes as follows:

$$ \hat{\mathcal{H}} := L^2(\Lambda^+, \rho^+) \times L^2(\Lambda^-, \rho^-) \times \mathbb{C}^N, $$

and is endowed with the norm

$$ \forall \hat{\varphi} := (\hat{\varphi}^{+}, \hat{\varphi}^{-}, (\hat{\varphi}^{n})_{n=1}^{N}) \in \hat{\mathcal{H}}, \quad \|\hat{\varphi}\|_{\hat{\mathcal{H}}}^{2} := \|\hat{\varphi}^{+}\|_{L^{2}(\Lambda^{+}, \rho^{+})}^{2} + \|\hat{\varphi}^{-}\|_{L^{2}(\Lambda^{-}, \rho^{-})}^{2} + \sum_{n=1}^{N} |\hat{\varphi}^{n}|^{2}, $$

where we used the same notations as (2.4) and (2.10) since the continuous spectrum is unchanged.

**THEOREM 4.1.** The generalized Fourier transform $\mathcal{F}$ defined for all $\varphi \in L^1(\mathbb{R}) \cap L^2(\mathbb{R})$ by

$$ \mathcal{F}\varphi := (\mathcal{F}^{-}\varphi, \mathcal{F}^{+}\varphi, (\mathcal{F}^{n}\varphi)_{n=1}^{N}) \text{ with, for } \lambda \in \Lambda^{\pm} \text{ and } n \in [1, N], $$

$$ \mathcal{F}^{\pm}\varphi(\lambda) := \int_{\mathbb{R}} \varphi(x) \overline{\Psi^{\pm}(\lambda, x)} \, dx \quad \text{and} \quad \mathcal{F}^{n}\varphi := \int_{\mathbb{R}} \varphi(x) \overline{\Psi^{n}(x)} \, dx, $$

extends by density to a unitary operator from $L^2(\mathbb{R})$ to $\hat{\mathcal{H}}$. It diagonalizes $A$ on $L^2(\mathbb{R})$, in the sense that

$$ \forall \varphi \in H^2(\mathbb{R}), \quad A\varphi = \mathcal{F}^{-1}\lambda \mathcal{F}\varphi, $$

where the notation $\lambda$ stands for the operator of multiplication by $\lambda$ in $\hat{\mathcal{H}}$. Moreover, the inverse transform $\mathcal{F}^{-1}$ writes explicitly as follows: for all $(\hat{\varphi}^{-}, \hat{\varphi}^{+}, (\hat{\varphi}^{n})_{n=1}^{N}) \in \hat{\mathcal{H}}$ such that $\hat{\varphi}^{\pm} \in L^1(\Lambda^{\pm}, \rho^{\pm})$, for all $x \in \mathbb{R}$,

$$ \mathcal{F}^{-1}[(\hat{\varphi}^{-}, \hat{\varphi}^{+}, (\hat{\varphi}^{n})_{n=1}^{N})](x) = \sum_{n=1}^{N} \Psi^{n}(x)\hat{\varphi}^{n} + \sum_{\pm} \int_{-\kappa_{\pm}^{2}}^{+\infty} \Psi^{\pm}(\lambda, x)\hat{\varphi}^{\pm}(\lambda)\rho^{\pm}(\lambda)d\lambda. $$

**Remark 4.2.** By construction [12, Lemma 2.4.8], the $\Psi^{\pm}$'s are uniformly bounded on $\Lambda^{\pm} \times \mathbb{R}$ so that the formulas make sense for $L^1$ functions (as mentioned in Remark 2.4).

**4.2. Half-space representation.** Let us point out that compared to the two-layered situation, two difficulties arise when generalizing the proof of Theorem 1.1. Firstly, we now have to handle the $L^2$ eigenfunctions in addition to the generalized ones. Secondly, since we do not have explicit expressions of the generalized eigenfunctions anymore, it is more technical to derive the necessary analyticity properties.

We start by handling the first difficulty and we show that since all the eigenvalues are negative, the eigenfunctions do not contribute to the half-plane representation of $L^2$ solutions of the Helmholtz equation. More precisely, the following proposition states that the half-plane representation of an $L^2$ solution to (1.6) still writes as in (2.16) without the components associated to the eigenfunctions $\Psi^n$.
**PROPOSITION 4.3.** *Considering H a half-plane as defined in (2.14), suppose that $u \in L^2(H)$ satisfies*

$$ (4.4) \qquad -\Delta u - \kappa^2 u = 0 \text{ in } H $$

in the distributional sense where $\kappa$ is defined as in subsection 4.1.

Then, writing $(\hat{\varphi}^{-}, \hat{\varphi}^{+}, (\hat{\varphi}^{n})_{n=1}^{N}) := \mathcal{F} u(\cdot, 0)$, we have

$$ (4.5) \qquad \forall n \in [1, N], \quad \hat{\varphi}^n = 0, $$

and,

$$ (4.6) \qquad \forall (x, y) \in \mathbb{R} \times [0, +\infty), \quad u(x, y) = \sum_{\pm} \int_{\mathbb{R}^{+}} \hat{\varphi}^{\pm}(\lambda) \Psi^{\pm}(\lambda, x) e^{-\sqrt{\lambda y}} \rho^{\pm}(\lambda) d\lambda, $$

with $\hat{\varphi}^{\pm}$ satisfying (2.17).

*Proof.* Following the same reasoning as in the proof of Proposition 2.5, we can define a family of $L^2$ functions $\hat{u}^n(Y) := \mathcal{F}^n u(\cdot, Y)$ for almost every $Y \in (-\varepsilon, +\infty)$ and for $n \in [1, N]$. Then applying the partial generalized Fourier transform in the x-direction to the Helmholtz equation (4.4), we have that for all $n \in [1, N]$,

$$ -\partial_y^2 \hat{u}^n + \lambda^n \hat{u}^n = 0 \text{ on } (-\varepsilon, +\infty) $$

in the distributional sense. Then, there exist constants $\hat{A}^n, \hat{B}^n \in \mathbb{C}$ such that

$$ \hat{u}^n(Y) = \hat{A}^n e^{-(Y+\varepsilon)\sqrt{\lambda^n}} + \hat{B}^n e^{+(Y+\varepsilon)\sqrt{\lambda^n}}. $$

Since $\hat{u}^n \in L^2(-\varepsilon, +\infty)$ and since all the eigenvalues are negative by virtue of (4.2), we have that $\hat{A}^n = 0$ and $\hat{B}^n = 0$.

The properties on $\hat{\varphi}^{\pm}$ are then derived as in Proposition 2.5. $\square$

The conclusion of Corollary 2.7 still holds for the hypotheses of Proposition 4.3 by virtue of Remark 4.2.

To proceed as in the proofs detailed in section 3, we need an additional analyticity result on the general eigenfunctions for which we cannot rely on explicit expressions anymore. Recalling $\mathbb{D}$ defined in (2.9), we have the following.

**PROPOSITION 4.4.** *The mappings $\lambda \to R^{\pm}(\lambda)$ and $\lambda \to T^{\pm}(\lambda)$ corresponding to the coefficients of the generalized eigenfunctions outside $(x^{-}, x^{+})$ as defined in (4.3) have meromorphic continuations in $\mathbb{D}_{N}$. Moreover, for any $x \in \mathbb{R}$, the mappings to the generalized eigenfunctions $\lambda \to \Psi^{\pm}(\lambda, x)$ also have meromorphic continuations in $\mathbb{D}_{N}$ with poles independent from $x$. Finally, the meromorphic continuations of $\Psi^{\pm}$ are bounded in any compact set of $\mathbb{D} \setminus P \times \mathbb{R}$ where $P$ denotes their set of poles.*

*Proof.* The idea is to prove that the $\Psi^{\pm}(\cdot, x)$'s are meromorphic for all $x \in [x^{-}, x^{+}]$ which directly yields the same property for the coefficients $R^{\pm}(\lambda)$ and $T^{\pm}(\lambda)$ and then for any $x \in \mathbb{R}$. The proof is given in Appendix A. $\square$

**4.3. Proof of Theorem 1.1.** We use the same notations as in the previous proofs. We can once again derive the following expression:

$$ \hat{\varphi}_{N}^{\pm}(\lambda) = \underbrace{\int_{-\infty}^{a_{\mathrm{NW}}} \varphi_{N}(x) \overline{\Psi_{N}^{+}(\lambda, x)} dx}_{=: \hat{\varphi}_{N,W}^{+}(\lambda)} + \underbrace{\int_{a_{\mathrm{NW}}}^{a_{\mathrm{NE}}} \varphi_{N}(x) \overline{\Psi_{N}^{+}(\lambda, x)} dx}_{=: \hat{\varphi}_{N,0}^{+}(\lambda)} + \underbrace{\int_{a_{\mathrm{NE}}}^{+\infty} \varphi_{N}(x) \overline{\Psi_{N}^{+}(\lambda, x)} dx}_{=: \hat{\varphi}_{N,E}^{+}(\lambda)} $$
By virtue of the definitions of the three stratified half-planes detailed in the introduction, we necessarily have $[x_N^-, x_N^+] \subset [a_{NW}, a_{NE}]$. Then, $\Psi_N^+$ can be expressed semi-explicitly as in (4.3) in both $\hat{\varphi}_{N,W}^+$ and $\hat{\varphi}_{N,E}^+$. The fact that there is no explicit expression of $\Psi_N^+$ in $(a_{NW}, a_{NE})$ does not matter. Indeed, since $\hat{\varphi}_{N,0}^+$ is defined by an integral on a bounded interval, by virtue of Proposition 4.4, we can use Morera's theorem as in section 3 to show that this term is meromorphic. Moreover, we also have $x_W(x) > x_W^+$ on $(-\infty, a_{NW})$ and $x_E(x) < x_E^-$ on $(a_{NE}, +\infty)$ so that the $\Psi_J^+$'s, $J \in \{E, W\}$, both have semi-explicit expressions of the form (4.3) where they are considered. This allows us to detail $\hat{\varphi}_{N,W}^+$ and $\hat{\varphi}_{N,E}^+$ as previously. Then, proceeding as in section 3 and using Proposition 4.4, we get that $\hat{\varphi}_N(\lambda)$ has a meromorphic extension to $\mathbb{D}_N \setminus (\Lambda_{N,W} \cup \Lambda_{N,E}^- \cup \Lambda_{N,E}^+)$. Consequently, we can define a neighbourhood of $(-\eta, +\infty)$, for some $\eta > 0$, where $\hat{\varphi}_N(\lambda)$ is analytic and conclude as previously.

**5. Conclusion.** In this paper, we proved that in media that can be described as the union of stratified half-planes, there does not exist non-trivial square-integrable solutions to the two-dimensional Helmholtz equation. The tools used in the proof led to considering angles between the stratifications greater than $\pi/2$. In particular, the absence of trapped modes at the junction of waveguides when the angle between at least two branches is smaller than $\pi/2$ is an open question. We conjecture that the result still holds and that this limitation is purely technical, due to the use of separation of variables in half-planes.

Interestingly, our result still holds without imposing that $k^2$ be positive everywhere. Indeed, our proof only requires the positivity of $K_{N,\pm}^2$, which means that $K_{E,+}^2$ and $K_{W,-}^2$ can be negative. In other words, even if waves do not propagate in the southern quadrants of $H_E$ and $H_W$, the presence of propagating waves in $H_N$ is sufficient to prevent the existence of trapped modes. Let us point out that without information coming from the southern quadrants of $H_E$ and $H_W$, we cannot conclude. Indeed there exist non-trivial $L^2$ solutions to the Helmholtz equation in a half-plane, see [3] for the homogeneous case.

Note that we could have considered instead of the Helmholtz equation (1.6) a more general equation of the type

$$-\mathrm{div}(\mu\nabla u) - \rho u = 0.$$

Our result remains valid as soon as $\mu$ and $\rho$ satisfy the hypotheses given on $k^2$ in this paper, provided that $\mu$ is bounded from below by a positive constant. As mentioned above, the case of negative $\rho$ in the southern part of the domain is also allowed.

Let us now consider the Helmholtz equation in a three-dimensional domain $\Omega \times \mathbb{R}$ where $\Omega \subset \mathbb{R}^2$ and $k = k(x, y)$ independent from the third coordinate $z$ both satisfy the hypotheses of subsection 1.1. Our result means that there cannot exist so-called guided waves, i.e. non-trivial solutions of the form

$$u(x, y, z) = \hat{u}(x, y)e^{i\xi z}$$

with $\hat{u} \in L^2(\Omega)$ and $\xi \in \mathbb{R}$ if $\xi^2 < \min(K_{N,+}^2, K_{N,-}^2)$.

A natural extension would be to apply our approach to three-dimensional domains that are the union of half-spaces (five in a right-angle configuration and four otherwise), such that in each half-space, $k$ depends only on the transverse coordinates. However, this extension is far from straightforward as adapting the proof raises several difficulties. In particular, Corollary 2.7 does not directly extend to higher dimensions. Again, it is possible that this limitation is only due to the technique that we use. In-
deed, in the homogeneous case, this difficulty was circumvented by a tricky use of the two-dimensional result (see [3]). Unfortunately, this approach does not extend to the heterogeneous case we are interested in here.

It could be interesting to investigate links with the results concerning the so-called Landis' conjecture [7] which also applies to the Helmholtz equation with non-constant coefficients in exterior domains. This conjecture provides an upper bound for the rate of exponential decay at infinity of trapped modes.

**Appendix A.** In this appendix, we first prove the existence and uniqueness of the generalized eigenfunctions $\Psi^{\pm}(\lambda, \cdot)$, introduced as the bounded solutions to (4.1) satisfying (4.3) for $\lambda \in \Lambda^{\pm}$, and then we prove Proposition 4.4.

To prove that the $\Psi^{\pm}$'s are uniquely defined, we notice that they can be defined as the solutions to (4.1) which satisfy the following Robin conditions at $x^{\pm}$. For $\Psi^{+}$ we have,

$$ (A.1) \qquad \begin{aligned} \partial_x \Psi^+(\lambda, x^-) + i\beta^-(\lambda)\Psi^+(\lambda, x^-) &= 0, \\ \partial_x \Psi^+(\lambda, x^+) - i\beta^+(\lambda)\Psi^+(\lambda, x^+) &= -2i\beta^+(\lambda)e^{-i\beta^+(\lambda)x^+}, \end{aligned} $$

and for $\Psi^-$ (notice that only the right-hand side changes),

$$ (A.2) \qquad \begin{aligned} \partial_x \Psi^-(\lambda, x^-) + i\beta^-(\lambda)\Psi^-(\lambda, x^-) &= 2i\beta^-(\lambda)e^{i\beta^-(\lambda)x^-}, \\ \partial_x \Psi^-(\lambda, x^+) - i\beta^+(\lambda)\Psi^-(\lambda, x^+) &= 0. \end{aligned} $$

The idea is to show constructively that the generalized eigenfunctions can be defined as linear combinations of entire canonic solutions to an ODE and derive the different properties from this description.

Let us then introduce, for $\lambda$ in $\mathbb{C}$, $s(\lambda, \cdot)$ and $c(\lambda, \cdot)$ the two solutions in $C^1(\mathbb{R})$ to

$$ -\partial_x^2 u(\lambda, x) - (\kappa^2(x) + \lambda)u(\lambda, x) = 0 \quad \text{in } \mathbb{R} $$

in the distributional sense, such that

$$ c(\lambda, x^-) = 1 \quad \text{and} \quad \partial_x c(\lambda, x^-) = 0, $$

$$ s(\lambda, x^-) = 0 \quad \text{and} \quad \partial_x s(\lambda, x^-) = 1. $$

It is well-known that for all $x \in \mathbb{R}$, $s(\cdot, x)$ and $c(\cdot, x)$ are entire functions (see for instance [9, Section 4.2]).

Since the $\Psi^{\pm}$'s satisfy the same equation as $c$ and $s$, there exist complex coefficients $A^{\pm}(\lambda)$ and $B^{\pm}(\lambda)$ such that

$$ (A.3) \qquad \forall \lambda \in \Lambda^{\pm}, \forall x \in \mathbb{R}, \quad \Psi^{\pm}(\lambda, x) = A^{\pm}(\lambda)s(\lambda, x) + B^{\pm}(\lambda)c(\lambda, x). $$

The Robin conditions (A.1) and (A.2) on $\Psi^{\pm}$ then yield the following system

$$ \left( \begin{array}{cc} 1 & i\beta^-(\lambda) \\ \partial_x s(\lambda, x^+) - i\beta^+(\lambda)s(\lambda, x^+) & \partial_x c(\lambda, x^+) - i\beta^+(\lambda)c(\lambda, x^+) \end{array} \right) \left( \begin{array}{c} A^{\pm}(\lambda) \\ B^{\pm}(\lambda) \end{array} \right) = L^{\pm}(\lambda), $$

where $L^{\pm}(\lambda)$ only depend on $\beta^{\pm}(\lambda)$.

To prove that the $\Psi^{\pm}$'s are uniquely defined for $\lambda \in \Lambda^{\pm}$, it suffices to prove that the determinant of the above matrix never vanishes. We give the proof for $\lambda \in (-\min(\kappa_+^2, \kappa_-^2), +\infty)$ where both $\beta^{\pm}(\lambda)$ are real. It can easily be extended to $\lambda \in (-\max(\kappa_+^2, \kappa_-^2), -\min(\kappa_+^2, \kappa_-^2))$ where one of the $\beta^{\pm}(\lambda)$'s is imaginary.
Let us assume by contradiction that the determinant vanishes for some $\lambda \in (-\min(\kappa_+^2, \kappa_-^2), +\infty)$. This yields, as both $s(\lambda, \cdot)$ and $c(\lambda, \cdot)$ are real-valued for all $\lambda \in \mathbb{R}$,

$$ (A.4) \quad \forall \lambda \in (-\min(\kappa_+^2, \kappa_-^2), +\infty), \quad \left\{ \begin{array}{l} s(\lambda, x^+) = \frac{\partial_x c(\lambda, x^+)}{\beta^-(\lambda)\beta^+(\lambda)}, \\ \partial_x s(\lambda, x^+) = -\frac{\beta^+(\lambda)c(\lambda, x^+)}{\beta^-(\lambda)}. \end{array} \right. $$

Consider now the wronskian of $c$ and $s$ defined by

$$ \forall \lambda \in \mathbb{C}, \forall x \in \mathbb{R}, \quad W(\lambda, x) := c(\lambda, x)\partial_x s(\lambda, x) - s(\lambda, x)\partial_x c(\lambda, x). $$

It is a constant function of $x$. By definition of $c$ and $s$, $W(\lambda, x^-) = 1$, but (A.4) yields

$$ \forall \lambda \in (-\min(\kappa_+^2, \kappa_-^2), +\infty), \quad W(\lambda, x^+) = -\frac{\beta^+(\lambda)(c(\lambda, x^+))^2}{\beta^-(\lambda)} - \frac{(\partial_x c(\lambda, x^+))^2}{\beta^-(\lambda)\beta^+(\lambda)} < 0 $$

which gives a contradiction.

Now, we prove Proposition 4.4. Notice that the coefficients of the matrix and the right-hand term can analytically be extended to $\mathbb{D}$. Since the determinant can vanish on $\mathbb{D}$ outside of $\Lambda^\pm$, the solutions of the linear system $(A^\pm(\lambda), B^\pm(\lambda))$ have meromorphic continuations and have poles whenever the determinant vanishes. It follows that for every $x \in [x^-, x^+]$, the $\Psi^\pm(\cdot, x)$'s now have a meromorphic continuation to $\mathbb{D}$ with poles independent from $x$.

Finally, the fact that the $\Psi^\pm$'s are bounded on any compact of $\mathbb{D} \times \mathbb{R}$ where they are analytic follows directly from their decomposition (A.3) since the mappings $c$ and $s$ are bounded on any compact of $\mathbb{C} \times \mathbb{R}$ [9, Theorem 4.5].

**Remark A.1.** The existence, uniqueness and meromorphicity of the $\Psi^\pm$'s can also be derived from scattering theory using the analytic Fredholm theorem. The well-posedness of the problems defining $\Psi^\pm$ is a consequence of the energy conservation and the poles of the meromorphic extensions correspond to the so-called scattering resonances [6].

## REFERENCES

[1] Y. M. BEREZANSKY, Z. G. SHEFTEL, AND G. F. US, *Functional Analysis, II*, vol. 86 of Operator Theory: Advances and Applications, Birkhäuser Basel, 1996, https://doi.org/10.1007/978-3-0348-9024-3.

[2] A.-S. BONNET-BEN DHIA, G. DAKHIA, C. HAZARD, AND L. CHORFI, *Diffraction by a defect in an open waveguide: a mathematical analysis based on a modal radiation condition*, SIAM Journal on Applied Mathematics, 70 (2009), pp. 677–693, https://doi.org/10.1137/080740155.

[3] A.-S. BONNET-BEN DHIA, S. FLISS, C. HAZARD, AND A. TONNOIR, *A Rellich type theorem for the Helmholtz equation in a conical domain*, Comptes Rendus Mathematique, 354 (2016), pp. 27–32, https://doi.org/10.1016/j.crma.2015.10.015.

[4] A.-S. BONNET-BEN DHIA, B. GOURSAUD, AND C. HAZARD, *Mathematical analysis of the junction of two acoustic open waveguides*, SIAM Journal on Applied Mathematics, 71 (2011), pp. 2048–2071, https://doi.org/10.1137/100811374.

[5] S. DEBIÈVRE AND D. W. PRAVICA, *Spectral analysis for optical fibres and stratified fluids II: Absence of eigenvalues*, Communications in Partial Differential Equations, 17 (1992), pp. 69–97, https://doi.org/10.1080/03605309208820835.

[6] S. DYATLOV AND M. ZWORSKI, *Mathematical Theory of Scattering Resonances*, American Mathematical Society, 2019.
[7] A. FERNÁNDEZ-BERTOLIN, D. STAN, AND L. RONCAL, *Landis' conjecture: a survey*, 2024, https://arxiv.org/abs/2412.00788.

[8] C. HAZARD, *On the absence of trapped modes in locally perturbed open waveguides*, IMA Journal of Applied Mathematics, 80 (2015), pp. 1049–1062, https://doi.org/10.1093/imamat/hxu046.

[9] A. KIRSCH, *An Introduction to the Mathematical Theory of Inverse Problems*, Springer New York, 2011, https://doi.org/10.1007/978-1-4419-8474-6.

[10] D. KREJČÍRIK AND V. LOTOREICHIK, *Quasi-conical domains with embedded eigenvalues*, Bulletin of the London Mathematical Society, 56 (2024), pp. 2969–2981, https://doi.org/10.1112/blms.13113.

[11] C. LINTON AND P. MCIVER, *Embedded trapped modes in water waves and acoustics*, Wave Motion, 45 (2007), pp. 16–29, https://doi.org/10.1016/j.wavemoti.2007.04.009.

[12] J. OTT, *Halfspace Matching: a Domain Decomposition Method for Scattering by 2D Open Waveguides*, PhD thesis, Karlsruher Institut für Technologie (KIT), 2017, https://doi.org/10.5445/IR/1000070898.

[13] V. PAGNEUX, *Trapped modes and edge resonances in acoustics and elasticity*, in Dynamic Localization Phenomena in Elasticity, Acoustics and Electromagnetism, R. V. Craster and J. Kaplunov, eds., Springer Vienna, 2013, pp. 181–223, https://doi.org/10.1007/978-3-7091-1619-7_5.

[14] F. RELLICH, *Über das asymptotische verhalten der lösungen von $\delta u + \lambda u = 0$ in unendlichen gebieten*, Jahresber. Dtsch. Math.-Ver., 53 (1943), pp. 57–65, http://eudml.org/doc/146328.

[15] F. SANTOSA AND R. MAGNANINI, *Wave propagation in a 2-d optical waveguide*, SIAM Journal on Applied Mathematics, 61 (2001), pp. 1237–1252, https://doi.org/10.1137/S003613999360489.

[16] M. SCHECHTER, *Operator Methods in Quantum Mechanics*, Elsevier North Holland, Inc., 1981.

[17] E. TITCHMARSH, *Eigenfunction expansions associated with second order differential equations*, Oxford Clarendon Press, 1946.

[18] E. V. VESALAINEN, *Rellich type theorems for unbounded domains*, Inverse Problems and Imaging, 8 (2014), pp. 865–883, https://doi.org/10.3934/ipi.2014.8.865.

[19] R. WEDER, *Absence of eigenvalues of the acoustic propagator in deformed wave guides*, The Rocky Mountain Journal of Mathematics, (1988), pp. 495–503.

[20] R. WEDER, *Spectral and Scattering Theory for Wave Propagation in Perturbed Stratified Media*, vol. 87 of Applied Mathematical Sciences, Springer-Verlag New York, 1991.