GENERATING OPERATORS AND NORMAL
DERIVATIVES

TOSHIYUKI KOBAYASHI AND MICHAEL PEVZNER

# 1 INTRODUCTION

In [5], we initiated a new line of investigation on branching problems
by introducing the concept of **generating operator**, and found its
closed formula for the family of Rankin–Cohen brackets ([4, 5]).

As an application of generating operators, we proposed in [1] a trick
of transferring **discrete data** into **continuous data**, and illustrated
these ideas by an $SL_2$ example, in particular, showing how the *generating operator* $T$ of the Rankin–Cohen brackets {$R_\ell\}$$_{\ell\in\mathbb{N}}$ yields various
families of non-local intertwining operators with continuous parame-
ter in different geometric settings from the setting where the original
operators are defined.

In this note, we explore these principles in a higher-dimensional set-
ting focusing on a simpler case where symmetry breaking operators
(SBOs for short) are given by normal derivatives ([3]). In this case,
the generating operators reduce to the shift operator, see Example 2.1.
This reduction makes the entire framework more comprehensive as all
the technicalities boil down to elementary computations. More pre-
cisely, we shall consider the case ($GL_{n+1}, GL_n$) through the following
process.

Step 1. Description of SBOs given by normal derivatives;

Step 2. Changing real forms;

Step 3. Finding the generating operators;

Step 4. Trick from discrete to continuous.

<footnote>2024年度RIMS 共同研究(公開型)表現論と調和解析のひろがりExpansion
in Representation Theory and Harmonic Analysis, 2024年6月11日(火)–6月14日
(金) 京都大学数理解析研究所(世話人:田中雄一郎氏),講演録.</footnote>
In this way, we obtain singular non-local SBOs for branching problems of different real forms from the original setting where all SBOs are differential operators (localness theorem, [2]). While one could construct the singular non-local SBOs directly by using the general theory [6, 7], our approach shows that the concept of the generating operators bridges normal derivatives and a simple form of non-local SBOs.

In other words, the generating operators provide us a guiding principle to single out the case where the singularities of the distribution kernels of non-local symmetry breaking operators are normal crossing.

## 2 REMINDERS

**2.1. Generating operators.** Suppose that $\Gamma(X)$ and $\Gamma(Y)$ are the spaces of functions on $X$ and $Y$, respectively. Given a family of linear operators $R_\ell: \Gamma(X) \to \Gamma(Y)$, we consider a formal power series

$$ (2.1) \quad T \equiv T(\{R_\ell\}; t) := \sum_{\ell=0}^{\infty} \frac{R_\ell}{\ell!} t^\ell \in \operatorname{Hom}(\Gamma(X), \Gamma(Y)) \otimes \mathbb{C}[t]. $$

When $X = \{\text{point}\}$, $R_\ell$ is identified with an element of $\Gamma(Y)$, and such a formal power series is called a *generating function*, which has been particularly prominent in the classical study of orthogonal polynomials for $\Gamma(Y) = \mathbb{C}[y]$.

When $X = Y$, $\operatorname{Hom}(\Gamma(X), \Gamma(Y)) \simeq \operatorname{End}(\Gamma(X))$ has a ring structure, and one may take $R_\ell$ to be the $\ell$-th power of a *single* operator $R$ on $X$. In this case, the operator $T$ in (2.1) may be written as $e^{tR}$ if the summation converges. For a self-adjoint operator $R$ with bounded eigenvalues from the above, the operator $e^{tR}$ has been intensively studied as the *semigroup* generated by $R$ for $\operatorname{Re} t > 0$: examples include

* the heat kernel for $R = \Delta$,

* the Hermite semigroup for $R = \frac{1}{4}(\Delta - |x|^2)$ on $L^2(\mathbb{R}^n)$,

* the Laguerre semigroup for $R = |x|\left(\frac{\Delta}{4} - 1\right)$ on $L^2(\mathbb{R}^n, \frac{1}{|x|}dx)$.

In [5], we addressed a new line of investigation in a more general setting where $X \neq \{\text{point}\}$ and $X \neq Y$. In this generality, we refer to $T$ in (2.1) as the *generating operator* for a family of operators $R_\ell: \Gamma(X) \to \Gamma(Y)$.
Here is a simple example of the generating operator for $X = \mathbb{R}^n \supset Y = \mathbb{R}^{n-1}$:

**Example 2.1 (normal derivative).** Let $X = \mathbb{R}^n = \{(x_0, \dots, x_{n-1}) = (x_0, x)\}$ and $Y = \mathbb{R}^{n-1} = \{x = (x_1, \dots, x_{n-1})\}$, the hyperplane $x_0 = 0$. We consider a countable family of differential operators

$$R_{\ell}: C^{\omega}(X) \to C^{\omega}(Y), \quad F \mapsto \text{Rest}_{x_0=0} \circ (\frac{\partial}{\partial x_0})^{\ell} F.$$

The Taylor series expansion of $F(x_0, x_1, \dots, x_n)$ along the normal direction $x_0$ shows that the generating operator $T$ of $\{R_{\ell}\}_{\ell \in \mathbb{N}}$ is given as

$$\begin{aligned} (TF)(x_1, \dots, x_{n-1}; t) &= \sum_{\ell=0}^{\infty} \frac{t^{\ell}}{\ell!} \frac{\partial^{\ell} F}{\partial x_0^{\ell}}(0, x_1, \dots, x_{n-1}) \\ &= F(t, x_1, \dots, x_{n-1}). \end{aligned}$$

Thus, $T$ is essentially the identity operator in this special case.

In general, symmetry breaking operators are not given by normal derivatives, and finding a closed formula of the generating operator is even more involved in such cases. The one for the Rankin–Cohen brackets in the specific setting where $(X, Y) = (\mathbb{C}^2, \mathbb{C})$ was obtained in [4, 5].

## 2.2. Trick from discrete to continuous.

According to the recipe proposed in [1, Sect.3.3], we define a “meromorphic continuation” of a countable family of differential operators $\{R_{\ell}\}_{\ell \in \mathbb{N}}$ through its generating operator $T = \sum_{\ell=0}^{\infty} \frac{t^{\ell}}{\ell!} R_{\ell}$ by the following scheme.

$$\{R_{\ell}\}_{\ell \in \mathbb{N}} \xrightarrow{\substack{(2.1) \\ -- \to}} T \xrightarrow{\substack{(2.2) \\ -- \to}} T_{\mu, \delta}.$$

To implement the procedure $T \dashrightarrow T_{\mu, \delta}$, we recall that the locally integrable functions $|x|^{\mu}$ and $|x|^{\mu} \operatorname{sgn} x$, initially defined for $\operatorname{Re} \mu > -1$ on $\mathbb{R}$, extend to tempered distributions depending meromorphically on $\mu \in \mathbb{C}$. The poles of this family are all simple, and are located at $\{-1, -3, -5, \dots, \}$ and $\{-2, -4, -6, \dots, \}$, respectively.
Building on the generating operator $T: \Gamma(X) \to \Gamma(Y) \otimes \mathbb{C}[t]$ of $\{R_\ell\}$, we consider, for $\mu \in \mathbb{C}$ and $\delta \in \{0, 1\} = \mathbb{Z}/2\mathbb{Z}$,

$$ (2.2) \qquad (T_{\mu,\delta}f)(z) = \langle |t|^\mu (\operatorname{sgn} t)^\delta, T f(z,t) \rangle \\ = \langle |t|^\mu (\operatorname{sgn} t)^\delta, \sum_{\ell=0}^\infty \frac{t^\ell}{\ell!} (R_\ell f)(z) \rangle. $$

We may expect that $T_{\mu,\delta}$ constitutes a family of operators from $\Gamma(X)$ to $\Gamma(Y)$ that depends meromorphically on $\mu$ if the formal power series (2.1) converges in an appropriate way.

Furthermore, let $\tilde{T}_{\mu,\delta}$ be renormalization of $T_{\mu,\delta}$ given by the distribution kernels

$$ \frac{1}{\Gamma(\frac{\mu+1}{2})} |x|^{\mu} \quad (\delta = 0), \qquad \frac{|x|^{\mu} \operatorname{sgn} x}{\Gamma(\frac{\mu}{2} + 1)} \quad (\delta = 1) $$

which depend holomorphically on $\mu$ in the entire complex plane $\mathbb{C}$.
Finally, we note that there are non-zero constants $C_m$ such that

$$ \left. \frac{1}{\Gamma(\frac{\mu+1}{2})} |x|^{\mu} \right|_{\mu=-2k-1} = C_{2k} \delta^{(2k)}, \\ \left. \frac{1}{\Gamma(\frac{\mu}{2}+1)} |x|^{\mu} \operatorname{sgn} x \right|_{\mu=-2k-2} = C_{2k+1} \delta^{(2k+1)}. $$

## 2.3. Differential SBOs, localness theorem and extension theorem.

For a pair of real reductive Lie groups $G \supset G'$, continuous symmetry breaking operators between two principal series representations of $G$ and $G'$ are generically given by integral transforms and their meromorphic continuations, see [6, 7] for example.

However, in some geometric models, only differential symmetry breaking operators exist. This phenomenon, formalized as ‘localness theorem’, persists even for vector-bundle valued principal series representations, see [7]. For our purpose, we briefly recall from [2, Section 5] a prototype of this phenomenon in the holomorphic setting as below.

**Fact 2.2** (Localness Theorems [2, Thm. 5.3]). Suppose that $G \supset G'$ is a pair of reductive Lie groups such that both of the associated Riemannian
symmetric spaces $Y = G'/K'$ and $X = G/K$ are Hermitian symmetric. Moreover, assume that the natural $G'$-equivariant embedding $\iota: Y \hookrightarrow X$ is holomorphic, see [2, (2.1)] for precision. Consider a $G$-equivariant holomorphic vector bundle $\mathcal{V}_X$ over $G/K$ associated to a $K$-module $V$, and a $G'$-equivariant one $\mathcal{W}_Y$ associated to a $K'$-module $W$. Then, any continuous $G'$-homomorphism from $\mathcal{O}(X, V)$ to $\mathcal{O}(Y, W)$ is given by a holomorphic differential operator with respect to $\iota: Y \hookrightarrow X$, that is,

$$ \mathrm{Diff}_{G'}^{\mathrm{hol}}(\mathcal{V}_X, \mathcal{W}_Y) = \mathrm{Hom}_{G'}(\mathcal{O}(X, \mathcal{V}_X), \mathcal{O}(Y, \mathcal{W}_Y)). $$

Let $X = G/K \hookrightarrow X_C = G_C/P_C$, $Y = G'/K' \hookrightarrow Y_C = G'_C/P'_C$ be Borel embeddings. We write $\mathcal{V}_{X_C}$ and $\mathcal{W}_{Y_C}$ for the $G_C$-and $G'_C$-equivariant holomorphic vector bundles over $X_C$ and $Y_C$, respectively.

**Fact 2.3** (Automatic Continuity theorem [2, Thm. 5.13]). In the setting of Fact 2.2 any differential symmetry breaking operator (or equivalently, any continuous $G'$-homomorphism) extends to a differential operator between holomorphic vector bundles over the compactifications $Y_C \hookrightarrow X_C$ of the Hermitian symmetric spaces $Y \hookrightarrow X$, namely, the injection

$$ (2.3) \qquad \mathrm{Diff}_{G_C'}^{\mathrm{hol}}(\mathcal{V}_{X_C}, \mathcal{W}_{Y_C}) \hookrightarrow \mathrm{Diff}_{G'}^{\mathrm{hol}}(\mathcal{V}_X, \mathcal{W}_Y) $$

is bijective.

## 2.4. Generalities: Normal derivatives as symmetry breaking operators.

In [3], we addressed a question of which pairs of Hermitian symmetric spaces $(X, Y) = (G/K, G'/K')$ do admit symmetry breaking operators that are given by normal derivatives. It turns out that such pairs are quite rare. We established a classification theorem in the case when $\mathrm{rank}_{\mathbb{R}} G/G' = 1$. Namely, there are six complex geometries arising from real symmetric pairs of split rank one, which we review here:

Here $\mathrm{Gr}_p(\mathbb{C}^n)$ is the Grassmanian of $p$-planes in $\mathbb{C}^n$,

$$ \mathbb{Q}^m \mathbb{C} := \{z \in \mathbb{P}^{m+1} \mathbb{C} : z_0^2 + \cdots + z_{m+1}^2 = 0\} $$

is the complex quadric, and

$$ \mathrm{IGr}_n(\mathbb{C}^{2n}) := \{V \subset \mathbb{C}^{2n} : \dim V = n, Q|_V \equiv 0\} $$
$$
(1) \quad \mathbb{P}^n \mathbb{C} \hookrightarrow \mathbb{P}^n \mathbb{C} \times \mathbb{P}^n \mathbb{C}
$$

$$
(2) \quad \mathrm{LGr}(\mathbb{C}^{2n-2}) \times \mathrm{LGr}(\mathbb{C}^2) \hookrightarrow \mathrm{LGr}(\mathbb{C}^{2n})
$$

$$
(3) \quad \mathbb{Q}^n \mathbb{C} \hookrightarrow \mathbb{Q}^{n+1} \mathbb{C}
$$

$$
(4) \quad \mathrm{Gr}_{p-1}(\mathbb{C}^{p+q}) \hookrightarrow \mathrm{Gr}_p(\mathbb{C}^{p+q})
$$

$$
(5) \quad \mathbb{P}^n \mathbb{C} \hookrightarrow \mathbb{Q}^{2n} \mathbb{C}
$$

$$
(6) \quad \mathrm{IGr}_{n-1}(\mathbb{C}^{2n-2}) \hookrightarrow \mathrm{IGr}_n(\mathbb{C}^{2n})
$$

<caption>TABLE 2.1. Equivariant embeddings of flag varieties</caption>

is the Grassmanian of isotropic subspaces of $\mathbb{C}^{2n}$ equipped with a non-
degenerate quadratic form $Q$, and

$$
\mathrm{LGr}_n(\mathbb{C}^{2n}) := \{V \subset \mathbb{C}^{2n} : \dim V = n, \omega|_{V \times V} \equiv 0\}
$$

is the Grassmanian of Lagrangian subspaces of $\mathbb{C}^{2n}$ equipped with a
symplectic form $\omega$.

Among them only three do admit normal derivatives as symmetry
breaking operators.

**Fact 2.4.** (1) Any continuous $G'$-homomorphism from $O(X, \mathcal{L}_\lambda)$ to $O(Y, W)$ is given by normal derivatives with respect to the equivariant embedding $Y \hookrightarrow X$ if the embedding $Y \hookrightarrow X$ is of type (4), (5) or (6) in Table 2.1.

(2) None of normal derivatives of positive order is a $G'$-homomorphism if the embedding $Y \hookrightarrow X$ is of type (1), (2) and (3) in Table 2.1.

Recall that if $E = E' \oplus E''$ is a direct sum of complex vector spaces and $\mathcal{V}_E := E \times V$ and $\mathcal{W}_{E'} := E' \times W$ is a direct product vector bundles over $E$ and $E'$, respectively, then $\mathcal{O}(E, \mathcal{V}_E) \simeq \mathcal{O}(E) \otimes V$, and $\mathcal{O}(E', \mathcal{W}_{E'}) \simeq \mathcal{O}(E') \otimes W$.

Take coordinates $y = (y_1, \cdots, y_p)$ in $E'$ and $z = (z_1, \cdots, z_n)$ in $E''$.
The subspace $E'$ is given by the condition $z = 0$ in $E = \{(y, z) : y \in E', z \in E''\}$.
A holomorphic differential operator $\tilde{T} : \mathcal{O}(E) \otimes V \to \mathcal{O}(E') \otimes W$, $f(y, z) \mapsto (\tilde{T}f)(y)$ is said to be a normal derivative with respect to the decomposition $E = E' \oplus E''$ if it is of the form

$$
(2.4) \quad (\tilde{T}f)(y) = \sum_{\alpha \in \mathbb{N}^q} T_\alpha(y) \left( \left. \frac{\partial^{|\alpha|} f(y, z) }{\partial z^\alpha} \right|_{z=0} \right),
$$

for some $T_\alpha \in \mathcal{O}(E') \otimes \operatorname{Hom}_C(V, W)$.
We write $\mathcal{N}\text{Diff}^{\text{hol}}(\mathcal{V}_E, \mathcal{W}_{E'})$ for the space of (holomorphic) normal derivatives. This notion depends on the direct sum decomposition $E = E' \oplus E''$ and we may apply it to the subsymmetric space $G'/K'$ in the Hermitian symmetric space $G/K$ using the fact that we have the following direct sum decomposition of $K'$-modules:

$$ (2.5) \qquad \mathbf{n}_{-} = \mathbf{n}_{-}' \oplus \mathbf{n}_{-}'' . $$

Since the trivialization of the vector bundle $G_{\mathbb{C}} \times_{P_{\mathbb{C}}} V$ is $K_{\mathbb{C}}$-equivariant, there is a natural isomorphism:

$$ \text{Hom}_{K'}(V, S(\mathbf{n}_{-}'') \otimes W) \xrightarrow{\sim} \mathcal{N}\text{Diff}_{K'}^{\text{const}}(\mathcal{V}_X, \mathcal{W}_Y). $$

It was proven in [3, Theorem 5.3] that in the case when $\dim V = 1$ and the spectral parameter of the homogeneous line bundle over $X = G/K$ satisfies a certain positivity condition all continuous $G'$-homomorphisms

$$ \mathcal{O}(X, \mathcal{L}_{\lambda}) \to \mathcal{O}(Y, W), $$

are given by normal derivatives with respect to the decomposition $\mathbf{n}_{-} = \mathbf{n}_{-}' \oplus \mathbf{n}_{-}''$ for any irreducible $K'$-module $W$ if and only if the symmetric pair $(\mathfrak{g}, \mathfrak{g}')$ is isomorphic to one of $(\mathfrak{su}(p, q), \mathfrak{s}(\mathfrak{u}(1) \oplus \mathfrak{u}(p-1, q)))$, $(\mathfrak{so}(2, 2n), \mathfrak{u}(1, n))$ or $(\mathfrak{so}^*(2n), \mathfrak{so}(2) \oplus \mathfrak{so}^*(2n-2))$, see Fact 2.4 (2).

3 TEST CASE $(G_{\mathbb{R}}, G'_{\mathbb{R}}) = (GL(n + 1, \mathbb{R}), GL(n, \mathbb{R}))$

We apply our strategy of meromorphic continuation to the generating operator and test it in the simplest setting for the pair $(GL(n + 1, \mathbb{R}), GL(n, \mathbb{R}))$.

### 3.1. Step 1: Description of symmetry breaking operators given by normal derivatives.

Let $(G, G') = (U(p, q), U(p-1, q))$. We realize the Hermitian symmetric spaces $G'/K'$ and $G/K$ as bounded complex symmetric domains:

$$ G'/K' \simeq \{Y \in M(p-1, q; \mathbb{C}) : I_{p-1} - YY^* \gg 0\}, $$

$$ G/K \simeq \{X \in M(p, q; \mathbb{C}) : I_p - XX^* \gg 0\}. $$
Let $P'_C$ and $P_C$ be maximal parabolic subgroups of $G'_C$ and $G_C$ with Levi subgroups $K'_C$ and $K_C$, respectively. Then the embedding $G'/K' \hookrightarrow G/K$ is naturally realized via

$$ \begin{array}{ccc} G'_C/P'_C & \longrightarrow & G_C/P_C \\ \cup & & \cup \\ M(p-1, q; \mathbb{C}) & \longrightarrow & M(p, q; \mathbb{C}) \\ \cup & & \cup \\ G'/K' & \longrightarrow & G/K. \end{array} $$

### 3.2. Step 2: Changing real form.

All symmetry breaking operators regarding the embedding $G'/K' \hookrightarrow G/K$ in the holomorphic setting are holomorphic differential operators by the localness theorem (Fact 2.2). Moreover, they extend to the whole complex flag varieties regarding the embedding $G'_C/P'_C \hookrightarrow G_C/P_C$ by the extension theorem (Fact 2.3). We restrict them to other real forms as follows. For $(G, G') = (U(p, q), U(p-1, q))$, we set $(G_R, G'_R) := (GL(p+q, \mathbb{R}), GL(p+q-1, \mathbb{R}))$ and $P_R := G_R \cap P_C$, $P'_R := G'_R \cap P'_C$, so that

$$ G/K \subset_{\text{open}} G_C/P_C \supset_{\text{real form}} G_R/P_R. $$

**Proposition 3.1.** *One has a natural morphism*

$$ \mathrm{Diff}_{G'}(\mathcal{O}(G/K, \mathcal{L}_{\lambda}), \mathcal{O}(G'/K', \mathcal{L}'_{\nu})) \hookrightarrow \mathrm{Diff}_{G'_R}(\mathrm{Ind}_{P_R}^{G_R}(\mathbb{C}_{\lambda}), \mathrm{Ind}_{P'_R}^{G'_R}(\mathbb{C}_{\nu})). $$

We shall see in Section 3.7 that this morphism is not necessarily surjective, see (3.8).

### 3.3. Degenerate principal series representations $\pi_{\lambda, \delta}$ of $G_R$.

For $\lambda, \nu \in \mathbb{C}$ and $\delta, \varepsilon \in \{0, 1\} \simeq \mathbb{Z}/2\mathbb{Z}$, we define characters of $P_R$ and $P'_R$ by

$$ \chi_{\lambda, \delta} \begin{pmatrix} A & 0 \\ C & D \end{pmatrix} := |\det D|^{\lambda} \operatorname{sgn}(\det D)^{\delta} \quad \text{for} \quad \begin{pmatrix} A & 0 \\ C & D \end{pmatrix} \in P_R \subset G_R. $$
$$ \chi'_{\nu, \epsilon} \begin{pmatrix} A & 0 \\ c & d \end{pmatrix} := |\det d|^\nu \operatorname{sgn}(\det d)^\epsilon \quad \text{for } \begin{pmatrix} A & 0 \\ c & d \end{pmatrix} \in P'_{\mathbb{R}} \subset G'_{\mathbb{R}}, $$

where $C \in M(q, p; \mathbb{R})$ and $c \in M(q, p-1; \mathbb{R})$.

For $g^{-1} = \begin{pmatrix} A & B \\ C & D \end{pmatrix} \in G_{\mathbb{R}} = GL(p+q, \mathbb{R})$, one has

$$ \begin{pmatrix} A & B \\ C & D \end{pmatrix} \begin{pmatrix} I_p & X \\ 0 & I_q \end{pmatrix} = \begin{pmatrix} I_p & (AX+B)(CX+D)^{-1} \\ 0 & I_q \end{pmatrix} \begin{pmatrix} A - (AX+B)(CX+D)^{-1}C & 0 \\ C & CX+D \end{pmatrix}, $$

therefore the representation $\Pi_{\lambda, \delta}$ of $G_{\mathbb{R}}$ induced from a character $\chi_{\lambda, \delta}$ of the parabolic subgroup $P_{\mathbb{R}}$ is given as a multiplier representation in the open Bruhat cell

$$ (\Pi_{\lambda, \delta}(g)F)(X) = |\det(CX+D)|^{-\lambda} (\operatorname{sgn}(\det(CX+D)))^{\delta} F((AX+B)(CX+D)^{-1}). $$

Our convention is to use an unnormalized induction without 'p-shift'.

It may be useful for later purpose to write a formula of the infinitesimal representation $d\Pi_{\lambda, \delta}$. For $1 \le i \le p$ and $p+1 \le j \le p+q$,

$$ (3.1) \qquad d\Pi_{\lambda,\delta}(E_{ii} - E_{jj}) = -\lambda \mathrm{Id} - \sum_{a=1}^{q} X_{ia} \frac{\partial}{\partial X_{ia}} - \sum_{b=1}^{p} X_{bj} \frac{\partial}{\partial X_{bj}}. $$

### 3.4. $(p, q) = (n, 1)$ case.

From now, we set $(p, q) = (n, 1)$. Our notation is as follows:

$G_{\mathbb{R}} = GL(n+1, \mathbb{R})$ acting $\mathbb{R}^{n+1}$ with standard basis $e_0, e_1, \dots, e_n$

$G'_{\mathbb{R}} = GL(n, \mathbb{R})$ is the stabilizer of $e_0$.

Let $\mu \in \mathbb{C}$ and $\delta \in \mathbb{Z}/2\mathbb{Z}$. A $C^\infty$-function $f(x)$ defined on $\mathbb{R}^{n+1} \setminus \{0\}$ is of homogeneous degree of $(\mu, \delta)$ if

$$ f(rx) = r^\mu f(x) \quad \text{for every } r > 0, $$

$$ f(-x) = (-1)^\delta f(x). $$

Let $\Xi_{\mu, \delta}^{\infty}(\mathbb{R}^{n+1})$ denote the vector space of smooth homogeneous functions of degree $(\mu, \delta)$. An analogous notation $\Xi_{\mu, \delta}^{-\infty}(\mathbb{R}^{n+1})$ will be used for generalized functions à la Gelfand so that $\Xi_{\mu, \delta}^{\infty}(\mathbb{R}^{n+1}) \subset \Xi_{\mu, \delta}^{-\infty}(\mathbb{R}^{n+1})$.
We identify the real flag variety $G_R/P_R$ with the real projective space

$$G_R/P_R \xrightarrow{\sim} \mathbb{P}^n \mathbb{R} = \{[x_0 : \cdots : x_n]\}, \quad gP_R \mapsto [g^t(0, \dots, 0, 1)].$$

Then the regular representation of $G_R$ on $\Xi_{-\lambda, \delta}^\infty$ is identified with the principal series representation $\Pi_{\lambda, \delta}$ on $C^\infty(G_R/P_R, \mathcal{L}_{\lambda, \delta})$.

We take a covering of $\mathbb{R}^{n+1} \setminus \{0\}$ by $V_i := \{(x_0, \dots, x_n) : x_i \neq 0\}$ for $0 \le i \le n$. Then $V_i/\mathbb{R}^\times \simeq \mathbb{R}^n$, and the inclusive map $V_i \hookrightarrow \mathbb{R}^n \setminus \{0\}$ induces coordinates $\iota_i : \mathbb{R}^n \hookrightarrow \mathbb{P}^n \mathbb{R}$. For $F \in \Xi_{-\lambda, \delta}^\infty(\mathbb{R}^{n+1})$, we set $f_i := \iota_i^* F$, that is,

$$(3.2) \qquad f_i(x_0, \dots, \hat{x}_i, \dots, x_n) := F(x_0, \dots, x_{i-1}, 1, x_i, \dots, x_n).$$

Then $\{f_k\}_{0 \le k \le n}$ satisfies

$$(3.3) \qquad f_i(x_0, \dots, \hat{x}_i, \dots, x_n) = |x_j|^{-\lambda} (\operatorname{sgn} x_j)^{\delta} f_j(z_0, \dots, \hat{z}_j, \dots, z_n)$$

on $\iota_i(\mathbb{R}^n) \cap \iota_j(\mathbb{R}^n) = (V_i \cap V_j)/\mathbb{R}^\times \subset \mathbb{P}^n \mathbb{R}$, where $z_k = \frac{x_k}{x_j}$ ($k \neq i, j$) and $z_i = \frac{1}{x_j}$. Then we have a one-to-one correspondence

$$F \leftrightarrow \{f_0, \dots, f_n\} \quad \text{subject to the relation (3.3).}$$

For instance,

$$f_n(x_0, \dots, x_n) = |x_0|^{-\lambda} (\operatorname{sgn} x_0)^{\delta} f_0(z_1, \dots, z_n),$$

where $z_k = \frac{x_k}{x_0}$ ($1 \le k \le n-1$) and $z_n = \frac{1}{x_0}$.

For $F \in \Xi_{-\lambda, \delta}^\infty(\mathbb{R}^{n+1}) \simeq C^\infty(G_R/P_R, \mathcal{L}_{\lambda, \delta})$, $f_n = \iota_n^* F$ gives the 'N-picture' of the principal series representation of $G_R$ defined on the open Bruhat cell $V_n/\mathbb{R}^\times \simeq \mathbb{R}^n$ (and similarly for $G'_R$):

$$G_R \stackrel{\Pi_{\lambda, \delta}}{\rightleftharpoons} C^\infty(G_R/P_R, \mathcal{L}_{\lambda, \delta}) \hookrightarrow \iota_n^* C^\infty(\mathbb{R}^n),$$

$$G'_R \stackrel{\pi_{\nu, \varepsilon}}{\rightleftharpoons} C^\infty(G'_R/P'_R, \mathcal{L}_{\nu, \varepsilon}) \hookrightarrow C^\infty(\mathbb{R}^{n-1}).$$

Here we used the letters $\Pi_{\lambda, \delta}$ for the $G_R$ and $\pi_{\nu, \varepsilon}$ for the $G'_R$-actions, respectively.

In the coordinates $\iota_n : \mathbb{R}^n \simeq V_n/\mathbb{R}^\times \hookrightarrow \mathbb{P}^n \mathbb{R}$, we set

$$(3.4) \qquad R_\ell := \operatorname{Rest}_{x_0=0} \circ (\frac{\partial}{\partial x_0})^\ell.$$
The normal derivatives $R_\ell$ are restrictions of the holomorphic differential SBOs defined on holomorphic line bundles over $\mathbb{P}^n\mathbb{C} \simeq G_\mathbb{C}/P_\mathbb{C}$, and give rise to symmetry breaking operators in the real setting as well:

**Proposition 3.2 (normal derivatives).** If $\nu - \lambda = \ell$ and $\delta - \varepsilon \equiv \ell \bmod 2$, then $\pi_{\nu,\varepsilon} \circ R_\ell = R_\ell \circ \Pi_{\lambda,\delta}$ as $G'_\mathbb{R}$-homomorphisms.

*Proof.* Let $g^{-1} = \begin{pmatrix} A & b \\ c & d \end{pmatrix} \in G'_\mathbb{R} = GL(n, \mathbb{R})$ where $A \in GL(n-1, \mathbb{R})$, $b, c \in \mathbb{R}^{n-1}$, $d \in \mathbb{R}$. Then for $f = \iota_n^*F$ where $F \in \Xi_{-\lambda,\delta}^\infty(\mathbb{R}^n)$,

$$ (R_\ell \circ \Pi_{\lambda,\delta}(g)f)(x) = \mathrm{Rest}_{x_0=0} \circ \left(\frac{\partial}{\partial x_0}\right)^\ell \left|(c, x) + d\right|^{-\lambda} \left(\mathrm{sgn}\left((c, x) + d\right)\right)^\delta f\left(\frac{x_0}{(c, x) + d}, \frac{Ax + b}{(c, x) + d}\right) $$
$$ = \left|(c, x) + d\right|^{-\lambda} \left(\mathrm{sgn}(c, x) + d\right)^\delta \mathrm{Rest}_{x_0=0} \circ \left(\frac{\partial}{\partial x_0}\right)^\ell f\left(\frac{x_0}{(c, x) + d}, \frac{Ax + b}{(c, x) + d}\right) $$
$$ = \left|(c, x) + d\right|^{-\lambda-\ell} \left(\mathrm{sgn}(c, x) + d\right)^{\delta-\ell} \frac{\partial^\ell f}{\partial x_0^\ell} \left(0, \frac{Ax + b}{(c, x) + d}\right) $$
$$ = (\pi_{\nu,\varepsilon}(s) \circ R_\ell f)(x). $$

□

### 3.5. Step 3: Generating function of SBOs {$R_\ell$}.

Since all $R_\ell$ are given by normal derivatives in our setting, the generating operator $T$ of {$R_\ell$}$_{\ell \in \mathbb{N}}$ is easy to find, as shown in Example 2.1. Explicitly, it is essentially the identity map, that is,

$$ (Tf)(x_2, \dots, x_n; t) = \sum_{\ell=0}^{\infty} \frac{t^\ell}{\ell!} (R_\ell f)(x_2, \dots, x_n) = f(t, x_2, \dots, x_n) $$

for $f_n \equiv f(x_1, x_2, \dots, x_n) \in C^\infty(G_\mathbb{R}/P_\mathbb{R}, \mathcal{L}_{\lambda,\delta})$ defined on $\iota_n(\mathbb{R}^n) \subset \mathbb{P}^n\mathbb{R}$.

### 3.6. Step 4: From discrete to continuous.

The operator $T_{\mu,\delta}$ constructed in (2.2) by the general idea [1] “from discrete to continuous” is a non-local operator as follows:

$$ (3.5) \qquad (T_{\mu,\kappa} F)(x_1, \dots, x_{n-1}, x_n) = \int_{\mathbb{R}} |x_0|^\mu (\mathrm{sgn} x_0)^\kappa F(x_0, x_1, \dots, x_{n-1}, x_n) dx_0 $$
in the homogeneous coordinates, for $F \in \Xi_{-\lambda,\delta}(\mathbb{R}^n)$. Clearly, $T_{\mu,\kappa}F$ is homogeneous of degree $(-\lambda+\mu+1, \delta+\kappa)$, whenever the integral makes sense. This yields a non-local symmetry breaking operator, which depends meromorphically on $\mu \in \mathbb{C}$. We write $\tilde{T}_{\mu,\kappa}$ for a normalization that depends holomorphically on $\mu \in \mathbb{C}$. Then we have the following.

**Proposition 3.3.** If $\lambda = \mu + \nu + 1$ and $\delta \equiv \varepsilon + \kappa \bmod 2$, then

$$ \pi_{\nu,\varepsilon} \circ \tilde{T}_{\mu,\kappa} = \tilde{T}_{\mu,\kappa} \circ \Pi_{\lambda,\delta}, $$

as $G'_{\mathbb{R}}$-homomorphisms.

*Proof.* It suffices to verify the statement for generic $\mu$. In the $N$-picture for $f_n = \iota_n^*F$ on $\iota_n(\mathbb{R}^n) \subset \mathbb{P}^n\mathbb{R}$ one has: (3.6)

$$ (T_{\mu,\kappa}f_n)(x_1, \dots, x_{n-1}) = \int_{\mathbb{R}} |x_0|^{\mu} (\operatorname{sgn} x_0)^{\kappa} f_n(x_0, x_1, \dots, x_{n-1}) dx_0. $$

Let $g^{-1} = \begin{pmatrix} A & b \\ c & d \end{pmatrix} \in G'_{\mathbb{R}} = GL(n, \mathbb{R})$ where $A \in GL(n-1, \mathbb{R})$, $b, c \in \mathbb{R}^{n-1}$, $d \in \mathbb{R}$. Then, $(T_{\mu,\kappa} \circ \Pi_{\lambda,\delta}(g)f_n)(x)$ amounts to

$$ \begin{aligned} & \int_{\mathbb{R}} |x_0|^{\mu} (\operatorname{sgn} x_0)^{\kappa} |(c, x) + d|^{-\lambda} (\operatorname{sgn}((c, x) + d))^{\delta} f_n\left(\frac{x_0}{(c, x) + d}, \frac{Ax + b}{(c, x) + d}\right) dx_0 \\ = & \int_{\mathbb{R}} |(c, x) + d|^{\mu - \lambda + 1} (\operatorname{sgn}((c, x) + d))^{\kappa + \delta} |t|^{\mu} (\operatorname{sgn} t)^{\kappa} f_n(t, \frac{Ax + b}{(c, x) + d}) dt \\ = & \pi_{\nu,\varepsilon}(g)(T_{\mu,\kappa}f_n)(x). \end{aligned} $$

□

### 3.7. From continuous to discrete.

This section explains a transfer from the family of differential SBOs $\{R_\ell\}$ to another family of differential SBOs $\{L_\ell\}$ via the meromorphic family $T_{\mu,\kappa}$ in (3.6). According to the transition

$$ \{R_\ell\} \rightsquigarrow \{T_{\mu,\kappa}\} \rightsquigarrow \{L_\ell\}, $$

the supports of the distribution kernels of these operators vary geometrically as

$$ X'_0 \rightsquigarrow X_1 \cup X_0 \cup X'_0 \rightsquigarrow X_0 $$

with the notation of the diagram in Section 4 below.
Since $F \in C^\infty(G_\mathbb{R}/P_\mathbb{R}, \mathcal{L}_{\lambda,\delta} \simeq \Xi_{-\lambda,\delta}^\infty(\mathbb{R}^{n+1})$ is of homogeneous degree $(-\lambda, \delta)$, the formula (3.5) of $T_{\mu,\kappa}F(x_1, \dots, x_n)$ amounts to :

$$ (3.7) \qquad \begin{aligned} & |x_n|^{\mu+1} (\operatorname{sgn} x_n)^\kappa \int_{\mathbb{R}} |t|^{\lambda-\mu-2} (\operatorname{sgn} t)^{-\delta+\kappa} F(x_n, tx_1, \dots, tx_{n-1}, tx_n) dt \\ &= |x_n|^{\mu+1} (\operatorname{sgn} x_n)^\kappa \int_{\mathbb{R}} |t|^{\nu-1} (\operatorname{sgn} t)^\varepsilon F(x_n, tx_1, \dots, tx_{n-1}, tx_n) dt. \end{aligned} $$

The integral kernel $|t|^{\nu-1}(\operatorname{sgn} t)^\varepsilon$ in the right-hand side has simple poles at $\nu = -\ell$ ($\ell \in \mathbb{N}$) if $\varepsilon \equiv \ell \bmod 2$, and its residue is given as a non-zero scalar multiple of $\delta^{(\ell)}(t)$, and therefore, $(T_{\mu,\kappa}F)(x_1, \dots, x_n)$ is proportional to

$$ \begin{aligned} & |x_n|^{\mu+1} (\operatorname{sgn} x_n)^\kappa \sum_{|\alpha|=\ell} \frac{\ell!}{\alpha!} x_1^{\alpha_1} \cdots x_n^{\alpha_n} \frac{\partial^{|\alpha|} F}{\partial x^\alpha} (x_n, 0, \dots, 0) \\ &= \sum_{|\alpha|=\ell} \frac{\ell!}{\alpha!} x_1^{\alpha_1} \cdots x_n^{\alpha_n} \frac{\partial^{|\alpha|} F}{\partial x^\alpha} (1, 0, \dots, 0), \end{aligned} $$

where $\alpha = (\alpha_1, \dots, \alpha_n) \in \mathbb{N}^n$. Here the last equality follows from that $\frac{\partial^{|\alpha|} F}{\partial x^\alpha}$ is of homogeneous degree $(-\lambda - \ell, \delta + \ell)$ and that, according to Proposition 3.3, $\lambda = \mu + \nu + 1 = \mu - \ell + 1$, $\delta \equiv \kappa + \varepsilon \equiv \kappa + \ell$. Define for $F(x_0, \dots, x_n) \in C^\infty(\mathbb{R}^{n+1})$

$$ (3.8) \qquad (L_\ell F)(x_1, \dots, x_n) := \sum_{|\alpha|=\ell} \frac{\ell!}{\alpha!} x_1^{\alpha_1} \cdots x_n^{\alpha_n} \frac{\partial^{|\alpha|} F}{\partial x^\alpha} (1, 0, \dots, 0). $$

Thus, we have obtained yet another family of differential SBOs $\{L_\ell\}$ induced by $\tilde{T}_{\lambda+\ell-1, \ell \bmod 2}$ for $\ell \in \mathbb{N}$:

$$ \begin{array}{r@{\quad}l} \text{L}_\ell: & C^\infty(G_\mathbb{R}/P_\mathbb{R}, \mathcal{L}_{\lambda,\delta}) \to C^\infty(G'_\mathbb{R}/P'_\mathbb{R}, \mathcal{L}_{-\ell,\ell \bmod 2}) \\ & \searrow \varkappa \qquad \searrow \varkappa \\ & \Xi^\infty_{-\lambda,\delta}(\mathbb{R}^{n+1}) \cdots \to \Xi^\infty_{\ell,\ell \bmod 2}(\mathbb{R}^n) \\ & \stackrel{\Psi}{\rightleftharpoons} \stackrel{\Psi}{\rightleftharpoons} \\ & F(x_0, \dots, x_n) \longmapsto \sum_{|\alpha|=\ell} \frac{\ell!}{\alpha!} x_1^{\alpha_1} \cdots x_n^{\alpha_n} \frac{\partial^{|\alpha|} F}{\partial x^\alpha} (1, 0, \dots, 0) \end{array} $$

The image of $L_\ell$ is a finite-dimensional subrepresentation $\mathrm{Pol}_\ell$, which consists of homogeneous polynomials of degree $\ell$ in $x_1, \dots, x_n$.
When $\ell = 0$, the subrepresentation $\text{Pol}_\ell$ is the trivial one-dimensional representation. Using the same trick as in [6, Thm. 14.9] we obtain the following corollary:

**Corollary 3.4.** For any $\lambda \in \mathbb{C}$ and $\delta \in \mathbb{Z}/2\mathbb{Z}$,

$$ \text{Hom}_{G_{\mathbb{R}}} (\Pi_{\lambda, \delta}, C^{\infty}(G_{\mathbb{R}}/G'_{\mathbb{R}})) \neq 0. $$

*Remark 3.5.* It is notable that the family $\{R_\ell\}_{\ell \in \mathbb{N}}$ arises from the holomorphic embedding of the Hermitian symmetric spaces $G'/K' \hookrightarrow G/K$, while $\{L_\ell\}_{\ell \in \mathbb{N}}$ does not because the operator $\text{Rest}_{z_n=0}$ does not produce functions on

$$ G'/K' = \{z \in \mathbb{P}^n \mathbb{C} : z_0 = 0, |z_1|^2 + \cdots + |z_{n-1}|^2 < |z_n|^2\}. $$

# 4 APPENDIX: ORBIT PICTURE $P'_R \backslash G_R / P_R$

In [7, Chap. 3], we proposed a general scheme for classifying all symmetry breaking operators. This scheme is based on the Hasse diagram of the double coset space $P'_R \backslash G_R / P_R$, with closed orbits producing differential SBOs ([2]). From this perspective, we examine the results discussed in the previous sections.

We recall our convention that $G'_R / P'_R$ is identified with the hypersurface $x_0 = 0$ in $G_R / P_R \simeq \mathbb{P}^n \mathbb{R} = \{[x_0 : \cdots : x_n]\}$. We define the following subsets of $G_R / P_R$ by

$$ X_n := \{[x_0 : \cdots : x_n] : x_0 \neq 0, (x_1, \ldots, x_{n-1}) \neq (0, \ldots, 0)\}, $$

$$ X_{n-1} := \{[x_0 : \cdots : x_n] : x_0 = 0, (x_1, \ldots, x_{n-1}) \neq (0, \ldots, 0)\}, $$

$$ X_1 := \{[x_0 : \cdots : x_n] : x_0 \neq 0, (x_1, \ldots, x_{n-1}) = (0, \ldots, 0), x_n \neq 0\}, $$

$$ X_0 := \{[x_0 : \cdots : x_n] : x_0 \neq 0, x_1 = \cdots = x_n = 0\}, $$

$$ X'_0 := \{[x_0 : \cdots : x_n] : x_0 = \cdots = x_{n-1} = 0, x_n \neq 0\}. $$

Then, $\dim X_j = j$, $\dim X'_0 = 0$ and $P'_R$ acts transitively on each of these sets. Thus one has an orbit decomposition of the action of $P'_R$ on $G_R / P_R$:

$$ \mathbb{P}^n \mathbb{R} = X_n \amalg X_{n-1} \amalg X_1 \amalg X_0 \amalg X'_0. $$

The closure relations among these orbits are represented by the following Hasse diagram:
<figure><img src="image_1.png" /></figure>

The two singletons $X_0$ and $X'_0$ are the fixed points by $P'_R$, however, they have significant differences, that is, $X_0$ is a fixed point of the whole group $G'_R$, while the $G'_R$-orbit through $X'_0$ is the $(n-1)$-dimensional variety $\mathbb{P}^{n-1}\mathbb{R} \simeq X_{n-1} \cup X'_0$.

The general isomorphism [6, Prop. 3.2] describing all symmetry breaking operators amounts to

$$ (4.1) \qquad \mathrm{Hom}_{G'_R}(\Pi_{\lambda,\delta}|_{G'_R}, \pi_{\nu,\varepsilon}) \simeq (\mathcal{D}'(G_R/P_R, \mathcal{L}_{-\lambda+2\rho,\delta}) \otimes \mathbb{C}_{\nu,\varepsilon})^{P'_R}, $$

where $2\rho = (-1, \dots, -1; n)$.

The procedure in Section 3.6 inflates the support of the distribution kernel of the SBOs appearing in the right-hand side of (4.1) from the singleton $X'_0$ to the closure $\overline{X_1} = X_1 \cup X_0 \cup X'_0 \simeq \mathbb{P}^1\mathbb{R}$ of the one-dimensional orbit $X_1$. The procedure developed in Section 3.7 goes into the usual direction of taking the residues of a meromorphic family of distributions, and consequently, it shrinks the support from $\overline{X_1} \simeq \mathbb{P}^1\mathbb{R}$ to the singleton $X_0$.

In the coordinates, the set $X'_0$ is at the origin in the open Bruhat cell $V_n/\mathbb{R}^\times \simeq \mathbb{R}^n \subset \mathbb{P}^n\mathbb{R}$, and differential operators $\{R_\ell\}_{\ell \in \mathbb{N}}$ are supported on the singleton $X'_0$ via the isomorphism (4.1).

On the other hand, the closed orbit $X_0 \in \mathbb{P}^n\mathbb{R}$ sits outside the open dense subset $\iota_n: V_n/\mathbb{R}^\times \simeq \mathbb{R}^n \hookrightarrow \mathbb{P}^n\mathbb{R}$ (Bruhat cell), but corresponds to the origin in the coordinates $\iota_0: V_0/\mathbb{R}^\times \simeq \mathbb{R}^n \hookrightarrow \mathbb{P}^n\mathbb{R}$, which we used in Section 3.7.

A detailed proof will appear elsewhere.
## REFERENCES

[1] T. Kobayashi, *Generating operators of symmetry breaking—from discrete to continuous*, Indagationes Mathematicæ, 2024. 18 pages. Published online 15 March 2024. DOI: 10.1016/j.indag.2024.03.007. Available also at arXiv: 2307.16587.

[2] T. Kobayashi, M. Pevzner, *Differential symmetry breaking operators: I. General theory and F-method*, Selecta Math. (N.S.), **22** (2016) no.2, 801–845.

[3] T. Kobayashi, M. Pevzner, *Differential symmetry breaking operators. II. Rankin-Cohen operators for symmetric pairs*, Selecta Mathematica (N.S.), **22** (2016), 847–911.

[4] T. Kobayashi, M. Pevzner, *A short proof for Rankin-Cohen brackets and generating operators*, in press. Lie Theory and its Applications in Physics, (ed. V. Dobrev), Springer Proceedings in Mathematics & Statistics **473**, (2024), 1–13. https://doi.org/10.1007/978-981-97-6453-2-1. Available also at arXiv: 2402.05363.

[5] T. Kobayashi, M. Pevzner, *A generating operator for Rankin-Cohen brackets*, preprint. 24 pp. arXiv: 2306.16800

[6] T. Kobayashi, B. Speh, *Symmetry Breaking for Representations of Rank One Orthogonal Groups, I*, Memoirs of Amer. Math. Soc. **238** no.1126, vi+112 pages, Amer. Math. Soc. 2015.

[7] T. Kobayashi, B. Speh, *Symmetry Breaking for Representations of Rank One Orthogonal Groups, II*, Lecture Notes in Math., **2234** Springer, 2018. xv+342 pages.

Toshiyuki KOBAYASHI

– Graduate School of Mathematical Sciences,
The University of Tokyo, 3-8-1 Komaba, Meguro, Tokyo, 153-8914, Japan.

– French-Japanese Laboratory in Mathematics and its Interactions,
FJ-LMI CNRS IRL2025, Tokyo, Japan.

E-mail: toshi@ms.u-tokyo.ac.jp

Michael PEVZNER

– LMR, Université de Reims-Champagne-Ardenne, CNRS UMR 9008, F-51687, Reims, France.

– French-Japanese Laboratory in Mathematics and its Interactions,
FJ-LMI CNRS IRL2025, Tokyo, Japan.

E-mail: pevzner@math.cnrs.fr