# Quantization commutes with reduction in the non-compact setting: the case of holomorphic discrete series

Paul-Emile Paradan

<figure><img src="image_1.png" /></figure>

## To cite this version:

Paul-Emile Paradan. Quantization commutes with reduction in the non-compact setting: the case of holomorphic discrete series. Journal of the European Mathematical Society, 2015, 17 (4), pp.955-990.
10.4171/JEMS/523 . hal-00663048

HAL Id: hal-00663048

https://hal.science/hal-00663048v1

Submitted on 25 Jan 2012

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
QUANTIZATION COMMUTES WITH REDUCTION IN THE
NON-COMPACT SETTING: THE CASE OF HOLOMORPHIC
DISCRETE SERIES

PAUL-EMILE PARADAN

ABSTRACT. In this paper we show that the multiplicities of holomorphic discrete series representations relatively to reductive subgroups satisfy the credo "Quantization commutes with reduction".

CONTENTS

<table><tr><td>1. Introduction</td><td>2</td></tr><tr><td>2. Hamiltonian actions of real reductive Lie groups</td><td>5</td></tr><tr><td>2.1. Proper actions</td><td>5</td></tr><tr><td>2.2. Proper moment map</td><td>7</td></tr><tr><td>2.3. Criterion</td><td>10</td></tr><tr><td>2.4. Kostant-Souriau line bundle</td><td>11</td></tr><tr><td>2.5. The case of elliptic orbits</td><td>11</td></tr><tr><td>3. Quantization commutes with reduction</td><td>13</td></tr><tr><td>3.1. Holomorphic coadjoint orbits</td><td>14</td></tr><tr><td>3.2. Holomorphic discrete series</td><td>16</td></tr><tr><td>3.3. Formal geometric quantization</td><td>16</td></tr><tr><td>3.4. Formal geometric quantization of holomorphic orbits</td><td>18</td></tr><tr><td>3.5. Multiplicities of the holomorphic discrete series</td><td>18</td></tr><tr><td>3.6. Jakobsen-Vergne's formula</td><td>21</td></tr><tr><td>3.7. Formal geometric quantization of G-actions</td><td>23</td></tr><tr><td>3.8. Geometric quantization of the slice Y</td><td>27</td></tr><tr><td>4. Transversally elliptic operators</td><td>28</td></tr><tr><td>4.1. Transversally elliptic operators</td><td>28</td></tr><tr><td>4.2. Riemann-Roch character</td><td>30</td></tr><tr><td>4.3. Definition of Q<sup>Φ</sup></td><td>30</td></tr><tr><td>4.4. Proof of Theorem 3.25 under Assumption A1</td><td>33</td></tr><tr><td>4.5. Proof of Theorem 3.30</td><td>36</td></tr><tr><td>4.6. Proof of Theorem 3.25 under Assumption A2</td><td>38</td></tr><tr><td>5. Appendix: proof of Lemma 3.28</td><td>38</td></tr><tr><td>References</td><td>40</td></tr></table>

Date: January 2012.

Keywords: holomorphic discrete series, moment map, reduction, geometric quantization, transversally elliptic symbol.

1991 Mathematics Subject Classification: 58F06, 57S15, 19L47.
# 1. INTRODUCTION

The orbit method, introduced by Kirillov in the 60's, proposes a correspondence between the irreducible unitary representations of a Lie group $G$ and its orbits in the coadjoint representation : the representation $\pi_G^O$ should be the geometric quantization of the Hamiltonian action of $G$ on the coadjoint orbit $O \subset g^*$. The important feature of this correspondence is the functoriality relatively to inclusion $H \hookrightarrow G$ of closed subgroups. It means that if we start with representations $\pi_G^O$ and $\pi_{O'}^H$ attached to the coadjoint orbits $O \subset g^*$ and $O' \subset h^*$, one expects that the multiplicity of $\pi_{O'}^H$, in the restriction $\pi_{O'}^G|_H$ can be computed in terms of the space

$$ (1) \qquad O \cap \pi_{b,g}^{-1}(O')/H, $$

where $\pi_{b,g} : g^* \to h^*$ denotes the canonical projection. The symplectic geometers recognise that (1) is a symplectic reduced space in the sense of Marsden-Weinstein, since $\pi_{b,g} : O \to h^*$ is the moment map relative to the Hamiltonian action of $H$ on $O$. Let us give some examples where this theory is known to be valid.

For simply connected nilpotent Lie groups, Kirillov [24] described the correspondence $O \mapsto \pi_G^O$, and Corwin-Greenleaf [10] proves its functoriality relatively to subgroup : the multiplicity appearing in the direct integral decomposition of $\pi_{O'}^G|_H$ is the cardinal of the reduced space (1).

For compact Lie group, G. Heckman [21] proved that the multiplicity was asymptotically given by the volume of the reduced space (1). Just after Guillemin and Sternberg [19] replaced this functoriality principle in a more geometric framework and proposed another version of this rule for a good quantization process: the quantization should commute with the reduction. This means that if $Q_H(M)$ is the geometric quantization of an Hamiltonian action of a compact Lie group $H$ on a symplectic manifold $M$, the multiplicity of the representation $\pi_{O'}^H$, in $Q_H(M)$ should be the (dimension of the) quantization of the reduced space $(\Phi_M^H)^{-1}(O')/H$. Here $\Phi_M^H : M \to h^*$ denotes the moment map.

A good quantization process for compact Lie group action on compact symplectic manifolds turns to be the equivariant index of a Dolbeault-Dirac operator [41, 43]. In the late 90's, Meinrenken and Meinrenken-Sjamaar proved that the principle of Guillemin-Sternberg works in this setting [31, 32]. Afterwards, this quantization procedure was extended to non-compact manifolds with a proper moment map by Ma-Zhang and the author [36, 29, 39]. See also the recent work of Duflo-Vergne on the multiplicities of the tempered representations relatively to compact subgroups [17].

The purpose of this article is to show that the quantization commutes with reduction principle holds in a case where the group of symmetry is a real reductive Lie group. Loosely speaking, we prove that if $\pi_G^O$ and $\pi_{O'}^H$ are holomorphic discrete series representations of real reductive Lie groups $H \subset G$, the multiplicity of $\pi_{O'}^H$, in the restriction $\pi_{O'}^G|_H$ is equal to the quantization of the reduced space (1).

We turn now to a description of the contents of the various sections, highlighting the main features.

In Section 2, we clarify previous work of Weinstein [46] and Duflo-Vargas [15, 16] concerning the Hamiltonian action of a connected reductive real Lie group $G$ on a symplectic manifold $M$. The main point is that if the action of $G$ on $M$ is proper and the moment map $\Phi_M^G : M \to g^*$ relative to this action is proper, then the image of $\Phi_M^G$ is contained in the strongly elliptic open subset $g_{se}^*$ and the manifold has a
decomposition

$$ (2) \qquad M = G \times_K Y. $$

Here $K$ is a maximal compact subgroup of $G$, and $Y$ is a closed $K$-invariant symplectic sub-manifold of $M$. Thanks to (2), we remark that the reduced space $(\Phi_M^G)^{-1}(\mathcal{O})/G$ is connected for any coadjoint orbit $\mathcal{O} \subset \mathfrak{g}^*$: this is a notable difference with the nilpotent case where the reduced space (1) can be disconnected.

The decomposition (2) will be the main ingredient of this paper to prove some quantization commutes with reduction phenomenon. Note that P. Hochs already used this idea when the manifold $Y$ is compact to get a quantization commutes with reduction theorem in the setting of KK-theory [22]. Hochs was working on some induction process, while we will use (2) to prove some functoriality relatively to a restriction procedure.

In this context, it is natural to look at the induced action of a reductive subgroup $G' \subset G$ on $M$, and we know then that we have another decomposition $M = G' \times_{K'} Y'$ if the moment map $\Phi_M^{G'}$ is proper. In Section 2.3, we give a criterion that insures the properness of $\Phi_M^{G'}$.

In Section 3, we turn onto a closed study of the holomorphic discrete series representations of a reductive Lie group $G$. Recall that the parametrization of these representations depends on the choice of an element $z$ in the center of the Lie algebra of $K$ such that the adjoint map $\mathrm{ad}(z)$ defines a complex structure on $\mathfrak{g}/\mathfrak{t}$. Let $T$ be a maximal torus in $K$, with Lie algebra $\mathfrak{t}$. The existence of element $z$ forces $\mathfrak{t}$ to be a Cartan sub-algebra of $\mathfrak{g}$, and it defines a closed cone $C_{\mathrm{hol}}^{\rho}(z) \subset \mathfrak{t}^*$. If $\mathfrak{t}^* \subset \mathfrak{t}^*$ is the weight lattice, we consider the subset

$$ \hat{G}_{\mathrm{hol}}(z) := C_{\mathrm{hol}}^{\rho}(z) \cap \Lambda_{+}^{*} $$

where $\Lambda_{+}^*$ is the set of dominant weights. The work of Harish-Chandra tells us that that we can attach an holomorphic discrete series representation $V_{\lambda}^G$ to any $\lambda \in \hat{G}_{\mathrm{hol}}(z)$.

In Section 3, we look at formal quantization procedures attached to the Hamiltonian action of $G$ on a symplectic manifold $M$. We suppose that the properness assumptions are satisfied and that the image of the moment map $\Phi_M^G$ is contained in $G \cdot C_{\mathrm{hol}}^{\rho}(z) \subset \mathfrak{g}_{se}^*$. Let us briefly recall the definition. We define the formal geometric quantization of the $G$-action on $M$ as the following formal sum

$$ (3) \qquad Q_G^{-\infty}(M) := \sum_{\lambda \in \hat{G}_{\mathrm{hol}}(z)} Q(M_{\lambda,G}) V_{\lambda}^G, $$

where $Q(M_{\lambda,G}) \in \mathbb{Z}$ is the quantization of the compact symplectic reduced space $M_{\lambda,G} := (\Phi_M^G)^{-1}(G \cdot \lambda)/G$.

Since the moment map $\Phi_M^K$ is proper, we can also define the formal geometric quantization of the $K$-action on $M$ as

$$ (4) \qquad Q_K^{-\infty}(M) := \sum_{\mu \in \Lambda_{+}^{*}} Q(M_{\mu,K}) V_{\mu}^{K}, $$

where $M_{\mu,K} := (\Phi_M^K)^{-1}(K \cdot \mu)/K$, and $V_{\mu}^K$ denotes the irreducible representation of $K$ with highest weight $\mu$. The formal quantization procedure $Q_K^{-\infty}$, together with its functorial properties, has been studied by Ma-Zhang and the author in [36, 29, 39].
Let $R^{-\infty}(G, z)$ be the $\mathbb{Z}$-module formed by the infinite sums $\sum_{\lambda \in \hat{G}_{\text{hol}}(z)} m_{\lambda} V_{\lambda}^{G}$ with $m_{\lambda} \in \mathbb{Z}$. We consider also the $\mathbb{Z}$-module $R^{-\infty}(K)$ formed by the infinite sums $\sum_{\mu \in \hat{\Lambda}^{*}} n_{\mu} V_{\mu}^{K}$ with $n_{\mu} \in \mathbb{Z}$. The following basic result will be an important tool in our paper (see Lemma 3.20).

**Lemma A** We have an injective restriction morphism $\mathbf{r}_{K,G} : R^{-\infty}(G, z) \to R^{-\infty}(K)$.

We can state one of our main result

**Theorem B** If Assumptions<fnref n="1" /> **A1** or **A2** are satisfied, we have the following relation

$$ \mathbf{r}_{K,G} (Q_G^{-\infty}(M)) = Q_K^{-\infty}(M). $$

The main tool for proving Theorem B, is the following relation

$$ Q_K^{-\infty}(M) = Q_K^{-\infty}(Y) \otimes S^{\bullet}(\mathfrak{p}) $$

where $Q_K^{-\infty}(Y)$ is the formal geometric quantization of the slice $Y$ and $S^{\bullet}(\mathfrak{p})$ is the symmetric algebra of the complex $K$-module $\mathfrak{p} := (\mathfrak{g}/\mathfrak{k}, \text{ad}(z))$.

Consider now a connected reductive subgroup $G' \subset G$ such that its Lie algebra $g'$ contains the element $z$. Let $\Phi_{G',\lambda}$ be the moment map relative to the Hamiltonian action of $G'$ on the coadjoint orbit $G \cdot \lambda$, $\lambda \in \hat{G}_{\text{hol}}(z)$. It is not difficult to see that $\Phi_{G',\lambda}$ is a proper map. Thanks to the work of T. Kobayashi [27] and Duflo-Vargas [15], we know that the representation $V_{\lambda}^{G}$ admits an admissible restriction to $G'$. It means that the restriction $V_{\lambda}^{G}|_{G'}$ is a discrete sum formed by holomorphic discrete series representations $V_{\mu}^{G'}$, $\mu \in \hat{G}_{\text{hol}}(z)$.

We can now state the major result of this paper (see Theorem 3.26).

**Theorem C** Let $\lambda \in \hat{G}_{\text{hol}}(z)$. We have the following relation

$$ V_{\lambda}^{G}|_{G'} = Q_{G'}^{-\infty}(G \cdot \lambda). $$

It means that for any $\mu \in \hat{G}'_{\text{hol}}(z)$, the multiplicity of the representation $V_{\mu}^{G'}$ in the restriction $V_{\lambda}^{G}|_{G'}$ is equal to the geometric quantization $Q((G \cdot \lambda)_{\mu,G'}) \in \mathbb{Z}$ of the (compact) reduced space $(G \cdot \lambda)_{\mu,G'}$.

In Section 3.7, we prove that the formal quantization process $Q_G^{-\infty}$ is functorial relatively to reductive subgroups.

**Theorem D** Suppose that Assumption **A2** holds. Then we have

$$ \mathbf{r}_{G',G} (Q_G^{-\infty}(M)) = Q_{G'}^{-\infty}(M). $$

where $\mathbf{r}_{G',G}$ is a restriction morphism.

In [23], Jakobsen-Vergne proposed another formula for the multiplicity of the representation $V_{\mu}^{G'}$ in the restriction $V_{\lambda}^{G}|_{G'}$. In Section 3.8, we explain how to recover their result from Theorem B.

Section 4 is devoted to the proofs of the main results of this paper. We use here previous work of the author on localization techniques in the setting of transversally elliptic operators.

<footnote n="1">¹See Assumptions 3.23.</footnote>
**Notations.** In this paper, $G$ will denoted a connected real reductive Lie group. We take here the convention of Knapp [26]. We have a Cartan involution $\Theta$ on $G$, such that the fixed point set $K := G^\Theta$ is a connected maximal compact subgroup. We have Cartan decompositions : at the level of Lie algebras $\mathfrak{g} = \mathfrak{k} \oplus \mathfrak{p}$ and at the level of the group $G \simeq K \times \exp(\mathfrak{p})$. We denote by $b$ a $G$-invariant non-degenerate bilinear form on $\mathfrak{g}$ that defines a $K$-invariant scalar product $(X, Y) := -b(X, \Theta(Y))$.

When $V$ and $V'$ are two representations of a group $H$, the multiplicity of $V$ in $V'$ will be denoted $[V : V']$.

## 2. HAMILTONIAN ACTIONS OF REAL REDUCTIVE LIE GROUPS

This section is mainly a synthesis of previous work by Weinstein [46], Duflo-Vargas [15, 16] and Hochs [22], except the criterion that we obtain in Section 2.3.

Let $G$ be a connected real reductive Lie group. We consider an Hamiltonian action of $G$ on a connected symplectic manifold $(M, \Omega_M)$. The corresponding moment map $\Phi_M^G : M \to \mathfrak{g}^*$ is defined (modulo a constant) by the relations

$$ (5) \qquad \iota(X_M)\Omega_M = -d\langle\Phi_M^G, X\rangle, \quad \forall X \in \mathfrak{g}, $$

where $X_M(m) := \frac{d}{ds}e^{-sX} \cdot m|_{s=0}$ is the vector field generated by $X \in \mathfrak{g}$.

Let $\mathfrak{g} = \mathfrak{k} \oplus \mathfrak{p}$ be a Cartan decomposition. Let $K \subset G$ be the maximal compact subgroup with Lie algebra $\mathfrak{k}$. Thus we have a decomposition

$$ \Phi_M^G = \Phi_M^K \oplus \Phi_M^\mathfrak{p} $$

where $\Phi_M^K : M \to \mathfrak{k}^*$ is a moment map relative to the action of $K$ on $(M, \Omega_M)$, and $\Phi_M^\mathfrak{p} : M \to \mathfrak{p}^*$ is $K$-equivariant.

The $K$-invariant scalar product $\|X\|^2 = -b(X, \Theta(X))$ on $\mathfrak{g}$ induces an identification $\xi \mapsto \tilde{\xi}$, $\mathfrak{g}^* \simeq \mathfrak{g}$ defined by $(\tilde{\xi}, X) := \langle \xi, X \rangle$ for $\xi \in \mathfrak{g}^*$ and $X \in \mathfrak{g}$. We still denote $\|\xi\|^2 := \|\tilde{\xi}\|^2$ the corresponding scalar product on $\mathfrak{g}^*$.

Let $\kappa^G$, $\kappa^K$ and $\kappa^\mathfrak{p}$ be respectively the Hamiltonian vectors fields of the $K$-invariant functions $\frac{1}{2}\|\Phi_M^G\|^2$, $\frac{1}{2}\|\Phi_M^K\|^2$, and $\frac{1}{2}\|\Phi_M^\mathfrak{p}\|^2$. The relations (5) give that

$$ (6) \qquad \kappa^{-}(m) = [\tilde{\Phi}_{M}^{-}(m)]_{M}(m), \quad \forall m \in M, $$

for $- \in \{G, K, \mathfrak{p}\}$.

### 2.1. Proper actions.
In this section we suppose **C1**: the action of $G$ on $M$ is proper<fnref n="2" />. We have then the fundamental fact.

**Lemma 2.1.** • The map $\Phi_M^\mathfrak{p} : M \to \mathfrak{p}^*$ is a $K$-equivariant submersion, so for any $a \in \mathfrak{p}^*$, the fiber $Y_a := (\Phi_M^\mathfrak{p})^{-1}(a)$ is either empty or a sub-manifold of $M$.

• The set of critical points of $\|\Phi_M^\mathfrak{p}\|^2 : M \to \mathbb{R}$ is $Y_0 := (\Phi_M^\mathfrak{p})^{-1}(0)$.

*Proof.* Let us prove the first point. Let $m \in M$. Since the tangent map $T\Phi_M^\mathfrak{p}(m) : T_mM \to \mathfrak{p}^*$ satisfies

$$ (7) \qquad \langle \mathbf{T}\Phi_M^\mathfrak{p}(m), X \rangle = -\iota(X_M)\Omega_M|_m, \quad \forall X \in \mathfrak{p}, $$

the orthogonal of the image of $\mathbf{T}\Phi_M^\mathfrak{p}(m)$ is equal to $\mathfrak{p}_m := \{X \in \mathfrak{p} | X_M(m) = 0\}$. As the action of $G$ on $M$ is proper, the stabilizer subgroup $G_m$ is compact. This forces $\mathfrak{p}_m = \text{Lie}(G_m) \cap \mathfrak{p}$ to be reduced to $\{0\}$. Thus $\mathbf{T}\Phi_M^\mathfrak{p}(m)$ is onto and the first

<footnote n="2">²For any compact subset $A$ of $M$ the subset $\{g \in G | g \cdot A \cap A \neq \emptyset\}$ is compact.</footnote>
<table><caption>TABLE 1. Strongly elliptic set</caption><thead><tr><th>G</th><th>g<sub>se</sub><sup>*</sup> ≠ </th></tr></thead><tbody><tr><td>GL(n, C)</td><td>no</td></tr><tr><td>O(n, C)</td><td>no</td></tr><tr><td>SL(n, R)</td><td>n = 2</td></tr><tr><td>SO<sub>o</sub>(p, q)</td><td>pq even</td></tr><tr><td>Sp(n, R)</td><td>yes</td></tr><tr><td>SO<sup>*</sup>(2n)</td><td>yes</td></tr><tr><td>U(p, q)</td><td>yes</td></tr><tr><td>Sp(p, q)</td><td>yes</td></tr></tbody></table>

point is proved. Let $m \in M$ be a critical point of $\|\Phi_M^p\|^2$. The Hamiltonian vector field $\kappa^p$ vanishes at $m$, and (6) tells you that $\Phi_M^p(m) \in \mathfrak{p}_m = \{0\}$. The second point is proved. $\square$

For the remaining part of this section, we consider the $K$-invariant sub-manifold

$$Y := Y_0 \subset M$$

that we suppose non empty. Let us consider the restriction $\Omega_Y$ of the symplectic structure $\Omega_M$ to $Y$. For $y \in Y$, let $\mathfrak{p} \cdot y = \{X_M(y), X \in \mathfrak{p}\} \subset \mathbf{T}_y M$. The tangent space $\mathbf{T}_y Y$ is by definition the kernel of $\mathbf{T}\Phi_{\mathfrak{p}}(y)$. Relations (7) show that

$$(8) \qquad \mathbf{T}_y Y = (\mathfrak{p} \cdot y)^{\perp}$$

where the orthogonal is taken relatively to the symplectic form. Hence the kernel of $\Omega_Y|_y$ is equal to $(\mathfrak{p} \cdot y)^{\perp} \cap \mathfrak{p} \cdot y$. For $X, X' \in \mathfrak{p}$ and $y \in Y$, we have

$$\begin{aligned} \Omega_M(X_M(y), X'_M(y)) &= \langle \Phi_M^G(y), [X, X'] \rangle \\ &= \langle [\Phi_M^K(y), X], X' \rangle. \end{aligned}$$

Hence $(\mathfrak{p} \cdot y)^{\perp} \cap \mathfrak{p} \cdot y \simeq \mathfrak{g}_{\xi} \cap \mathfrak{p}$ for $\xi = \Phi_M^K(y)$. Note that for $\xi \in \mathfrak{k}$, we have $\mathfrak{g}_{\xi} = \mathfrak{g}_{\xi} \cap \mathfrak{k} \oplus \mathfrak{g}_{\xi} \cap \mathfrak{p}$. We have then proved

**Lemma 2.2.** Let $y \in Y$. The 2-form $\Omega_Y|_y$ is non degenerated if and only if $\mathfrak{g}_{\xi} \subset \mathfrak{k}$ for $\xi = \Phi_M^K(y)$.

We have a canonical $G$-equivariant map $\pi : G \times_K Y \to M$ that sends $[g, y]$ to $g \cdot y$. Following Weinstein [46], we consider the $G$-invariant open subset

$$(9) \qquad \mathfrak{g}_{se}^* = \{\xi \in \mathfrak{g}^* | G_{\xi} \text{ is compact}\}$$

of strongly elliptic elements.

**Example 2.3.** For the group $G = SL_2(\mathbb{R})$, the set $\mathfrak{g}_{se}^*$ is equal, through the trace, to the cone $\{X \in sl_2(\mathbb{R}) | \det(X) > 0\}$.

Let $\mathfrak{t}$ be the Lie algebra of a maximal torus in $K$. Weinstein proves that the open subset $\mathfrak{g}_{se}^*$ is non-empty if and only if $\mathfrak{t}$ is a Cartan sub-algebra of $\mathfrak{g}$ [46]. See the Table 1. We note that $\mathfrak{k}_{se}^* := \mathfrak{g}_{se}^* \cap \mathfrak{k}^*$ is equal to $\{\xi \in \mathfrak{k}^* | G_{\xi} \subset K\}$ and that

$$(10) \qquad \mathfrak{g}_{se}^* = \mathrm{Ad}^*(G) \cdot \mathfrak{k}_{se}^*.$$

Let us consider the invariant open subsets $M_{se} = (\Phi_M^G)^{-1}(\mathfrak{g}_{se}^*) \subset M$ and $Y_{se} := Y \cap M_{se} \subset Y$.
**Lemma 2.4.** • The 2-form $\Omega_Y$ is non-degenerated on $Y_{se}$.

• The action of the group $K$ on $(Y_{se}, \Omega_{Y_{se}})$ is Hamiltonian, with moment map $\Phi_{Y_{se}}^K$ that is equal to the restriction of $\Phi_M^G$ to $Y_{se}$.

• The map $\pi$ induces a $G$-equivariant diffeomorphism $\pi_{se} : G \times_K Y_{se} \to M_{se}$.

*Proof.* The first point is a direct consequence of Lemma 2.2. The second point is immediate. Let us check the last point.

Relation (10) shows that $\pi_{se}$ is onto. Let $[g_0, y_0]$, $[g_1, y_1]$ such that $g_0 \cdot y_0 = g_1 \cdot y_1$. Then by taking the image by the moment map, we get $\mathrm{Ad}^*(g_0)\xi_0 = \mathrm{Ad}^*(g_1)\xi_1$ where the $\xi_k = \Phi_M^K(y_k)$ belong to $\mathfrak{t}_{se}^*$. Let $h = g_1^{-1}g_0 \in G$. We have $\mathrm{Ad}^*(h)\xi_0 = \xi_1$, and $\mathrm{Ad}^*(\Theta(h))\xi_0 = \xi_1$ by taking the Cartan involution. Finally $h^{-1}\Theta(h) \in G_{\xi_0}$. Since $G_{\xi_0} \subset K$, we get that $h \in K$, and finally that $[g_0, y_0] = [g_1, y_1]$ in $G \times_K Y_{se}$. $\square$

Let us denote by $\Omega_{M_{se}}$ the restriction of the symplectic form $\Omega_M$ on the open subset $M_{se}$. We will now finish this section by giving a simple expression of the pull-back $\pi_{se}^*(\Omega_{M_{se}}) \in \mathcal{A}^2(G \times_K Y_{se})$.

Let $\theta^G \in \mathcal{A}^1(G) \otimes \mathfrak{g}$ be the canonical connexion 1-form relative to the $G$-action by right translations : $\iota(X^r)\theta^G = X$, $\forall X \in \mathfrak{g}$, where $X^r(g) = \frac{d}{dt}(ge^{tX})|_0$. Let $\theta^K \in \mathcal{A}^1(G) \otimes \mathfrak{k}$, the composition of $\theta^G$ with the orthogonal projection $X \to X_\mathfrak{k}$ from $\mathfrak{g}$ to $\mathfrak{k}$. We will use the $G \times K$-invariant 1-form on $G \times Y_{se}$ defined by $\langle \theta^K, \Phi_{Y_{se}}^K \rangle$.

Note that the space of differentials forms on $G \times_K Y_{se}$ admits a canonical identification with the space of $K$-basic differentials forms on $G \times Y_{se}$.

**Proposition 2.5.** The 2-form $\pi_{se}^*(\Omega_{M_{se}})$ is equal to the $K$-basic, $G$-invariant, 2-form $\Omega_{Y_{se}} - d\langle\Phi_{Y_{se}}^K, \theta^K\rangle$.

*Proof.* Let $\pi_1 : G \times Y_{se} \to M_{se}$ the map that factorizes $\pi_{se}$. By $G$-invariance, we need only to show that $\pi_1^*(\Omega_{M_{se}})$ equals $\Omega_{Y_{se}} - d\langle\Phi_{Y_{se}}^K, \theta^K\rangle$ at the point $(1, y) \in G \times Y_{se}$.

Let $(X', v'), (X, v) \in \mathfrak{g} \times \mathbf{T}_y = \mathbf{T}_{(1,y)}(G \times Y_{se})$. We have

$$ 
\begin{align*}
\pi_1^*(\Omega_{M_{se}})\big((X', v'), (X, v)\big) & \\
&= \Omega_M(-X'_M(y) + v', -X_M(y) + v) \\
&= \Omega_M(v', v) + \Omega_M(X'_M(y), X_M(y)) - \Omega_M(X'_M(y), v) + \Omega_M(X_M(y), v') \\
&= \Omega_{Y_{se}}(v', v) + \underbrace{\langle\Phi_{Y_{se}}^K(y), [X', X]_\mathfrak{k}\rangle}_{\mathbf{A}} + \underbrace{d\langle\Phi_{Y_{se}}^K, X_\mathfrak{k}'\rangle|_y(v) - d\langle\Phi_{Y_{se}}^K, X_\mathfrak{k}\rangle|_y(v')}_{\mathbf{B}} \\
&= \Omega_{Y_{se}}(v', v) - d\langle\Phi_{Y_{se}}^K, \theta^K\rangle\big((X', v'), (X, v)\big).
\end{align*}
$$

The last equality is due to the fact that $\mathbf{A} = -\langle\Phi_{Y_{se}}^K(y), d\theta^K|_1\rangle(X', X)$ since $d\theta^K((X')^r, (X)^r) = -[X', X]_\mathfrak{k}$, and $\mathbf{B} = -\langle d\Phi_{Y_{se}}^K, \theta^K\rangle((X', v'), (X, v))$. $\square$

<figure><img src="image_3.png" /></figure>

2.2. Proper moment map. In this section we study the Hamiltonian actions of a real reductive group $G$ on a symplectic manifold $(M, \Omega_M)$ that meet the following condition:

C1 The action of $G$ on $M$ is proper,

C2 The moment map $\Phi_M^G : M \to \mathfrak{g}^*$ is a proper map$^3$.

<footnote>$^3$For any compact subset $B \subset \mathfrak{g}^*$ the fiber $\Phi_M^{-1}(B)$ is a compact subset of $M$.</footnote>
The condition C2 imposes that the image of $\Phi_M^G$ is a closed subset of $\mathfrak{g}^*$. Let $\tilde{A}$ be a compact subset of Image($\Phi_G$), and let $A = (\Phi_M^G)^{-1}(\tilde{A})$ be the corresponding compact subset of $M$. We see then that, $\forall g \in G$

$$g \cdot A \cap A \neq \emptyset \iff g \cdot \tilde{A} \cap \tilde{A} \neq \emptyset.$$

Condition C1 tell us then that $\{g \in G | g \cdot A \cap A \neq \emptyset\}$ is compact, hence $\{g \in G | g \cdot \tilde{A} \cap \tilde{A} \neq \emptyset\}$ is compact for any compact set $\tilde{A}$ in the image of $\Phi_M^G$. By taking $\tilde{A}$ equal to a point, we get the following

**Lemma 2.6.** Under the conditions C1 and C2, the image of $\Phi_M^G$ is contained in the open subset $\mathfrak{g}_{se}^* = \{\xi \in \mathfrak{g}^* | G_\xi$ is compact$\}$ of strongly elliptic elements. In particular, the image of $\Phi_M^G$ does not contain $0 \in \mathfrak{g}^*$.

The previous Lemma gives a strong condition on the reductive Lie group $G$: it may acts in an Hamiltonian fashion on a symplectic manifold, properly and with a proper moment map only if $\mathfrak{g}_{se}^* \neq \emptyset$. So $G$ can not be for example $SL_n(\mathbb{R})$ for $n \ge 3$ or a complex reductive Lie group (see Table 1).

If we use the last section we see that $M = M_{se}$. We summarize with the following

**Proposition 2.7.** • The set $Y$ is a $K$-invariant symplectic sub-manifold of $M$, with proper moment map $\Phi_Y^K$ equal to the restriction of $\Phi_M^G$ to $Y$.

• The manifold $G \times_K Y$ carries an induced symplectic structure $\Omega_Y - d\langle\Phi_Y^K, \theta^K\rangle$. The corresponding moment map is $[g, y] \mapsto g \cdot \Phi_Y^K(y)$.

• The map $\pi: G \times_K Y \to M$ is a $G$-equivariant diffeomorphism of Hamiltonian $G$-manifolds.

• The manifold $Y$ is connected.

*Proof.* Thanks to the Cartan decomposition, the third point implies that $\mathfrak{p} \times Y \simeq M$ and then the last point follows. $\square$

Let $\mathfrak{t}$ be the Lie algebra of a maximal torus $T$ in $K$. Note that $\mathfrak{g}_{se}^* \neq \emptyset$ is equivalent to the fact that $\mathfrak{t}$ is a Cartan sub-algebra of $\mathfrak{g}$. Let $\mathfrak{g}_{se}^* = \mathfrak{g}_{se}^* \cap \mathfrak{t}^*$ and $\mathfrak{t}_{se}^* = \mathfrak{g}_{se}^* \cap \mathfrak{t}^*$. We have $\mathfrak{g}_{se}^* = \mathrm{Ad}^*(G) \cdot \mathfrak{t}_{se}^* = \mathrm{Ad}^*(G) \cdot \mathfrak{t}_{se}^*$.

Let $\wedge^* \subset \mathfrak{t}^*$ be the weight lattice : $\alpha \in \wedge^*$ if $i\alpha$ is the differential of a character of $T$. Let $\mathfrak{R} \subset \wedge^*$ be the set of roots for the action of $T$ on $\mathfrak{g} \otimes \mathbb{C}$. We have $\mathfrak{R} = \mathfrak{R}_c \cup \mathfrak{R}_n$ where $\mathfrak{R}_c$ and $\mathfrak{R}_n$ are respectively the set of roots for the action of $T$ on $\mathfrak{k} \otimes \mathbb{C}$ and $\mathfrak{p} \otimes \mathbb{C}$. We fix a system of positive roots $\mathfrak{R}_c^+$ in $\mathfrak{R}_c$: let $\mathfrak{t}_+^* \subset \mathfrak{t}^*$ be the corresponding Weyl chamber. Let $W = W(K, T)$ be the Weyl group. We have then

$$ \begin{aligned} \mathfrak{t}_{se}^* &= W \cdot (\mathfrak{t}_{se}^* \cap \mathfrak{t}_+^*) &&= W \cdot \{\xi \in \mathfrak{t}_+^* | (\xi, \alpha) \neq (0, 0), \forall \alpha \in \mathfrak{R}_n\} \\ &= W \cdot (C_1 \cup \cdots \cup C_N), \end{aligned} $$

where each $C_j$ is an open cone of the Weyl chamber.

We recover the following result due to Weinstein [46].

**Theorem 2.8.** • The Kirwan set $\Delta_K(Y) := \mathrm{Image}(\Phi_Y^K) \cap \mathfrak{t}_+^*$ is a closed convex locally polyhedral subset contained in one cone $C_j$.

• We have $\mathrm{Image}(\Phi_M^G)/\mathrm{Ad}^*(G) \simeq \Delta_K(Y)$.
Proof. Since the moment map $\Phi_M^K$ is proper and $Y$ is connected, the Convexity Theorem [2, 18, 25, 28] tells us that the Kirwan set $\Delta_K(Y)$ is a closed, convex, locally polyhedral, subset of the Weyl chamber. On the other hand, we know that the image of $\Phi_M^K$ belongs to $t_{se}^*$. Then $\Delta_K(Y) \subset C_1 \cup \cdots \cup C_N$, but since $\Delta_K(Y)$ is convex we have $\Delta_K(Y) \subset C_j$ for a unique cone $C_j$. The last point is obvious since the isomorphism $\pi: G \times_K Y \to M$ satisfies $\Phi_M^G \circ \pi([g, y]) = g \cdot \Phi_Y^K(y)$. $\square$

We finish this section, with the following

**Theorem 2.9.** Let $(M, \Omega_M, \Phi_M^G)$ be an Hamiltonian $G$-manifold.

* If the $G$-action on $M$ is proper, $\Phi_M^G$ is proper if and only if $\Phi_M^K$ is proper.

* Under the conditions C1 and C2, we have

$$ \emptyset \neq \mathrm{Cr}(\|\Phi_M^G\|^2) = \mathrm{Cr}(\|\Phi_M^K\|^2) = \mathrm{Cr}(\|\Phi_Y^K\|^2) \subset Y. $$

*Proof.* Let us prove the first point. As $\|\Phi_M^G\| \ge \|\Phi_M^K\|$ one implication trivially holds. Suppose now that $\Phi_M^K$ is proper. Thanks to Propositions 2.7 and 2.8, we know that $M = G \times_K Y$ where $Y$ is a $K$-Hamiltonian manifold, with proper moment map $\Phi_Y^K$, and with Kirwan set $\Delta_K(Y)$ being a closed set in $t_{se}^*$. Let $R > 0$. We consider

* $M_{\le R} = \{m \in M \mid \|\Phi_M^K(m)\|^2 \le R\}$,

* $Y_{\le R} = \{y \in Y \mid \|\Phi_Y^K(y)\|^2 \le R\}$ which is a compact subset of $Y$,

* $\bar{K} = \Delta_K(Y) \cap \{\xi \in t^* \mid \|\xi\|^2 \le R\}$ which is a compact subset of $t_{se}^*$,

* $c(\mathcal{K}) = \inf_{\substack{\xi \in \bar{K} \\ \alpha \in \mathfrak{R}_n}} \frac{|(\alpha, \xi)|^2}{2\|\xi\|}$ which is strictly positive.

We have to show that $M_{\le R}$ is a compact subset of $M$. Take $m = [ke^X, y]$, with $k \in K$ and $X \in \mathfrak{p}$. Since $\Phi_M^G(m) = ke^X \cdot \Phi_Y^K(y)$, we have

$$ \|\Phi_M^K(m)\|^2 \ge -b(\Phi_M^G(m), \Phi_M^G(m)) = \|\Phi_Y^K(y)\|^2 $$

$$ \|\Phi_M^K(m)\|^2 = \| [e^X \cdot \Phi_Y^K(y)]_{t^*} \|^2. $$

Hence if $m = [ke^X, y] \in M_{\le R}$, we have $y \in Y_{\le R}$ and then $\Phi_Y^K(y) = k_o \cdot \xi$ for some $k_o \in K$ and $\xi_o \in \mathcal{K}$. Then we have, for $X' = k_o^{-1} \cdot X \in \mathfrak{p}$,

$$ \begin{aligned} \|\Phi_M^K(m)\| &= \| [e^{X'} \cdot \xi_o]_{t^*} \| \ge \frac{1}{\|\xi_o\|} (e^{X'} \cdot \xi_o, \xi_o) \\ &= \frac{1}{\|\xi_o\|} \sum_{n \in \mathbb{N}} \frac{1}{2n!} \|\mathrm{ad}^*(X')^n \xi_o\|^2 \\ &\ge \frac{1}{2\|\xi_o\|} \|\mathrm{ad}^*(X')\xi_o\|^2 \ge c(\mathcal{K}) \|X\|^2. \end{aligned} $$

Thus if $m = [ke^X, y] \in M_{\le R}$, the vector $X$ is bounded and $y$ belongs to the compact subset $Y_{\le R}$. This proves that $M_{\le R}$ is compact.

Let us concentrate to the last point. First we note that since the map $\|\Phi_M^G\|^2: M \to \mathbb{R}$ is proper, its infimum is reached, and so $\mathrm{Cr}(\|\Phi_M^G\|^2) \ne \emptyset$. Let $- \in \{G, K\}$. Thanks to (6), we know that

$$ m \in \mathrm{Cr}(\|\Phi_M^-\|^2) \iff \kappa^-(m) = 0 \iff \tilde{\Phi}_M^-(m) \in \mathfrak{g}_m. $$

Since $\mathfrak{g}_m \subset \mathfrak{g}_\xi$ with $\Phi_M^G(m) = \xi = \xi_t \oplus \xi_p$, we have $m \in \mathrm{Cr}(\|\Phi_M^-\|^2)$ only if $[\tilde{\xi}_p, \tilde{\xi}] = 0$. Since $\xi$ is strongly elliptic the last condition imposes that $\xi_p = 0$. We
have proved that $\mathrm{Cr}(\|\Phi_M^K\|^2)$ and $\mathrm{Cr}(\|\Phi_M^G\|^2)$ are both contained in $\{\Phi_M^P = 0\} = Y$. We have $\kappa^G = \kappa^K + \kappa^P$ and the vector field $\kappa^P$ vanishes on $Y$. Finally we see that

$$ \mathrm{Cr}(\|\Phi_M^G\|^2) = \mathrm{Cr}(\|\Phi_M^K\|^2) = \{y \in Y \mid [\tilde{\Phi}_M^K(y)]_M(y) = 0\} \\ = \mathrm{Cr}(\|\Phi_Y^K\|^2). $$

The last equality is due to the fact that $\Phi_Y^K$ is the restriction of $\Phi_M^K$ to $Y$. $\square$

<figure><img src="image_4.png" /></figure>

**2.3. Criterion.** We have seen in Theorem 2.9 a situation where the properness property of the moment maps $\Phi_M^G$ and $\Phi_M^K$ are equivalent. In this section, we start with a symplectic manifold $(M, \Omega_M)$ admitting an Hamiltonian action of a compact connected Lie group $K$. We suppose that the moment map $\Phi_M^K$ is proper. Let $K' \subset K$ be a closed subgroup. The aim of the section is to give a criterion under which the induced moment map $\Phi_M^{K'}$ is still proper. We start by recalling basic facts concerning the notion of asymptotic cone.

To any non-empty subset $C$ of a real vector space $E$, we define its asymptotic cone $\mathrm{As}(C) \subset E$ as the set formed by the limits $y = \lim_{k \to \infty} t_k y_k$ where $(t_k)$ is a sequence of non-negative reals converging to 0 and $y_k \in C$. Note that $\mathrm{As}(C) = \{0\}$ if and only if $C$ is compact.

We recall the following basic facts.

**Lemma 2.10.** Let $C_i$, $i = 0, 1$ be closed and convex subsets of $E$.

* We have $C_i + \mathrm{As}(C_i) \subset C_i$.

* If $C_0 \cap C_1$ is non-empty we have $\mathrm{As}(C_0) \cap \mathrm{As}(C_1) = \mathrm{As}(C_0 \cap C_1)$.

* If $C_0 \cap C_1$ is non-empty and compact, we have $\mathrm{As}(C_0) \cap \mathrm{As}(C_1) = \{0\}$

*Proof.* Let us check the first point. Take $z \in C_i$ and $y = \lim_{k \to \infty} t_k y_k$ an element of $\mathrm{As}(C_i)$. Then $z + y = \lim_{k \to \infty} (1 - t_k)z + t_k y_k$. Since $(1 - t_k)z + t_k y_k \in C_i$ if $t_k \le 1$, we know that $z + y \in C_i$ since $C_i$ is closed.

The inclusion $\mathrm{As}(C_0 \cap C_1) \subset \mathrm{As}(C_0) \cap \mathrm{As}(C_1)$ follows from the inclusions $C_0 \cap C_1 \subset C_i$. Let $z \in C_0 \cap C_1$ and $y \in \mathrm{As}(C_0) \cap \mathrm{As}(C_1)$. Thanks to the first point we know that $z + \mathbb{R}^\ge 0 y \subset C_0 \cap C_1$. Then $y = \lim_{t \to 0^+} t(z + t^{-1}y) \in \mathrm{As}(C_0 \cap C_1)$. The second point is proved and the last point is a direct consequence of the second one. $\square$

<figure><img src="image_5.png" /></figure>

The following Proposition is a useful tool for finding proper moment map. For a closed subgroup $K'$ of $K$, we denote $\pi_{\mathfrak{k}', \mathfrak{k}} : \mathfrak{k}^* \to (\mathfrak{k}')^*$ the projection which is the dual of the inclusion $\mathfrak{k}' \hookrightarrow \mathfrak{k}$. The kernel $\pi_{\mathfrak{k}', \mathfrak{k}}(0)$ is denoted $(\mathfrak{k}')^\perp$.

**Proposition 2.11.** Let $(M, \Omega_M)$ be an Hamiltonian $K$-manifold with a proper moment map $\Phi_M^K$. Let $\Delta_K(M)$ be its Kirwan polyhedron. Let $K' \subset K$ be a closed subgroup. Then the following statement are equivalent

a) the moment map $\Phi_M^{K'} = \pi_{\mathfrak{k}', \mathfrak{k}} \circ \Phi_M^K$ is proper,

b) $\mathrm{As}(\Delta_K(M)) \cap K \cdot (\mathfrak{k}')^\perp = \{0\}$,

c) there exists $\varepsilon > 0$, such that the inequality $\|\Phi_M^{K'}\| \ge \varepsilon \|\Phi_M^K\| - \varepsilon^{-1}$ holds on $M$.

*Proof.* If c) does not hold we have a sequence $m_i \in M$ such that $\|\Phi_M^{K'}(m_i)\| \le \frac{1}{i} \|\Phi_M^K(m_i)\| - i$, for all $i \ge 1$. Then $\|\Phi_M^K(m_i)\|$ tends to infinity and $\frac{\|\Phi_M^{K'}(m_i)\|}{\|\Phi_M^K(m_i)\|}$ tends to zero. We write $\Phi_M^K(m_i) = k_i \cdot y_i$ with $k_i \in K$ and $y_i \in \Delta_K(M)$. The
sequence $\pi_{\mathfrak{t}',\mathfrak{t}}(k_i \cdot \frac{y_i}{\|y_i\|})$ converges to 0. Here we can assume that the sequence $k_i$ converge to $k \in K$, and that the sequence $\frac{y_i}{\|y_i\|}$ converge to $y \in \text{As}(\Delta_K(M))$. We get then that $\pi_{\mathfrak{t}',\mathfrak{t}}(k \cdot y) = 0$. In other words, $y$ is a non-zero element in $\text{As}(\Delta_K(M)) \cap K \cdot (\mathfrak{t}')^{\perp}$. We have proved $b) \implies c)$.

The implication $c) \implies a)$ is obvious. Let us prove the last implication $a) \implies b)$. First we note that the properness of $\Phi_M^K$ implies that the projection $\pi_{\mathfrak{t}',\mathfrak{t}}$ is proper when restricted to the closed subset $\text{Image}(\Phi_M^K) = K \cdot \Delta_K(M)$. Let $k \in K$ and $\xi_o \in k \cdot \Delta_K(M)$. Then

$$k \cdot \Delta_K(M) \cap \xi_o + (\mathfrak{t}')^{\perp} \subset \text{Image}(\Phi_M^K) \cap \pi_{\mathfrak{t}',\mathfrak{t}}^{-1}(\pi_{\mathfrak{t}',\mathfrak{t}}(\xi_o))$$

is non-empty and compact. If we apply the last point of Lemma 2.10 to the closed and convex sets $k \cdot \Delta_K(M)$ and $\xi_o + (\mathfrak{t}')^{\perp}$ we get that

$$\text{As}(k \cdot \Delta_K(M)) \cap \text{As}(\xi_o + \mathfrak{t}^{\perp}) = k \cdot \text{As}(\Delta_K(M)) \cap (\mathfrak{t}')^{\perp}$$

is reduced to $\{0\}$. So we have proved that $\text{As}(\Delta_K(M)) \cap k \cdot (\mathfrak{t}')^{\perp} = \{0\}$ for any $k \in K$. $\square$

**Remark 2.12.** When $M$ is a symplectic vector space $(E, \Omega_E)$, the moment map $\Phi_E^K : E \to \mathfrak{t}^*$ is quadratic. Then $\Phi_E^K$ is proper if and only if $(\Phi_E^K)^{-1}(0) = \{0\}$.

## 2.4. Kostant-Souriau line bundle.
In the Kostant-Souriau framework, an Hamiltonian $G$-manifold $(M, \Omega_M, \Phi_M^G)$ is pre-quantized if there is an equivariant Hermitian line bundle $L_M$ with an invariant Hermitian connection $\nabla_M$ such that

$$(11) \qquad \mathcal{L}(X) - \iota(X_M)\nabla_M = i\langle\Phi_M^G, X\rangle \quad \text{and} \quad (\nabla_M)^2 = -i\Omega_M,$$

for every $X \in \mathfrak{g}$.

The data $(L_M, \nabla_M)$ is also called a Kostant-Souriau line bundle. Note that conditions (11) imply via the equivariant Bianchi formula the relations (5).

We suppose now that conditions C1 and C2 hold. Then $M = G \times_K Y$ where $Y \subset M$ is the $K$-invariant symplectic sub-manifold defined in Section 2.2. Let $(L_M, \nabla_M)$ be a Kostant-Souriau line bundle on $M$. We denote $L_Y$ the restriction of the line bundle $L_M$ on $Y$. The connection $\nabla_M$ induces a $K$-invariant connection $\nabla_Y$ on $L_Y \to Y$, and we check easily that $(L_Y, \nabla_Y)$ is a Kostant-Souriau line bundle on $Y$.

Reciprocally, if $(L_Y, \nabla_Y)$ is a Kostant-Souriau line bundle on $(Y, \Omega_Y, \Phi_Y^K)$, we define on $M$ the line bundle $L_M := (G \times L_Y)/K$ equipped with the connection

$$\nabla_M := \nabla_Y + d^G + i\langle\Phi_Y^K, \theta^K\rangle,$$

where $d^G$ is the de Rham differential on $G$. Since $\Omega_M = \Omega_Y - d\langle\Phi_Y^K, \theta^K\rangle$, we check easily that $(L_M, \nabla_M)$ is a $G$-equivariant Kostant-Souriau line bundle on $(M, \Omega_M, \Phi_M^G)$.

## 2.5. The case of elliptic orbits.
In this section, we consider the examples given by the elliptic coadjoint orbits of $G$, that is $M := G \cdot \lambda$ for some $\lambda \in \mathfrak{k}^*$. The Kirillov-Kostant-Souriau symplectic structure $\Omega_M$ is defined by the relation

$$\Omega_M|_m(X_M)_m, Y_M)_m = \langle m, [X, Y]\rangle,$$

for $m \in M$ and $X, Y \in \mathfrak{g}$. The corresponding moment map relatively to the action of $G$ on $G \cdot \lambda$ is the inclusion $\Phi_M^G : G \cdot \lambda \hookrightarrow \mathfrak{g}^*$.

**Lemma 2.13.** The moment maps $\Phi_M^G$ and $\Phi_M^K$ are proper.
*Proof.* The inclusion $\Phi_M^G$ is proper since the elliptic orbit $M = G \cdot \lambda$ is closed in $g^*$. If we use the relations $\|\lambda\|^2 = -b(\Phi_M^G, \Phi_M^G) = \|\Phi_M^K\|^2 - \|\Phi_M^p\|^2$ and $\|\Phi_M^G\|^2 = \|\Phi_M^K\|^2 + \|\Phi_M^p\|^2$, we get $\|\Phi_M^K\|^2 = \frac{1}{2}(\|\lambda\|^2 + \|\Phi_M^G\|^2)$. The properness of $\|\Phi_M^K\|^2$ then follows. $\square$

We work now with an elliptic coadjoint orbit of $G$, $G \cdot \lambda$, such that the stabilizer subgroup $G_\lambda$ is compact. Then the action of $G$ (and those of any closed subgroup) on $G \cdot \lambda$ is proper.

Let $t$ be the Lie algebra of a maximal torus $T$ in $K$. Our hypothesis concerning the compactness of $G_\lambda$ imposes $t$ to be a Cartan sub-algebra of $g$. Let $\mathfrak{R}$ be the set of roots for the action of $t$ on $g \otimes \mathbb{C}$. We have $\mathfrak{R} = \mathfrak{R}_c \cup \mathfrak{R}_n$ where $\mathfrak{R}_c$ and $\mathfrak{R}_n$ are respectively the set of roots for the action of $t$ on $\mathfrak{k} \otimes \mathbb{C}$ and $\mathfrak{p} \otimes \mathbb{C}$. For the remaining part of this section, we fix a system of positive roots $\mathfrak{R}_{c,+}$ in $\mathfrak{R}_c$: let $t_+^* \subset t^*$ be the corresponding Weyl chamber.

So $\lambda$ is chosen in the Weyl chamber $t_+^*$, away from the non-compact wall : $(\alpha, \lambda) \neq 0$ for all $\alpha \in \mathfrak{R}_n$. Thanks to Lemma 2.13, we know that the moment map $\Phi_{G_\lambda}^K$ relative to a maximal compact subgroup $K \subset G$ is proper. The Convexity theorem tells us that the set

$$ \Delta_K(G \cdot \lambda) := \text{Image}(\Phi_M^K) \cap t_+^* $$

is a closed convex locally polyhedral subset of $t^*$. The results of Duflo-Heckman-Vergne [14] shows that in fact $\Delta_K(G \cdot \lambda)$ is defined by a finite number of inequalities. In this paper, we call $\Delta_K(G \cdot \lambda)$ the Kirwan polyhedron.

We consider now a connected reductive subgroup $G' \subset G$, such that a Cartan involution $\Theta$ for $G$ leaves $G'$ invariant. Then we have Cartan decompositions $g' = t' \oplus p'$ and $g = \mathfrak{k} \oplus \mathfrak{p}$, with $t' \subset \mathfrak{k}$ and $p' \subset \mathfrak{p}$. Let $K \subset G$ and $K' \subset G'$ be the corresponding maximal subgroups.

Consider now the action of $G'$ on $(G \cdot \lambda, \Omega_{G \cdot \lambda})$. The moment map $\Phi_{G_\lambda}^{G'}$ is the composition of the inclusion $G \cdot \lambda \hookrightarrow g^*$ with the orthogonal projection $\pi_{g',g} : g^* \to (g')^*$. Note that $Y := (\Phi_M^G)^{-1}(t^*)$ is non-empty since it contains $\pi_{g',g}(\lambda)$.

We are looking to connected reductive subgroups $G' \subset G$ such that the moment map $\Phi_{G_\lambda}^{G'}$ is proper. Theorem 2.9 shows that is equivalent to look at compact subgroups $K' \subset K$ such that the moment map $\Phi_{G_\lambda}^{K'}$ is proper. Thanks to Proposition 2.11, we have the following criterium

**Proposition 2.14.** • The moment map $\Phi_{G_\lambda}^{G'}$ is proper if and only if $\text{As}(\Delta_K(G \cdot \lambda)) \cap K \cdot (t')^\perp = \{0\}$.

We want to stress a property which is peculiar to the reductive Lie groups (in comparison with the nilpotent one).

**Proposition 2.15.** Let $O \subset g^*$ be a strongly elliptic orbit. Let $G' \subset G$ be a connected reductive subgroup such that $\Phi_O^{G'} = \pi_{g',g} : O \to (g')^*$ is proper. Then for any coadjoint orbit $O' \subset (g')^*$ the reduced space

$$ (12) \qquad O \cap \pi_{g',g}^{-1}(O')/G' $$

is connected.

*Proof.* Since $\Phi_O^{G'}$ is proper, we have a decomposition $O = G' \times_{K'} Y'$, where $Y'$ is a connected sub-manifold. Then $O \cap \pi_{g',g}^{-1}(O')$ is empty if $O'$ is not elliptic. And if
$O' = G' \cdot \mu$ with $\mu \in (\mathfrak{t}')^*$, we see that $O \cap \pi_{\mathfrak{g}',\mathfrak{g}}^{-1}(O')/G' \simeq (\Phi_Y^K)^{-1}(K \cdot \mu)/K$ which is connected since $\Phi_Y^K$ is proper (see the Convexity Theorem [2, 18, 25, 28]). $\square$

In general the Kirwan polyhedron $\Delta_K(G \cdot \lambda)$ is not known, but we can use at least the following observation. Let $\mathfrak{R}_n(\lambda)$ be the set of non-compact roots $\alpha$ such that $(\alpha, \lambda) > 0$. Let us consider the following cone in $\mathfrak{t}^*$:

$$ \mathcal{C}(\lambda) := \sum_{\alpha \in \mathfrak{R}_n(\lambda)} \mathbb{R}^{\ge 0} \alpha. $$

**Lemma 2.16.** For the Kirwan polyhedron we have

$$ \Delta_K(G \cdot \lambda) \subset \{\lambda + \mathcal{C}(\lambda)\}, \quad \text{and then} \quad \text{As } (\Delta_K(G \cdot \lambda)) \subset \mathcal{C}(\lambda). $$

**Corollary 2.17.** The moment map $\Phi_{G,\lambda}^{G'}$ is proper if $\mathcal{C}(\lambda) \cap K \cdot (\mathfrak{t}')^{\perp} = \{0\}$.

*Proof.* Let $C_\lambda$ be the cone tangent to $\Delta_K(G \cdot \lambda)$ at $\lambda$:

$$ C_{\lambda} = \mathbb{R}^{\ge 0} \cdot \{\xi - \lambda, \xi \in \Delta_{K}(G \cdot \lambda)\} \subset \mathfrak{t}^{*}. $$

We have to show that $C_\lambda$ is contained in $\mathcal{C}(\lambda)$. Thanks to the result of Sjamaar [40], we know that $C_\lambda$ is determined by a local Hamiltonian model near $K \cdot \lambda \subset G \cdot \lambda$.

The maximal torus $T$ of $K$ is still a maximal torus for the stabiliser subgroup $K_\lambda$: let $t_{\lambda,+}^*$ be a Weyl chamber for $(K_\lambda, T)$ which contains $t_+^*$. Here, we consider the vector space $\mathfrak{p}$ equipped with the linear symplectic structure $\Omega_\lambda(X, Y) := \langle \lambda, [X, Y] \rangle$. The group $K_\lambda$ acts in a Hamiltonian fashion on $(\mathfrak{p}, \Omega_\lambda)$. Let us denote by

$$ \Delta_{K_\lambda}(\mathfrak{p}) \subset \mathfrak{t}_{\lambda,+}^* $$

the corresponding Kirwan polytope (which is a rational cone). Since the stabiliser of the point $\lambda \in M := G \cdot \lambda$ coincides with the stabiliser subgroup $K_\lambda$ of its image by the moment map $\Phi_M^K$, the local form of Marle [30] and Guillemin and Sternberg [20] tells us that $M$ is symplectomorphic with $K \times_{K_\lambda} \mathfrak{p}$ in a neighbourhood of $K \cdot \lambda$. Theorem 6.5 of [40] tells us then that $C_\lambda = \Delta_{K_\lambda}(\mathfrak{p})$.

Let us consider the Hamiltonian action of the torus $T$ on $(\mathfrak{p}, \Omega_\lambda)$. Let $J_\lambda$ be an invariant complex structure on $\mathfrak{p}$ which is compatible with $\Omega_\lambda$: we can check that the weights of the $T$-action on $(\mathfrak{p}, J_\lambda)$ are $-\alpha$, for $\alpha \in \mathfrak{R}_n(\lambda)$. Hence the image $\Delta_T(\mathfrak{p})$ of the moment map is equal to the cone generated by the weights $\alpha \in \mathfrak{R}_n(\lambda)$. Finally we have proved that

$$ C_{\lambda} = \Delta_{K_{\lambda}}(\mathfrak{p}) \subset \Delta_{T}(\mathfrak{p}) = \mathcal{C}(\lambda). $$

$\square$

### 3. QUANTIZATION COMMUTES WITH REDUCTION

Let $G$ be a connected real reductive Lie group and let $K$ be a maximal connected compact subgroup. Let $c_\mathfrak{t}, c_\mathfrak{g}$ be respectively the center of $\mathfrak{t}$ and $\mathfrak{g}$. In all the section we assume that the group $G$ satisfies the following condition

$$ (13) \qquad Z_{\mathfrak{g}}(c_{\mathfrak{t}}) = \mathfrak{t}, $$

i.e. the centralizer of $c_\mathfrak{t}$ in $\mathfrak{g}$ coincides with $\mathfrak{t}$. Hence $c_\mathfrak{g} \subset c_\mathfrak{t} \subset \mathfrak{t}$.

We make the choice of a maximal torus $T$ in $K$ with Lie algebra $\mathfrak{t}$. Note that (13) forces $\mathfrak{t}$ to be a Cartan sub-algebra of $\mathfrak{g}$. Let $\mathfrak{R} = \mathfrak{R}_c \cup \mathfrak{R}_n$ be the set of roots. We fix a system of positive roots $\mathfrak{R}_c^+$ in $\mathfrak{R}_c$. We know also that (13) imposes the
existence of elements $z \in c_t$ such that $\mathrm{ad}(z)$ defines a complex structure on $\mathfrak{p}$ (see Section 9 in [26]). For such element $z$, we define

$$ \mathfrak{R}_n(z) := \{ \alpha \in \mathfrak{R}_n | \langle \alpha, z \rangle = 1 \}. $$

which is invariant relatively to the action of the Weyl group $W_K$. The union $\mathfrak{R}_c^+ \cup \mathfrak{R}_n(z)$ defines then a system of positive roots in $\mathfrak{R}$.

We will be interested to the closed $W_K$-invariant cones in $\mathfrak{t}^*$

$$ C_{\mathrm{hol}}(z) := \{ \xi \in \mathfrak{t}^* | (\beta, \xi) \ge 0, \forall \beta \in \mathfrak{R}_n(z) \}, $$

$$ C(z) := \sum_{\beta \in \mathfrak{R}_n(z)} \mathbb{R}^{\ge 0} \beta. $$

We recall some basic facts about them.

**Lemma 3.1.** *We have the following inclusions*

$$ (14) \qquad C(z) \subset C_{\mathrm{hol}}(z) \subset \{ \xi \in \mathfrak{t}^* | \langle \xi, z \rangle \ge 0 \}. $$

*Proof.* Since $(\beta_0, \beta_1) \ge 0$ for any $\beta_k \in \mathfrak{R}_n(z)$, we see that $C(z) \subset C_{\mathrm{hol}}(z)$. For $\xi \in \mathfrak{t}^*$, we have $\langle \xi, z \rangle = -b(\tilde{\xi}, z) = 2 \sum_{\beta \in \mathfrak{R}_n(z)} \langle \beta, \tilde{\xi} \rangle$ with $\langle \beta, \tilde{\xi} \rangle = (\beta, \xi)$. Then $\xi \in C_{\mathrm{hol}}(z)$ implies $\langle \xi, z \rangle \ge 0$. $\square$

**Example 3.2.** *We have the following classical examples:*

<table><thead><tr><th>G</th><th>K</th><th>p</th></tr></thead><tbody><tr><td>Sp(n, ℝ)</td><td>U(n)</td><td>S<sup>2</sup>(ℝ<sup>n</sup>)</td></tr><tr><td>SO<sup>*</sup>(2n)</td><td>U(n)</td><td>Λ<sup>2</sup>ℝ<sup>n</sup></td></tr><tr><td>SO<sub>o</sub>(2, n)</td><td>SO(2) × SO(n)</td><td>ℝ<sup>n</sup></td></tr><tr><td>U(p, q)</td><td>U(p) × U(p)</td><td>M<sub>p,q</sub>(ℝ)</td></tr></tbody></table>

3.1. **Holomorphic coadjoint orbits.** The holomorphic coadjoint orbits are $G \cdot \lambda$ with $\lambda$ in the interior of $C_{\mathrm{hol}}(z)$. These symplectic manifolds possess a $G$-invariant (integrable) complex structure $J_\lambda$ which is compatible with the symplectic structure $\Omega_{G \cdot \lambda}$ (see [38]). Hence $(G \cdot \lambda, \Omega_{G \cdot \lambda}, J_\lambda)$ is a Kähler manifold when $\lambda \in \mathrm{Interior}(C_{\mathrm{hol}}(z))$.

The real $K$-module $\mathfrak{p}$ is equipped with the invariant linear symplectic structure $\Omega_{\mathfrak{p}}(A, B) := -b(z, [A, B])$. We have two families of Hamiltonian $K$-manifold : $K \cdot \lambda \times \mathfrak{p}$ and $G \cdot \lambda$ for $\lambda \in C_{\mathrm{hol}}(z)$. We start with the fundamental fact.

**Proposition 3.3.** Let $\lambda \in \mathrm{Interior}(C_{\mathrm{hol}}(z))$. We have

$$ \text{a)} \quad \Delta_K(G \cdot \lambda) \subset \lambda + C(z) \subset C_{\mathrm{hol}}(z), $$

$$ \text{b)} \quad \Delta_K(G \cdot \lambda) = \Delta_K(K \cdot \lambda \times \mathfrak{p}), $$

$$ \text{c)} \quad \text{As } (\Delta_K(G \cdot \lambda)) = \Delta_K(\mathfrak{p}). $$

*Proof.* Point a) is the translation of Lemma 2.16 since the cone $C(\lambda)$ is equal to $C(z)$. Point b) is proved in [38]. Another proof is given by Deltour in [13], by showing the stronger result that the Hamiltonian $K$-manifolds $G \cdot \lambda$ and $K \cdot \lambda \times \mathfrak{p}$ are symplectomorphic. The point c) follows easily from b). $\square$

**Remark 3.4.** When $G$ is one of the groups appearing in Example 3.2, the generators of the cone $\Delta_K(\mathfrak{p})$ can be defined in term of strongly orthogonal roots (see Section 5 in [38]). Note also that Deltour has completely described the facet of the polytopes $\Delta_K(G \cdot \lambda)$ when $\mathrm{Interior}(C_{\mathrm{hol}}(z))$ (see [12]).
<table><caption>TABLE 2. Involution $\sigma$ such that $\sigma(z) = z$</caption><thead><tr><th>G</th><th>G<sup>σ</sup></th></tr></thead><tbody><tr><td>Sp(n, ℝ)</td><td>Sp(p, ℝ) × Sp(n − p, ℝ)</td></tr><tr><td>Sp(n, ℝ)</td><td>U(p, n − p)</td></tr><tr><td>SO(2, 2n)</td><td>U(1, n)</td></tr><tr><td>SO(2, n)</td><td>SO(2, p) × SO(n − p)</td></tr><tr><td>SO<sup>*</sup>(2n)</td><td>U(p, n − p)</td></tr><tr><td>SO<sup>*</sup>(2n)</td><td>SO<sup>*</sup>(2p) × SO<sup>*</sup>(2n − 2p)</td></tr><tr><td>U(n, n)</td><td>Sp(n, ℝ)</td></tr><tr><td>U(n, n)</td><td>SO<sup>*</sup>(2n)</td></tr><tr><td>U(p, q)</td><td>U(i, j) × U(p − i, q − j)</td></tr></tbody></table>

Let $S^{\bullet}(\mathfrak{p})$ be the symmetric algebra of the complex $K$-module $(\mathfrak{p}, \text{ad}(z))$: it is an admissible representation of $K$. Let $K'$ be a closed connected subgroup of $K$. We denote by $\Phi_{G \cdot \Lambda}^{K'}$ and by $\Phi_{\mathfrak{p}}^{K'}$ the corresponding moment maps. We have the following

**Proposition 3.5.** Let $\lambda \in \text{Interior}(C_{\text{hol}}(z))$. The following assertions are equivalent

a) $\Phi_{G \cdot \lambda}^{K'} : G \cdot \lambda \to (\mathfrak{k}')^*$ is a proper map,

b) $\Delta_K(\mathfrak{p}) \cap K \cdot (\mathfrak{k}')^{\perp} = \{0\}$,

c) $\Phi_{\mathfrak{p}}^{K'} : \mathfrak{p} \to (\mathfrak{k}')^*$ is a proper map,

d) $\{\Phi_{\mathfrak{p}}^{K'} = 0\}$ is reduced to $\{0\}$,

e) $S^{\bullet}(\mathfrak{p})$ is an admissible representation of $K'$.

*Proof.* The equivalences a) $\Leftrightarrow$ b) and b) $\Leftrightarrow$ c) follow from Propositions 2.11 and 3.3. The other equivalences c) $\Leftrightarrow$ d) $\Leftrightarrow$ e) are proved in [36][Section 5]. $\square$

Let us consider the moment map $\Phi_{\mathfrak{p}}^{K} : \mathfrak{p} \to \mathfrak{k}^{*}$. Via the identification $\mathfrak{k}^{*} \simeq \mathfrak{k}$, the moment map $\Phi_{\mathfrak{p}}^{K}$ is defined by

$$ \Phi_{\mathfrak{p}}^{K}(X) = -[X, [z, X]], \quad X \in \mathfrak{p}. $$

Hence we see that $\langle \Phi_{\mathfrak{p}}^{K}, z \rangle : \mathfrak{p} \to \mathbb{R}$ is a proper map taking positive values. This simple fact and Proposition 3.5 gives us the following

**Corollary 3.6.** Let $G'$ be a connected reductive subgroup of $G$, and let $\lambda \in \text{Interior}(C_{\text{hol}}(z))$. The moment map $\Phi_{G \cdot \Lambda}^{G'}$ is proper when the Lie algebra $\mathfrak{g}'$ contains $\mathbb{R}z$.

**Example 3.7.** The condition $\mathbb{R}z \subset \mathfrak{g}'$ is fulfilled in the following cases:

(1) $G' = \text{SO}_o(2, p) \subset G = \text{SO}_o(2, n)$ for $0 \le p \le n$,

(2) $G'$ is the identity component of $G^{\sigma}$, where $\sigma$ is an involution of $G$ such that $\sigma(z) = z$ (see Table 2),

(3) $G'$ is the diagonal in $G := G' \times \cdots \times G'$.
3.2. **Holomorphic discrete series.** Let $\Lambda^* \subset \mathfrak{t}^*$ be the lattice of characters of $T$. We know that the set $\Lambda_+^* := \Lambda^* \cap \mathfrak{t}_+^*$ parametrizes the set $\hat{K}$ of irreducible representations of $K$: for any $\mu \in \Lambda_+^*$, we denote $V_\mu^K$ the irreducible representation of $K$ with highest weight $\mu$.

We will be interested in $C_{\text{hol}}^{\rho}(z) = 2\rho_n(z) + C_{\text{hol}}(z) \subset C_{\text{hol}}(z)$ where $2\rho_n(z)$ is the sum of the roots of $\mathfrak{R}_n(z)$. Let us denote

$$ \hat{G}_{\text{hol}}(z) := \Lambda_+^* \cap C_{\text{hol}}^{\rho}(z). $$

**Theorem 3.8 (Harish-Chandra).** For any $\lambda \in \hat{G}_{\text{hol}}(z)$, there exists an irreducible unitary representation of $G$, denoted $V_\lambda^G$, such that the vector space of $K$-finite vectors is

$$ V_{\lambda}^{G}|_{K} := V_{\lambda}^{K} \otimes S^{\bullet}(\mathfrak{p}). $$

Here $S^{\bullet}(\mathfrak{p})$ is the symmetric algebra of the complex vector space $(\mathfrak{p}, \text{ad}(z))$.

3.3. **Formal geometric quantization.** Let us first recall the definition of the geometric quantization of a smooth and compact Hamiltonian manifold. Then we show a way of extending the notion of geometric quantization to the case of a non-compact Hamiltonian manifold.

Let $K$ be a compact connected Lie group. Let $(M, \Omega_M, \Phi_M^K)$ be a Hamiltonian $K$-manifold which is pre-quantized by the Hermitian line bundle $L_M$ (see Section 2.4).

Let us recall the notion of geometric quantization when $M$ is **compact**. Choose a $K$-invariant almost complex structure $J$ on $M$ which is compatible with $\Omega_M$ in the sense that the symmetric bilinear form $\Omega_M(\cdot, J\cdot)$ is a Riemannian metric. Let $\overline{\partial}_{L_M}$ be the Dolbeault operator with coefficients in $L$, and let $\overline{\partial}_{L_M}^*$ be its (formal) adjoint. The *Dolbeault-Dirac operator* on $M$ with coefficients in $L_M$ is $D_{L_M} = \sqrt{2}(\overline{\partial}_{L_M} + \overline{\partial}_{L_M}^*)$, considered as an elliptic operator from $\mathcal{A}^{0,\text{even}}(M, L_M)$ to $\mathcal{A}^{0,\text{odd}}(M, L_M)$. Let $R(K)$ be the representation ring of $K$.

**Definition 3.9.** The geometric quantization of a compact Hamiltonian $K$-manifold $(M, \Omega_M, \Phi_M^K)$ is the element $\mathcal{Q}_K(M) \in R(K)$ defined as the equivariant index of the Dolbeault-Dirac operator $D_{L_M}$.

Let us consider the case of a **proper** pre-quantized Hamiltonian $K$-manifold $M$: the manifold is (perhaps) **non-compact** but the moment map $\Phi_M^K: M \to \mathfrak{t}^*$ is supposed to be proper. In this setting, we have two ways of extending the geometric quantization procedure.

**First way:** $\mathcal{Q}_K^{-\infty}$. One defines the *formal geometric quantization* of $M$ as an element $\mathcal{Q}_K^{-\infty}(M)$ that belongs to $R^{-\infty}(K) := \text{hom}_{\mathbb{Z}}(R(K), \mathbb{Z})$ [45, 36, 29, 39]. Let us recall the definition.

For any $\mu \in \hat{K}$ which is a regular value of the moment map $\Phi$, the reduced space<fnref n="4" /> (or symplectic quotient)

$$ (15) \qquad M_{\mu} := (\Phi_{M}^{K})^{-1}(K \cdot \mu)/K $$

is a **compact** orbifold equipped with a symplectic structure $\Omega_\mu$. Moreover $L_\mu := (L|_{(\Phi_M^K)^{-1}(\mu)} \otimes \mathbb{C}_{-\mu})/K_\mu$ is a Kostant-Souriau line orbibundle over $(M_\mu, \Omega_\mu)$. The definition of the index of the Dolbeault-Dirac operator carries over to the orbifold

<footnote n="4">⁴The symplectic quotient will be denoted $M_{\mu,K}$ when we need more precise notations.</footnote>
case, hence $Q(M_\mu) \in \mathbb{Z}$ is defined. This notion of geometric quantization extends further to the case of singular symplectic quotients [32, 34]. So the integer $Q(M_\mu) \in \mathbb{Z}$ is well defined for every $\mu \in \hat{K}$: in particular $Q(M_\mu) = 0$ if $\mu$ is not in the Kirwan polytope $\Delta_K(M)$.

**Definition 3.10.** Let $(M, \Omega_M, \Phi_M^K)$ be a proper Hamiltonian $K$-manifold which is pre-quantized by a Kostant-Souriau line bundle $L$. The formal quantization of $(M, \Omega_M, \Phi_M^K)$ is the element of $R^{-\infty}(K)$ defined by

$$Q_K^{-\infty}(M) = \sum_{\mu \in \hat{K}} Q(M_\mu) V_\mu^K.$$

When $M$ is compact, the fact that

$$(16) \qquad Q_K(M) = Q_K^{-\infty}(M)$$

is known as the “quantization commutes with reduction” Theorem. This was conjectured by Guillemin-Sternberg in [19] and was first proved by Meinrenken [31] and Meinrenken-Sjamaar [32]. Other proofs of (16) were also given by Tian-Zhang [42] and the author [34]. For complete references on the subject the reader should consult [41, 43].

One of the main features of the formal geometric quantization $Q^{-\infty}$ is summarized in the following

**Theorem 3.11 ([36]).** • **Restriction to subgroup.** Let $M$ be a pre-quantized Hamiltonian $K$-manifold which is proper. Let $H \subset K$ be a closed connected Lie subgroup such that $M$ is still proper as a Hamiltonian $H$-manifold. Then $Q_K^{-\infty}(M)$ is $H$-admissible and we have $Q_K^{-\infty}(M)|_H = Q_H^{-\infty}(M)$ in $R^{-\infty}(H)$.

• **Product.** Let $M$ and $N$ be pre-quantized Hamiltonian $K$-manifolds with $M$ is proper and $N$ is compact. Then $M \times N$ is a proper pre-quantized Hamiltonian $K$-manifold and we have $Q_K^{-\infty}(M \times N) = Q_K^{-\infty}(M) \cdot Q_K(N)$ in $R^{-\infty}(K)$.

**Second way :** $Q_K^\Phi$. When $M$ is a proper pre-quantized Hamiltonian $K$-manifold, we can define another formal geometric quantization of $M$ through a non-abelian localization procedure à la Witten [47]. In [29, 36], one proves that an element

$$(17) \qquad Q_K^\Phi(M) \in R^{-\infty}(K)$$

is well-defined by localizing the index of the Dolbeault-Dirac operator $D_{LM}$ on the set $\mathrm{Cr}(\|\Phi_M^K\|^2)$ of critical points of the square of the moment map.

The crucial result is that these two procedures coincides [29, 36].

**Theorem 3.12 (Ma-Zhang, Paraná).** Let $M$ be a proper pre-quantized Hamiltonian $K$-manifold. Then, the following equality

$$(18) \qquad Q_K^{-\infty}(M) = Q_K^{\Phi}(M).$$

holds in $R^{-\infty}(K)$.
**3.4. Formal geometric quantization of holomorphic orbits.** Let us come back to the holomorphic discrete representation $V_\lambda^G$. Consider a coadjoint orbit $G \cdot \lambda$ for $\lambda \in \Lambda_+^*$ in the interior of the chamber $C_{\text{hol}}(z)$, so that $\lambda$ is strongly elliptic. The action of $G$ on $G \cdot \lambda$ is Hamiltonian, and the line bundle

$$L := G \times_{K_\lambda} C_\lambda$$

is a Kostant-Souriau line bundle over $G \cdot \lambda \simeq G/K_\lambda$. Here $C_\lambda$ denotes the 1-dimensional representation of the stabilizer subgroup $K_\lambda$ that can be attached to the weight $\lambda$.

Thanks to Lemma 2.13, we know that the moment map $\Phi_{G \cdot \lambda}^K$ relatively to the action of $K$ on $G \cdot \lambda$ is proper. Hence the reduced spaces

$$(G \cdot \lambda)_\mu := (\Phi_{G \cdot \lambda}^K)^{-1}(K \cdot \mu)/K.$$

are compact for any $\mu \in \Lambda_+^*$, and the generalized character $Q_K^\Phi(G \cdot \lambda) \in R^{-\infty}(K)$ is well defined. We have proved in [35, 38] the following

**Theorem 3.13.** Let $\lambda \in \Lambda_+^* \cap \text{Interior}(C_{\text{hol}}(z))$. The following equality

$$Q_K^\Phi(G \cdot \lambda) = V_\lambda^K \otimes S^\bullet(p)$$

holds in $R^{-\infty}(K)$.

This result will be generalized in (21). It shows that $Q_K^\Phi(G \cdot \lambda)$ coincides with the vector space of $K$-finite vector of the holomorphic discrete representation $V_\lambda^G$ when $\lambda \in C_{\text{hol}}^\rho(z)$. Note that for $\lambda \in \text{Interior}(C_{\text{hol}}(z)) \setminus C_{\text{hol}}^\rho(z)$, the generalized character $Q_K^\Phi(G \cdot \lambda)$ can not be associated to an holomorphic discrete representation of $G$.

Theorems 3.13 and 3.12, gives us the following informations concerning the $K$-multiplicities.

**Corollary 3.14.** Let $\lambda \in \Lambda_+^* \cap \text{Interior}(C_{\text{hol}}(z))$, and $\mu \in \Lambda_+^*$.

* The multiplicity of $V_\mu^K$ in $V_\lambda^K \otimes S^\bullet(p)$ is equal to the quantization of the reduced space $(G \cdot \lambda)_\mu$.

* If $[V_\mu^K : V_\lambda^K \otimes S^\bullet(p)] \neq 0$ then $\mu \in \lambda + C(z) \subset C_{\text{hol}}(z)$. The last condition imposes that $\|\mu\| > \|\lambda\|$ or $\mu = \lambda$.

*Proof.* The first point is a consequence of the equality $Q_K^\Phi(G \cdot \lambda) = Q_K^{-\infty}(G \cdot \lambda)$. We know then that if $[V_\mu^K : V_\lambda^K \otimes S^\bullet(p)] \neq 0$, then $\mu$ belongs to the Kirwan polytope $\Delta_K(G \cdot \lambda)$. But we know after Lemma 2.16 that $\Delta_K(G \cdot \lambda) \subset \lambda + C(z)$ : so $\mu = \lambda + \sum_{\beta \in \mathfrak{R}_n(z)} x_\beta \beta$ with $x_\beta \ge 0$. Finally, we have

$$\begin{align*}
\|\mu\|^2 &= \|\lambda\|^2 + \| \sum_{\beta \in \mathfrak{R}_n(z)} x_\beta \beta \|^2 + 2 \sum_{\beta \in \mathfrak{R}_n(z)} x_\beta \underbrace{x_\beta (\beta, \lambda)}_{\ge 0} \\
&\ge \|\lambda\|^2,
\end{align*}$$

and we have $\|\mu\|^2 = \|\lambda\|^2$ only if $\mu = \lambda$. $\square$

<figure><img src="image_6.png" /></figure>

**3.5. Multiplicities of the holomorphic discrete series.** We consider now a connected reductive subgroup $G' \subset G$ such that $z \in g'$. Then it is easy to check that $G'$ satisfies (13). Let $K' \subset K$ be the maximal compact subgroup in $G'$, and let $T' \subset T$ be a maximal torus in $K'$. Let $C_{\text{hol}}(z), C_{\text{hol}}^\rho(z) \subset t^*$ and $C_{\text{hol}}'(z), C_{\text{hol}}'^\rho(z) \subset (t')^*$ be the corresponding convex cone. For $l \in \{t, t, g\}$, we denote $\pi_{l',l} : l^* \to (l')^*$ the canonical projection. We have the following important fact.
**Proposition 3.15.** We have the following relations

$$ (a) \qquad \pi_{t',t}(C_{\text{hol}}(z)) \subset C'_{\text{hol}}(z), $$

$$ (b) \qquad \pi_{t',t}(K \cdot C_{\text{hol}}(z)) \subset K' \cdot C'_{\text{hol}}(z), $$

$$ (c) \qquad \pi_{t',t}(K \cdot C_{\text{hol}}^{\rho}(z)) \subset K' \cdot C_{\text{hol}}^{\rho'}(z), $$

$$ (d) \qquad \pi_{g',g}(G \cdot C_{\text{hol}}^{\rho}(z)) \subset G' \cdot C_{\text{hol}}^{\rho'}(z). $$

*Proof.* Let $\alpha \in t^*$ be a non-compact root of $(g, t)$. Let $g_\alpha \subset p \otimes \mathbb{C}$ the corresponding 1-dimensional weight space. Then we know that there exists $h_\alpha \in i[g_\alpha, \overline{g_\alpha}] \cap t$ such that $\alpha = -b(h_\alpha, \cdot)$. Note that the half-line $\mathbb{R}^{>0}h_\alpha$ does not depend of the bilinear form $b$, and that the condition $(\alpha, \xi) \ge 0$ is equivalent to $\langle \xi, h_\alpha \rangle \ge 0$ for any $\xi \in t^*$.

Let $\alpha \in t^*$ be a non-compact root of $(g, t)$ such that its restriction $\alpha' = \pi_{t',t}(\alpha)$ is a non-compact root of $(g', t')$. Since the 1-dimensional weight spaces $g_\alpha$ and $g_{\alpha'}$ coincide we have $\mathbb{R}^{>0}h_\alpha = \mathbb{R}^{>0}h_{\alpha'} \subset t'$. Then the condition $\langle \xi, h_\alpha \rangle \ge 0$ is equivalent to $\langle \pi_{t',t}(\xi), h_{\alpha'} \rangle \ge 0$. Finally we have proved the point (a) : if one has $\langle \xi, h_\alpha \rangle \ge 0$ for any positive non-compact root of $(g, t)$, then $\langle \pi_{t',t}(\xi), h_{\alpha'} \rangle \ge 0$ holds for any positive non-compact root of $(g', t')$.

Let $\xi \in C_{\text{hol}}(z)$ and $\xi' \in \pi_{t',t}(K \cdot \xi) \cap (t')^*$. Then $\xi' \in \pi_{t',t} \circ \pi_{t,t}(K \cdot \xi)$. By the Convexity theorem [2, 18, 25, 28], we know that $\pi_{t,t}(K \cdot \xi)$ is equal to the convex hull of $W_K\xi$. But $\xi$ belongs to the $W_K$-invariant convex cone $C_{\text{hol}}(z)$, and then $\pi_{t,t}(K \cdot \xi) \subset C_{\text{hol}}(z)$. Finally $\xi' \in \pi_{t',t}(C_{\text{hol}}(z)) \subset C'_{\text{hol}}(z)$ thanks to the point (a).

Let $\xi \in C_{\text{hol}}(z)$. Since $2\rho_n(z)$ is $K$-invariant, we have $K \cdot (2\rho_n(z) + \xi) = 2\rho_n(z) + K \cdot \xi$. Thanks to the point (b), we see that

$$ \begin{aligned} \pi_{t',t}(K \cdot (2\rho_n(z) + \xi)) &= \pi_{t',t}(2\rho_n(z)) + \pi_{t',t}(K \cdot \xi) \\ &\subset K' \cdot (\pi_{t',t}(2\rho_n(z)) + C'_{\text{hol}}(z)). \end{aligned} $$

The $K$-invariant term $\pi_{t',t}(2\rho_n(z))$ belongs to $(t')^*$ and is equal to $2\rho'_n(z) + \pi_{t',t}(A)$ where $A$ is the sum of the positive non-compact roots $\alpha$ such that $g_\alpha$ is not included in $p' \otimes \mathbb{C}$. Hence $A \in C_{\text{hol}}(z)$ and thanks to point (a) its projection $\pi_{t',t}(A)$ belongs to $C'_{\text{hol}}(z)$. The point (c) is then proved.

Let $\lambda \in C_{\text{hol}}^{\rho}(z)$. The coadjoint orbit $G \cdot \lambda$ is contained in $g_{\lambda}^*$, and the moment map $\Phi_{G,\lambda}^{G'}$ is proper since $z \in g'$ (see Corollary 3.6). Then, we know that

$$ \pi_{g',g}(G \cdot \lambda) = \text{Image}(\Phi_{G,\lambda}^{G'}) = G' \cdot \left( \pi_{g',g}(G \cdot \lambda) \cap (t')^* \right) $$

and

$$ \begin{aligned} \pi_{g',g}(G \cdot \lambda) \cap (t')^* &\subset \pi_{t',t} \circ \pi_{t,g}(G \cdot \lambda) \\ &\subset \pi_{t',t}(K \cdot \Delta_K(G \cdot \lambda)) \\ &\subset \pi_{t',t}(K \cdot C_{\text{hol}}^{\rho}(z)) \quad [1] \\ &\subset K' \cdot C_{\text{hol}}^{\rho'}(z). \quad [2] \end{aligned} $$

Equality [1] is due to the fact that $\Delta_K(G \cdot \lambda) \subset \lambda + C(z) \subset C_{\text{hol}}^{\rho}(z)$ when $\lambda \in C_{\text{hol}}^{\rho}(z)$ (see Lemma 3.3). Equality [2] corresponds to $c$. $\square$

**Remark 3.16.** When the Lie algebra $g$ is simple the set $G \cdot C_{\text{hol}}(z) \subset g_{se}^*$ is a maximal closed convex $G$-invariant cone. See [33, 44].
We finish the section by considering the restriction of the irreducible representation $V_\lambda^G$ to the reductive subgroup $G'$. We will denote $\hat{K}_{\text{hol}}(z) \subset \hat{K}$ the subset $\wedge_+^* \cap C_{\text{hol}}^\rho(z)$. We see that $\hat{K}_{\text{hol}}(z)$ and $\hat{G}_{\text{hol}}(z)$ are the same set but they parametrize representations of different groups ($K$ and $G$ respectively).

We start with the

**Proposition 3.17.** Let $b \in \hat{K}_{\text{hol}}(z)$, $\lambda \in \hat{G}_{\text{hol}}(z)$ and $\mu \in \hat{K}'$. We have

* If $[V_\mu^{K'} : V_b^K]_{K'} \neq 0$ then $\mu \in \hat{K}'_{\text{hol}}(z)$.

* If $[V_\mu^{K'} : V_\lambda^G]_{K'} \neq 0$ then $\mu \in \hat{G}'_{\text{hol}}(z)$.

*Proof.* We use here the "Restriction to subgroup" property of Theorem 3.11.

For the first point, we know after the Borel-Weil Theorem that $V_b^K = Q_K(K \cdot b)$, and then $V_b^K|_{K'} = Q_{K'}(K \cdot b)$. Then $[V_\mu^{K'} : V_b^K]_{K'} \neq 0$ only if $b$ belongs to $\text{Image}(\Phi_{K \cdot b}^{K'}) = \pi_{\mathfrak{t}',\mathfrak{t}}(K \cdot b) \subset K' \cdot C_{\text{hol}}^{\rho}(z)$. But $K' \cdot C_{\text{hol}}^{\rho}(z)(t')^* = C_{\text{hol}}^{\rho}(z)$ since $C_{\text{hol}}^{\rho}(z)$ is $W_{K'}$-invariant. We have proved finally that $b \in \hat{K}'_{\text{hol}}(z)$.

For the second point, it works the same. We know that $V_\lambda^G|_{K'} = Q_{K'}^\Phi(G \cdot \lambda)$. Hence $V_\mu^{K'}$ occurs in the restriction $V_\lambda^G|_{K'}$ only if $\mu$ belongs to the image of the moment map $\Phi_{G \cdot \lambda}^{K'} = \pi_{\mathfrak{t}',\mathfrak{t}} \circ \Phi_{G \cdot \lambda}^K$. Since $\Delta_K(G \cdot \lambda) \subset C_{\text{hol}}^\rho(z)$ (see Lemma 3.3), we have

$$ \text{Image}(\Phi_{G \cdot \lambda}^{K'}) = \pi_{\mathfrak{t}',\mathfrak{t}}(\text{Image}(\Phi_{G \cdot \lambda}^K)) = \pi_{\mathfrak{t}',\mathfrak{t}}(K \cdot \Delta_K(G \cdot \lambda)) \\ \subset \pi_{\mathfrak{t}',\mathfrak{t}}(K \cdot C_{\text{hol}}^\rho(z)) \subset K' \cdot C_{\text{hol}}^{\rho}(z), $$

where the last inclusion is point (c) of Proposition 3.15. We have then proved that $V_\mu^{K'}$ occurs in the restriction $V_\lambda^G|_{K'}$ only if

$$ \mu \in \text{Image}(\Phi_{G \cdot \lambda}^{K'}) \cap (t')^* \subset (K' \cdot C_{\text{hol}}^{\rho}(z)) \cap (t')^* = C_{\text{hol}}^{\rho}(z). $$

The last equality is due to the fact that $C_{\text{hol}}^{\rho}(z)$ is a $W_{K'}$-invariant subset of $(t')^*$. $\square$

<figure><img src="image_7.png" /></figure>

We denote by $\hat{G}'$ the unitary dual of $G'$, and by $\hat{G}'_d$ the subset of classes of square integrable irreducible unitary representations. The elements of $\hat{G}'_d$ are called discrete series representations of $G'$ and $\hat{G}'_d$ contains the holomorphic ones : $\hat{G}'_{\text{hol}}(z) \hookrightarrow \hat{G}'_d$.

Since the moment map $\Phi_{G \cdot \lambda}^{G'}$ is proper, we know, thanks to the work of T. Kobayashi [27] and Duflo-Vargas [15], that the unitary representation $V_\lambda^G$ is discretely admissible relatively to $G'$. It means that we have an Hilbertian direct sum

$$ V_\lambda^G|_{G'} = \bigoplus_{\Pi \in \hat{G}'_d} m_\lambda(\Pi) \Pi $$

where the multiplicities $m_\lambda(\Pi)$ are finite. In fact, we can be more precise.

**Proposition 3.18.** Let $\lambda \in \hat{G}_{\text{hol}}(z)$. The multiplicity $m_\lambda(\Pi)$ is non-zero only if $\Pi = V_\mu^{G'}$ for some $\mu \in \hat{G}'_{\text{hol}}(z)$. This means that we have

$$ V_\lambda^G|_{G'} = \bigoplus_{\mu \in \hat{G}'_{\text{hol}}(z)} m_\lambda(\mu) V_\mu^{G'}, $$

with $m_\lambda(\mu)$ finite for any $\mu$.
Proof. Recall the parametrization of $\hat{G}'_d$ given by Harish-Chandra. Let $(\wedge')^* \subset (t')^*$ be the weight lattice. Let $\mathfrak{R}(\mathfrak{g}', t')^+$ be a choice of positive roots and let $\rho'$ be half the sum of its elements. The set $(\wedge')^*+\rho'$ does not depend of the choice of $\mathfrak{R}(\mathfrak{g}', t')^+$: we denote it by $(\wedge')_\rho^*$. Let $(t')_+^* \subset (t')^*$ be the Weyl chamber corresponding to the choice of a set $\mathfrak{R}(t', t')^+$ of positive roots.

The discrete series representations of $G'$ are parametrized by

$$ \hat{G}'_d := \{\mu \in (t')^*, \mathfrak{g}' - \text{regular} \} \cap (\wedge')_\rho^* \cap (t')_+^*. $$

At $\mu \in \hat{G}'_d$, Harish-Chandra associates a square integrable unitary representation $\Pi_\mu^{G'} : \mu$ is the Harish-Chandra parameter, and

$$ \mu_B := \mu - \rho'_c + \rho'_n(\mu) $$

is the corresponding Blattner parameter. Here $\rho'_n(\mu)$ is associated to $\mathfrak{R}'_n(\mu)$. It is a classical fact that $\mu_B \in (\wedge')^* \cap (t')_+^*$, and that the representation $V_{\mu_B}^{K'}$ occurs in $\Pi_\mu^{G'}|_{K'}$ with multiplicity one : $\mu_B$ is the minimal $K'$-types of $\Pi_\mu^{G'}$ in the sense of Vogan. Moreover the map $\mu \mapsto \mu_B$ induces a bijection between $\hat{G}'_d \cap C_{\text{hol}}(z)$ and $\hat{G}'_{\text{hol}}(z)$ and we have $\Pi_\mu^{G'} = V_{\mu_B}^{G'}$.

Let $\mu \in \hat{G}'_d$ such that $[\Pi_\mu^{G'} : V_\lambda^G|_{G'}] \neq 0$. The Proposition will be proved if we check that $\mu \in C_{\text{hol}}(z)$. Since $[V_{\mu_B}^{K'} : \Pi_\mu^{G'}|_{K'}] = 1$, we have $[V_{\mu_B}^{K'} : V_\lambda^G|_{K'}] \neq 0$. Thanks to Proposition 3.17, we have then $\mu_B \in C_{\text{hol}}^{\rho}(z) : \mu_B = 2\rho'_n(z) + \xi$ with $\xi \in C_{\text{hol}}(z)$. Hence

$$ \mu = (\rho'_{n}(z) + \rho'_{c}) + (\rho'_{n}(z) - \rho'_{n}(\mu)) + \xi. $$

The term $\rho' := \rho_n(z) + \rho'_c$ is associated to the choice of positive roots $\mathfrak{R}(\mathfrak{g}', t')^+ := \mathfrak{R}(t', t')^+ \cup \mathfrak{R}'_n(z)$. Thus we have $(\rho', \alpha) > 0$ for any $\alpha \in \mathfrak{R}'_n(z)$.

The term $\rho'_n(z) - \rho'_n(\mu)$ is equal to the sum

$$ \sum_{\substack{(\alpha, z)>0 \\ (\alpha, \mu)<0}} \alpha $$

and then $(\rho'_n(z) - \rho'_n(\mu), \alpha) \ge 0$ for any $\alpha \in \mathfrak{R}'_n(z)$. Finally we have proved that

$$ (\mu, \alpha) = \underbrace{(\rho'_{n}(z) + \rho'_{c}, \alpha)}_{>0} + \underbrace{(\rho'_{n}(z) - \rho'_{n}(\mu), \alpha)}_{\ge 0} + \underbrace{(\xi, \alpha)}_{\ge 0} $$

is positive for any $\alpha \in \mathfrak{R}'_n(z)$, thus $\mu \in C_{\text{hol}}(z)$. $\square$

**3.6. Jakobsen-Vergne's formula.** The aim of this section is to give a direct proof of the following result of Jakobsen-Vergne [23].

**Theorem 3.19 (Jakobsen-Vergne).** The multiplicity $m_\lambda(\mu)$ is equal to the multiplicity of the representation of $V_\mu^{K'}$ in $S^\bullet(\mathfrak{p}/\mathfrak{p}') \otimes V_\lambda^K|_{K'}$.

Let us denote

(19) \qquad $R^{-\infty}(G, z)$

the $\mathbb{Z}$-module formed by the infinite sum $\sum_{\lambda \in \hat{G}_{\text{hol}}(z)} m_\lambda V_\lambda^G$ with $m_\lambda \in \mathbb{Z}$. Similarly, we define $R^{-\infty}(K, z) \subset R^{-\infty}(K)$ as the sub-module formed by the infinite sum $\sum_{\mu \in \wedge_+^*} n_\mu V_\mu^K$ where $n_\mu \in \mathbb{Z}$ is non-zero only if $\mu \in \hat{K}_{\text{hol}}(z)$.

We have the following basic result.
**Lemma 3.20.** • The restriction to K defines a morphism

$$ (20) \qquad \mathbf{r}_{K,G} : R^{-\infty}(G, z) \to R^{-\infty}(K, z) $$

that is injective.

• The product by $S^\bullet(\mathfrak{p})$ defines a map from $R^{-\infty}(K, z)$ into itself.

*Proof.* Let us prove the first point. Thanks to Corollary 3.14, we have $V_\lambda^G = \sum_{\mu \in \Lambda_+^*} Q((G \cdot \lambda)_\mu) V_\mu^K$. Then

$$ \begin{aligned} \mathbf{r}_{K,G} \left( \sum_{\lambda \in \hat{G}_{\mathrm{hol}}(z)} m_\lambda V_\lambda^G \right) &:= \sum_{\lambda \in \hat{G}_{\mathrm{hol}}(z)} m_\lambda V_\lambda^G |_K \\ &= \sum_{\mu \in \Lambda_+^*} \left( \sum_{\lambda \in \hat{G}_{\mathrm{hol}}(z)} m_\lambda Q((G \cdot \lambda)_\mu) \right) V_\mu^K. \end{aligned} $$

We know that $Q((G \cdot \lambda)_\mu) \neq 0$ only if $\|\lambda\| \le \|\mu\|$ and $\mu \in \mathcal{C}(z)$. Hence the sum $n_\mu := \sum_{\lambda \in \hat{G}_{\mathrm{hol}}(z)} m_\lambda Q((G \cdot \lambda)_\mu)$ has a finite number of non-zero term and $n_\mu \neq 0$ only if $\mu \in \mathcal{C}_{\mathrm{hol}}(z)$.

Let $A = \sum_{\lambda \in \hat{G}_{\mathrm{hol}}(z)} m_\lambda V_\lambda^G$ be a non-zero element in $R^{-\infty}(G, z)$. Let $\lambda_A \in \hat{G}_{\mathrm{hol}}(z)$ such that $\|\lambda_A\|$ is minimal among the set $\{\|\lambda\| \mid m_\lambda \neq 0\}$. Let $\mathbf{r}_{K,G}(A) = \sum_\mu n_\mu V_\mu^K$. Then

$$ n_{\lambda_A} := m_{\lambda_A} + \sum_{\lambda \neq \lambda_A} m_{\lambda} Q((G \cdot \lambda)_{\lambda_A}). $$

But $m_\lambda = 0$ if $\|\lambda\| < \|\lambda_A\|$ and $Q((G \cdot \lambda)_{\lambda_A}) = 0$ if $\lambda \neq \lambda_A$ and $\|\lambda\| \ge \|\lambda_A\|$ (see second point of Corollary 3.14). We have checked that $n_{\lambda_A} = m_{\lambda_A} \neq 0$ and then $\mathbf{r}_{K,G}(A) \neq 0$.

Let us check the second point. Let $A = \sum_{\mu \in \Lambda_+^*} n_\mu V_\mu^K \in R^{-\infty}(K, z)$. Then

$$ \begin{aligned} A \otimes S^\bullet(\mathfrak{p}) &= \sum_{\mu \in \mathcal{C}(z)} n_\mu V_\mu^K \otimes S^\bullet(\mathfrak{p}) \\ &= \sum_{\theta} \left( \sum_{\mu \in \mathcal{C}(z)} n_\mu Q((G \cdot \mu)_\theta) \right) V_\theta^K. \end{aligned} $$

Like before, the term $Q((G \cdot \mu)_\theta)$ is non-zero only if $\|\mu\| \le \|\theta\|$ and $\theta \in \mathcal{C}(z)$. Hence the sum $\sum_{\mu \in \mathcal{C}(z)} n_\mu Q((G \cdot \mu)_\theta)$ has a finite number of non-zero term and is non-zero only if $\theta \in \mathcal{C}(z)$. $\square$

Let us consider the similar morphism $\mathbf{r}_{K',G'} : R^{-\infty}(G', z) \to R^{-\infty}(K', z)$ for the reductive subgroup $G'$. We consider the following elements of $R^{-\infty}(G', z)$:

$$ \begin{aligned} V_\lambda^G|_{G'} &= \sum_{\mu \in \hat{G}'_{\mathrm{hol}}(z)} m_\lambda(\mu) V_\mu^{G'}, \quad \text{and} \\ \delta &:= \sum_{\mu \in \hat{G}'_{\mathrm{hol}}(z)} n_\lambda(\mu) V_\mu^{G'}, \end{aligned} $$

where $n_\lambda(\mu) := [V_\mu^{K'} : S^\bullet(\mathfrak{p}/\mathfrak{p}') \otimes V_\lambda^K]_{K'}$. Theorem 3.19 will be proved if we check that $\mathbf{r}_{K',G'}(V_\lambda^G|_{G'}) = \mathbf{r}_{K',G'}(\delta)$. But $\mathbf{r}_{K',G'}(V_\lambda^G|_{G'}) = V_\lambda^G|_{K'} = S^\bullet(\mathfrak{p}) \otimes V_\lambda^K|_{K'}$,
and

$$
\begin{align*}
\mathbf{r}_{K', G'}(\delta) &= \sum_{\mu \in \hat{G}_{\text{hol}}'(z)} n_{\lambda}(\mu) V_{\mu}^{G'}|_{K'} \\
&= S^{\bullet}(\mathbf{p}') \otimes \left( \sum_{\mu \in \hat{G}_{\text{hol}}'(z)} n_{\lambda}(\mu) V_{\mu}^{K'} \right) \quad [1] \\
&= S^{\bullet}(\mathbf{p}') \otimes \left( \sum_{\mu \in \hat{G}_{\text{hol}}'(z)} [V_{\mu}^{K'} : S^{\bullet}(\mathbf{p}/\mathbf{p}') \otimes V_{\lambda}^{K}|_{K'}] V_{\mu}^{K'} \right) \\
&= S^{\bullet}(\mathbf{p}) \otimes V_{\lambda}^{K}|_{K'}
\end{align*}
$$

The second point of Lemma 3.20 insures that the product in [1] is well-defined. We need to explain the last equality. Note that $[V_{\mu}^{K'}, S^{\bullet}(\mathbf{p}/\mathbf{p}') \otimes V_{\lambda}^{K}|_{K'}] \neq 0$ implies $[V_{\mu}^{K'}, S^{\bullet}(\mathbf{p}) \otimes V_{\lambda}^{K}|_{K'}] \neq 0$ and then $\mu \in \hat{G}_{\text{hol}}'(z)$ (see Proposition 3.17). This insures that the sum

$$
\sum_{\mu \in \hat{G}'_{\text{hol}}(z)} [V_{\mu}^{K'} : S^{\bullet}(\mathbf{p}/\mathbf{p}') \otimes V_{\lambda}^{K}|_{K'}] V_{\mu}^{K'} \in R^{-\infty}(K', z)
$$

is equal to $S^{\bullet}(\mathbf{p}/\mathbf{p}') \otimes V_{\lambda}^{K}|_{K'}$.

3.7. Formal geometric quantization of G-actions. In this section we consider the Hamiltonian action of a connected real reductive Lie group G on a symplectic manifold (M, Ω_M). We suppose that the action of G on M is proper and that the moment map Φ_M^G : M → g* is proper. We know that we have a global slice Y ⊂ M such that

$M \simeq G \times_K Y,$

and that the G-orbits in the image of Φ<sub>M</sub><sup>G</sup> are parametrized by the Kirwan polytope
Δ<sub>K</sub>(Y).

Let us suppose the existence of a G-equivariant pre-quantum line bundle $L_M \to M$. Note that $L_M$ is completely determined by its restriction $L_Y \to Y$ to the sub-manifold $Y$: here $L_Y$ is a K-equivariant pre-quantum line bundle over $(Y, \Omega_Y)$. For any dominant weight $\mu$, we see that the reduce space

$$
M_{\mu,G} := (\Phi_M^G)^{-1}(G \cdot \mu)/G
$$

coincides with $Y_{\mu,K} := (\Phi_M^K)^{-1}(K \cdot \mu)/K$. Hence its quantization

$$
\mathcal{Q}(M_{\mu,G}) := \mathcal{Q}(Y_{\mu,K}) \in \mathbb{Z}
$$

is well-defined (see Section 3.3).

We suppose also that G satisfies (13), and we fix a complex structure ad(z) on
**p**. Let C<sup>ρ</sup><sub>hol</sub>(z) ⊂ **t**<sup>*</sup> be the corresponding cone.

**Lemma 3.21.** Let Let (M, Ω_M, Φ_M^G) be a Hamiltonian manifold. Suppose that the image of Φ_M^G is contained in G · C^ρ_{hol}(z) ⊂ g^*_{se}. Then :

(1) the Kirwan polytopes $\Delta_K(Y) \subset \Delta_K(M)$ are contained in $C_{\text{hol}}^\rho(z)$,

(2) the functions $\langle \Phi_Y^K, z \rangle$ and $\langle \Phi_M^K, z \rangle$ take strictly positive values.

Proof. The Kirwan polytope $\Delta_K(M) = \pi_{\mathfrak{t},\mathfrak{g}}(\text{Image}(\Phi_M^G)) \cap \mathfrak{t}_+^*$ is contained in

$$
\bigcup_{\lambda \in \mathcal{C}_{\text{hol}}^{\rho}(z)} \Delta_{\kappa}(G \cdot \lambda) \subset \mathcal{C}_{\text{hol}}^{\rho}(z),
$$
where the last inclusion is a consequence of point (a) in Proposition 3.3. The first point is proved. Hence we get, thanks to (14), the following relations

$$ \text{Image}(\langle \Phi_Y^K, z \rangle) \subset \text{Image}(\langle \Phi_M^K, z \rangle) \subset \langle \Delta_K(M), z \rangle \subset \langle C_{\text{hol}}^\rho(z), z \rangle \subset [c, +\infty[ \\ \text{with } c = \langle \rho(z), z \rangle = \frac{1}{2} \dim \mathfrak{p}. \quad \square $$

We have the following notion of formal geometric quantization that extends the case of compact Lie group actions.

**Definition 3.22.** Let $(M, \Omega_M, \Phi_M^G)$ be a pre-quantized Hamiltonian manifold, such that the moment map $\Phi_M^G$ is a proper map from $M$ into $G \cdot C_{\text{hol}}^\rho(z)$. Then we define the formal geometric quantization of $M$ as the following element of $R^{-\infty}(G, z)$:

$$ Q_G^{-\infty}(M) := \sum_{\mu \in \hat{G}_{\text{hol}}(z)} Q(M_{\mu,G}) V_{\mu}^G. $$

Let $\mathbf{r}_{K,G}: R^{-\infty}(G, z) \to R^{-\infty}(K, z)$ be the restriction morphism defined in Lemma 3.20. Recall that in the setting of Definition 3.22, the moment map $\Phi_M^K$ is proper (see Theorem 2.9). Then the formal geometric quantization of $M$ relatively to the $K$-action is well-defined : $Q_K^{-\infty}(M) \in R^{-\infty}(K)$.

We have proved in Theorem 2.9, that the sets of critical points of the function $\|\Phi_M^G\|^2$, $\|\Phi_M^K\|^2$ and $\|\Phi_Y^K\|^2$ are equal. We will be interested by one of the following hypothesis:

**Assumption 3.23.** • A1 The set $\text{Cr}(\|\Phi_M^G\|^2)$ is compact.
• A2 The map $\langle \Phi_M^G, z \rangle : M \to \mathbb{R}$ is proper.

In the following Lemma, we exhibit examples where the Assumptions **A1** or **A2** are satisfied.

**Lemma 3.24.** • Suppose that we are in the algebraic setting: the manifold $M$ is real algebraic and the map $\Phi_M^G$ is a proper algebraic map. Then $\text{Cr}(\|\Phi_M^G\|^2)$ is compact.

• Suppose that the Lie algebra $\mathfrak{g}$ is simple. Then, in the context of Definition 3.22, the map $\langle \Phi_M^G, z \rangle : M \to \mathbb{R}$ is proper.

*Proof.* Let us prove the first point. The map $\varphi := \|\Phi_M^G\|^2 : M \to \mathbb{R}$ is a real algebraic map on a real algebraic manifold. Thus the set $\text{Cr}(\varphi)$ is an algebraic variety, and by a standard Theorem of Whitney, it as a finite number of connected components $C_1, \cdots, C_p$. Each $C_i$ is contained in $\varphi^{-1}(\varphi(C_i))$ which is compact since $\varphi$ is proper. The proof is completed.

For the second point we use the result of Proposition 2.11, and the facts that, since $\mathfrak{g}$ is simple, $[\mathfrak{p}, \mathfrak{p}] = \mathfrak{k}$ and the center $\mathfrak{c}_\mathfrak{k}$ of $\mathfrak{k}$ is reduced to $\mathbb{R}z$.

The function $\langle \Phi_M^G, z \rangle$, which is the moment map for the $S^1$-action, is proper if and only if $\text{As}(\Delta_K(M)) \cap (\mathbb{R}z)^\perp = \{0\}$. Since $\Delta_K(M) \subset C_{\text{hol}}^\rho(z)$ (see Lemma 3.21), it is sufficient to prove that $C_{\text{hol}}(z) \cap (\mathbb{R}z)^\perp = \{0\}$. Let $\xi \in C_{\text{hol}}(z)$. We have $\langle \xi, z \rangle = -b(\tilde{\xi}, z) = 2 \sum_{\beta \in \mathfrak{R}_n(z)} \langle \beta, \tilde{\xi} \rangle$ with $\langle \beta, \tilde{\xi} \rangle = (\beta, \xi) \ge 0$. If $\langle \xi, z \rangle = 0$, we must have $(\beta, \xi) = 0, \forall \beta \in \mathfrak{R}_n(z)$ or equivalently $[\tilde{\xi}, \mathfrak{p}] = 0$. Then $\tilde{\xi}$ commutes with all elements in $[\mathfrak{p}, \mathfrak{p}] = \mathfrak{k}$, i.e. $\tilde{\xi} \in \mathfrak{c}_\mathfrak{k} = \mathbb{R}z$. Finally, we have proved that $\xi \in (\mathbb{R}z)^\perp$ and $\tilde{\xi} \in \mathbb{R}z$, hence $\xi = 0$. □
We can now state the main result of this section.

**Theorem 3.25.** If Assumptions **A1** or **A2** are satisfied, we have the following relation

$$ \mathbf{r}_{K,G}(Q_G^{-\infty}(M)) = Q_K^{-\infty}(M). $$

*Proof.* We have

$$ \begin{aligned} \mathbf{r}_{K,G}(Q_G^{-\infty}(M)) &= \sum_{\mu \in \hat{G}_{\text{hol}}(z)} Q(M_{\mu,G}) V_{\mu}^G|_K \\ &= \left( \sum_{\mu \in \Lambda_+^* \cap C_{\text{hol}}^\rho(z)} Q(Y_\mu) V_\mu^K \right) \otimes S^\bullet(\mathfrak{p}) \quad [1] \\ &= Q_K^{-\infty}(Y) \otimes S^\bullet(\mathfrak{p}) \quad [2]. \end{aligned} $$

Note that the product in [1] and [2] are well defined thanks to Lemma 3.20. In [2] we use the fact that $Q_K^{-\infty}(Y) = \sum_{\mu \in \Lambda_+^* \cap C_{\text{hol}}^\rho(z)} Q(Y_\mu) V_\mu^K$ since by hypothesis $\Delta_K(Y) \subset C_{\text{hol}}^\rho(z)$. So Theorem 3.25 follows from the following equality

$$ (21) \qquad Q_K^{-\infty}(G \times_K Y) = Q_K^{-\infty}(Y) \otimes S^\bullet(\mathfrak{p}), $$

that will be proved in Sections 4.4 and 4.6. $\square$

<figure><img src="image_8.png" /></figure>

We consider now a connected reductive subgroup $G' \subset G$ such that $z \in g'$. The coadjoint orbit $G \cdot \lambda$ is pre-quantized when $\lambda \in \hat{G}_{\text{hol}}(z)$ and we have obviously $Q_G^{-\infty}(G \cdot \lambda) = V_\lambda^G$. The moment map $\Phi_{G,\lambda} : G \cdot \lambda \to (g')^*$ relative to the $G'$-action on $G \cdot \lambda$ is proper. In fact we have more : the map $\langle \Phi_{G,\lambda}^G, z \rangle : G \cdot \lambda \to \mathbb{R}$ is proper, thus Assumption **A2** holds.

We are interested in the compact reduced spaces

$$ (G \cdot \lambda)_{\mu,G'} := (\Phi_{G,\lambda}^{G'})^{-1}(G' \cdot \mu)/G', $$

for $\mu' \in \hat{G}_{\text{hol}}(z)$. We are now able to prove the following

**Theorem 3.26.** Let $\lambda \in \hat{G}_{\text{hol}}(z)$. Then we have the following relation

$$ V_{\lambda}^{G}|_{G'} = Q_{G'}^{-\infty}(G \cdot \lambda) $$

in $R^{-\infty}(G', z)$. It means that for any $\mu \in \hat{G}_{\text{hol}}'(z)$, the multiplicity of the representation $V_\mu^{G'}$ in the restriction $V_\lambda^G|_{G'}$ is equal to the geometric quantization

$$ \mathcal{Q}((G \cdot \lambda)_{\mu,G'}) \in \mathbb{Z} $$

of the (compact) reduced space $(G \cdot \lambda)_{\mu,G'}$.

*Proof.* Since the restriction morphism $\mathbf{r}_{K',G'} : R^{-\infty}(G', z) \to R^{-\infty}(K', z)$ is injective (see Lemma 3.20) it suffices to prove that

$$ (22) \qquad \mathbf{r}_{K',G'}(V_{\lambda}^{G}|_{G'}) = \mathbf{r}_{K',G'}(Q_{G'}^{-\infty}(G \cdot \lambda)). $$

But the left hand side of (22) is equal to the restriction $V_\lambda^G|_{K'}$, while the right hand side is equal to $Q_{K'}^{-\infty}(G \cdot \lambda)$ thanks to Theorem 3.25. Theorem 3.13 tells us that $Q_K^{-\infty}(G \cdot \lambda) = V_\lambda^G|_K$ and the functoriality of the quantization process $Q^{-\infty}$ (see Theorem 3.11) insures that the restriction $V_\lambda^G|_{K'} = Q_K^{-\infty}(G \cdot \lambda)|_{K'}$ is equal to $Q_{K'}^{-\infty}(G \cdot \lambda)$. $\square$

<figure><img src="image_9.png" /></figure>
We can finish this section by extending the functoriality of the quantization process $Q^{-\infty}$ relatively to the restrictions.

**Definition 3.27.** An element $m := \sum_{\lambda \in \hat{G}_{\text{hol}}(z)} m_{\lambda} V_{\lambda}^{G} \in R^{-\infty}(G, z)$ is admissible relatively to $G'$ if the projection $\pi_{\mathfrak{g}', \mathfrak{g}}$ is proper when restricted to the subset $G$. Support($m$) $\subset G \cdot C_{\text{hol}}^{\rho}(z)$, where $\text{Support}(m) = \{\lambda | m_{\lambda} \neq 0\} \subset \hat{G}_{\text{hol}}(z)$. The same definition holds for the couple $(K', K)$.

When $m \in R^{-\infty}(G, z)$ is $G'$-admissible, we can define its restriction

$$ \mathbf{r}_{G',G}(m) := \sum_{\lambda \in \hat{G}_{\text{hol}}(z)} m_{\lambda} V_{\mu}^{G}|_{G'} \\ = \sum_{\mu \in \hat{G}'_{\text{hol}}(z)} \left( \sum_{\lambda \in \hat{G}_{\text{hol}}(z)} m_{\lambda} Q((G \cdot \lambda)_{\mu,G'}) \right) V_{\mu}^{G'} \in R^{-\infty}(G', z). $$

Note that for any $\mu \in \hat{G}'_{\text{hol}}(z)$ the sum $\sum_{\lambda \in \hat{G}_{\text{hol}}(z)} m_{\lambda} Q((G \cdot \lambda)_{\mu,G'})$ has only a finite number of non-zero term. Similarly, when $n = \sum_{b \in \hat{K}_{\text{hol}}(z)} n_b V_b^K \in R^{-\infty}(K, z)$ is $K'$-admissible, we can define its restriction

$$ \mathbf{r}_{K',K}(n) := \sum_{b \in \hat{K}_{\text{hol}}(z)} n_b V_b^K|_{K'} \\ = \sum_{\mu \in \hat{K}'_{\text{hol}}(z)} \left( \sum_{b \in \hat{K}_{\text{hol}}(z)} n_b Q((K \cdot b)_{a,K'}) \right) V_a^{K'} \in R^{-\infty}(K', z). $$

We will use the following Lemma that will be proved in the Appendix.

**Lemma 3.28.** Let $m \in R^{-\infty}(G, z)$ that is $G'$-admissible. Then $\mathbf{r}_{K,G}(m) \in R^{-\infty}(K, z)$ is $K'$-admissible and the following relation

$$ \mathbf{r}_{K',K} \circ \mathbf{r}_{K,G}(m) = \mathbf{r}_{K',G'} \circ \mathbf{r}_{G',G}(m) $$

holds in $R^{-\infty}(K', z)$.

We finish this section with the following

**Theorem 3.29.** Let $(M, \Omega_M, \Phi_M^G)$ be a pre-quantized Hamiltonian manifold. Suppose that Image$(\Phi_M^G) \subset G \cdot C_{\text{hol}}(z)$, and that the map $\langle \Phi_M^G, z \rangle$ is proper. Let $G'$ be a reductive subgroup such that $z \in \mathfrak{g}'$. Then:

* The map $\Phi_M^{G'}$ is proper and Image$(\Phi_M^{G'}) \subset G' \cdot C_{\text{hol}}^{\rho}(z)$. So $Q_{G'}^{-\infty}(M) \in R^{-\infty}(G', z)$ is well defined.

* The element $Q_G^{-\infty}(M) \in R^{-\infty}(G, z)$ is $G'$-admissible and we have

$$ \mathbf{r}_{G',G} (Q_G^{-\infty}(M)) = Q_{G'}^{-\infty}(M). $$

*Proof.* The map $\Phi_M^{G'}$ is proper since $\langle \Phi_M^G, z \rangle$ is proper. The point concerning the image of $\Phi_M^{G'}$ is a consequence of point (d) in Proposition 3.15. Let $m = Q_G^{-\infty}(M) \in R^{-\infty}(G, z)$. By definition Support($m$) is contained in Image$(\Phi_M^G)$, and then $G \cdot$ Support($m$) $\subset$ Image$(\Phi_M^G)$. Since the moment map $\Phi_M^{G'}$ is proper, we know that the projection $\pi_{\mathfrak{g}', \mathfrak{g}}$ is proper when restricted to Image$(\Phi_M^G)$. This implies that $m \in R^{-\infty}(G, z)$ is $G'$-admissible.
We have then

$$
\begin{align*}
\mathbf{r}_{K',G'} \circ \mathbf{r}_{G',G} (Q_G^{-\infty}(M)) &= \mathbf{r}_{K',K} \circ \mathbf{r}_{K,G} (Q_G^{-\infty}(M)) \quad [1] \\
&= \mathbf{r}_{K',K} (Q_K^{-\infty}(M)) \quad [2] \\
&= Q_{K'}^{-\infty}(M). \quad [3]
\end{align*}
$$

Here [1] follows from Lemma 3.28, [2] follows from Theorem 3.25 and [3] is the
consequence of Theorem 3.11. We have checked that

$$
\mathbf{r}_{K', G'} \circ \mathbf{r}_{G', G} (Q_G^{-\infty}(M)) = Q_{K'}^{-\infty}(M) = \mathbf{r}_{K', G'} (Q_{G'}^{-\infty}(M)).
$$

Since the map $\mathbf{r}_{K',G'}$ is injective, it follows that $\mathbf{r}_{G',G} (Q_G^{-\infty}(M)) = Q_{G'}^{-\infty}(M)$. $\square$

3.8. **Geometric quantization of the slice Y.** Let $\lambda \in \hat{G}'_{\text{hol}}(z)$. Consider$^5$ the coadjoint orbit $G' \cdot \lambda$ associated to the holomorphic discrete series representation $V_{\lambda}^{G'}$. Let $G$ be a reductive subgroup of $G'$ such that $z \in \mathfrak{g}$. We know that we have a geometric decomposition

$$
G' \cdot \lambda = G \times_K Y
$$

where $Y \subset G' \cdot \lambda$ is a closed $K$-invariant symplectic sub-manifold.

We have two ways of computing the multiplicity of $m_\lambda(\mu)$ of $V_\mu^G$ in $V_\lambda^{G'}$. First, after Jakobsen-Vergne, we know that

$$
m_{\lambda}(\mu) = [V_{\mu}^{K} : S^{\bullet}(\mathfrak{p}'/\mathfrak{p}) \otimes V_{\lambda}^{K'}|_{K}],
$$

and Theorem 3.26 tells us also that

$$
m_{\lambda}(\mu) = \mathcal{Q}((G' \cdot \lambda)_{\mu, G}) = \mathcal{Q}(Y_{\mu, G}).
$$

We would like to understand a priori why $\mathcal{Q}(Y_{\mu,K}) = [V_\mu^K : S^\bullet(\mathfrak{p}'/\mathfrak{p}) \otimes V_\lambda^{K'}|_K]$ for any $\mu \in \hat{K}_{\text{hol}}(z)$, or equivalently why we have the relation

$$
(23) \qquad \mathcal{Q}_K^{-\infty}(Y) = S^{\bullet}(\mathfrak{p}'/\mathfrak{p}) \otimes \mathcal{V}_{\lambda}^{K'}|_K.
$$

Note that Assumption **A2** holds in this setting : the map $\langle \Phi_{G'.\lambda}^G, z \rangle$ is proper.

Let us consider a more general situation. Let (M, Ω<sub>M</sub>, Φ<sub>M</sub><sup>G</sup>) be a pre-quantized Hamiltonian G-manifold. We suppose the G-action proper, and that the moment map Φ<sub>M</sub><sup>G</sup> takes values in G ⋅ C<sub>hol</sub><sup>ρ</sup>(z). We suppose furthermore that Assumption **A2** holds. Let Y ⊂ M be the symplectic slice. The aim of this section is to compute Q<sub>K</sub><sup>-∞</sup>(Y) in a way similar to (23).

Let $\mathcal{X}$ be a connected component of $Y^z$. Let us fix a $K$-invariant almost complex structure on $\mathcal{X}$ which is compatible with the symplectic structure. Let

$$
\mathrm{RR}^K(\mathcal{X}, -)
$$

be the corresponding Riemann-Roch character (see Section 4.2). Recall that, if $L_{\mathcal{X}}$ denotes the restriction of the Kostant-Souriau line bundle $L_M$ on $\mathcal{X}$, we have $\mathcal{Q}_K(\mathcal{X}) = \mathrm{RR}^K(\mathcal{X}, L_{\mathcal{X}})$.

Let $\mathcal{N}_{\mathcal{X}} \to \mathcal{X}$ be the normal bundle of $\mathcal{X}$ in $Y$: it inherits a complex structure $J_{\mathcal{X}}$ and a linear endomorphism $\mathcal{L}(z)$ on the fibres. We have a decomposition $\mathcal{N}_{\mathcal{X}} =$

<footnote n="5">⁵In this section, we interchange the role of the groups *G* and *G'* in order to minimize the primes in the notation.</footnote>
$\sum_{a \in \mathbb{R}} \mathcal{N}_{\mathcal{X}}^{a}$ where $\mathcal{N}_{\mathcal{X}}^{a} = \{v \in \mathcal{N}_{\mathcal{X}} | \mathcal{L}(z)v = aJ_{\mathcal{X}}(v)\}$ is a sub-bundle of $\mathcal{N}_{\mathcal{X}}$. We define the vector bundle $\mathcal{N}_{\mathcal{X}}^{\pm,z} := \sum_{\pm a > 0} \mathcal{N}_{\mathcal{X}}$ and

$$|\mathcal{N}|^z = \mathcal{N}_{\mathcal{X}}^{\pm,z} \oplus \overline{\mathcal{N}_{\mathcal{X}}^{-,z}}.$$

**Theorem 3.30.** We have the following equality in $R^{-\infty}(K)$:

$$Q_K^{-\infty}(Y) = \sum_{\mathcal{X}} (-1)^{r_{\mathcal{X}}} \mathrm{RR}^K(\mathcal{X}, L_{\mathcal{X}} \otimes \det(\mathcal{N}_{\mathcal{X}}^{+,z}) \otimes S^{\bullet}(|\mathcal{N}_{\mathcal{X}}|^z)),$$

where $r_{\mathcal{X}}$ is the complex rank of $\mathcal{N}_{\mathcal{X}}^{+,z}$.

The proof will be given in Section 4.5.

Let us explain how the formulas of Jakobsen-Vergne can be recover with Theorem 3.30. When $M = G' \cdot \lambda$, the sub-manifolds $Y^z$ and $M^z$ are both equal to $K' \cdot \lambda$. The restriction of the Kostant-Souriau line bundle $L_M \to M$ on $Y^z$ is $[\mathbb{C}_{\lambda}] := K' \times_{K'_{\lambda}} \mathbb{C}_{\lambda} \to K' \cdot \lambda$. Relation (8) tells us that the normal bundle $N_1$ of $Y$ in $M$ is equal to the trivial bundle $\mathfrak{p} \times Y$, and the normal bundle $N_2$ of $Y^z$ in $M$ is equal to $Y^z \times \mathfrak{p}'$. Hence the normal bundle of $Y^z$ in $Y$ is

$$\mathcal{N} = \mathcal{N}_2/(\mathcal{N}_1|_{Y^z}) = Y^z \times (\mathfrak{p}'/\mathfrak{p}).$$

We check that $\mathcal{N}^{+,z} = 0$: this is due to the fact that the function $\langle \Phi_{G' \cdot \lambda, z}^G \rangle$ takes its minimal value on $Y^z = K' \cdot \lambda$ (see Lemma 7.3 in [34]). So $|\mathcal{N}|^z = \overline{\mathcal{N}}$ is the trivial complex bundle with fiber $(\mathfrak{p}'/\mathfrak{p}, \mathrm{ad}(z))$. Theorem 3.30 gives

$$ \begin{aligned} Q_K^{-\infty}(Y) &= \mathrm{RR}^K(K' \cdot \lambda, [\mathbb{C}_{\lambda}] \otimes S^{\bullet}(\mathfrak{p}'/\mathfrak{p})) \\ &= \mathrm{RR}^{K'}(K' \cdot \lambda, [\mathbb{C}_{\lambda}])|_K \otimes S^{\bullet}(\mathfrak{p}'/\mathfrak{p}) \\ &= V_{\lambda}^{K'}|_K \otimes S^{\bullet}(\mathfrak{p}'/\mathfrak{p}). \end{aligned} \quad [1] $$

In [1], we use that $\mathrm{RR}^{K'}(K' \cdot \lambda, [\mathbb{C}_{\lambda}]) = V_{\lambda}^{K'}$ thanks to the Borel-Weil theorem.

## 4. TRANSVERSALLY ELLIPTIC OPERATORS

The aim of this section is to give a proof of Theorem 3.25 and 3.30. In the first section, we briefly introduce the material we need from the theory of transversally elliptic operator. And in Section 4.3 we recall the definition of the geometric quantization process $Q^{\Phi}$. In the rest of this paper, $K$ will denoted a connected compact Lie group.

### 4.1. Transversally elliptic operators.
Here we give the basic definitions from the theory of transversally elliptic symbols (or operators) defined by Atiyah-Singer in [1]. For an axiomatic treatment of the index morphism see Berline-Vergne [8, 9] and Paranadan-Vergne [37]. For a short introduction see [34].

Let $\mathcal{X}$ be a *compact* $K$-manifold. Let $p : \mathcal{TX} \to \mathcal{X}$ be the projection, and let $(-, -)_\mathcal{X}$ be a $K$-invariant Riemannian metric. If $E^0, E^1$ are $K$-equivariant complex vector bundles over $\mathcal{X}$, a $K$-equivariant morphism

$$\sigma \in \Gamma(\mathcal{TX}, \mathrm{hom}(p^*E^0, p^*E^1))$$

is called a *symbol* on $\mathcal{X}$. The subset of all $(x, v) \in \mathcal{TX}$ where $\sigma(x, v) : E_x^0 \to E_x^1$ is not invertible is called the *characteristic set* of $\sigma$, and is denoted by $\mathrm{Char}(\sigma)$.

<footnote n="6">⁶The map $\sigma(x, v)$ will be also denoted $\sigma|_x(v)$</footnote>
In the following, the product of a symbol $\sigma$ by a complex vector bundle $F \to M$, is the symbol $\sigma \otimes F$ defined by $\sigma \otimes F(x, v) = \sigma(x, v) \otimes \mathrm{Id}_{F_x}$ from $E_x^0 \otimes F_x$ to $E_x^1 \otimes F_x$. Note that $\mathrm{Char}(\sigma \otimes F) = \mathrm{Char}(\sigma)$.

Let $\mathbf{T}_K\mathcal{X}$ be the following subset of $\mathbf{T}\mathcal{X}$:

$$ \mathbf{T}_K\mathcal{X} = \{(x, v) \in \mathbf{T}\mathcal{X}, (v, X_{\mathcal{X}}(x))_x = 0 \text{ for all } X \in \mathfrak{k}\}. $$

A symbol $\sigma$ is *elliptic* if $\sigma$ is invertible outside a compact subset of $\mathbf{T}\mathcal{X}$ (i.e. $\mathrm{Char}(\sigma)$ is compact), and is *K-transversally elliptic* if the restriction of $\sigma$ to $\mathbf{T}_K\mathcal{X}$ is invertible outside a compact subset of $\mathbf{T}_K\mathcal{X}$ (i.e. $\mathrm{Char}(\sigma) \cap \mathbf{T}_K\mathcal{X}$ is compact). An elliptic symbol $\sigma$ defines an element in the equivariant $\mathbf{K}^0$-theory of $\mathbf{T}\mathcal{X}$ with compact support, which is denoted by $\mathbf{K}_K^0(\mathbf{T}\mathcal{X})$, and the index of $\sigma$ is a virtual finite dimensional representation of $K$, that we denote $\mathrm{Index}_{\mathcal{X}}^K(\sigma) \in R(K)$ [3, 4, 5, 6].

A K-transversally elliptic symbol $\sigma$ defines an element of $\mathbf{K}_K^0(\mathbf{T}_K\mathcal{X})$, and the index of $\sigma$ is defined as a trace class virtual representation of $K$, that we still denote $\mathrm{Index}_{\mathcal{X}}^K(\sigma) \in R^{-\infty}(K)$ [1].

Using the excision property, one can easily show that the index map $\mathrm{Index}_{\mathcal{U}}^K: \mathbf{K}_K^0(\mathbf{T}_K\mathcal{U}) \to R^{-\infty}(K)$ is still defined when $\mathcal{U}$ is a K-invariant relatively compact open subset of a K-manifold (see [34][section 3.1]).

Suppose now that the group $K$ is equal to the product $K_1 \times K_2$. An intermediate notion between the “ellipticity” and “$K_1 \times K_2$-transversal ellipticity” is the “$K_1$-transversal ellipticity”. When a $K_1 \times K_2$-equivariant symbol $\sigma$ is $K_1$-transversally elliptic, its index $\mathrm{Index}_{\mathcal{X}}^{K_1 \times K_2}(\sigma) \in R^{-\infty}(K_1 \times K_2)$, viewed as a generalized function on $K_1 \times K_2$, is smooth relatively to the variable in $K_2$ [1, 9, 37]. It implies that:

* $\mathrm{Index}_{\mathcal{X}}^{K_1 \times K_2}(\sigma) = \sum_{\lambda \in \widehat{K_1}} \theta_{\lambda} \otimes V_{\lambda}^{K_1}$ with $\theta_{\lambda} \in R(K_2)$,

* we can restrict $\mathrm{Index}_{\mathcal{X}}^{K_1 \times K_2}(\sigma)$ to the subgroup $K_1$ and

$$ (24) \qquad \mathrm{Index}_{\mathcal{X}}^{K_1 \times K_2}(\sigma)|_{K_1} = \sum_{\lambda \in \widehat{K_1}} \dim(\theta_{\lambda}) V_{\lambda}^{K_1} = \mathrm{Index}_{\mathcal{X}}^{K_1}(\sigma). $$

Here $\dim : R(K_2) \to \mathbb{Z}$ is the morphism induced by the restriction to $1 \in K_2$.

Let us recall the multiplicative property of the index map for the product of manifolds that was proved by Atiyah-Singer in [1]. Consider a compact Lie group $K_2$ acting on two manifolds $\mathcal{X}_1$ and $\mathcal{X}_2$, and assume that another compact Lie group $K_1$ acts on $\mathcal{X}_1$ commuting with the action of $K_2$. The external product of complexes on $\mathbf{T}\mathcal{X}_1$ and $\mathbf{T}\mathcal{X}_2$ induces a multiplication (see [1]):

$$ \circ : \mathbf{K}_{K_1 \times K_2}^0(\mathbf{T}_{K_1}\mathcal{X}_1) \times \mathbf{K}_{K_2}^0(\mathbf{T}_{K_2}\mathcal{X}_2) \to \mathbf{K}_{K_1 \times K_2}^0(\mathbf{T}_{K_1 \times K_2}(\mathcal{X}_1 \times \mathcal{X}_2)). $$

Let us recall the definition of this external product. For $k = 1, 2$, we consider equivariant morphisms$^7$ $\sigma_k : \mathcal{E}_k^+ \to \mathcal{E}_k^-$ on $\mathbf{T}\mathcal{X}_k$. We consider the equivariant morphism on $\mathbf{T}(\mathcal{X}_1 \times \mathcal{X}_2)$

$$ \sigma_1 \odot \sigma_2 : \mathcal{E}_1^+ \otimes \mathcal{E}_2^+ \oplus \mathcal{E}_1^- \otimes \mathcal{E}_2^- \to \mathcal{E}_1^- \otimes \mathcal{E}_2^+ \oplus \mathcal{E}_1^+ \otimes \mathcal{E}_2^- $$

defined by

$$ (25) \qquad \sigma_1 \odot \sigma_2 = \begin{pmatrix} \sigma_1 \otimes \mathrm{Id} & -\mathrm{Id} \otimes \sigma_2^* \\ \mathrm{Id} \otimes \sigma_2 & \sigma_1^* \otimes \mathrm{Id} \end{pmatrix}. $$

<footnote>$^7$In order to simplify the notation, we do not make the distinctions between vector bundles on $\mathbf{T}\mathcal{X}$ and on $\mathcal{X}$.</footnote>
We see that the set $Char(\sigma_1 \odot \sigma_2) \subset T\mathcal{X}_1 \times T\mathcal{X}_2$ is equal to $Char(\sigma_1) \times Char(\sigma_2)$.
We suppose now that the morphisms $\sigma_k$ are respectively $K_k$-transversally elliptic.
Since $T_{K_1 \times K_2}(\mathcal{X}_1 \times \mathcal{X}_2) \neq T_{K_1}\mathcal{X}_1 \times T_{K_2}\mathcal{X}_2$, the morphism $\sigma_1 \odot \sigma_2$ is not necessarily
$K_1 \times K_2$-transversally elliptic. Nevertheless, if $\sigma_2$ is taken almost homogeneous, then
the morphism $\sigma_1 \odot \sigma_2$ is $K_1 \times K_2$-transversally elliptic (see [37]). So the exterior
product $a_1 \odot a_2$ is the $k$-theory class defined by $\sigma_1 \odot \sigma_2$, where $a_k = [\sigma_k]$ and $\sigma_2$ is
taken almost homogeneous.

The following property is a useful tool (see [1][Lecture 3] and [37]).

**Theorem 4.1 (Multiplicative property).** For any $[\sigma_1] \in K_{K_1 \times K_2}^0(T_{K_1}\mathcal{X}_1)$ and any $[\sigma_2] \in K_{K_2}^0(T_{K_2}\mathcal{X}_2)$ we have

$$ \mathrm{Index}_{\mathcal{X}_1 \times \mathcal{X}_2}^{K_1 \times K_2} ([\sigma_1] \odot [\sigma_2]) = \mathrm{Index}_{\mathcal{X}_1}^{K_1 \times K_2} ([\sigma_1]) \otimes \mathrm{Index}_{\mathcal{X}_2}^{K_2} ([\sigma_2]). $$

4.2. **Riemann-Roch character.** Let $M$ be a compact $K$-manifold equipped with an invariant almost complex structure $J$. Let $p : TM \to M$ be the projection. The complex vector bundle $(T^*M)^{0,1}$ is $K$-equivariantly identified with the tangent bundle $TM$ equipped with the complex structure $J$. Let $h$ be the Hermitian structure on $(TM, J)$ defined by : $h(v, w) = \Omega(v, Jw) - i\Omega(v, w)$ for $v, w \in TM$. The symbol

$$ \mathrm{Thom}(M, J) \in \Gamma(TM, \mathrm{hom}(p^*(\wedge_{\mathbb{C}}^{\mathrm{even}}TM), p^*(\wedge_{\mathbb{C}}^{\mathrm{odd}}TM))) $$

at $(m, v) \in TM$ is equal to the Clifford map

$$ (26) \qquad \mathbf{c}_m(v) : \wedge_{\mathbb{C}}^{\mathrm{even}} \mathbf{T}_m M \to \wedge_{\mathbb{C}}^{\mathrm{odd}} \mathbf{T}_m M, $$

where $\mathbf{c}_m(v).w = v \wedge w - \iota(v)w$ for $w \in \wedge_{\mathbb{C}} \mathbf{T}_m M$. Here $\iota(v) : \wedge_{\mathbb{C}} \mathbf{T}_m M \to \wedge_{\mathbb{C}} \mathbf{T}_m^{-1} M$ denotes the contraction map relative to $h$. Since $\mathbf{c}_m(v)^2 = -\|v\|^2 \mathrm{Id}$, the map $\mathbf{c}_m(v)$ is invertible for all $v \neq 0$. Hence the characteristic set of $\mathrm{Thom}(M, J)$ corresponds to the 0-section of $TM$.

**Definition 4.2.** To any $K$-equivariant complex vector bundle $E \to M$, we associate its Riemann-Roch character

$$ \mathrm{RR}^K(M, E) := \mathrm{Index}_M^K(\mathrm{Thom}(M, J) \otimes E) \in \mathbb{R}(K). $$

**Remark 4.3.** The character $\mathrm{RR}^K(M, E)$ is equal to the equivariant index of the Dolbeault-Dirac operator $\mathcal{D}_E := \sqrt{2}(\overline{\partial}_E + \overline{\partial}_E^*)$, since $\mathrm{Thom}(M, J) \otimes E$ corresponds to the principal symbol of $\mathcal{D}_E$ (see [7][Proposition 3.67]).

4.3. **Definition of $\mathcal{Q}^\Phi$.** Let $(M, \Omega_M, \Phi_M^K)$ be a compact Hamiltonian $K$-manifold pre-quantized by an equivariant line bundle $L_M$. Let $J$ be an invariant almost complex structure compatible with $\Omega$. Let $\mathrm{RR}^K(M, -)$ be the corresponding Riemann-Roch character. The topological index of $\mathrm{Thom}(M, J) \otimes L_M \in K_K^0(TM)$ is equal to the analytical index of the Dolbeault-Dirac operator $\sqrt{2}(\overline{\partial}_{L_M} + \overline{\partial}_{L_M}^*)$:

$$ (27) \qquad \mathcal{Q}_K(M) = \mathrm{RR}^K(M, L_M). $$

When $M$ is not compact the topological index of $\mathrm{Thom}(M, J) \otimes L_M$ is not defined. In order to extend the notion of geometric quantization to this setting we deform the symbol $\mathrm{Thom}(M, J) \otimes L$ in the “Witten” way [34, 35, 29]. Consider
the identification $\xi \mapsto \tilde{\xi}, \mathfrak{k}^* \to \mathfrak{k}$ defined by a $K$-invariant scalar product on $\mathfrak{k}^*$. We define the Kirwan vector field on $M$:

$$ (28) \qquad \kappa_m = \left( \widetilde{\Phi_M^K}(m) \right)_M (m), \quad m \in M. $$

**Definition 4.4.** The symbol $\text{Thom}(M, J) \otimes L$ pushed by the vector field $\kappa$ is the symbol $\mathbf{c}^\kappa$ defined by the relation

$$ \mathbf{c}^{\kappa}|_{m}(v) = \text{Thom}(M, J) \otimes L|_{m}(v - \kappa_{m}) $$

for any $(m, v) \in \mathbf{T}M$. More generally, if $E \to M$ is an equivariant complex vector bundle, one defines the symbol $\mathbf{c}_E^\kappa$ with the same relation (with $E$ at the place of $L$).

Note that $\mathbf{c}^\kappa|_m(v)$ is invertible except if $v = \kappa_m$. If furthermore $v$ belongs to the subset $\mathbf{T}_K M$ of tangent vectors orthogonal to the $K$-orbits, then $v = 0$ and $\kappa_m = 0$. Indeed $\kappa_m$ is tangent to $K \cdot m$ while $v$ is orthogonal.

Since $\kappa$ is the Hamiltonian vector field of the function $\frac{1}{2}\|\Phi_M^K\|^2$, the set of zeros of $\kappa$ coincides with the set of critical points of $\|\Phi_M^K\|^2$. Finally we have

$$ \text{Char}(\mathbf{c}^{\kappa}) \cap \mathbf{T}_K M \simeq \text{Cr}(\|\Phi_M^K\|^2). $$

In general $\text{Cr}(\|\Phi_M^K\|^2)$ is not compact, so $\mathbf{c}^\kappa$ does not define a transversally elliptic symbol on $M$. In order to define a kind of index of $\mathbf{c}^\kappa$, we proceed as follows. For any invariant open relatively compact subset $U \subset M$ the set $\text{Char}(\mathbf{c}^\kappa|_U) \cap \mathbf{T}_K U \simeq \text{Cr}(\|\Phi\|^2) \cap U$ is compact when

$$ (29) \qquad \partial U \cap \text{Cr}(\|\Phi\|^2) = \emptyset. $$

When (29) holds we denote

$$ (30) \qquad \mathcal{Q}_K^\Phi(U) := \text{Index}_U^K(\mathbf{c}^\kappa|_U) \quad \in \quad R_{tc}^{-\infty}(K) $$

the equivariant index of the transversally elliptic symbol $\mathbf{c}^\kappa|_U$.

Let us recall the description of the critical points of $\|\Phi_M^K\|^2$, when the moment map $\Phi_M^K$ is proper. We know that $m \in \text{Cr}(\|\Phi_M^K\|^2)$ if and only if $\tilde{\beta}_M(m) = 0$ for $\beta = \Phi(m)$. Hence the set $\text{Cr}(\|\Phi_M^K\|^2)$ has the following decomposition

$$ \text{Cr}(\|\Phi_M^K\|^2) = \bigcup_{\beta \in \mathfrak{k}^*} M^{\tilde{\beta}} \cap (\Phi_M^K)^{-1}(\beta) = \bigcup_{\beta \in \mathcal{B}} \underbrace{K \cdot (M^{\tilde{\beta}} \cap (\Phi_M^K)^{-1}(\beta))}_{Z_{\beta}}, $$

where $\mathcal{B}$ is a subset of the Weyl chamber $\mathfrak{k}^*$. We denote by $B_r \subset \mathfrak{k}^*$ the open ball $\{\xi \in \mathfrak{k}^* | \|\xi\| < r\}$. The following Proposition is proved in [39].

**Proposition 4.5.** • For any $r > 0$, the set $\mathcal{B} \cap B_r$ is finite.

• The set of singular values of $\|\Phi_M^K\|^2 : M \to \mathbb{R}$ forms a sequence $0 \le r_1 < r_2 < \dots < r_k < \dots$ which is finite if and only if $\text{Cr}(\|\Phi_M^K\|^2)$ is compact. In the other case $\lim_{k \to \infty} r_k = \infty$.

For any $\beta \in \mathcal{B}$, we consider a relatively compact open invariant neighbourhood $\mathcal{U}_\beta$ of $Z_\beta$ such that $\text{Cr}(\|\Phi_M^K\|^2) \cap \overline{\mathcal{U}_\beta} = Z_\beta$. The excision property tell us that the generalized character $\mathcal{Q}_K^\Phi(\mathcal{U}_\beta) = \text{Index}_{\mathcal{U}_\beta}^K(\mathbf{c}^\kappa|_{\mathcal{U}_\beta})$ does not depend of the choice of $\mathcal{U}_\beta$. In order to simplify the notations we consider the following
**Definition 4.6.** • We denote $Q_K^\beta(M) \in R_{tc}^{-\infty}(K)$ the equivariant index$^8$ of the transversally elliptic symbol $c^\kappa|_{u_\beta}$.

• When $E \to M$ is an equivariant complex vector bundle, we denote $RR_\beta^K(M, E)$ the equivariant index of the transversally elliptic symbol $c_E^\kappa|_{u_\beta}$.

The following crucial property is proved in [29, 39].

**Theorem 4.7.** A representation $V_\lambda^K$ occurs in the generalized character $Q_K^\beta(M) \in R^{-\infty}(K)$ only if $\|\lambda\| \ge \|\beta\|$.

**Definition 4.8.** The generalized character $Q_K^\Phi(M) \in R^{-\infty}(K)$ is defined by

$$ (31) \qquad Q_K^\Phi(M) = \sum_{\beta \in \mathcal{B}} Q_K^\beta(M). $$

The sum (31) converges in $R^{-\infty}(K)$ since we know after Theorem 4.7 that the multiplicity of $V_\lambda^K$ in $Q_K^\beta(M)$ is zero when $\|\beta\| > \|\lambda\|$.

We finish this section, by recalling a result that will be needed in Section 4.5. Suppose that $\mathfrak{k} = \mathfrak{k}_1 \oplus \mathfrak{k}_2$ where $[\mathfrak{k}_1, \mathfrak{k}_2] = 0$ and $\mathfrak{k}_i$ are the Lie algebras of closed connected subgroups $K_i$. We assume that the moment map $\Phi_M^{K_1}: M \to \mathfrak{k}_1^*$ relative to the $K_1$-action is proper. Let us explain how we can use the $K$-invariant proper map $\|\Phi_M^{K_1}\|^2$ instead of $\|\Phi_M^K\|^2$ in order to defined the the geometric quantization $Q_K^\Phi(M)$.

Let us choose $\mathfrak{t} = \mathfrak{t}_1 \oplus \mathfrak{t}_2$ such that $\mathfrak{t}_i \subset \mathfrak{t}_i$ is a maximal abelian sub-algebras. We start a decomposition

$$ (32) \qquad \mathrm{Cr}(\|\Phi_M^{K_1}\|^2) = \bigcup_{\beta \in \mathcal{B}_1} \underbrace{K \cdot (M^{\tilde{\beta}} \cap (\Phi_M^{K_1})^{-1}(\beta))}_{Z_\beta^1}, $$

with $\mathcal{B}_1 \subset \mathfrak{t}_1^*$.

Let $\kappa_1$ be the Hamiltonian vector field of $\frac{1}{2}\|\Phi_M^{K_1}\|^2$, and let $c^{\kappa_1}$ be the corresponding pushed symbol. For any $\beta \in \mathcal{B}_1$, we consider a relatively compact open $K$-invariant neighbourhood $\mathcal{U}_\beta^1$ of $Z_\beta^1$ such that $\mathrm{Cr}(\|\Phi_M^{K_1}\|^2) \cap \mathcal{U}_\beta^1 = Z_\beta^1$. We denote $Q_K^{\beta,1}(M) \in R^{-\infty}(K)$ the equivariant index of the $K_1$-transversally elliptic symbol $c^{\kappa_1}|_{\mathcal{U}_\beta^1}$. Theorem 4.7 admits the following extension

**Theorem 4.9.** A representation $V_\lambda^K$ occurs in the generalized character $Q_K^{\beta,1}(M)$ only if $\|\lambda_1\| \ge \|\beta\|$. Here $\lambda \in \Lambda^* \subset \mathfrak{t}^*$ is decomposed in $\lambda = \lambda_1 \oplus \lambda_2$ with $\lambda_i \in \mathfrak{t}_i^*$.

Like in Definition 4.8, we can define the generalized character $Q_K^{\Phi_1}(M) \in R^{-\infty}(K)$ by

$$ (33) \qquad Q_K^{\Phi_1}(M) = \sum_{\beta \in \mathcal{B}_1} Q_K^{\beta,1}(M). $$

In [39][Section 4.1], we prove the following

**Theorem 4.10.** Let $(M, \Omega_M, \Phi_M^K)$ be a proper Hamiltonian $K$-manifold that is pre-quantized. If the moment map $\Phi_M^{K_1}: M \to \mathfrak{k}_1^*$ is proper, we have

$$ Q_K^\Phi(M) = Q_K^{\Phi_1}(M) \qquad \text{in } R^{-\infty}(K). $$

<footnote>$^8$The index of $c^\kappa|_{u_\beta}$ was denoted $RR_\beta^K(M, L)$ in [34]</footnote>
4.4. **Proof of Theorem 3.25 under Assumption A1.** In this section we consider the manifold $M = G \times_K Y$, where $(Y, \Omega_Y, \Phi_Y^K)$ is a Hamiltonian $K$-manifold pre-quantized by a line bundle $L_Y$. We suppose that the moment map $\Phi_Y^K$ is proper, and that the Kirwan polytope $\Delta_K(Y)$ is contained in the cone $C_{\text{hol}}^{\rho}(z) \subset t_{\text{se}}^*$.

Then on $M$, we have an induced $G$-invariant symplectic form $\Omega_M$ and a moment map $\Phi_M^G : M \to \mathfrak{g}^*$ defined by $\Phi_M^G([g, y]) = g \cdot \Phi_M^K(y)$. We know that the line bundle $L_M = (G \times L_Y)/K$ pre-quantizes the Hamiltonian manifold $(M, \Omega_M, \Phi_M^G)$. Let us consider the $K$-action on $M$: the moment map $\Phi_M^K$ is also proper.

We are then in a setting where the formal geometric quantization of $M$ and $Y$ relatively to the $K$-action are well defined: $\mathcal{Q}_K^\Phi(M)$, $\mathcal{Q}_K^\Phi(Y) \in \mathbb{R}^{-\infty}(K)$. The aim of this section is to prove that

$$ (34) \qquad \mathcal{Q}_K^\Phi(M) = \mathcal{Q}_K^\Phi(Y) \otimes S^\bullet(\mathfrak{p}), $$

when $(M, \Omega_M, \Phi_M^G)$ satisfies Assumption A1. Then the set (see Theorem 2.9)

$$ \mathrm{Cr}(\|\Phi_M^G\|^2) = \mathrm{Cr}(\|\Phi_M^K\|^2) = \mathrm{Cr}(\|\Phi_Y^K\|^2) = \bigcup_{\beta \in \mathcal{B}} \underbrace{K \cdot (Y^{\tilde{\beta}} \cap (\Phi_Y^K)^{-1}(\beta))}_{Z_\beta} $$

is compact: the parametrizing set $\mathcal{B}$ is finite. So we have $\mathcal{Q}_K^\Phi(M) = \sum_{\beta \in \mathcal{B}} \mathcal{Q}_K^\beta(M)$ and $\mathcal{Q}_K^\Phi(Y) = \sum_{\beta \in \mathcal{B}} \mathcal{Q}_K^\beta(Y)$, and we reduce to show the following

**Theorem 4.11.** For any $\beta \in \mathcal{B}$, the following relation

$$ (35) \qquad \mathcal{Q}_K^\beta(M) = \mathcal{Q}_K^\beta(Y) \otimes S^\bullet(\mathfrak{p}), $$

holds in $\mathbb{R}^{-\infty}(K)$.

*Proof.* Let $\kappa_M$ be the Kirwan vector field on $M$ associated to the moment map $\Phi_M^K$. Let $J_M$ a $K$-invariant almost complex structure compatible with $\Omega_M$, and let $\mathcal{U}_\beta \subset M$ be a (small) neighbourhood of $Z_\beta$ in $M$.

The symbol Thom$(M, J_M) \otimes L_M$ pushed by the vector field $\kappa_M$ is denoted $c_M^\kappa$. By definition $\mathcal{Q}_K^\beta(M)$ is the equivariant index of the $K$-transversally elliptic symbol $c_M^\kappa|_{u_\beta}$. Note that $\mathcal{Q}_K^\beta(M)$ does not depend on the choice of the neighbourhood $\mathcal{U}_\beta$ nor on the choice of the almost complex structure on $\mathcal{U}_\beta \subset M$.

We use the $K$-diffeomorphism $\varphi : \mathfrak{p} \times Y \simeq M$ defined by $\varphi(X, y) = [e^X, y]$. The Kirwan vector field $\kappa_{\mathfrak{p} \times Y} := \varphi^*(\kappa_M)$ is defined by the relations : $\kappa_{\mathfrak{p} \times Y}(X, y) = (\kappa_1(X, y), \kappa_2(X, y)) \in \mathbb{T}_{\mathfrak{p}} \times Y$ where$^9$

$$ \kappa_1(X, y) = A_Y(y), \quad \kappa_2(X, y) = -[A, X] \quad \text{and} \quad A = [e^X \cdot \widetilde{\Phi_Y^K}(y)]_t. $$

The Kostant-Souriau line bundle $\varphi^*(L_M)$ is $K$-diffeomorphic with $L_Y$ since $Y$ is a deformation retract of $\mathfrak{p} \times Y$. Let us compute the pull-back of the symplectic form $\Omega_{\mathfrak{p} \times Y} = \varphi^*(\Omega_M)$ at $(0, y)$. For $v, v' \in \mathbb{T}_Y Y$ and $\eta, \eta' \in \mathbb{T}_0 \mathfrak{p} = \mathfrak{p}$, we have

$$ \begin{aligned} \Omega_{\mathfrak{p} \times Y}(\eta \oplus v, \eta' \oplus v') &= \Omega_M(v \oplus \eta \cdot y, v' \oplus \eta' \cdot y) \\ &= \Omega_Y(v, v') + \langle \Phi_Y^K(y), [\eta, \eta'] \rangle. \end{aligned} $$

**Lemma 4.12.** $\langle \xi, [\eta, \mathrm{ad}(z)\eta] \rangle = -([\tilde{\xi}, \eta], [z, \eta]) < 0$ for any $\xi \in K \cdot C_{\text{hol}}^{\rho}(z)$ and any $\eta \in \mathfrak{p} \setminus \{0\}$.

<footnote>$^9[Z]_t$ and $[X]_p$ are respectively the $t$ and $p$ components of $Z \in \mathfrak{g}$.</footnote>
*Proof.* Recall that the scalar product on $\mathfrak{g}$ is defined by $(X, Y) = -b(X, \Theta(Y))$. Hence

$$ 
\begin{align*}
\langle \xi, [\eta, \mathrm{ad}(z)\eta] \rangle &= -b(\tilde{\xi}, \Theta([\eta, \mathrm{ad}(z)\eta])) \\
&= (\mathrm{ad}(z)\mathrm{ad}(\tilde{\xi})\eta, \eta) \\
&= (\mathrm{ad}(z)\mathrm{ad}(\tilde{\xi}')\eta', \eta')
\end{align*}
$$

where $\xi = k \cdot \xi'$ with $\xi' \in C_{\mathrm{hol}}^{\rho}(z)$ and $\eta = k \cdot \eta'$ for some $k \in K$. We can then check that the symmetric endomorphism $\mathrm{ad}(z)\mathrm{ad}(\tilde{\xi}') : \mathfrak{p} \to \mathfrak{p}$ is negative definite when $\xi' \in C_{\mathrm{hol}}^{\rho}(z)$: the Lemma is proved. $\square$

If $J_Y$ is a $K$-invariant almost complex structure on $Y$ compatible with $\Omega_Y$, the last Lemma tells us that $(-\mathrm{ad}(z), J_Y)$ is a $K$-invariant almost complex structure on $\mathfrak{p} \times Y$ compatible with $\Omega_{\mathfrak{p} \times Y}$ in a neighbourhood of $Y$.

Let us fix $U_\beta$, such that $\varphi^{-1}(U_\beta) = B_r \times V_\beta$ where $V_\beta$ is a neighbourhood of $Z_\beta$ in $Y$ and $B_r := \{X \in \mathfrak{p} \mid \|X\| < r\}$. The almost complex structure $J_M$ on $U_\beta$ defined by $\varphi^*(J_M) = (-\mathrm{ad}(z), J_Y)$ is compatible with $\Omega_M$ if $V_\beta$ and $B_r$ are small enough. Finally we see that the symbol $\varphi^*(\mathbf{c}_M^\kappa|_{U_\beta})$ is equal to the product $\sigma_1 \odot \sigma_2|_{B_r \times V_\beta}$, where

$$ \sigma_2(X, y; \eta, v) = \mathbf{c}(v - \kappa_1(X, y)), \quad (X, y; \eta, v) \in \mathbf{T}(\mathfrak{p} \times Y), $$

acts on $\wedge_{\mathbb{C}}^* \mathbf{T}_Y Y \otimes L_Y$, and

$$ \sigma_1(X, y; \eta, v) = \mathbf{c}(\eta - \kappa_2(X, y)), \quad (X, y; \eta, v) \in \mathbf{T}(\mathfrak{p} \times Y), $$

acts on $\wedge_{\mathbb{C}}^* \mathfrak{p}^-$ (here $\mathfrak{p}^-$ denotes the complex $K$-module $(\mathfrak{p}, -\mathrm{ad}(z)))$.

Let $\kappa_Y$ be the Kirwan vector field on $Y$ associated to the moment map $\Phi_Y^K$. We denoted $\mathbf{c}_Y^\kappa$, the symbol Thom$(Y, J_Y) \otimes L_Y$ pushed by the vector field $\kappa_Y$. By definition $\mathcal{Q}_K^\beta(Y)$ is the equivariant index of the $K$-transversally elliptic symbol $\mathbf{c}_Y^\kappa|_{V_\beta}$.

The Atiyah symbol $\mathrm{At}_\mathfrak{p}$ on $\mathfrak{p}$ is defined by the following relations : for $(X, \eta) \in \mathbf{T}\mathfrak{p}$,

$$ (36) \qquad \mathrm{At}_{\mathfrak{p}}(X, \eta) := \mathbf{c}(\eta + [z, X]) : \wedge_{\mathbb{C}}^{\mathrm{even}} \mathfrak{p}^- \to \wedge_{\mathbb{C}}^{\mathrm{odd}} \mathfrak{p}^-. $$

**Lemma 4.13.** The symbols $\sigma_1 \odot \sigma_2|_{B_r \times V_\beta}$ and $\mathrm{At}_\mathfrak{p} \odot \mathbf{c}_Y^\kappa|_{B_r \times V_\beta}$ define the same class in $\mathbf{K}_K^0(\mathbf{T}_K(B_r \times V_\beta))$.

*Proof.* We consider the paths $s \in [0, 1] \mapsto A^s := [e^{sX} \cdot \widetilde{\Phi}_Y^K(y)]_s$, $\kappa_1^s(X, y) = A_Y^s(y)$, and $\kappa_2^s(X, y) = -[A^s, X]$. We define then the paths at the level of symbols : $\sigma_1^s$ and $\sigma_2^s$. We check that

$$ \mathrm{Char}(\sigma_1^s \odot \sigma_2^s) \cap \mathbf{T}_K(Y \times \mathfrak{p}) = \{(X, y; v, \eta) \mid v = A_Y^s(y) = 0, \text{ and } \eta = [A^s, X] = 0\}. $$

But since $\Phi_Y^K(y) \in \mathfrak{k}_{se}^*$, the condition $[A^s, X] = [e^X \cdot \widetilde{\Phi}_Y^K(y), X]_\mathfrak{p} = 0$ forces $X$ to be equal to 0. Hence we get

$$ \mathrm{Char}(\sigma_1^s \odot \sigma_2^s) \cap \mathbf{T}_K(Y \times \mathfrak{p}) \simeq \mathrm{Cr}(\|\Phi_Y^K\|^2) \times \{0\}, \quad \forall s \in [0, 1]. $$

We have proved that $s \in [0, 1] \mapsto \sigma_1^s \odot \sigma_2^s|_{B_r \times V_\beta}$ is an homotopy of transversally elliptic symbols: $\sigma_1 \odot \sigma_2$ and $\sigma_1^0 \odot \sigma_2^0$ define the same class in $\mathbf{K}_K^0(\mathbf{T}_K(B_r \times V_\beta))$.

We see that $\sigma_2^0 = \mathbf{c}_Y^\kappa$ and we have

$$ \sigma_1^0(X, y; \eta, v) = \mathbf{c}(\eta + [\widetilde{\Phi}_Y^K(y), X]). $$
We consider another path of symbols

$$
\tau^t(X, y; \eta, v) = \mathbf{c}(\eta + [t\tilde{\Phi}_Y^K(y) + (1-t)z, X]), \quad t \in [0, 1].
$$

We check that if $(X, y; \eta, v) \in \mathrm{Char}(\tau^t \odot \mathbf{c}_Y^\kappa) \cap \mathbf{T}_K(\mathbf{p} \times Y)$ then the vector $\eta \oplus v \in \mathbf{T}_{(X,y)}(\mathbf{p} \times Y)$ is orthogonal to the vector field generated by $\widetilde{\Phi}_Y^K(y)$ and we have moreover $v = \kappa_Y(y)$ and $\eta = -[t\Phi_Y^K(y) + (1-t)z, X]$. Thus

$$
\begin{align*}
0 &= \|\kappa_Y(y)\|^2 + \underbrace{([t\tilde{\xi} + (1-t)z, X], [\tilde{\xi}, X])}_{\delta} \\
&= \|\kappa_Y(y)\|^2 + t \underbrace{\|[\tilde{\xi}, X]\|^2 + (1-t) \underbrace{([z, X], [\tilde{\xi}, X])}_{\delta}}_{\delta}
\end{align*}
$$

where $\xi = \Phi_Y^K(y) \in K \cdot C_{\mathrm{hol}}^\rho(z)$. Since $\xi$ is strongly elliptic and thanks to Lemma 4.12, we know that the term $\delta$ is strictly positive if $X \neq 0$, thus $\kappa_Y(y) = 0$ and $X = 0$.

We have proved that $t \in [0, 1] \mapsto \tau^t \odot \mathbf{c}_Y^\kappa|_{B_r \times V_\beta}$ is an homotopy of $K$-transversally elliptic symbols: $\sigma_1^0 \odot \sigma_2^0$ and $\mathrm{At}_p \odot \mathbf{c}_Y^\kappa$ define the same class in $\mathbf{K}_K^0(\mathbf{T}_K(B_r \times V_\beta))$. $\square$

At this stage we know that

$$
Q_K^\beta(M) = \mathrm{Index}_{B_r \times V_\beta}^K (\mathrm{At}_p|_{B_r} \odot \mathbf{c}_Y^\kappa|_{V_\beta}) \in \mathbb{R}^{-\infty}(K).
$$

Since $\mathbf{c}_Y^\kappa \odot \mathrm{At}_p$ is also $K$-transversally elliptic on $V_\beta \times \mathbf{p}$, the excision property gives
also

$$
Q_K^\beta(M) = \mathrm{Index}_{V_\beta \times \mathbf{p}}^K (\mathrm{At}_p \odot \mathbf{c}_Y^\kappa |_{V_\beta}) \in \mathbb{R}^{-\infty}(K).
$$

Let $S^1$ be the circle subgroup of $K$ with Lie algebra equal to $\mathbb{R}z$. We can consider
$\mathfrak{p}$ as a $S^1 \times K$-manifold. We note that the Atiyah symbol $\mathrm{At}_{\mathfrak{p}}$ is $S^1 \times K$-equivariant
and $S^1$-transversally elliptic. Its index is computed in [1], see also [34][Section 5].
We have the following relation

$$
\mathrm{Index}_{\mathfrak{p}}^{S^1 \times K}(\mathrm{At}_{\mathfrak{p}}) = S^{\bullet}(\mathfrak{p})
$$

in $R^{-\infty}(S^1 \times K)$.

So we have two classes $\mathrm{At}_{\mathfrak{p}} \in \mathbf{K}_{S^1 \times K}^0(\mathbf{T}_{S^1} \mathfrak{p})$, and $\mathbf{c}_Y^\kappa|_{V_\beta} \in \mathbf{K}_K^0(\mathbf{T}_K V_\beta)$. By the
*multiplicative property* (see Theorem 4.1) we know that their product $\mathrm{At}_{\mathfrak{p}} \odot \mathbf{c}_Y^\kappa|_{V_\beta} \in$
$\mathbf{K}_{K \times S^1}^0(\mathbf{T}_{K \times S^1}(Y \times \mathfrak{p}))$ has the following $S^1 \times K$-equivariant index

$$
\begin{align*}
\mathrm{Index}_{\mathfrak{p} \times V_{\beta}}^{S^1 \times K} (\mathrm{At}_{\mathfrak{p}} \odot \mathbf{c}_{Y}^{\kappa} |_{V_{\beta}}) &= \mathrm{Index}_{\mathfrak{p}}^{S^1 \times K} (\mathrm{At}_{\mathfrak{p}}) \otimes \mathrm{Index}_{V_{\beta}}^{K} (\mathbf{c}_{Y}^{\kappa} |_{V_{\beta}}) \\
&= S^{\bullet}(\mathfrak{p}) \otimes \mathrm{Index}_{V_{\beta}}^{K} (\mathbf{c}_{Y}^{\kappa} |_{V_{\beta}}) \\
&= S^{\bullet}(\mathfrak{p}) \otimes Q_{K}^{\beta}(Y) \quad \in \mathbb{R}^{-\infty}(S^1 \times K).
\end{align*}
$$

Finally, thanks to the *restriction property* (24), we know that

$$
Q_K^\beta(M) = \mathrm{Index}_{\mathfrak{p} \times V_\beta}^K (\mathrm{At}_{\mathfrak{p}} \odot \mathbf{c}_Y^\kappa |_{V_\beta}) \in \mathbb{R}^{-\infty}(K)
$$

is equal to the restriction of

$$
\mathrm{Index}_{\mathfrak{p} \times V_{\beta}}^{S^1 \times K} (\mathrm{At}_{\mathfrak{p}} \odot \mathbf{c}_{Y}^{\kappa} |_{V_{\beta}}) = S^{\bullet}(\mathfrak{p}) \otimes Q_{K}^{\beta}(Y) \in \mathbb{R}^{-\infty}(S^{1} \times K)
$$

to the subgroup $K \hookrightarrow S^1 \times K$. The Theorem is then proved. $\square$
**Remark 4.14.** The Assumption A1 is used because we don't how to prove the equality

$$ \sum_{\beta \in \mathcal{B}} (Q_K^{\beta}(Y) \otimes S^{\bullet}(\mathfrak{p})) = \left( \sum_{\beta \in \mathcal{B}} Q_K^{\beta}(Y) \right) \otimes S^{\bullet}(\mathfrak{p}) $$

when the set $\mathcal{B}$ is not finite, e.g. the set $\mathrm{Cr}(\|\Phi_M^K\|^2)$ is non-compact.

**4.5. Proof of Theorem 3.30.** Here we work with a pre-quantized Hamiltonian $K$-manifold $(P, \Omega_P, \Phi_P^K)$, and we assume that the map $\langle \Phi_P^K, z \rangle$ is proper. Here $\mathbb{R}z$ is the Lie algebra of a circle subgroup $S^1 \subset K$ contained in the center of $K$.

We are in the context of Theorem 4.10. We have a decomposition $\mathfrak{k} = \mathfrak{k}_1 \oplus \mathfrak{k}_2$ where $\mathfrak{k}_1 := \mathbb{R}z$ and $\mathfrak{k}_2$ are ideals of $\mathfrak{k}$ and the moment map $\langle \Phi_P^K, z \rangle$ relative to the $S^1$-action is proper. Then we have the following equality

$$ (37) \qquad Q_K^{-\infty}(M) = Q_K^{\Phi}(P) = Q_K^{\langle \Phi, z \rangle}(P) \quad \in \mathbb{R}^{-\infty}(K) $$

where the right hand side is computed via a localization procedure on the set $\mathrm{Cr}(\varphi_P)$ of critical points of the proper map $\varphi_P := (\langle \Phi_P^K, z \rangle)^2$. We note that

$$ \mathrm{Cr}(\varphi_P) = \varphi_P^{-1}(0) \cup P^z. $$

We are interested in the following cases

(1) $P$ is a proper Hamiltonian $G$-manifold $(M, \Omega_M, \Phi_M^G)$ with a moment map taking values in $G \cdot C_{\mathrm{hol}}^\rho(z)$, and which satisfies Assumption A2.

(2) $P$ is the symplectic slice $Y$ of the former case $M := G \times_K Y$.

Thanks to Lemma 3.21, we know that in the two cases described above, the proper map $\varphi_P$ is strictly positive : hence $\varphi_P^{-1}(0) = \emptyset$. Let us compute the generalized character $Q_K^{\langle \Phi, z \rangle}(P)$ in this case.

Let $\kappa_\varphi$ be the Hamiltonian vector field of $\frac{1}{2}\varphi_P$. The symbol $\mathrm{Thom}(P, J_P) \otimes L_P$ pushed by the vector field $\kappa_\varphi$ is denoted $c_P^\varphi$. Let $\mathcal{B}_P$ the set of connected component of $P^z$. For any $\mathcal{X} \in \mathcal{B}_P$, we consider a relatively compact open $K$-invariant neighbourhood $\mathcal{U}_\mathcal{X}$ of $\mathcal{X}$ such that $\mathrm{Cr}(\varphi_P) \cap \overline{\mathcal{U}_\mathcal{X}} = \mathcal{X}$. We denote $Q_K^\mathcal{X}(P) \in \mathbb{R}^{-\infty}(K)$ the equivariant index of the $S^1$-transversally elliptic symbol $c^\varphi|_{\mathcal{U}_\mathcal{X}}$.

When $\varphi_P^{-1}(0) = \emptyset$, the generalized character $Q_K^{\langle \Phi, z \rangle}(P)$ is defined by the relation

$$ (38) \qquad Q_K^{\langle \Phi, z \rangle}(P) = \sum_{\mathcal{X} \in \mathcal{B}_P} Q_K^{\mathcal{X}}(P) \in \mathbb{R}^{-\infty}(K). $$

For $\mathcal{X} \in \mathcal{B}_P$, we denote

• $L_\mathcal{X}$ the restriction of the Kostant-Souriau line bundle $L_P$ on $\mathcal{X}$,

• $\mathcal{N}_\mathcal{X}$ the normal bundle of $\mathcal{X}$ in $P$, and $|\mathcal{N}_\mathcal{X}|^z, \mathcal{N}_\mathcal{X}^{+,z}$ are the $z$-polarized versions (see Section 3.8).

If we use (37) and (38), the proof of Theorem 3.30 is reduced to the following

**Proposition 4.15.** We have the following equality in $\mathbb{R}^{-\infty}(K)$:

$$ (39) \qquad Q_K^\mathcal{X}(P) = (-1)^{r_\mathcal{X}} \mathrm{RR}^K (\mathcal{X}, L_\mathcal{X} \otimes \det(\mathcal{N}_\mathcal{X}^{+,z}) \otimes S^\bullet(|\mathcal{N}_\mathcal{X}|^z)), $$
where $r_\mathcal{X}$ is the complex rank of $\mathcal{N}_\mathcal{X}^{+,z}$.

*Proof.* Relations (5) show that $\kappa_\varphi = \langle \Phi_P^K, z \rangle z_P$. Since $\langle \Phi_P^K, z \rangle > 0$ in a neighbourhood of $\mathcal{U}_\mathcal{X}$, we can replace $\kappa_\varphi$ by the vector field $z_P$ without changing the index of
the corresponding transversally elliptic operator. This means that $Q_K^\chi(M)$ is equal to the index of $\sigma^z|_{\mathcal{U}_\chi}$, where the symbol $\sigma^z$ is defined by : for $(m, v) \in \mathbf{T}P$,

$$
(40) \quad \sigma^z(m, v) := \mathbf{c}(v - z_P(m)) : \wedge_{\mathbb{C}}^{\text{even}} \mathbf{T}_m P \otimes L_P|_m \to \wedge_{\mathbb{C}}^{\text{odd}} \mathbf{T}_m P \otimes L_P|_m.
$$

We have proved in [34][Theorem 5.8], that the index of $\sigma^z|_{\mathcal{U}_X}$ is equal to the right hand side of (39). Hence the proof is completed. $\square$

We want now to clarify the convergence of the sum that appears in (38), when $P^z$ is non-compact.

Let $T$ be a maximal torus in $K$: it contains the circle subgroup $S^1$. Let $\Lambda \subset \mathfrak{t}$ be the lattice which is the kernel of exp : $\mathfrak{t} \to T$. Let $z_o \in \mathbb{R}^{>0}z \cap \Lambda$ that generates the sub-lattice $\mathbb{R}z \cap \Lambda$: the torus $S^1$ acts on an irreducible representation $V_\mu^K$ through the character $t \mapsto t^n$ with $n = \frac{\langle \mu, z_o \rangle}{2\pi} \in \mathbb{Z}$. We have then a graduation $R(K) = \sum_{n \in \mathbb{Z}} R_n(K)$ where $R_n(K)$ is the group generated by the representations $V_\mu^K$ such that $\frac{\langle \mu, z_o \rangle}{2\pi} = n$. We see that $R_n(K) \cdot R_m(K) \subset R_{n+m}(K)$.

For any $n \in \mathbb{Z}$, we denote $R_{\ge n}(K)$ (resp. $R_{\ge n}^{-\infty}(K)$) the subgroup formed by the
finite (resp. infinite) sum $\sum_{l \ge n} E_l$ where $E_l \in R_l(K)$. We have the following basic
lemma

**Lemma 4.16.** • If $A \in R_{\ge n}^{-\infty}(K)$ and $B \in R_{\ge m}^{-\infty}(K)$, then the product $A \cdot B$ is well-defined and belongs to $R_{\ge n+m}^{-\infty}(K)$.

* An infinite sum $\sum_{n \ge 0} A_n$, with $A_n \in R_{\ge n}^{-\infty}(K)$, converges in $R_{\ge 0}^{-\infty}(K)$.

Proof. The proof is left to the reader.

<figure><img src="image_10.png" /></figure>

For $\mathcal{X} \in \mathcal{B}_P$, the action of $S^1$ is trivial on $\mathcal{X}$, and relation (11) shows that $S^1$ acts on the fibres of Kostant-Souriau line bundle $L_{\mathcal{X}}$ through the character $t \mapsto t^{n(\mathcal{X})}$, where $n(\mathcal{X}) = \frac{\langle \Phi_P^K(\mathcal{X}), z_o \rangle}{2\pi}$ is a strictly positive integer.

**Proposition 4.17.** • The generalized character $Q_K^\chi(P)$ belongs to $R_{\ge n(\chi)}^{-\infty}(K)$.

* The sum $\sum_{\chi \in \mathcal{B}_P} Q_K^\chi(P)$ converges in $R_{\ge 0}^{-\infty}(K)$.

*Proof.* The generalized character $Q_K^\chi(P)$ is equal to the sum $(-1)^{r(\chi)} \sum_{p \ge 0} E_p$, with

$$
E_p = \mathrm{RR}^K(\mathcal{X}, L_\mathcal{X} \otimes \det(\mathcal{N}_\mathcal{X}^{+,z}) \otimes S^p(|\mathcal{N}_\mathcal{X}|^z)) \in \mathcal{R}(K).
$$

Since the group $S^1$ acts on the fibres of the polarized bundles $\mathcal{N}_{\mathcal{X}}^{+,z}$ and $|\mathcal{N}_{\mathcal{X}}|^z$
through characters $t^n$ with $n > 0$, we see that $E_p \in R_{\ge n(\mathcal{X})+p}(K)$. Hence $Q_K^\chi(P) =$
$(-1)^{r(\mathcal{X})} \sum_{p \ge 0} E_p$ converges in $R_{\ge n(\mathcal{X})}^{-\infty}(K)$.

For the second point we see that $\sum_{\chi \in \mathcal{B}_P} Q_K^\chi(P) = \sum_{n \ge 0} A_n$ with

$$
A_n = \sum_{n(\mathcal{X})=n} Q_K^{\mathcal{X}}(P) \in R_{\ge n}^{-\infty}(K).
$$

The former sum is finite (and then well-defined) because the map $\langle \Phi_P^K, z_o \rangle$ is proper
: for any $C > 0$, we have only a finite number of $\mathcal{X} \in \mathcal{B}_P$ such that $\langle \Phi_P^K(\mathcal{X}), z_o \rangle \le C$.
The last point is proved.
$\square$

<figure><img src="image_11.png" /></figure>
4.6. **Proof of Theorem 3.25 under Assumption A2.** The proof of Theorem 3.25, in the case where Assumption A2 is satisfied, follows directly of the results of the Section 4.5 applied to the following two cases:

(1) $(M, \Omega_M, \Phi_M^G)$ is a proper Hamiltonian $G$-manifold with a moment map taking values in $G \cdot C_{\text{hol}}^\rho(z)$, and which satisfies Assumption A2.

(2) $Y$ is the symplectic slice of the former case $M := G \times_K Y$.

Note that the fixed point set $M^z$ and $Y^z$ coincide. For a connected component $\mathcal{X}$ of $Y^z$, let $\mathcal{N}_{\mathcal{X}}$ (resp. $\mathcal{N}_{\mathcal{X}}'$) be the normal bundle of $\mathcal{X}$ in $M$ (resp. $Y$). Since the normal bundle of $Y$ in $M$ is the trivial bundle $Y \times \mathfrak{p}$, we have $\mathcal{N}_{\mathcal{X}} = \mathcal{N}_{\mathcal{X}}' \oplus \mathfrak{p}$. A small computation shows that

$$ \mathcal{N}_{\mathcal{X}}^{+,z} = (\mathcal{N}_{\mathcal{X}}')^{+,z} \quad \text{and} \quad |\mathcal{N}_{\mathcal{X}}|^{+,z} = |\mathcal{N}_{\mathcal{X}}'|^{+,z} \oplus (\mathfrak{p}, \text{ad}(z)). $$

Finally (37) and Proposition 39 gives

$$ \begin{align*} Q_K^{-\infty}(M) &= \sum_{\mathcal{X}} (-1)^{r \times} \text{RR}^K (\mathcal{X}, L_{\mathcal{X}} \otimes \det(\mathcal{N}_{\mathcal{X}}^{+,z}) \otimes S^\bullet(|\mathcal{N}_{\mathcal{X}}|^z)) \\ &= \sum_{\mathcal{X}} (-1)^{r \times} \text{RR}^K (\mathcal{X}, L_{\mathcal{X}} \otimes \det(\mathcal{N}_{\mathcal{X}}')^{+,z} \otimes S^\bullet(|\mathcal{N}_{\mathcal{X}}'|^z) \otimes S^\bullet(\mathfrak{p})) \\ &= \left( \sum_{\mathcal{X}} (-1)^{r \times} \text{RR}^K (\mathcal{X}, L_{\mathcal{X}} \otimes \det(\mathcal{N}_{\mathcal{X}}')^{+,z} \otimes S^\bullet(|\mathcal{N}_{\mathcal{X}}'|^z)) \right) \otimes S^\bullet(\mathfrak{p}) \\ &= Q_K^{-\infty}(Y) \otimes S^\bullet(\mathfrak{p}). \end{align*} $$

We know that the term $\sum_{\mathcal{X}} (-1)^{r \times} \text{RR}^K (\mathcal{X}, L_{\mathcal{X}} \otimes \det(\mathcal{N}_{\mathcal{X}}')^{+,z} \otimes S^\bullet(|\mathcal{N}_{\mathcal{X}}'|^z))$ belongs to $R_{\ge 0}^{-\infty}(K)$ (see Proposition 4.17). We see also that $S^\bullet(\mathfrak{p}) \in R_{\ge 0}^{-\infty}(K)$. Hence their product is well-defined (see Lemma 4.16).

5. APPENDIX: PROOF OF LEMMA 3.28

Let $m = \sum_{\mu \in \hat{G}_{\text{hol}}(z)} m_{\mu} V_{\mu}^{G} \in R^{-\infty}(G, z)$ that is $G'$-admissible : the map $\pi_{g',g}: G \cdot \text{Support}(m) \to (g')^*$ is proper. We start with the

**Lemma 5.1.** • There exists a closed subset $C_m \subset C_{\text{hol}}^{\rho}(z)$ such that $\pi_{g',g}(G \cdot \text{Support}(m)) = G' \cdot C_m$.

• The projection $\pi_{g',g'} : G' \cdot C_m \to (g')^*$ is proper.

• The projection $\pi_{g',g} : \pi_{g,g}(G \cdot \text{Support}(m)) \to (g')^*$ is proper.

*Proof.* First we note that $G \cdot \text{Support}(m)$ is closed in $g^*$. Thanks to point (d) of Proposition 3.15, we know that $\pi_{g',g}(G \cdot \text{Support}(m)) = G' \cdot C_m$, with $C_m = \pi_{g',g}(G \cdot \text{Support}(m)) \cap (t')^* \subset C_{\text{hol}}^{\rho}(z)$. The properness assumption forces $G' \cdot C_m$ (resp. $C_m$) to be closed in $(g')^*$ (resp. $C_{\text{hol}}^{\rho}(z)$). The first point is proved.

Let $B_R := \{\xi' \in (t')^* | \|\xi'\| \le R\}$, and consider $e^{X'} \cdot \lambda' \in G' \cdot C_m \cap \pi_{g',g'}^{-1}(B_R)$. Since $\|\pi_{g',g'}(e^{X'} \cdot \lambda')\| \ge \|\lambda'\|$, we see that $\lambda'$ belongs to a compact subset $K$ of $C_{\text{hol}}^{\rho}(z)$. Then there exists a constant $c(K) > 0$ such that $\|\pi_{g',g'}(e^Y \cdot \lambda')\| \ge c(K)\|X'\|^2$ for any $Y \in \mathfrak{p}'$ and $\lambda' \in K$ (see the proof of Proposition 2.9). Finally, we have proved that $e^{X'} \cdot \lambda'$ belongs to a compact subset of $G' \cdot C_m$.

Let $\eta \in G \cdot \text{Support}(m)$ such that $\xi = \pi_{g,g}(\eta) \in \pi_{g',g'}^{-1}(B_R)$. Then $\pi_{g',g}(\xi) = \pi_{g',g}(\eta) = \pi_{g',g'} \circ \pi_{g,g}(\eta) \in B_R$. Thanks to the former point, we know that $\pi_{g',g}(\eta)$ is bounded. Since $\pi_{g',g}: G \cdot \text{Support}(m) \to (g')^*$ is proper, it implies that $\eta$ and $\xi = \pi_{g,g}(\eta)$ are bounded.
<figure><img src="image_12.png" /></figure>

Consider now the restriction of $m \in R^{-\infty}(G, z)$ to $K$ and $G'$. The generalized
character $n := \mathbf{r}_{K,G}(m) = \sum_{\lambda \in \hat{K}_{\text{hol}}(z)} n_{\lambda} V_{\lambda}^{K}$, is defined by the relations

$$
n_{\lambda} = \sum_{\mu \in \hat{G}_{\text{hol}}(z)} m_{\mu} \mathcal{Q}((G \cdot \mu)_{\lambda,K}),
$$

and $q := \mathbf{r}_{G',G}(m) = \sum_{\nu \in \hat{G}'_{\text{hol}}(z)} q_{\nu} V^{G'}_{\nu}$, is defined by the relations

$$
q_{\nu} = \sum_{\mu \in \hat{G}_{\text{hol}}(z)} m_{\mu} \mathcal{Q}((G \cdot \mu)_{\nu, G'}),
$$

We see that $\lambda \in \text{Support}(n)$ only if there exists $\mu$ such that $m_\mu \mathcal{Q}((G \cdot \mu)_{\lambda,K}) \neq 0$:
hence

$$
\mathrm{Support}(n) \subset \pi_{\mathfrak{t},\mathfrak{g}} (G \cdot \mathrm{Support}(m)).
$$

Since $\pi_{\mathfrak{t}',\mathfrak{t}}$ is proper on $\pi_{\mathfrak{t},\mathfrak{g}}(G \cdot \text{Support}(m))$, we have that $\pi_{\mathfrak{t}',\mathfrak{t}}$ is proper on $K \cdot$
$\text{Support}(n)$. Thus $n \in R^{-\infty}(K, z)$ is $K'$-admissible.

We will now compare the following two elements of $R^{-\infty}(K', z) : A := \mathbf{r}_{K',K}(n) =$
$\sum_{\delta \in \hat{K}'_{\text{hol}}(z)} a_{\delta} V^{K'}_{\delta}$ and $B := \mathbf{r}_{K',G'}(q) = \sum_{\delta \in \hat{K}'_{\text{hol}}(z)} b_{\delta} V^{K'}_{\delta}$.

By definition, we have

$$
\begin{align*}
a_{\delta} &= \sum_{\lambda \in \hat{K}_{\text{hol}}(z)} n_{\lambda} \mathcal{Q}((K \cdot \lambda)_{\delta, K'}) \\
&= \sum_{\mu \in \hat{G}_{\text{hol}}(z)} m_{\mu} \underbrace{\left( \sum_{\lambda \in \hat{K}_{\text{hol}}(z)} \mathcal{Q}((G \cdot \mu)_{\lambda, K}) \mathcal{Q}((K \cdot \lambda)_{\delta, K'}) \right)}_{X_{\mu, \delta}},
\end{align*}
$$

and

$$
\begin{align*}
b_{\delta} &= \sum_{\nu \in \hat{G}'_{\text{hol}}(z)} q_{\nu} \mathcal{Q}((G' \cdot \nu)_{\delta, K'}) \\
&= \sum_{\mu \in \hat{G}_{\text{hol}}(z)} m_{\mu} \underbrace{\left( \sum_{\nu \in \hat{G}_{\text{hol}}(z)} \mathcal{Q}((G' \cdot \nu)_{\delta, K'}) \mathcal{Q}((G \cdot \mu)_{\nu, G'}) \right)}_{Y_{\mu, \delta}}.
\end{align*}
$$

We have

$$
\begin{align*}
X_{\mu, \delta} &= \sum_{\lambda \in \hat{K}_{\text{hol}}(z)} \mathcal{Q}((G \cdot \mu)_{\lambda, K}) \mathcal{Q}((K \cdot \lambda)_{\delta, K'}) \\
&= \sum_{\lambda \in \hat{K}_{\text{hol}}(z)} [V_{\lambda}^{K} : V_{\mu}^{G}|_{K}] \cdot [V_{\delta}^{K'} : V_{\lambda}^{K'}|_{K}] \\
&= [V_{\delta}^{K'} : V_{\mu}^{G}|_{K'}]
\end{align*}
$$

which is finite if $m_\mu \ne 0$. Similarly, we check that $Y_{\mu,\delta} = [V_\delta^{K'} : V_\mu^G]_{K'}$.

Finally, we have proved that $\mathbf{r}_{K',K} \circ \mathbf{r}_{K,G}(m) = A = B = \mathbf{r}_{K',G'} \circ \mathbf{r}_{G',G}(m)$.
REFERENCES

[1] M.F. ATIYAH, Elliptic operators and compact groups, Springer, 1974. Lecture notes in Mathematics, **401**.

[2] M.F. ATIYAH, *Convexity and commuting Hamiltonians*, Bull. London Math. Soc., **14**, 1982, p. 1-15.

[3] M.F. ATIYAH, G.B. SEGAL, *The index of elliptic operators II*, Ann. Math. **87**, 1968, p. 531-545.

[4] M.F. ATIYAH, I.M. SINGER, *The index of elliptic operators I*, Ann. Math. **87**, 1968, p. 484-530.

[5] M.F. ATIYAH, I.M. SINGER, *The index of elliptic operators III*, Ann. Math. **87**, 1968, p. 546-604.

[6] M.F. ATIYAH, I.M. SINGER, *The index of elliptic operators IV*, Ann. Math. **93**, 1971, p. 139-141.

[7] N. BERLINE, E. GETZLER and M. VERGNE, *Heat kernels and Dirac operators*, Grundlehren, vol. 298, Springer, Berlin, 1991.

[8] N. BERLINE and M. VERGNE, The Chern character of a transversally elliptic symbol and the equivariant index, *Invent. Math.*, **124**, 1996, p. 11-49.

[9] N. BERLINE and M. VERGNE, L'indice équivariant des opérateurs transversalement elliptiques, *Invent. Math.*, **124**, 1996, p. 51-101.

[10] L. CORWIN and F.P. GREENLEAF, Spectrum and multiplicities for restrictions of unitary representations in nilpotent Lie groups, *Pacific Journal of Mathematics*, **135**, 1988, p. 233-267.

[11] G. DELTOUR, Propriétés symplectiques et hamiltoniennes des orbites coadjointes holomorphes, Thèse de doctorat, University Montpellier 2, 2010. arXiv:1101.3849

[12] G. DELTOUR, Kirwan polyhedron of holomorphic coadjoint orbits, *Transformation Groups*, 2012. DOI: 10.1007/s00031-011-9164-3. arXiv:1104.1909

[13] G. DELTOUR, On a generalization of a theorem of McDuff, arXiv:1102.4723.

[14] M. DUFLO, G. HECKMAN and M. VERGNE, Projection dorbites, formule de Kirillov et formule de Blattner, *Mem. Soc. Math. France*, **15**, 1984, p. 65128.

[15] M. DUFLO and J.A. VARGAS, Proper map and multiplicities, 2007, preprint.

[16] M. DUFLO and J.A. VARGAS, Branching laws for square integrable representations, *Proc. Japan Acad.*, **86**, 2010, p. 49-54.

[17] M. DUFLO and M. VERGNE, Kirillov's formula and GuilleminSternberg conjecture, *C. R. A. S.*, **349**, 2011, p. 1213-1217.

[18] V. GUILLEMIN and S. STERNBERG, Convexity properties of the moment mapping, *Invent. Math.*, **67**, 1982, p. 491-513.

[19] V. GUILLEMIN and S. STERNBERG, Geometric quantization and multiplicities of group representations, *Invent. Math.*, **67**, 1982, p. 515-538.

[20] V. GUILLEMIN and S. STERNBERG, A normal form for the moment map, in *Differential Geometric Methods in Mathematical Physics* (S. Sternberg, ed.), Reidel Publishing Company, Dordrecht, 1984.

[21] G. HECKMAN, Projection of orbits and asymptotic behavior of multiplicities for compact connected Lie groups, *Invent. Math.*, **67**, 1982, p. 333-356.

[22] P. HOCHS, Quantisation commutes with reduction at discrete series representations of semisimple groups, *Advances in Mathematics*, **222**, 2009, p. 862-919.

[23] H.P. JAKOBSEN and M. VERGNE, Restrictions and expansions of holomorphic representations, *J. Functional Analysis*, **34**, 1979, p. 29-53.

[24] A.A. KIRILLOV, Unitary representations of nilpotent Lie groups, *Uspekhi Mat. Nauk.*, **17**, 1962, p. 53-104.

[25] F. KIRWAN, Convexity properties of the moment mapping III, *Invent. Math.*, **77**, 1984, p. 547-552.

[26] A.W. KNAPP, *Lie groups beyond an introduction*, Progress in Math., **140**, Birkhäuser, Springer, 2004.

[27] T. KOBAYASHI, Discrete decomposability of the restriction of $A_q(\lambda)$ with respect to reductive subgroups and its applications, *Invent. Math.*, **117**, 1994, p. 181-205.

[28] E. LERMAN, E. MEINRENKEN, S. TOLMAN and C. WOODWARD, Non-Abelian convexity by symplectic cuts, *Topology*, **37**, 1998, p. 245-259.

[29] X. MA and W. ZHANG, Geometric quantization for proper moment map, arXiv:0812.3989
[30] C.-M. MARLE, Modèle daction hamiltonienne dun groupe de Lie sur une variété symplectique. *Rendiconti del Seminario Matematico*, **43**, 1985, p. 227251. Università e Politecnico, Torino.

[31] E. MEINRENKEN, Symplectic surgery and the Spin<sup>c</sup>-Dirac operator, *Advances in Math.*, **134**, 1998, p. 240-277.

[32] E. MEINRENKEN, R. SJAMAAR, Singular reduction and quantization, *Topology*, **38**, 1999, p. 699-762.

[33] S.M. PANEITZ, Determination of invariant convex cones in simple Lie algebras, *Arkiv för Matematik*, **21**, 1983, p. 217-228.

[34] P-E. PARADAN, Localization of the Riemann-Roch character, *J. Funct. Anal.* **187**, 2001, p. 442-509.

[35] P-E. PARADAN, Spin<sup>c</sup> quantization and the K-multiplicities of the discrete series, *Annales Scientifiques de l'E. N. S.*, **36**, 2003, p. 805-845.

[36] P-E. PARADAN, Formal geometric quantization, *Ann. Inst. Fourier* **59**, 2009, p. 199-238.

[37] P-E. PARADAN M. VERGNE, Index of transversally elliptic operators, *Astérique, Soc. Math. Fr.*, **328**, 2009, p. 297-338.

[38] P-E. PARADAN, Multiplicities of the discrete series, 38 pages. ArXiv:0812.0059.

[39] P-E. PARADAN, Formal geometric quantization 2, *Pacific Journal of Mathematics*, **253**, 2011, p. 169211.

[40] R. SJAMAAR, Convexity properties of the moment mapping re-examined, *Adv. in Math.*, **138**, 1998, p. 46-91.

[41] R. SJAMAAR, Symplectic reduction and Riemann-Roch formulas for multiplicities, *Bull. Amer. Math. Soc.* **33**, 1996, p. 327-338.

[42] Y. TIAN, W. ZHANG, An analytic proof of the geometric quantization conjecture of Guillemin-Sternberg, *Invent. Math.*, **132**, 1998, p. 229-259.

[43] M. VERGNE, Quantification géométrique et réduction symplectique, *Séminaire Bourbaki* **888**, 2001.

[44] E.B. VINBERG, Invariant convex cones and orderings in Lie groups, *Funct. Anal. and its Applications*, **14**, 1980, p. 1-10.

[45] J. WEITSMAN, Non-abelian symplectic cuts and the geometric quantization of noncompact manifolds. *Lett. Math. Phys.*, **56**, 2001, no. 1, p. 31-40.

[46] A. WEINSTEIN, Poisson geometry of discrete series orbits, and momentum convexity for noncompact group actions, *Lett. Math. Phys.*, **56**, 2001, p. 1730.

[47] E. WITTEN, Two dimensional gauge theories revisited, *J. Geom. Phys.* **9**, 1992, p. 303-368.

INSTITUT DE MATHÉMATIQUES ET DE MODÉLISATION DE MONTPELLIER (I3M), UNIVERSITÉ MONTPELLIER 2

E-mail address: Paul-Emile.Paradan@math.univ-montp2.fr