THEORY OF DISCRETELY DECOMPOSABLE
RESTRICTIONS OF UNITARY REPRESENTATIONS OFSEMISIMPLE LIE GROUPS AND SOME APPLICATION ∗Toshiyuki KOBAYASHIRIMS, Kyoto University
Abstract. Branching problems ask how an irreducible representation of a groupdecomposes when restricted to a subgroup. This exposition surveys new aspects ofbranching problems for unitary representations of reductive Lie groups.
The first half is written from the representation theoretic viewpoint. After anobservation on the wild features of branching problems for non-compact subgroups
in a general setting, we introduce the notion of admissible restrictions as a goodframework that enjoys two properties: finiteness of multiplicities and discretenessof spectrum. A criterion for admissible restrictions is presented, of which the ideaof proof stems from microlocal analysis and algebraic geometry. In this framework,
we present a finite multiplicity theorem. Furthermore, an exclusive law of discrete
spectrum is formulated for inductions and restrictions.
The second half deals with applications. Once we know the non-existence ofcontinuous spectrum in the restrictions, we could expect an algebraic approach tobranching problems. In this framework, new branching formulas have been recently
obtained in various settings, among which we present an example, namely, a generalization of the Kostant-Schmid formula to non-compact subgroups. Finally, we
mention some applications of discretely decomposable branching laws to other fields
of mathematics. The topics include(1) topological properties of modular varieties in locally symmetric spaces,(2) a construction of new discrete series representations for non-Riemannian nonsymmetric homogeneous spaces.
We end the exposition by a brief discussion on the mystery between tessellationof non-Riemannian homogeneous spaces and branching problems of unitary representations.

2000 Mathematics Subject Classification: 22E46, 43A85, 11F67, 53C50,
53D20.
Keywords and Phrases: Unitary representation, branching law, reductive Liegroup, restriction, discrete series, homogeneous space, semisimple Lie group.

∗ This article is an English translation of “Sugaku”, (1999) 51 (4), pp. 337–356.

It was theSpring Prize Lecture to the Mathematical Society of Japan in 1999.
Typeset by AMS-TEX
1

2

TOSHIYUKI KOBAYASHI

Table of Contents§0. Introduction§1. Analytic theory of admissible restrictions
1.A. Branching laws of unitary representations
1.B. Discretely decomposable restrictions as a “nice framework”
1.C. A sufficient condition for discretely decomposable restrictions§2. Algebraic theory of discretely decomposable restrictions
2.A. Algebraic reformulation of discrete decomposability
2.B. A necessary condition for discretely decomposable restrictions— an approach by associated varieties
2.C. Three more theorems on discretely decomposable restrictions§3. New aspect of representation theory related to branching problems
3.A. Finite multiplicity conjectures
3.B. A generalization of the Kostant-Schmid formula to semisimple symmetricpairs
3.C. Unipotent representations and discrete branching laws
3.D. Quantization of elliptic orbits and discrete branching laws
§4. Applications of admissible restrictions
4.A. Branching laws and geometry
4.B. A vanishing theorem for modular varieties
4.C. Construction of new discrete series representations for non-symmetric spaces
4.D. Discontinuous groups versus restrictions of unitary representations§0. Introduction
In my opinion, one of the most fascinating features in representation theory of
Lie groups arises from the “outside”, namely, through various interactions with different fields of mathematics and physics, including partial differential equations, differential geometry, algebraic geometry, functional analysis, combinatorics, number
theory, etc. Furthermore, such interactions are still growing actively and sometimes
show up unexpectedly.
However, if we look at the “inside” of representation theory itself by forgettinginteractions with other branches of mathematics, what remains as central problems?
From the viewpoint of “analysis and synthesis”, we may emphasize the following
two problems:
Problem 1.
Understand irreducible representations.
Find and classify the “smallest” objects.
Problem 2.
Decompose a given representation into irreducible ones.
How is a given representation built from the “smallest” objects?
In traditional chemistry or physics of condensed matter, Problem 1 would correspond to the “classification of atoms” [or elementary particles, · · · ] (the level
depends on what we regard as “smallest”), while Problem 2 would correspond to
the “analysis and synthesis” of molecules [or of atoms, · · · ].
Let us consider Problems 1 and 2 for Lie groups and their representation theory.
First, the “smallest objects” for Lie groups should be simple Lie groups such as
SL(n, R) and SU (p, q), and one-dimensional Abelian Lie groups such as R and S 1 .

BRANCHING PROBLEMS OF UNITARY REPRESENTATIONS

3

Simple Lie groups (or slightly more generally, reductive Lie groups) are the groups
that we shall deal with throughout this article. Simple Lie groups were infinitesimally classified by É. Cartan (1894 for complex Lie groups, 1914 for real Lie groups)
after a pioneering work of Killing from 1888 to 1890. We recall that semisimple Liegroups are locally isomorphic to the direct product of simple Lie groups; reductive
Lie groups are locally isomorphic to the direct product of semisimple Lie groups
and Abelian Lie groups.
Next, let us consider Problems 1 and 2 for “representations”. Then, the smallest objects should be irreducible representations (we need to use an appropriate
category of representations because there are subtle topological problems in dealing with infinite dimensional representations). Problem 1 asks the classification of
irreducible (unitary) representations, which contains the following subproblems:· Construction of irreducible representations.
 · Finding a complete set of invariants of representations, so that they can

separate different irreducible representations from one another.

· Understanding these invariants.

A classical example of invariants of representations π is the character Trace(π).
If π is infinite dimensional, then the character Trace(π) is no more a continuousfunction on a group G in general. Harish-Chandra justified it as a distribution for asuitable class of representations π (e.g. π is an irreducible unitary representation ofa reductive Lie group G). The asymptotic K-support ASK (π) and the associatedvariety Vg (π) are also useful invariants of representations of a reductive Lie group G,
which we shall explain and apply in branching problems in §1 and §2, respectively.
The classification of irreducible unitary representations of simple Lie groups has
been a long-standing problem. More than half a century has passed since thepioneering work of Bargmann and the Gel’fand school in the 1940s, and there has
been a large development by Vogan and some others, particularly in the 1980s(we refer to the textbook [26] by Knapp and Vogan for a guide to some recent
literatures). The unitary dual has been classified for some groups such as GL(n, F)(F = R, C, H), but it has not been classified for some other groups such as O(p, q)
and Sp(n, R) (p, q, n ≥ 3).
Second, let us consider Problem 2. We begin with some examples of the decomposition of representations. They are closely related to classical mathematicalproblems such as:1) Spectral theory of unitary operators. This is equivalent to the irreducible decomposition of a unitary representation of Z on a Hilbert space.
2) The theory of reduction of matrices to Jordan normal forms of matrices. This
corresponds to the decomposition of finite dimensional representations of Z onCn .
3) The Fourier transform. We may regard this as the irreducible decompositionof the regular representation of the Abelian Lie group R on L2 (R).
4) The Fourier series expansion. We may regard this as the irreducible decomposition of the regular representation of the torus group S 1 on L2 (S 1 ).
All of the above examples correspond to the decompositions of representationsof Abelian groups, Z, R and S 1 . How about non-Abelian groups, such as SL(n, R)?
We consider two important settings where questions of decomposing representations arise naturally: Let G be a group, and G0 its subgroup.

4

TOSHIYUKI KOBAYASHI

Problem 2-A (Decomposition of induction) Given an irreducible representation τ of a subgroup G0 , decompose the induced representation IndG
G0 τ into irreducibles of G.
Problem 2-B (Decomposition of restriction) Given an irreducible representation π of G, decompose the restriction π|G0 into irreducibles of a subgroup G0 .
For a compact G, these two problems are related to each other by the Frobenius
reciprocity. For a non-compact G, we do not know a strong analog of the Frobenius
reciprocity; however, the comparison of these problems may help us to get a feelingof the current status on them. (So, we shall compare these problems occasionallyin this article.)
Problem 2-A corresponds to the Plancherel type theorem for the homogeneous space G/G0 if τ = 1 (the one-dimensional trivial representation), namely,
to find the formula of the irreducible decomposition of the regular representation
L2 (G/G0 ). For general τ , Problem 2-A deals with L2 -harmonic analysis on a Gequivariant vector bundle over the homogeneous space G/G0 .
The formula of the irreducible decomposition in Problem 2-B is called a branching law. The decomposition of the tensor product of two representations is an
example of branching laws. Branching laws for certain groups arise in quantum
mechanics as a description of breaking symmetries.
In this paper, we shall focus on Problem 2-B, namely, on branching problems.
We are interested in the branching problem of the restriction π|G0 in a general
setting where both G and G0 are reductive Lie groups and π is an irreducible
unitary representation. This setting contains many important cases indeed, but isperhaps too general to expect strong results (at least, now). Our initial project is
to single out a nice category of branching problems, in which we could study deeply
and explicitly the restriction of unitary representations. For this purpose, we shallobserve some of major difficulties present in branching problems in a general setting(see §1.A).
In order to clarify our viewpoint of this article, we begin with an elementary
example of branching laws of finite dimensional representations: Let C[x, y, z] be
the polynomial ring of three variables. We write Pk for its subspace consisting of
homogeneous polynomials of degree k. For instance, dimC P3 = 10. We arrange
elements of P3 in the descending order of z j as follows:(0.1)
P3 = C{z 3 } + C{xz 2 , yz 2 } + C{x2 z, xyz, y 2 z} + C{x3 , x2 y, xy 2 , y 3 }' C{z 3 } + C{x, y} ⊗ C{z 2 } + C{x2 , xy, y 2 } ⊗ C{z} + C{x3 , x2 y, xy 2 , y 3 }.
Then, G := GL(3, C) acts on the left side of (0.1) naturally and irreducibly, and sodoes G0 := GL(2, C) × GL(1, C) on each summand of the right side. Thus, we canregard (0.1) as the branching law of the restriction of an irreducible representationof G with respect to the subgroup G0 .
Let us pin down the dimensions of (0.1), by forgetting explicit representation
spaces:10 = 1 + 2 + 3 + 4= (1 × 1) + (1 × 2) + (1 × 3) + (1 × 4).

BRANCHING PROBLEMS OF UNITARY REPRESENTATIONS

5

The second equality indicates that each irreducible representation of G0 occurs
multiplicity free. We shall compare this formula with branching laws of infinite
dimensional representations (see (0.4), (0.5) and (0.6) below).
Next, consider the direct sum decomposition of the Hilbert space L2 (S 1 ) by the
Fourier series expansion:
L2 (S 1 ) '

(0.2)

X⊕

√

Ce −1nx ,f (x) 7→ {fˆ(n)}n∈Z .

n∈Z

Let G = SL(2, R). Then one can define an action of G on the left side of (0.2)
as an irreducible unitary representation (a principal series representation) by
π(g) : L2 (R) → L2 (R),where g −1 =f (x) 7→ |cx + d|−1 f (ax + b)cx + da b, via the identificationc d∼

L2 (S 1 ) → L2 (R),θθF 7→ | cos |−1 F (tan ).
22Then we may interpret (0.2) as a branching law when restricted from G = SL(2, R)
to G0 = SO(2) (see [50], for detailed formulas).
Similarly, let us consider the direct integral decomposition of the Hilbert space2
L (R) by the Fourier transform:
Z
√
2
(0.3)
L (R) '
Ce −1ξx dξ f (x) 7→ f˜(ξ).
RThen, we may interpret this formula as the branching law when restricted fromSL(2, R) to a unipotent subgroup G0 (consisting of strictly upper triangular matrices) which is isomorphic to R. Namely, G acts on the left side L2 (R) as anirreducible unitary representation,√and the subgroup G0 acts irreducibly on each
one-dimensional representation Ce −1ξx on the right side.
As a “coarse” information on (0.2) and (0.3), we pin down the multiplicities andthe dimensions of irreducible components, respectively as follows:
Dimensions
(0.4)
(0.5)

∞ = · · · + (1 × 1) + (1 × 1) + (1 × 1) + · · ·Z∞ = (1 × 1)dξSpectrum(purely discrete),(purely continuous).

R

Since our concern in this article is with non-Abelian and non-compact groups,
we need to deal mostly with infinite dimensional representations. Then, the featureof the dimension formula (0.4) will be loosely stated in the following generalization:(0.6)∞ = · · · + (finite × ∞) + (finite × ∞) + (finite × ∞) + · · · .

Here, the left side of (0.6) is controlled by G, and the right side indicates thateach irreducible (infinite dimensional) representation of G0 occurs discretely with

6

TOSHIYUKI KOBAYASHI

finite multiplicity. A branching formula with this feature will be called a G0 admissible restriction (see Definition 1.1).
The main theme of this paper is admissible restrictions of unitary representations, namely, branching laws without continuous spectrum and with discrete
spectrum of finite multiplicity. We shall ask:
When does the restriction π|G0 become G0 -admissible?
This is the case if G0 is a maximal compact subgroup (a fundamental theoremof Harish-Chandra; see Example 1.2). This is also often the case if π is a unitary
highest weight representation (see Definition 3.5). Here, among irreducible
unitary representations of a reductive group, unitary highest weight representations are rather special and have been studied extensively and understood best.
Typical examples are holomorphic discrete series representations (see §3.B). The
Segal-Shale-Weil representation splits into two irreducible representations of themetaplectic group M p(n, R) (the double covering group of the symplectic group
Sp(n, R)), and each of them is also a unitary highest weight representation. These
representations are infinite dimensional, but it turns out that they are relatively“small” compared to non-highest weight representations. They have a nature of“one-sided infinity” something like the half line [0, ∞) which has the “bottom” 0.
This is in contrast to a “both-sided infinity” (−∞, ∞). By this one-sided property, unitary highest weight modules π tend to be discretely decomposable whenrestricted to a subgroup (see Example 1.3, Theorem 3.6). We note that the minimum element (such as 0 in [0, ∞)) corresponds to a highest weight vector of π,which may be interpreted as a vacuum vector in quantum mechanics.
On the other hand, most of the irreducible unitary representations are not “onesided”, namely, there are no highest weight vectors. In other words, unitary highestb What shall weweight representations are rather rare among the unitary dual G.
expect for the spectrum in the branching laws of “general” infinite dimensional
representations ? Does it happen that the restriction π|G0 is G0 -admissible?
As we have seen for the irreducible decomposition of the regular representationof R on L2 (R) in (0.3), branching laws usually contain continuous spectrum, whenrestricted to non-compact subgroups. It is no wonder that most people did not
pay attention on the possibility of the non-existence of continuous spectrum in the
branching law of the restriction π|G0 in a general case where π is a non-highest
weight representation and G0 is non-compact.
In 1988, inspired by the theory of discontinuous groups for pseudo-Riemannian
homogeneous spaces (see [41], [84] for an exposition), I found explicit branchinglaws of some (non-holomorphic) discrete series representations with respect to noncompact subgroups. The branching laws are not very complicated and still discretely decomposable, and I was curious about a mysterious phenomenon of discrete
decomposability even in such a general setting ([28]). My proof of the branchinglaws was based on the theory of harmonic analysis on semisimple symmetric spaces([14], [68]) and vector bundles on them ([31]), and on the algebraic theory of Zuckerman and Vogan’s derived functor modules ([74], [75], [76]), both of which developed
largely in the 1980s (see also the references in [26], [36]).
These new branching laws became the first breakthrough in our study of discretely decomposable restrictions ([31], [35], [36], [37], [44], [49], [50]). We shall
explain its flavor in §4.D.5.

BRANCHING PROBLEMS OF UNITARY REPRESENTATIONS

7

Different from our original methods in [28], we shall adopt in this article theapproach of [37] and [46], where global analysis on homogeneous spaces has arelatively small role. That is, we shall study discretely decomposable restrictions
as a problem inside representation theory (see §§1 – 3), and then apply the theoryof restrictions as a method to study global analysis on homogeneous spaces (see§4.C).
This exposition is organized as follows. First, we formulate and give basic results
on discrete decomposable branching laws from an analytic aspect (§1) and from analgebraic aspect (§2). Next, we explain some more results on discretely decomposable restrictions, possible directions for further developments and new perspectivesof unitary representation theory relevant to branching problems in §3. In the latter
half of this paper, we give an outline of some applications of discretely decomposable branching laws to other areas of mathematics. Most results here have beendeveloped in the last five years. The applications explained in §4 range from number theory to discontinuous groups and to global analysis on homogeneous spaces.
Each application in §4 can be read independently.
§1. Analytic theory of admissible restrictions
Throughout this paper, we shall assume that a reductive Lie group G is a linear
group or its finite covering. Without loss of generality, we shall assume that a real
reductive linear group G is realized as a closed subgroup of GL(N, R) satisfying thefollowing two conditions:(i) The number of connected components of G is at most finite.
(ii) G is stable under the transpose operation of matrices (namely, tG = G).
Here are classical examples of linear reductive Lie groups G:
G =GL(n, R), SL(n, R), O(p, q), U (p, q), Sp(p, q), Sp(n, R),
SU ∗ (2n), SO∗ (2n), GL(n, C), SL(n, C), SO(n, C), Sp(n, C).
Here, we note that GL(n, C) can be realized in GL(2n, R) such that it is stable
under the transpose operation of 2n × 2n matrices.
Suppose G is a linear reductive Lie group satisfying the above conditions (i) and(ii). We put
K := G ∩ O(N ).
Then, K is a maximal compact subgroup of G. We write g, k for the Lie algebrasof G, K. A reductive Lie group G is a semisimple Lie group if the center of g is{0}.
b be the set of the (unitary) equivalence classes of irreducible unitary repLet G
b is called the unitary dual of G.
resentations of G. Then G

1.A. Branching laws of unitary representations.
b and a subgroup G0 of G, we consider the problem of finding theGiven π ∈ Gbranching law of the restriction π|G0 , namely, the decomposition formula of π into
irreducible representations of G0 .
For compact G, any irreducible unitary representation π is finite dimensional.
Then, the branching problem is theoretically solvable in the sense that any particular case can be done, because there exists an algorithm to obtain branching laws,

8

TOSHIYUKI KOBAYASHI

based on Weyl’s character formula. Of course, such an algorithm often involves
complicated combinatorial problems.
On the other hand, for the branching law π|G0 of an infinite dimensional unitary
representation π, no general algorithm is known if G ⊃ G0 are (non-compact)
reductive Lie groups.
As a matter of fact, branching laws of unitary representations of semisimple
Lie groups have not been studied systematically except for some special cases (although some of the special cases are already rich and very interesting). Here is anobservation about difficulties to find branching laws of infinite dimensional unitary
representations.
b is constructed as a usual induced representation (e.g. a principal series
1) If π ∈ G
representation), then by using the classical theory of Mackey ([58]) the branchinglaw is reduced to another (usually difficult) problem of harmonic analysis, thatis, to find the Plancherel-type theorem for a homogeneous space (see Problem
2-A). Only recently, the latter problem has been solved under the assumptionthat the homogeneous space is a semisimple symmetric space1 (see [12], [68]).
But, the homogeneous spaces arising from the branching problems are usuallymuch more general than semisimple symmetric spaces and the Plancherel-type
theorem for such spaces is far from being understood. (Even a subproblem suchas Problem 4.C.1 is very difficult.)
2) Some of irreducible unitary representations of semisimple Lie groups cannot berealized as usual induced representations. Discrete series representations are an
example. In this case, there is no known general method to find branching lawsof the restriction π|G0 for non-compact G0 . (We note that the Mackey theory
does not work in this case.)
3) Branching laws when restricted to non-compact subgroups often contain both
discrete and continuous spectrum. Usually, purely algebraic methods do notwork if continuous spectrum occurs.
4) Even worse, multiplicities of irreducible unitary representations of G0 occurringin branching laws can be infinite even if G0 is a maximal reductive subgroup of
G (e.g. (G, G0 ) is a semisimple symmetric pair), a problem to which we paid
much attention in [28]. Infinite multiplicities in branching laws can happen evenin the decomposition of tensor product representations, that is, the restriction
with respect to a diagonally embedded group G1 in the direct product group
G1 × G1 .
5) Fix a group G and its subgroup G0 . By the Frobenius reciprocity, the following
two problems are equivalent if G is compact:bi) To find branching laws of the restriction π|G0 for all π ∈ G.
ii) To find irreducible decompositions of the induced representation IndG
G0 (τ )
0c
for all τ ∈ G .
Loosely, we might suppose that the above two problems (i) and (ii) have comparable difficulties also for non-compact G0 . If so, let us compare our current
knowledge on the problems (i) and (ii). As for (ii), very little has been studied
in the case dim τ = ∞ (see [28], §1.3), although there are some successful cases,
namely, where dim τ = 1 and (G, G0 ) is a symmetric pair, as we have already mentioned. In summary, (ii) is still far from being solved because we need to deal with
1 Irreducible semisimple symmetric spaces were classified by M. Berger on the level of Liealgebras [6].

BRANCHING PROBLEMS OF UNITARY REPRESENTATIONS

9c0 which are mostly infinite dimensional representations. Likewise, (i) is far
all τ ∈ G
from being solved in general. Thus, we believe it is reasonable to attack branchingproblems (i) by limiting ourselves to more special and nicer cases.
1.B. Discretely decomposable restrictions as a “nice framework”.
As we have discussed so far, general branching problems involve too many, andtoo much different a type of difficulties when dealing with infinite dimensional
representations of reductive Lie groups.
So, our strategy is first to find a good framework among general branching problems, and then to initiate a deeper and detailed study in this framework. Such aframework should cover at least some important cases of branching problems where
Mackey’s classical theory does not apply (e.g., π is a discrete series representationof G). Furthermore, the following nature is desirable:a) The framework is rich in new interesting examples, which are also useful in some
applications.
b) In such a framework, we could find branching laws explicitly, or at least there
exists an algebraic algorithm to find branching laws.
From this viewpoint, the author proposed the following Definition 1.1 in [35] and[37] with emphasis on the case of non-compact subgroups: Let G ⊃ G0 be reductiveb We define the multiplicity of τ ∈ G
c0 in the discrete spectrum
Lie groups, and π ∈ G.
of the restriction π|G0 by the dimension of continuous G0 -intertwining operators:mπ (τ ) := dim HomG0 (τ, π|G0 ).
Definition 1.1 (analytic definition of discretely decomposable restriction). We
say that the restriction π|G0 is G0 -admissible if the restriction π|G0 splits into a
discrete direct sum of irreducible unitary representations of G0 and if mπ (τ ) < ∞c0 .
for any τ ∈ G
If π|G0 is G0 -admissible, then we have a unitary equivalence of G0 -modules:(1.1.1)π|G0 'X⊕mπ (τ )τ(a discrete direct sum of Hilbert spaces).

c0
τ ∈G

Here,

P⊕

denotes the Hilbert completion of an algebraic direct sum. In particular,

the formula (1.1.1) means that there is no continuous spectrum in the branching
law of π|G0 .
The significance of Definition 1.1 is that there are “new” examples. However, weshall start with “old” examples of admissible restrictions.
The following theorem of Harish-Chandra is fundamental in representation theory of reductive Lie groups, and has enabled us to study unitary representations
by purely algebraic methods (the so-called theory of (gC , K)-modules or HarishChandra modules). This theorem may be regarded as a special example of Definition 1.1 (the case G0 = K):10

TOSHIYUKI KOBAYASHIb
Example 1.2 (Harish-Chandra [17], see also [78], Theorem 3.4.1). For any π ∈ G,2
the restriction π|K is K-admissible .
In the Introduction, we have seen an example of branching laws which is givenby the Fourier series expansion (see (0.2)). This is a special case of Example 1.2
applied to (G, K) = (SL(2, R), SO(2)) (multiplicity is free in this case).
The theta-correspondence plays an important role in number theory of automorphic forms. The following result of Howe [20] presents another example ofG0 -admissible restrictions, where G0 is non-compact:
Example 1.3 (discrete decomposability in the theta-correspondence). Let G be
the metaplectic group M p(n, R). Suppose that (G, G0 ) is a reductive dual pair,namely, G0 = G01 G02 is a reductive subgroup in G such that G01 and G02 are eachother’s centralizers in G. If G01 or G02 is compact and if π is the Segal-Shale-Weilrepresentation of G, then the restriction π|G0 decomposes discretely with multiplicity free, in particular, it is G0 -admissible. The branching laws produce a lot of
irreducible unitary highest weight representations (see Kashiwara and Vergne [24],for example, for some explicit branching laws).
In Definition 1.1, we have formulated analytically the condition that there
is no continuous spectrum in the branching law. We shall also formulate thenotion of “discrete decomposability” algebraically in terms of (gC , K)-modules(Definition 2.3). There is a slight difference between these two definitions of discrete
decomposable restrictions, especially, we allow the multiplicity to be infinite in our
definition of algebraic discrete decomposability in §2. Then, the study of thisdifference gives rise to a finite multiplicity theorem ([49], see Theorem 3.2 in §3.A):“discreteness (in the spectrum) ⇒ finiteness (of multiplicity)”
for the restriction of discrete series representations with respect to semisimple symmetric pairs (see also Conjecture 3.4 in §3.A in a more general setting).
So far, we have given two examples of G0 -admissible restrictions, namely, Examples 1.2 and 1.3. In both cases, we made strong assumptions: in Example 1.2 G0 is
compact, while in Example 1.3 π has a non-zero highest weight vector (we called
this property “one-sided infinity” in the Introduction; such representations are veryb
special among G).
Our formulation (Definition 1.1) was intended to seek for new settings where thebranching law π|G0 is G0 -admissible, beyond Examples 1.2 and 1.3. The criterion
below (see Theorem 1.5) assures that there are quite rich examples of G0 -admissible
restrictions π|G0 even though G0 is non-compact and π is not a highest weight representation. Thus, a number of branching problems in this framework are newly
obtained, which should to be accessible by purely algebraic methods. Examples
of explicit (discretely decomposable) branching laws of discrete series representations (or more generally, Zuckerman-Vogan derived functor modules Aq (λ)) havebeen found with respect to symmetric pairs such as (O(p, q), O(p − r, q) × O(r)),(O(2p, 2q), U (p, q)) and so on, in this new “nice” framework (see [35] Part I, [37]).
2 This property is usually called “admissible”. The terminology in Definition 1.1 is named afterit.

BRANCHING PROBLEMS OF UNITARY REPRESENTATIONS

11

1.C. A sufficient condition for discretely decomposable restrictions.
Let G be a real reductive linear group. For simplicity, we assume that G isconnected. Then, a maximal compact subgroup K of G is also connected. Since K iscompact, any irreducible unitary representation of K is finite dimensional. As usual,b for the set of equivalence classes of irreducible unitary representations
we write Kof K. Let k be the Lie algebra of K, and take a maximal Abelian subspace t of k.
b is identified with the set of dominant
We fix a positive system ∆+ (k, t). Then Kb will be
integral weights by the √Cartan-Weyl highest weight theory. Hereafter, K∗
regarded as a subset of −1t .
b we define two subsets ofDefinition
1.4 (asymptotic K-support [25]). For π ∈ G,√∗
−1t as follows:b : HomK (τ, π|K ) 6= {0}},SuppK (π) := {τ ∈ K
ASK (π) := SuppK (π)∞.

Here, for a subset S of the Euclidean space, the limit cone S∞ (see [23]) is a
closed cone defined by:
S∞ := { lim n yn : yn ∈ S, n → 0}.
n→∞

The closed cone ASK (π) is called the asymptotic K-support of the representationπ.
Let G ⊃ G0 be a pair of reductive Lie groups in the sense that both G and G0
satisfy the conditions (i) and (ii) stated at the beginning of §1. Then, the adjoint
representation Ad : G → GL(g) is completely reducible when restricted to G0 . If
we set K 0 := K ∩ G0 , then K 0 is a maximal compact subgroup of G0 .
Example.
1) For (G, G0 ) = (GL(n, C), GL(n, R)), we have (K, K 0 ) = (U (n), O(n)).
2) For (G, G0 ) = (GL(n, C), U (p, q)), we have (K, K 0 ) = (U (n), U (p) × U (q)).
Here, p + q = n.
We write g0 , k0 for the Lie algebras of G0 , K 0 . We fix a K-invariant innerproduct on k. With respect to this inner product, we define (k0 )⊥ as the orthogonal∗
complement of k0 in k. We also identify t∗ with t, and then√ regard t as a subspaceof k. In particular, we can regard ASK (π) as a subset of −1k.
Here is a sufficient condition for the restriction π|G0 to be G0 -admissible:
Theorem 1.5 (a sufficient condition for G0 -admissible restriction; [44]). Let G ⊃b If
G0 be a pair of reductive Lie groups, and π ∈ G.
(1.5.1)

ASK (π) ∩√−1 Ad(K)(k0 )⊥ = {0},then the restriction π|K 0 is K 0 -admissible. In particular, the restriction π|G0 is G0 admissible, namely, π|G0 decomposes discretely with finite multiplicities (Definition
1.1).
Assumption (1.5.1) is obviously satisfied if ASK (π) = {0} or if Ad(K)(k0 )⊥ ={0}. First of all, let us explain these two special cases in Examples 1.6 and 1.7.

12

TOSHIYUKI KOBAYASHI

Example 1.6. If G0 = K, then k0 = k and therefore (k0 )⊥ = {0}. Hence, we have
Ad(K)(k0 )⊥ = {0}.
(In fact, it is easy to see that Ad(K)(k0 )⊥ = {0} if and only if G0 ⊃ K.) Then,b The conclusion ofassumption (1.5.1) is automatically fulfilled for any π ∈ G.
Theorem 1.5 in this special case is the admissibility theorem due to Harish-Chandra
as stated in Example 1.2.
b we have
Example 1.7. For π ∈ G,
ASK (π) = {0} if and only if dim π < ∞.
Thus, if dim π < ∞ then assumption (1.5.1) is obviously satisfied for any subgroup
G0 . The conclusion of Theorem 1.5 in this special case follows also from an easycomplete reducibility result of finite dimensional unitary representations.
Here is a non-trivial example of Theorem 1.5:
Example 1.8. Let (G, G0 ) = (U (2, 2), Sp(1, 1)). We note that the pair (G, G0 ) is
locally isomorphic to (S 1 × SO(4, 2), SO(4, 1)). There are 18 series of irreducible
unitary representations of G with regular and integral infinitesimal character by
a result of Salamanca Riba. Among those 18 series, 6 can be realized in closed
subspaces of L2 (G), namely, they are Harish-Chandra’s discrete series representations. Among the 18 series, there are 12 series (2 of them being discrete series) ofirreducible unitary representations of G that satisfy the condition (1.5.1) (see [37]for details). In particular, there is no continuous spectrum in the branching lawsof the restriction π|G0 if π belongs to these 12 series. Conversely, the remaining
18 − 12 = 6 series of irreducible unitary representations of G are not algebraicallydiscretely decomposable (Definition 2.3) when restricted to G0 (see [49]).
For some important representations π such as discrete series representations, we
can compute ASK (π) in terms of the root data [44]. See [42] for actual computations
to apply Theorem 1.5.
The converse direction of Theorem 1.5 will be discussed in §2.B.
The idea of the proof of Theorem 1.5 is to capture the existence of continuous
spectrum as a “size3 of infinite dimensional representations”, by looking at theasymptotic behavior of K-types. In [44], this was carried out by extending the workof Kashiwara-Vergne and Howe in the 70s on the microlocal study of characters ofrepresentations. Here, the asymptotic K-support ASK (π) plays a role of “size” of π
by means of the wave front set of the character Trace(π). Before finding a general
method in [44], we took a different and more algebraic approach in proving theG0 -admissibility of the restriction π|G0 in a special case where π is a ZuckermanVogan derived functor module. See [31], Proposition 4.1.3 in the case where K 0 isof the form of the direct product K10 × K20 ; see also [37], Theorem 1.5 in the casewhere (G, G0 ) is a semisimple symmetric pair. Gross and Wallach [16] also studied
3 The representation space is an infinite dimensional, separable Hilbert space, which is unique

as a topological vector space. However, the size of representations may be “different” if we takegroup actions into account. Associated varieties (in §2) and their dimensions (Gel’fand-Kirillov
dimensions) are also useful to measure ‘size’ of representations.

BRANCHING PROBLEMS OF UNITARY REPRESENTATIONS

13

G0 -admissible restrictions in the case where K 0 is of the form K10 × K20 , especiallywhen K10 ' SU (2).
In §2 below, we shall reformulate Definition 1.1 in terms of (gC , K)-modules, andobtain a necessary condition for the discrete decomposability. Namely, we introduce
the notion of infinitesimally discrete decomposability of the restriction (seeDefinition 2.3), and estimate the size of infinite dimensional representations by
their associate varieties [77], which originally arose from the D-module theory. If
the restriction π|G0 is discretely decomposable, then we may consider analogous
notion of (gC , K)-modules à la Harish-Chandra and Lepowski in a more general
setting, namely, an algebraic direct sum of infinite dimensional representations(symbolically, a notion of (gC , G0 )-modules).
We note that Theorem 1.5 gives a sufficient condition not only for the G0 admissibility of the restriction π|G0 but also for the infinitesimally discrete decomposability of the restriction π|G0 , which is the object of §2.
§2. Algebraic theory of discretely decomposable restrictions
2.A. Algebraic reformulation of discrete decomposability.
Let us start with an example where there is only continuous spectrum in theirreducible decomposition. As an opposite extremal case, this example (Example
2.1) serves us as a hint to find an algebraic definition of discretely decomposable
restrictions.
Example 2.1 (Wiener subspace). Let V be a subspace of L2 (R). We say V is
R-invariant if V satisfies:
f (x − a) ∈ V for any f (x) ∈ V and any a ∈ R.
For a measurable set E of R, we write L2 (E) for the closed subspace of L2 (R), consisting of all L2 -functions supported on E. Then, the image of the Fourier transformof L2 (E), denoted by F(L2 (E)), is a closed R-invariant subspace. Conversely, it
is known that any closed R-invariant subspace is of the form F(L2 (E)) for somemeasurable set E. Then, given any non-zero closed R-invariant subspace V , there
exists an infinite decreasing sequence of closed R-invariant subspaces {Vj }:V % V 1 % V2 % · · ·(to see this, it is enough to take a sequence of measurable sets Ej of E such that E %E1 % E2 % · · · , and then to define Vj := F(L2 (Ej ))). This property is equivalent
to the fact that there is no discrete spectrum in the irreducible decomposition of
the regular representation L2 (R) of R. (Of course, the latter property follows also
from the irreducible decomposition of L2 (R) by means of the Fourier transform (see(0.3)).)
In summary, Example 2.1 relates the non-existence of discrete spectrum withthe existence of an infinite decreasing sequence of invariant subspaces. Next, weshall relate the non-existence of continuous spectrum with the existence of an infinite increasing sequence of invariant subspaces. Here is an algebraic formulation:
Definition 2.2 ([49], Definition 1.1). Let g be a Lie algebra, and X a g-module.
We say the g-module X is discretely decomposable if there is an increasing sequenceof g-submodules:
X0 ⊂ X1 ⊂ X2 ⊂ · · ·14

TOSHIYUKI KOBAYASHI

such that the S
following two properties are satisfied:∞(2.2.1) X = m=0 Xm .
(2.2.2) Each Xm is of finite length as a g-module.
We note that irreducible representations of a finite dimensional Lie algebra areusually infinite dimensional. In the above definition, we have infinite dimensional
modules Xm in mind.
Let (π, H) be an irreducible unitary representation of G, and X a subspaceof the Hilbert space H consisting of K-finite vectors. Then, X is dense in H,
and has the g-module structure (the differential representation), in addition to the
K-module structure. The above gC ∪ K-module (πK , X) is called the underlying(gC , K)-module of π.
Next, let G ⊃ G0 be a pair of reductive Lie groups so that K 0 := K ∩ G0 is a
maximal compact subgroup of G0 . We apply Definition 2.2 to the restriction to G0 .
Definition 2.3 (algebraic definition of discretely decomposable restriction). Letb We say that the restriction π|G0 is g0 -discretely decomposable or infinitesπ ∈ G.
imally discretely decomposable if the underlying (gC , K)-module πK is discretely
decomposable as a g0 -module in the sense of Definition 2.2.
It might look strange at a first glance that Definition 2.2 gives the notion of“discrete decomposition”. In fact, the terminology is named after the following:
Theorem 2.4 (characterization of infinitesimally discretely decomposable restriction; [49]). Let (G, G0 ) be a pair of reductive Lie groups, π an irreducible unitary
representation of G, and (πK , X) its underlying (gC , K)-module. Then the following three conditions on the triple (G, G0 , π) are equivalent:(i) The restriction π|G0 is infinitesimally discretely decomposable (Definition 2.3).
(ii) The (gC , K)-module (πK , X) is isomorphic to an algebraic direct sum of irreducible (g0C , K 0 )-modules (discrete branching law):M(2.4.1)X'
nπ (Y ) Y (an algebraic direct sum).
Y

Here, the sum is taken over all irreducible (g0C , K 0 )-modules Y , andnπ (Y ) := dim Hom(g0C ,K 0 ) (Y, X)
is the multiplicity of Y occurring in X.
(iii) There exists an irreducible (g0C , K 0 )-module Y such that
Hom(g0C ,K 0 ) (Y, X) 6= {0}.
We note that the multiplicity nπ (Y ) may or may not be infinite in Theorem 2.4.
The point of the condition (iii) is that only a single representation Y is used (forthis, we have assumed that π is irreducible).
Moreover, the following theorem holds:
Theorem 2.5 (infinitesimal ⇒ Hilbert discrete decomposition). If the restrictionb is infinitesimally discretely decomposable, then we have the following
π|G0 of π ∈ G
c0 :equality for any τ ∈ G(2.5.1)

dim Hom(g0C ,K 0 ) (τK 0 , πK ) = dim HomG0 (τ, π|G0 ).

BRANCHING PROBLEMS OF UNITARY REPRESENTATIONS

15

We fix mπ (τ ) by (2.5.1). Then, the restriction π|G0 of the unitary representation
π is decomposed into irreducibles of G0 without continuous spectrum:π|G0 'X⊕mπ (τ )τ(a discrete direct sum of Hilbert spaces).

c0
τ ∈G

We note that the right-hand side of (2.5.1) is the dimension of continuous G0 intertwining operators, while no topology is specified in the left-hand side of (2.5.1).
In general, we havethe left-hand side of (2.5.1) ≤ the right-hand side of (2.5.1)
without the assumption of infinitesimally discrete decomposability.
We should keep in mind that we have not imposed the condition nπ (Y ) < ∞in the definition of infinitesimally discrete decomposability (Definition 2.3), while
we imposed the finiteness of the multiplicities in the definition of G0 -admissible
restriction (Definition 1.1). As we have pointed out in [49], [50] (see also §1.B), themultiplicities tend to be finite if the restriction has no continuous spectrum (seeTheorem 3.2 and Conjecture 3.4 in §3.A for a precise formulation).
The next (easy) example follows immediately from the equivalent definitions of
infinitesimally discrete decomposability (Theorem 2.4). We note that G0 = {e} isallowed in the example below.
Example 2.6. If G0 is a compact subgroup, then the restriction π|G0 is infinitesibmally discretely decomposable for any π ∈ G.
2.B. A necessary condition for discretely decomposable restrictions— an approach by associated varieties.
Loosely, the associated variety of a g-module π is an algebraic variety thatapproximates the representation π by means of the graded ring gr U (gC ). Since
gr U (gC ) is isomorphic to a polynomial ring, one can use a standard technique ofalgebraic geometry. It turns out that associated varieties are useful for the studyof infinitesimally discretely decomposable restrictions ([35] Part II, [49]).
Let us recall briefly the definition of associated varieties of g-modules (see Vogan[77] for more details). Let X be a finitely generated module (over C) of a Lie
algebra g. The associated variety of X, denoted by Vg (X), is defined similarly
to the characteristic varieties of D-modules as follows (see [9]). Let gC be thecomplexification of g. Then, X becomes a U (gC )-module by the universality of theenveloping algebra U (gC ). Take a natural filtration
C = U0 (gC ) ⊂ U1 (gC ) ⊂ U2 (gC ) ⊂ · · ·of U (gC ) (corresponding to degrees of partial differential operators). Let X0 be
a finite dimensional subspace of X that generates X as a U (gC )-module and put
Xn := Un (gC ) · X0 . Then X0 ⊂ X1 ⊂SX2 ⊂ · · · gives a filtration of X such thatUi (gC )Xj ⊂ Xi+j . We note that X = n Xn . We put
gr U (gC ) :=MnUn (gC )/Un−1 (gC ),gr X :=Mn

Xn /Xn−1 .

16

TOSHIYUKI KOBAYASHI

Then, the graded module gr X carries naturally a gr U (gC )-module structure. The
gr U (gC )-module, gr X, is regarded as an approximation of the U (gC )-module X.
The enveloping algebra U (gC ) is a non-commutative algebra provided g is noncommutative, while the graded ring gr U (gC ) is isomorphic to the symmetric algebra S(gC ) by the Poincaré-Birkhoff-Witt theorem, and then is isomorphic to thepolynomial algebra over g∗C . In particular, gr U (gC ) is commutative. The characteristic variety of an S(gC )-module gr X is called the associated variety of X, andwill be written as Vg (X). That is,
Vg (X) := {λ ∈ g∗C : f (λ) = 0

for any f ∈ Anngr U (gC ) (gr X)}

where Anngr U (gC ) (gr X) is the annihilator ideal of gr X.
So far, X is just a finitely generated module of the Lie algebra of g. From now, let(πK , X) be the underlying (gC , K)-module of an irreducible unitary representation
π of a reductive Lie group G. Then we have:
Theorem 2.7 (Vogan, [77]). The associated variety Vg (X) is a KC -invariant algebraic variety contained in the nilpotent cone of g∗C . Furthermore, Vg (X) ⊂ (gC /kC )∗Here we recall that the nilpotent cone of g∗C is the algebraic variety that is identified with the closed subset{X ∈ gC : ad X is nilpotent}
under the isomorphism gC ' g∗C . Suppose G0 is a subgroup of G. Let
prg→g0 : g∗C → (g0C )∗be the projection dual to the inclusion of Lie algebras g0C ,→ gC . Here is a lower
estimate of the associated varieties of irreducible summands in the branching law:
Theorem 2.8 (associated varieties of irreducible summands, [49]). Suppose that
there is an irreducible (g0C , K 0 )-module Y such that Hom(g0C ,K 0 ) (Y, X) 6= {0}. Then,(2.8.1)

prg→g0 (Vg (X)) ⊂ Vg0 (Y ).

To see the meaning of Theorem 2.8, let us consider the simplest case, that is,the case where Y is finite dimensional. Then, it follows from the definition of an
associated variety that
Vg0 (Y ) = {0}.
Consequently, (2.8.1) is equivalent toVg (X) ⊂ (gC /g0C )∗ .
Now, we consider two cases in the following example: G0 is compact or noncompact.

BRANCHING PROBLEMS OF UNITARY REPRESENTATIONS

17

Example 2.9. 1) If G0 is compact, then obviously there exists a finite dimensional
irreducible (g0C , K 0 )-module Y satisfying Hom(g0C ,K 0 ) (Y, X) 6= {0}. In particular, if
G0 = K, then we have
Vg (X) ⊂ (gC /kC )∗ ' pC
by Theorem 2.8. This result was proved previously by Vogan (see Theorem 2.7).
2) Suppose G is a simple Lie group. If G0 is non-compact, thenprg→g0 (Ad(KC )v) 6= {0}for any non-zero nilpotent element v ∈ pC . If dim X = ∞, then Vg (X) 6= {0}, andin particular, Vg (X) contains Ad(KC )v for some non-zero nilpotent element v inpC . Hence, prg→g0 (Vg (X)) 6= {0}. Therefore, it follows from Theorem 2.8 that
Hom(g0C ,K 0 ) (Y, X) = {0}for any finite dimensional irreducible (g0C , K 0 )-module Y . See also [49], Corollary 3.9
for a relation to Moore’s ergodicity theorem [62].
The following criterion ([49], Corollary 3.4) is useful and is readily deduced from
Theorem 2.8.
Theorem 2.10 (a necessary condition for discretely decomposable restrictions).
b If the restriction π|G0
Let (πK , X) be the underlying (gC , K)-module of π ∈ G.
is infinitesimally discretely decomposable, then prg→g0 (Vg (X)) is contained in thenilpotent cone of (g0C )∗ .
2.C. Three more theorems on discretely decomposable restrictions.
In this subsection, we state three direct consequences of Theorem 2.10.
We recall an obvious fact that the restriction to a compact subgroup is always
discretely decomposable (Example 2.6). For example:
Example 2.11 (the restriction which is always discretely decomposable). Let(G, G0 ) = (SL(n, C), SU (n)). Then, the restriction π|G0 is infinitesimally discretely
decomposable for any irreducible unitary representation π of G.
Theorem 2.10 leads us to an opposite extremal case:
Theorem 2.12 (the restriction which is never discretely decomposable). Let(G, G0 ) = (SL(n, C), SL(n, R)). Then, the restriction π|G0 is not infinitesimallydiscretely decomposable for any irreducible unitary representation π of G except for
π = 1.
So, with regard to infinitesimally discrete decomposability, two real forms SU (n)and SL(n, R) of SL(n, C) have completely different feature. Other real forms such
as SU (p, n − p) (1 ≤ p ≤ n − 1) are intermediate. A real form G0 of a complex
reductive Lie group G is called normal or split if rank G = R-rank G0 . For example,
SL(n, R) is a normal real form of SL(n, C), while SU (p, n − p) is not a normal real
form except for the case (p, n) = (1, 2).
Combining Theorem 2.10 with the following lemma:
prg→g0 (Ad(KC )v) contains a non-zero semisimple element for
any non-zero nilpotent element v ∈ pC if G0 is a normal real formof a complex reductive Lie group G,
we can generalize Theorem 2.12 as follows (see [49], Theorem 8.1):

18

TOSHIYUKI KOBAYASHI

Theorem 2.13 (the restriction is never discretely decomposable). Let G be a complex reductive Lie group, and G0 its normal real form. Then, the restriction π|G0 is
not infinitesimally discretely decomposable for any infinite dimensional irreducible
unitary representation π of G.
A second application of Theorem 2.10 deals with the relation between the discreteness in the induced representation (discrete series representation) and the discreteness in the restriction (discrete decomposable restriction). It turns out thatthey cannot stand together for symmetric pairs. Here is a statement:
Theorem 2.14 (the exclusive law of discrete spectrum for the restriction andthe induction). Let (G, G0 ) be an irreducible symmetric pair such that G is nonb Then both (i) and (ii) cannot occur simultaneously.
compact. Let π ∈ G.
(i) The restriction π|G0 is infinitesimally discretely decomposable .
(ii) π is a discrete series representation for the homogeneous space G/G0 (i.e.
HomG (π, L2 (G/G0 )) 6= {0}).
We refer to [49] for the proof. At a first glance, this result might look strange, but
it is another thing that one might expect as a Frobenius reciprocity-type theorem
for infinite dimensional representations.
Here is a very special example of Theorem 2.14.
Example 2.15. Let (G, G0 ) be a Riemannian symmetric pair, namely, G0 is a maximal compact subgroup K. Then, we recall the following two well-known results:b1) The restriction π|K is infinitesimally discretely decomposable for any π ∈ G.
2) (Harish-Chandra, Helgason) There is no discrete series representation for the
Riemannian symmetric space G/K (see §4.C for the definition of discrete series
representations for a homogeneous space).
(1) is obvious (see Example 2.6), but (2) is non-trivial. (2) is equivalent to thefact that discrete series representations for the group manifold G do not have (nonzero) K-fixed vectors. The point here is that Theorem 2.14 asserts a non-trivial
implication (1) ⇒ (2).
As a simplest case of Example 2.15, let us consider the setting(G, K) = (SL(2, R), SO(2)).
Then, (1) corresponds to the discreteness of the Fourier series expansion (e.g. (0.2)
in §0), and (2) means the fact that the Laplace-Bertram operator on the Poincaré
upper half plane has no L2 -spectrum. Thus, even for SL(2, R), Theorem 2.14 gives
a new relationship between these two results.
Our proof of Theorem 2.14 uses the estimate of the associated varieties for infinitesimally discretely decomposable restrictions (Theorem 2.8). A more direct
approach to Theorem 2.14 would be preferable, which might give a better understanding of this mysterious relation.
Another aspect of Theorem 2.14 is that it clarifies a representation-theoretic
background for the following antithesis between vanishing and non-vanishing theorems of modular symbols in arithmetic quotients of Riemannian symmetric spaces
induced from the morphism Γ0 \G0 /K 0 → Γ\G/K (see §4.B for details):
1) A non-vanishing theorem [73] due to Tong and Wang (some twisted case)by using a discrete series representation π for a semisimple symmetric space G/G0 .
(Discrete spectrum in the induction.)

BRANCHING PROBLEMS OF UNITARY REPRESENTATIONS

192) A vanishing theorem [52] due to Kobayashi and Oda by using the discrete
decomposability of the restriction π|G0 (cf. Theorem 1.5). (Discrete decomposability of the restriction.)A third application of Theorem 2.10 is about the comparison of irreducible constituents of the restriction π|G0 . As we shall mention in Remark 2.17, an analogous
result fails if there exists continuous spectrum in the branching law.
Theorem 2.16 (irreducible summands have the same associated varieties). Let X
be an irreducible (gC , K)-module. For any irreducible (g0C , K 0 )-modules Y1 and Y2
such that
Hom(g0C ,K 0 ) (Yj , X) 6= {0} (j = 1, 2),
their associated varieties are the same:(2.16.1)

Vg0 (Y1 ) = Vg0 (Y2 ).

Remark 2.17. If the branching law contains continuous spectrum, then representations of different Gel’fand-Kirillov dimensions may occur in the restriction π|G0 as
discrete spectrum. For example, the Plancherel theorem for the semisimple symmetric space Sp(n, R)/GL(n, R) is equivalent to the decomposition of the tensor
product of two degenerate principal series representations of Sp(n, R) with suitableparameters by the Mackey theory (for example, see [36], Proposition 6.1). In particular, this is a special case of branching laws. On the other hand, by using theFlensted-Jensen construction ([14]), one can prove that there exist discrete series
representations for the semisimple symmetric space Sp(n, R)/GL(n, R) with different associated varieties. Hence, (2.16.1) fails. In other words, we have given acounterexample of the following wrong statement:b If τ1 , τ2 ∈ G
c0 satisfy
False “Theorem” 2.160 . Let π ∈ G.
HomG0 (τj , π|G0 ) 6= {0}(j = 1, 2),then
Vg0 (Y1 ) = Vg0 (Y2 ),where Yj (j = 1, 2) are the underlying (g0C , K 0 )-modules of τj .
§3. New aspect of representation theory
related to branching problems
It turns out that there exist fairly rich examples of discretely decomposable
restrictions owing to the criterion in Theorem 1.5. Consequently, many branchingproblems arise, to which not much attention has been paid before, and on which
we can now expect a deeper and explicit study by algebraic methods.
In this section, we shall explain briefly some of recent topics related to discrete
branching laws.
3.A. Finite multiplicity conjecture.

20

TOSHIYUKI KOBAYASHI

Conjecture 3.1 (Wallach, see [79]). Let (G, G0 ) be a semisimple symmetric pair.
If π is a discrete series representation for G, thendim HomG0 (τ, π|G0 ) < ∞(3.1.1)c0 .
for any τ ∈ G

For example, the admissibility theorem of Harish-Chandra (Example 1.2) assertsthat Conjecture 3.1 holds if G0 is compact. On the other hand, if G0 is compact,
then the restriction π|G0 is obviously infinitesimally discretely decomposable. It is
proved in [49] that (3.1.1) still holds by assuming only the condition that π|G0 is
infinitesimally discretely decomposable :Theorem 3.2 (discreteness ⇒ finite multiplicity). Let (G, G0 ) be a semisimple
symmetric pair. For any Zuckerman-Vogan derived functor (gC , K)-module 4 X(more precisely, cohomologically induced from a finite dimensional representationin the good range of parameters), and for any irreducible(g0C , K 0 )-module Y , we have
dim Hom(g0C ,K 0 ) (Y, X) < ∞.
Since the underlying (gC , K)-module of any discrete series representation π isexpressed as a Zuckerman-Vogan derived functor module, we have the following
corollary:
Corollary 3.3 (a proof of Conjecture 3.1 in the discrete decomposable case).
If the restriction π|G0 is infinitesimally discretely decomposable, Conjecture 3.1
is true.
Remark (infinite multiplicity). Even if (G, G0 ) is a semisimple symmetric pair, themultiplicity of discrete spectrum in the restriction π|G0 can be infinite, namely,
dim HomG0 (τ, π|G0 ) = ∞c0 and π ∈ G.
b
for some τ ∈ GThis can happen if the branching law of the restriction π|G0 contains continuous
spectrum. Different from the result due to Corwin-Greenleaf in the case of nilpotent
Lie groups, the situation of semisimple Lie groups is more delicate. For instance,
we proved in [50] that there is an example of (G, G0 , π) such that(c0 ,dim HomG0 (τ1 , π|G0 ) = ∞ for some τ1 ∈ G

c0 ,0 < dim HomG0 (τ2 , π|G0 ) < ∞ for some τ2 ∈ G
where π is an irreducible unitary representation of SO(5, C) and G0 = SO(3, 2).
Building on Theorem 3.2, we proposed the following conjecture:
4 These representations are algebraic analog of a generalized Borel-Weil-Bott theorem on (possi-

bly non-compact) complex homogeneous manifolds (see §3.D). They are often denoted by Aq (λ).
See [26], [75] for an algebraic explanation, and [36] for a survey of geometric approach due to
Schmid and Wong.

BRANCHING PROBLEMS OF UNITARY REPRESENTATIONS

21

Conjecture 3.4 (see [50], Conjecture C). Let (G, G0 ) be a semisimple symmetricb If the restriction π|G0 is infinitesimally discretely decomposable,
pair, and π ∈ G.
then
c0 .
dim HomG0 (τ, π|G0 ) < ∞for any τ ∈ G
As we saw in (2.5.1), this conjecture also implies
dim Hom(g0C ,K 0 ) (Y, πK ) < ∞for any irreducible (g0C , K 0 )-module Y .

To end this subsection, we would like to mention an analytic aspect of Corollary 3.3:
If one realizes the representation π in a geometric way, then Corollary 3.3 may giverise to an example of the following phenomenon: “In a system of non-holonomic
partial differential equations, local solutions are possibly infinite dimensional, but
global solutions are possibly finite dimensional.”5
3.B. A generalization of the Kostant-Schmid formula to semisimple symmetric pairs.
In the framework of discretely decomposable restrictions, an algebraic approach
could work effectively in branching problems. Furthermore, if the multiplicity isfree, then one could expect a simple and detailed study of branching laws. In thissubsection, we shall explain such examples. More precisely, in the setting below(where (G, G0 ) is a semisimple symmetric pair), it turns out that the branchinglaw is discrete by Theorem 1.5, and that the multiplicity of each irreducible representation is free owing to the multiplicity-one theorem in [47]. In particular, therestriction π|G0 is G0 -admissible. Then, we shall give a new explicit branching lawthat generalizes the Kostant-Schmid formula [71] to the setting of non-compact
subgroups. This subsection is taken from [47].
Throughout this subsection, let G be a non-compact simple Lie group of Hermitian type. This means that G is a Lie group locally isomorphic to one ofSU (p, q), SO(n, 2), Sp(n, R), SO∗ (2n), E6(−14) , E7(−25) .
Then, the complexified Lie algebra gC := g ⊗R C is decomposed into irreducible
modules under the adjoint action of K as follows:gC = kC ⊕ p+ ⊕ p− .
Let t be a maximal Abelian subspace of k, and fix a positive system ∆+ (k, t).
b We sayDefinition 3.5 (unitary highest weight representation). Let (π, V ) ∈ G.
+(π, V ) is an irreducible unitary highest weight representation, if V p 6= {0},where we put
V ∞ := the set of smooth vectors of V,+

V p := {v ∈ V ∞ : dπ(X)v = 0 for any X ∈ p+ }.
5 It is remarkable that the dimension of the space of global solutions on a non-compact

manifold becomes finite in this case. The relation between the dimension of global solutions andthe underlying geometry might be interesting to study, as is Atiyah-Singer’s index theorem for
elliptic differential operators on compact manifolds.

22

TOSHIYUKI KOBAYASHI++

Then, K acts on V p because Ad(K) stabilizes p+ . It turns out that V p is irreducible as a K-module. Furthermore, an irreducible highest weight representation+(π, V ) of G is determined uniquely by the K-module structure on V p . We
√ writeG
V (µ) for the irreducible highest weight representation (π, V ) of G, if µ ∈ −1t∗ is+a highest weight of the K-module V p with respect to the positive system ∆+ (k, t).
We use this notation also for other Lie groups G0 of Hermitian type.
+We say that V is of scalar type if dim V p = 1. If the highest weight representation V is realized in a closed subspace of L2 (G), we say V is a holomorphic discrete
series representation. Holomorphic discrete series representations were discoveredin an early stage of unitary representation theory by Harish-Chandra and have been
best-understood among discrete series representations of G.
Suppose an involution τ ∈ Aut(G) stabilizes K and acts holomorphically on the
Hermitian symmetric space G/K. We define a subgroup of G by
Gτ := {g ∈ G : τ g = g}.
Analogous notation V τ will be applied to denote the set of fixed points of τ if τ
acts on a vector space V . We define the subgroupG0 := Gτ0 ,by the connected component of Gτ containing the identity. Then G0 is a reductive
subgroup, and (G, G0 ) forms a semisimple symmetric pair. For example, the pairs(Sp(n, R), U (p, q)) and (Sp(n, R), Sp(p, R) × Sp(q, R)) (p + q = n) are the cases.
Let{ν1 , ν2 , . . . , νk }be a maximal set of strongly orthogonal roots in ∆((p+ )−τ , tτ ). Then one can showk = R- rank G/Gτ ,the real rank of a semisimple symmetric space G/Gτ , or equivalently, of G/G0 . For
example, k = min(p, q) if (G, G0 ) = (Sp(n, R), Sp(p, R) × Sp(q, R)).
Theorem 3.6 (a generalization of the Kostant-Schmid formula; [47]). In the above
b be a holomorphic discrete series representation of scalar type.
setting, let V G (µ) ∈ G
Then we have the following branching law of the restriction V G (µ)|G0 :(3.6.1)

V G (µ)|G0 'X⊕
a1 ≥···≥ak ≥0
aj ∈N

0

V G (µ|tτ −kX

aj νj ),

j=1

where the right side is a discrete direct sum of irreducible unitary representationsof G0 .
0

In the above theorem, if G0 is non-compact, then each V G (∗) is infinite dimensional.
Theorem 3.6 includes the following known results as special cases:(i) The formula due to Hua(classical)-Kostant(unpublished)-Schmid [71] corresponds to the case where G0 is a maximal compact group K. Then V K (∗) isfinite dimensional.
(ii) G0 is non-compact. Some special cases have been known, including the cases
G = SU (2, 2), SU (2, 1). See, for example, Jakobsen and Vergne ([22]) andXie ([79]).

BRANCHING PROBLEMS OF UNITARY REPRESENTATIONS

23

3.C. Unipotent representations and discrete branching laws.
There is another special case where the restriction has been studied extensively,that is, the local theta-correspondence (cf. Example 1.3). Then the restriction π|G0
is concerned with the case where π is the Segal-Shale-Weil representation of themetaplectic group G = M p(n, R) and (G, G0 ) is a reductive dual pair. The Weilrepresentation is an example of the minimal unipotent representation of the split
group G of type C. Branching laws of unipotent representations of other groups
have been studied in the last decade. Some explicit branching laws of unipotent
representations include:•) G is of type D, due to Kobayashi and Ørsted (discretely decomposable branching laws, 1991). See also [53], [88] for a more general case with continuous
spectrum.
•) G is an exceptional group with real rank 4, due to Gross and Wallach (discretely decomposable branching law, [16]),•) G is of type E, due to J-S. Li (discretely decomposable branching law, [57]).
As above, these branching laws have been studied mainly in the case where they
are discretely decomposable. Here are some advantages of discrete decompositions:(3.C.1) From the viewpoint of finding explicit branching laws, branching laws areless difficult to find, if there is no continuous spectrum, because one can use algebraic techniques.
c0 (smaller group), discrete spectrum(3.C.2) From the viewpoint of the study of G
is useful because it gives an explicit construction of irreducible unitary representations of the subgroup G0 .
b (larger group), discrete branching(3.C.3) From the viewpoint of the study of G
laws give a clue to study representations of G in terms of G0 (e.g., a special case
G0 = K gives a theory of (gC , K)-modules).
The study of branching laws of unipotent representations is still in an earlystage, and there seems to be much room for further developments. For example,
the following directions of research may be considered:a) A finer study of unipotent representations by means of branching laws (see(3.C.3)).
b) Construction of singular unitary representations as irreducible summands (see(3.C.2)).
c) Global analysis on manifolds (especially, on homogeneous spaces) arising frombranching laws of unipotent representations (e.g. [88]).
d) Combinatorial problems arising from algebraic study of discretely branchinglaws (cf. (3.C.1)).
3.D. Representations as a quantization of elliptic orbits and discrete
branching laws.
An elliptic orbit is an adjoint orbit of G through an element X such that ad(X)is diagonalizable with purely imaginary eigenvalues. Any elliptic orbit carries a
G-invariant pseudo-Kähler structure, and its “geometric quantization” gives an irreducible unitary representation of G, as was suggested by the Kirillov-Kostantorbit method, and as was proved by Schmid and Wong combined with algebraic
results due to Vogan, Wallach and Zuckerman, under certain regular and integral conditions on X. We note that its Harish-Chandra module is expressed as
Zuckerman-Vogan’s derived functor (gC , K)-modules (sometimes called Aq (λ), see

24

TOSHIYUKI KOBAYASHI[26]).
In the previous exposition [36], we gave a survey on the construction of these representations in details from the viewpoint of geometric quantization and discussed
the discrete decomposability of the restriction to subgroups.
We also wrote in [35], Part I and in [37], some explicit branching laws of (small)
discrete series and some more general representations Aq (λ) for classical symmetricpairs (G, G0 ) such as(G, G0 ) = (SO(p, q), SO(m) × SO(p − m, q)), (O(2p, 2q), U (p, q))in the framework of G0 -admissible restrictions. So we do not repeat ourselves here.
Let us just mention some new progress after [36] was written. By using Theorems
1.5 and 2.10, we now have a necessary and sufficient condition for the restriction π|G0 to be infinitesimally discretely decomposable if π is Zuckerman-Vogan’s
derived functor module. This was proved in [49], which strengthens the result in
the previous exposition ([36], Theorem 6.5).
Discretely decomposability of the restriction of Zuckerman-Vogan’s derived functor modules will be particularly important in applications in §4.B and §4.C below.
§4. Applications of admissible restrictions
4.A. Branching laws and geometry.
Historically, branching problems of unitary representations have been motivated,
not only by representation theory itself, but also by other fields, for instance, mathematical description of breaking symmetries in quantum mechanics, theta correspondence in automorphic forms and so on.
In this section, we shall discuss new interactions between branching problems
and related fields, which have been discovered in 1990s, especially, connected withdiscrete branching laws. The following principle was advocated in [37]:If representations help in the understanding of objects,
so do branching laws of representations in that of morphisms.
By simplifying settings for the exposition here, this principle may be explained
as follows. First, without group actions, let us consider the correspondences:geometry of X⇔

function space Γ(X),

map f : Y → X⇔

pullback f ∗ : Γ(X) → Γ(Y ).

Next, let G0 be a subgroup of G. Suppose that G acts on X and G0 on Y so that f
is G0 -equivariant. Then, the above correspondences are enriched by group actions:geometry of G-space X⇔

representation of G on Γ(X),G0 -equivariant map f : Y → X ⇔ restriction of the representation Γ(X) to G0+ G0 -intertwining map f ∗ : Γ(X) → Γ(Y ).
Thus, the knowledge of the restriction of representations of G to G0 should betransferred to some information on the original map f .
In this section, we shall illustrate this principle without technical details in thesettings where restrictions of unitary representations appear in a somewhat unexpected way. We shall also try to explain how and why discretely decomposable
restrictions play a crucial role there.

BRANCHING PROBLEMS OF UNITARY REPRESENTATIONS

25

4.B. A vanishing theorem for modular varieties.
In this subsection, we explain an application of our criterion for discretely decomposable restrictions (Theorem 1.5) to a differential geometric problem on modular
varieties. Roughly speaking, a modular symbol is the homology class in a locally
Riemannian symmetric space (sometimes called a Clifford-Klein form) determinedby the cycle induced by a subgroup. For example, a geodesic cycle in a closed
Riemann surface with genus ≥ 2 represents a modular symbol.
More generally, we consider the following setting:G0 ⊂ G : a pair of connected linear reductive Lie groups,
K 0 ⊂ K: maximal compact subgroups of G0 ⊂ G,
Γ0 ⊂ Γ : cocompact torsion-free discrete subgroups of G0 ⊂ G,
such that K 0 = K ∩ G0 and Γ0 = Γ ∩ G0 . Then, both of the double cosets
X := Γ\G/K and Y := Γ0 \G0 /K 0
are compact, orientable, locally Riemannian symmetric spaces. The inclusion G0 ,→G induces a natural map:
ι : Y → X.
The image ι(Y ) is called a modular variety. It is totally geodesic in the Riemannian
manifold X because the subgroup G0 is reductive in G. We put
m = dim Y (= dim G0 /K 0 ).
Then, the fundamental class [Y ] generates the homology group Hm (Y ; Z) of degree
m. Consider the induced homomorphism of homology groups of degree m:ι∗ : Hm (Y ; Z) → Hm (X; Z).
The modular symbol is defined to be the image ι∗ [Y ] ∈ Hm (X; Z) (see Ash andBorel [2]). Though its definition is simple, the understanding of modular symbols
is usually difficult.
In order to see how the discrete decomposability of the restriction G ↓ G0 (see§§1 and 2) affects a topological property of modular varieties, we consider a special
example where(G, G0 ) = (SO0 (2n, 2), SO0 (2n, 1)).
Then, dim G0 /K 0 = 2n, and the modular symbol ι∗ [Y ] is an element of the homology
group of degree 2n, of a 4n-dimensional locally Riemannian symmetric space
X = Γ\SO0 (2n, 2)/(SO(2n) × SO(2)),
of which the universal covering is the bounded symmetric domain of type IV. In
particular, X is a Kähler manifold. Since X is compact and orientable, we can regard the modular symbol as an element M(Y ) of the cohomology group H 2n (X; C)
via the Poincaré duality. Let
Mp,q (Y ) ∈ H p,q (X; C)26

TOSHIYUKI KOBAYASHI

be its Hodge component of type (p, q) such that p + q = 2n. Then, since X isKähler, we haveM(Y ) =

X

Mp,q (Y ).

p+q=2nThus, the Hodge components {Mp,q (Y ) : p + q = 2n} give a finer structure ofmodular symbols. Takayuki Oda conjectured a vanishing theorem of the middle
Hodge component Mn,n (Y ), from a viewpoint of automorphic forms (see [66]). Thisconjecture has been solved in [52] by using our criterion of discrete decomposable
restrictions (Theorem 1.5).

Theorem 4.1 (Hodge component of the modular symbol). There exists a universal
element η in the cohomology group H n,n (X; C), such that

Mn,n (Y ) =volume of Yη.
volume of X

Here, “universal” means that the element η is given in terms of Lie algebras explicitly, and does not depend on Γ. In particular, η is contained in the image of thecanonical map
H n,n (g, K; C) → H n,n (X, C).

Theorem 4.1 follows from a general vanishing theorem [52] of modular symbols
for a pair (G, G0 ) of reductive groups. The key point there is that the integrationof a harmonic form ω over Y becomes zero if ω comes from an infinite dimensional
irreducible unitary representation π of G such that the restriction π|G0 is discretely
decomposable. Then, for the latter condition, we can use a criterion given in
Theorem 1.5.
More precisely, what we need in the above special case is the following
b if
representation-theoretic result: the restriction π|K 0 is K 0 -admissible for any π ∈ Gn,n
H (g, K; πK ) 6= 0. This statement follows from an easy computation of Theorem
1.5.
Instead of an explanation of further technical details, we give the flavor of theproof by the following diagram that compares relevant results where the discreteness of spectrum plays an important role in the understanding of topology inother settings (see [42] for more details).

BRANCHING PROBLEMS OF UNITARY REPRESENTATIONS

Discreteness of spectrum⇒

X: compact Riemannian manifold
Laplacian ∆X has only discrete
spectrum in L2 (X)⇒27

Geometry ⇔ Function spaces(representations)

Hodge theory
topology ⇔ harmonic forms⇓ refinement

L2 (Γ\G) is decomposedbinto a discrete direct sum of G
(Gel’fand-Piateski-Shapiro, [15])⇒Matsushima-Murakami formula[8], [61] (1960s)⇓ (object ⇒ morphism)

Criterion for discrete decomposable
restrictions of Aq (λ) to subgroups(Kobayashi, [44])⇒Vanishing theoremof a modular symbol(Kobayashi-Oda, [52])Here Matsushima-Murakami formula describes the cohomology groups of a locally
Riemannian symmetric space in a representation-theoretic way (study of objects),
and Oda-Kobayashi vanishing theorem concerns maps between locally Riemanniansymmetric spaces (study of morphisms).
4.C. Application to non-commutative harmonic analysis:
Construction of new discrete series representations
for non-symmetric spaces.
Suppose a homogeneous space G/H carries a G-invariant Borel measure. Thisis the case if (G, H) is a pair of reductive Lie groups in the sense that G is areductive Lie group and H is a closed subgroup which is reductive in G. Then
we have a natural unitary representation of G on the Hilbert space L2 (G/H). We
b is realized in a closed
say π is a discrete series representation for G/H, if π ∈ G
2subspace of L (G/H), equivalently, if the space of continuous intertwining operators
HomG (π, L2 (G/H)) 6= {0}. Harish-Chandra’s discrete series representation π isthe case where H = {e}. A discrete series representation corresponds to a discrete
spectrum in the Plancherel formula, namely, the irreducible decomposition of theunitary representation L2 (G/H).
Discrete series representations are regarded as realizations of irreducible representations, through which infinite dimensional representation theory interacts in a
lively way with global analysis. Discrete series representations are not only a fundamental object in non-commutative harmonic analysis, but also play importantroles in the following topics.
(1) Construction of tempered series representations. Plancherel theorems forgroup manifolds and semisimple symmetric spaces have been discovered by
Harish-Chandra, Delorme, van den Ban, Oshima, Schlichtkrull and others.
The support of the Plancherel formula consists of tempered representations
which are obtained as the cuspidal parabolic induction of discrete series
representations for smaller symmetric spaces ([12], [18], [68]).

28

TOSHIYUKI KOBAYASHI(2) Topology of locally Riemannian symmetric spaces (e.g. a non-vanishingtheorem of modular symbol due to Tong and Wang [73]).
(3) Obstruction of the injectivity of Lp -Pompeiu problem in integral geometry;see [11] for simply connected solvable Lie groups; see [72] for SL(2, R); see[45], Theorem 1.2.17 for general reductive Lie groups and their homogeneous
spaces.
(4) A part of “isolated” irreducible unitary representations in the Fell topology(for example, [70]).
4.C.1. Current status on discrete series representations.
As the above observation indicates, the following is one of fundamental problems
in non-commutative harmonic analysis:
Problem 4.C.1.
1) Find a condition on the pair of groups (G, H) under which there exists a discrete
series representation for the homogeneous space G/H.
2) If they exist, construct and classify discrete series representations for G/H.
These problems have not been solved in the general setting where (G, H) is apair of real reductive Lie groups. Here are some special cases where answers are
known:i) The case where G/H is a semisimple Lie group, namely, H = {e}. Discrete
series representations exist if and only if rank G = rank K. All discrete series
representations are classified by Harish-Chandra. Geometric constructions of discrete series representations were studied by Atiyah, Hotta, Langlands, Okamoto,Parthasarathy, Schmid and others. Algebraic constructions were also studied by
Enright, Vogan, Wallach, Zuckerman and others.
ii) The case where G/H is a semisimple symmetric space. Discrete series representations exist if and only if rank G/H = rank K/(H ∩ K). They have been
constructed by Flensted-Jensen and Oshima-Matsuki. The classification has been
almost done, but certain subtle problems such as non-vanishing conditions andmultiplicity-one conjecture have not been completed, at least not in the literature.
The group case (i) can be regarded as a special case of (ii) by putting (G, H) =(G1 × G1 , diag G1 ). We refer to a survey [36], §4 and the references therein for (i)and (ii), and for more general results (e.g., vector bundle cases).
4.C.2. Relation to the discrete branching law.
The known methods used in (i) or (ii) are powerful, but they valid only in limited
cases, namely, only for semisimple symmetric spaces. For instance, one of the key
methods for the analysis on symmetric spaces is the Flensted-Jensen duality [14],which can be defined only when G/H is a symmetric space. Consequently, we needto a completely new machinery for the analysis on a more general homogeneous
space.
To construct new discrete series representations, our idea here is based on the
restriction of unitary representations. We will explain a rough idea of how the proof
goes (see [37] and [46] for precise formulation and some concrete examples):Step a) Suppose G/H is a homogeneous space, on which we want to constructe H
e for which
discrete series representations. We embed G/H into a larger space G/eeharmonic analysis is well-understood (e.g., G/H is a group manifold or a symmetric

BRANCHING PROBLEMS OF UNITARY REPRESENTATIONS

29

space):e H.
e
ι : G/H ,→ G/

Then, we consider the pullback of functions (or possibly, after taking finitely many
normal derivatives):
e H)
e → C ∞ (G/H),ι∗ : C ∞ (G/f 7→ f ◦ ι.

ee H)e and pick up a funcStep b) Take an irreducible G-representation
π in C ∞ (G/e He that belongs to a representation space of π. We expand the retion f on G/b as representations
striction ι∗ f into irreducible components (say, (ι∗ f )λ for λ ∈ G)of the subgroup G according to the branching law of the restriction π|G , and thenestimate the asymptotic behavior of each component (ι∗ f )λ along the submanifold
G/H at infinity (see [46]).
Step c) Find the asymptotic behavior of the measure on G/H, and compare thee He (see [43]).
asymptotic behavior of measures at infinity between G/H and G/
If the image of ι is a generic orbit (e.g., a principal type orbit in the sense of
Richardson), we do not need steps (b) and (c), and we can construct discrete series
representations by an elementary argument (this case was previously carried out
for some special homogeneous spaces in [21], [35], [37], [56]). For instance, by thisapproach, one can reduce the classification problem of discrete series representationsof some non-symmetric spherical homogeneous spaces G/H, such as
G/H = SU (n + 1, n)/Sp(n, R), G2 (R)/SL(3, R), · · · ,to that of discrete spectrum of some branching laws of discrete series representationse He ([37]). However, it often happens that non-generic orbitsof a (larger) space G/
give more interesting examples (in other words, some homogeneous spaces G/He He only as non-generic orbits, and then steps (b) and (c)can be embedded into G/
become necessary.
In step (b), if the branching law is discretely decomposable when restricted
e to G, then one can prove that the asymptotic behavior of each irreduciblefrom G
component (ι∗ f )λ has a nice decay inherited from that of the function f (see [46],e H,e then each§3). Consequently, if f satisfies an appropriate asymptotic decay on G/∗
non-zero irreducible component (ι f )λ generates a discrete series representation for
G/H. We remark that the assumption of discrete decomposability is crucial becausesuch a nice decay cannot be expected if π|G contains continuous spectrum.
Let us give a few comments on step (c). We note that a reductive homogeneous
space G/H does not always have a polar coordinate (for a special G/H such as asymmetric space, there is a polar coordinate G = KAH and accordingly we have an
integration formula on G/H essentially on the Abelian group A). The absence ofpolar coordinates causes difficulties in explicitly describing the asymptotic behaviorof the G-invariant measure on G/H at infinity. The idea of [47] for step (c) for anon-symmetric space G/H is to avoid working on the pseudo-Riemannian manifold
G/H (right coset space) itself, but to lift it to a larger dimensional non-homogeneous
space K ×(p/p∩h) where we can employ a comparison theorem of negatively curved
Riemannian manifold K\G (left coset space).

30

TOSHIYUKI KOBAYASHI

By using steps (a), (b) and (c), we can prove that there exist infinitely many
discrete series representations on homogeneous spaces such as:
G/H = Sp(2n, R)/(Sp(n0 , C) × GL(n1 , C) × · · · × GL(nk , C)),
where (n0 , n1 , · · · , nk ) is an arbitrary partition of n. We note that the above G/H
is a symmetric space if and only if
n1 = n2 = · · · = nk = 0,in the above example. The existence of discrete series representations were previously known by Flensted-Jensen [14] only in this special case.
It should be noted that the above approach deals with a family of homogeneous
spaces of G simultaneously, rather than a single homogeneous space alone. Thee H.
epoint here is that different homogeneous spaces of G can arise as G-orbits on G/
For instance, the above homogeneous space
G/H = Sp(2n, R)/(Sp(n0 , C) × GL(n1 , C) × · · · × GL(nk , C)),for an arbitrary partitionn = n0 + · · · + nk
arises as a G-orbit one H
e = (Sp(2n, R) × Sp(2n, R))/ diag(Sp(2n, R))G/(see [46]). Then we can “treat simultaneously” these homogeneous spaces by using
e to G. For instance, this explains
branching laws of unitary representations from G
the phenomenon that the same representations can occur as discrete series representations on certain different homogeneous spaces.
e H
e has recently been studied by Iida and Matsuki
The orbit decomposition G\G/[60]. Their description is useful in finding how G/H is embedded into a larger space
e H.
e
G/
The above approach not only gives new discrete series representations on nonsymmetric homogeneous spaces as stated, but also gives a new viewpoint evenfor analysis on symmetric spaces, where there is already extensive work in theliterature. For example, we can prove (without using serious results on semisimple
symmetric spaces) the following new geometric theorem, by making use of thecriterion of discretely decomposable restrictions (Theorems 1.5 and 2.4):
Theorem 4.2 (a necessary and sufficient condition for the existence of holomorphic discrete series representation on symmetric spaces). Suppose G/H is a noncompact irreducible symmetric space. Then the following two conditions on (G, H)
are equivalent:(i) There exist unitary highest weight representations of G that can be realized as
discrete series representations for G/H.
(ii) H/(H ∩ K) is a real form of the complex manifold G/K.

BRANCHING PROBLEMS OF UNITARY REPRESENTATIONS

31

Example 4.3. Let G/H = (SL(2, R) × SL(2, R))/ diag(SL(2, R)). We put D :={z ∈ C : |z| < 1}, the unit disc. Then the natural embedding H/(H ∩ K) ,→ G/K
is realized as the following map:
D ,→ D × D, z 7→ (z, z̄).
In particular, H/(H ∩ K) is a real form of G/K; that is, the geometric condition(ii) is satisfied. Then, Theorem 4.2 gives a new explanation of the well-known factthat there exist holomorphic discrete series representations of a group manifold
G/H ' SL(2, R).
The result (i) ⇒ (ii) in Theorem 4.2 is new. The opposite direction (ii) ⇒ (i)was previously proved by a completely different method (Ólafsson-Ørsted [67]).
4.D. Discontinuous groups versus restrictions of unitary representations.
4.D.1. Discretely decomposable restrictions revisited.
Let U (H) be the group of unitary operators on a Hilbert space H, and we
consider an irreducible unitary representation of G realized on H, namely, a grouphomomorphism
π : G → U (H).
The restriction to a subgroup G0 of G is nothing but the composition of the following
group homomorphisms:(4.4.1)π

G0 ⊂ G → U (H).

If G0 is compact, then the restriction π|G0 is always discretely decomposable. We
have seen in §1 that the restriction π|G0 can be discretely decomposable even when
G0 is non-compact. We may discuss the discrete decomposability of the restriction
π|G0 from the following viewpoint:“In the infinite dimensional group U (H),the image of a non-compact Lie group of G0
may behave as if it were a compact group.”
4.D.2. Properly discontinuous actions of discrete groups.
Let us consider a different setting. Let Γ be a topological group acting continuously on a manifold M . We define a subset of Γ by
ΓS := {γ ∈ Γ : γ · S ∩ S 6= ∅},for a subset S of M . The action of Γ on M is said to be proper [69] if ΓS iscompact for any compact subset S in M ; properly discontinuous if ΓS is finite forany compact subset S in M . We note that Γ acts properly discontinuously if andonly if Γ is discrete and acts properly on M .
A typical example of properly discontinuous actions is the covering transformation of the fundamental group on the universal covering space. Conversely, if a
torsion-free discrete group Γ acts on a manifold M properly discontinuously, thenthere is a natural manifold structure on the space of Γ-orbits, denoted by Γ\M ,and the quotient map M → Γ\M becomes a covering map.

32

TOSHIYUKI KOBAYASHIWe observe that the action of Γ on M is written as a homomorphism fromΓ to the group Diffeo(M ) of diffeomorphisms of M , or the group Homeo(M ) of
homeomorphisms of M :Γ → Homeo(M ).
The action of a finite group are always properly discontinuous. Then, again, properly discontinuous actions may be discussed from the following viewpoint“In the infinite dimensional group Homeo(M ),the image of a discrete group Γ
may behave as if it were a compact (or finite) group. ”
4.D.3. Restriction of unitary representations and proper actions.
The above two cases are summarized as follows:
Actions of finite groups are obviously properly discontinuous. It can happen
that actions of infinite groups are still properly discontinuous, such as coveringtransformations.
On the other hand, branching laws of compact groups are always discretely
decomposable, and it can happen that branching laws with respect to non-compact
subgroups are still discretely decomposable.
Comparing these two examples, one might ask the following question:
Question. Is there any relation between “discrete decomposability of branchinglaws of unitary representations of Lie groups” and “properly discontinuous actionsof discrete groups”?
As a special case, let us consider the setting where a Lie group G acts transitively
on M . Let H be the isotropy subgroup at a point x0 of M . Then we have a natural
homeomorphism G/H ' M, gH 7→ gx0 . Then a discrete subgroup Γ of G acts on
M by the left translation. This means that we have a group homomorphism:(4.4.2)

Γ ⊂ G → Diffeo(G/H).

We shall compare the two settings (4.4.1) and (4.4.2) below.
4.D.4. Criteria— discretely decomposable representations
and properly discontinuous actions.
Without proof and precise formulation, we give a brief summary of the following
known criteria:(i) In the setting (4.4.1), a sufficient condition for the discrete decomposability of
the restriction π|G0 was given in Theorem 1.5, roughly in the following form:(the cone determined by G0 ) ∩ (the cone determined by π) = {0}.
(ii) In the setting (4.4.2) (with Γ replaced by G0 ), a necessary and sufficient condition
for a reductive subgroup G0 on the homogeneous space G/H to act properly wasproved by the author in 1989 [27]:(the linear subspace determined by G0 )∩ (the linear subspace determined by H)= {0} modulo the action of a finite group (Weyl group).

BRANCHING PROBLEMS OF UNITARY REPRESENTATIONS

33(iii) In the setting (4.4.2), a necessary and sufficient condition for the action of adiscrete subgroup Γ on the homogeneous manifold G/H to be properly discontinuous is given in the following form:{a subset determined by Γ} ∩ {a tube determined by H}
is relatively compact modulo the action of a finite group (Weyl group).
This criterion generalizes [27] and was proved independently by Benoist [4] andKobayashi [39].
4.D.5. Actions of discrete groups and branching lawsof unitary representations.
In the previous subsection, (i) concerns an analytic representation theory (discreteness of spectrum), and (ii) and (iii) concern topological problems (proper actions). Accordingly, the objects and methods employed there are completely different. However, the criteria themselves are apparently similar to one another, andmay suggest a relationship in the following diagram:(iii) Discrete version:

Properly discontinuous actionsof discrete groups↓↑(ii) Continuous version:

Proper actions of connected Lie groups↓↑(i) Representation theory:

Discrete decomposable restrictions of
unitary representations of Lie groups

In fact, a first non-trivial example (see [28]) of the discrete branching laws of
the restriction π|G0 (i.e., π is not a highest weight representation, and G0 is notcompact) was inspired by the above diagram (see also §0). More precisely, the ideaof [28] consists of the following four steps:
Step a) First, take a uniform lattice6 Γ for the semisimple symmetric space G/H(see [27] for the construction).
Step b) Take the Zariski closure of Γ in G, for which we write G0 .
b for G/H (see [14] for the
Step c) Take a discrete series representation π(∈ G)construction).
Step d) Find branching laws of the restriction π|G0 .
6 In contrast to Borel’s theorem [7] on the existence of a uniform lattice of an arbitrary Rie-

mannian symmetric space G/H, there does not always exist a uniform lattice for a non-Riemannian
homogeneous manifold (for example, a semisimple symmetric space). Even the so-called CalabiMarkus phenomenon [10] occurs if H is non-compact. We refer to [41] for an exposition to
discontinuous groups on non-Riemannian homogeneous manifolds, developed rapidly in the last
decade.

34

TOSHIYUKI KOBAYASHI

A simplest example is given in the following setting:
G/H = SU (2, 2)/U (2, 1) (an open subset of P 3 C),
G0 = Sp(1, 1) ' Spin(4, 1) (de Sitter group),Γ is a uniform lattice of G0 = Sp(1, 1),
π is a discrete series representation for G/H.
We note that the above representation π has the Gel’fand-Kirillov dimension 5
and is not a highest weight representation. Then the restriction π|G0 turns out tobe discretely decomposable (see [36], Example 3.3). As we mentioned (see Theorem 1.5), the discrete decomposability of the restriction was later formulated interms of representation theory, apart from the above setting on global analysis on
homogeneous spaces.
The above setting also gives an example of an interesting Riemannian structure g
on a 6-dimensional simply connected manifold M = G/H: (M, g) is a non-compact
covering of a compact Riemannian manifold, and there exist L2 -eigenfunctions of
the Laplace-Beltram operator (related problems have been studied by Sunada [29],[64]).
With regard to Step (a), the following has been studied intensively in the last
decade:Problem. Does a pseudo-Riemannian homogeneous manifold admit a uniform lattice?
Various approaches to this problem include the structural results of Lie groups,
a criterion of proper actions of non-compact Lie groups, cohomology groups of discrete groups, characteristic classes, symplectic geometry, ergodic theory, and therestriction of unitary representations, etc., by Benoist, Corllette, Labourie, Margulis, Oh, Ono, Witte, Zimmer and the author after a general theory ([27], 1989)(see [4], [5], [30], [32], [33], [34], [39], [41], [48], [59], [65], [80]). In particular, a
recent method due to Margulis [59] is based on the asymptotic behavior of matrix
coefficients of the restrictions of unitary representations, which should merit further study, as it might strengthen a tie between unitary representations anddiscontinuous groups.
In the above diagram, the last ↓↑ is just a guiding principle, and a rigorous
formulation is not known. It is mysterious to me if there is an intrinsic interaction between properly discontinuous actions and branching problems of unitary
representations, especially discretely decomposable restrictions.

References
[1]
[2]
[3]
[4]
[5]

J. Adams, Discrete spectrum of the dual reductive pair (O(p, q), Sp(2m)), Invent. Math. 74(1984), 449–475.
A. Ash and A. Borel, Generalized modular symbols, Cohomology of arithmetic groups andautomorphic forms, Lecture Notes in Math., vol. 1447, Springer, 1990, pp. 57-76.
M. F. Atiyah, The Harish-Chandra character, Representation theory of Lie groups, London
Math. Soc. Lecture Note Series, vol. 34, Cambridge Univ. Press, 1979, pp. 176–181.
Y. Benoist, Actions propres sur les espaces homogènes réductifs, Annals of Math. 144 (1996),
315–347.
Y. Benoist and F. Labourie, Sur les espaces homogenes modeles de varietes compactes, Inst.
Hautes Etudes Sci. Publ. Math. 76 (1992), 99–109.

BRANCHING PROBLEMS OF UNITARY REPRESENTATIONS[6]35

M. Berger, Les espaces symétriques non compacts, Ann. Sci. Ecole Norm. Sup. (3) 74 (1957),
85–177.
[7] A. Borel, Compact Clifford-Klein forms of symmetric spaces, Topology 2 (1963), 111-122.
[8] A. Borel and N. Wallach, Continuous Cohomology, Discrete Subgroups, and Representationsof Reductive Groups, Ann. Math. Stud. 94, Princeton U.P., 1980.
[9] W. Borho and J. L. Brylinski, Differential operators on homogeneous spaces I, Invent. Math.
69 (1982), 437-476, Part III, ibid. 80 (1985), 1-68.
[10] E. Calabi and L. Markus, Relativistic space forms, Annals of Math. 75 (1962), 63–76.
[11] A. L. Carey, E. Kaniuth and W. Moran, The Pompeiu problem for groups, Math. Proc. Camb.
Phil. Soc. 109 (1991), 45-58.
[12] P. Delorme, Formule de Plancherel pour les espaces symétriques réductifs, Annals of Math.
147 (1998), 412–452.
[13] T. Enright, R. Howe and N. Wallach, A classification of unitary highest weight modules,
Representation theory of reductive groups, Progress in Math., vol. 40, Birkhäuser, 1983,
pp. 97–143.
[14] M. Flensted-Jensen, Discrete series for semisimple symmetric spaces, Annals of Math. 111(1980), 253–311.
[15] I. M. Gel’fand, M. I. Graev and I. Piateski-Shapiro, Representation Theory and Automorphic
Functions, Saunders Math. Books, 1969.
[16] B. H. Gross and N. R. Wallach, A distinguished family of unitary representations for the
exceptional groups of real rank = 4, Lie Theory and Geometry, In honor of Bertram Kostant,
Progress in Math., vol. 123, Birkhäuser, 1994, pp. 289–304.
[17] Harish-Chandra, Representations of semi-simple Lie groups, I, III, Trans. A. M. S. 75, (1953),
185–243; 76, (1954), 234–253; IV, Amer. J. Math. 77, (1955), 743–777.
[18] Harish-Chandra, Harmonic analysis on real reductive groups, I, J. Funct. Anal. 19, (1975),
104–204; II, Invent. Math. 36, (1976), 1–55; III, Annals of Math. 104, (1976), 117–201.
[19] H. Hecht and W. Schmid, A proof of Blattner’s conjecture, Invent. Math. 31 (1976), 129–154.
[20] R. Howe, θ-series and invariant theory, Proc. Symp. Pure Math. 33 (1979), Amer. Math.
Soc., 275–285.
[21] J-S. Huang, Harmonic analysis on compact polar homogeneous spaces, Pacific J. Math. 175(1996), 553—569.
[22] H. P. Jakobsen and M. Vergne, Restrictions and expansions of holomorphic representations,
J. Funct. Anal. 34 (1979), 29–53.
[23] M. Kashiwara, T. Kawai and T. Kimura, Foundations of Algebraic Analysis, Princeton Math.
Series, vol. 37, Princeton Univ. Press, 1986.
[24] M. Kashiwara and M. Vergne, On the Segal-Shale-Weil representations and harmonic polynomials, Invent. Math. 44 (1978), 1–47.
[25] M. Kashiwara and M. Vergne, K-types and singular spectrum, Lect. Notes in Math., vol. 728,
Springer, 1979, pp. 177–200.
[26] A. Knapp and D. Vogan, Cohomological Induction and Unitary Representations, Princeton
U.P., 1995.
[27] T. Kobayashi, Proper action on a homogeneous space of reductive type, Math. Ann. 285(1989), 249–263.
[28] T. Kobayashi, Unitary representations realized in L2 -sections of vector bundles over semisimple symmetric spaces, Proceedings at the 27-28th Symp. of Functional Analysis and Real
Analysis (1989), Math. Soc. Japan, 39–54.
[29] T. Kobayashi, K. Ono and T. Sunada, Periodic Schrödinger operators on a manifold, Forum
Mathematicum 1 (1989), 69–79.
[30] T. Kobayashi and K. Ono, Note on Hirzebruch’s proportionality principle, J. Fac. Sci. Univ.
Tokyo 37-1 (1990), 71–87.
[31] T. Kobayashi, Singular Unitary Representations and Discrete Series for Indefinite Stiefel
Manifolds U (p, q; F)/U (p−m, q; F), vol. 462, Memoirs of Amer. Math. Soc., 1992, (106 pages).
[32] T. Kobayashi, A necessary condition for the existence of compact Clifford-Klein forms ofhomogeneous spaces of reductive type, Duke Math. J. 67 (1992), 653–664.
[33] T. Kobayashi, Discontinuous groups acting on homogeneous spaces of reductive type, Proc.
of ICM-90 Satellite Conference on “Representation Theory of Lie Groups and Lie Algebras”
at Fuji-Kawaguchiko, World Scientific, 1992, pp. 59–75.

36

TOSHIYUKI KOBAYASHI[34] T. Kobayashi, On discontinuous groups acting on homogeneous spaces with non-compact
isotropy subgroups, J. Geom. Physics 12 (1993), 133–144.
[35] T. Kobayashi, The restriction of Aq (λ) to reductive subgroups, Part I, Proc. Japan Acad. 69(1993), 262–267; Part II, ibid. 71 1995, 24–26.
[36] T. Kobayashi, Harmonic analysis on homogeneous manifolds of reductive type and unitary
representation theory, Sugaku 46 (1994), 124–143; English transl., Amer. Math. Soc. Transl.
(2) 183 (1998), A.M.S, 1–31.
[37] T. Kobayashi, Discrete decomposability of the restriction of Aq (λ) with respect to reductive
subgroups and its applications, Invent. Math. 117 (1994), 181–205.
[38] T. Kobayashi, Introduction to harmonic analysis on spherical homogeneous spaces, Proceedings of the 3rd summer school on number theory, Homogeneous Spaces and Automorphic
Forms (F. Sato, eds.), 1995, pp. 22–41. (Japanese)[39] T. Kobayashi, Criterion of proper actions on homogeneous spaces of reductive groups, J. Lie
Theory 6 (1996), 147–163.
[40] T. Kobayashi, On the restriction of unitary representations and their applications, Proceedings of the Symposium of Representation Theory held at Mikawa-Heights (November 19–22,
1996) (Takenaka, eds.), 1996, pp. 131–141.
[41] T. Kobayashi, Discontinuous groups and Clifford-Klein forms of pseudo-Riemannian homogeneous manifolds, Algebraic and Analytic Methods in Representation Theory (H.
Schlichtkrull and B. Ørsted, ed.), Perspectives in Mathematics 17, Academic Press, 1996,
pp. 99–165, Lecture Notes of the European School on Group Theory 1994.
[42] T. Kobayashi, Vanishing theorem for modular symbols on locally symmetric spaces (Lecture notes in Japanese, taken by S. Ishikawa), Proceedings of “Representation theory andrelated topics” (held at Kurashiki, November 1996) (N. Shimeno, ed.), 1996, pp. 1–16,
http://www.merry.xmath.ous.ac.jp/math /Nov5th1996.html.
[43] T. Kobayashi, Invariant measures on homogeneous manifolds of reductive type, J. Reine undAngew. Math. 490 (1997), 37–53.
[44] T. Kobayashi, Discrete decomposability of the restriction of Aq (λ) with respect to reductive
subgroups II — micro-local analysis and asymptotic K-support, Annals of Math. 147 (1998),
709–729.
[45] T. Kobayashi, Convex domains and Fourier transform on spaces of constant curvature,
Travaux en Cours (P. Torasso, ed.), Hermann, Paris, 112 pages (to appear).
[46] T. Kobayashi, Discrete series representations for the orbit spaces arising from two involutionsof real reductive Lie groups, J. Funct. Anal. 152 (1998), 100–135.
[47] T. Kobayashi, Multiplicity free branching laws for unitary highest weight modules, Proceedings of the Symposium on Representation Theory held at Saga, Kyushu (K. Mimachi, ed.),1997, pp. 7–13.
[48] T. Kobayashi, Deformation of compact Clifford-Klein forms of indefinite-Riemannian homogeneous manifolds, Math. Ann. 310 (1998), 394–408.
[49] T. Kobayashi, Discrete decomposability of the restriction of Aq (λ) with respect to reductive
subgroups III — restriction of Harish-Chandra modules and associated varieties, Invent.
Math. 131 (1998), 229–256.
[50] T. Kobayashi, Discretely decomposable restrictions of unitary representations of reductive
Lie groups —examples and conjectures., Analysis on homogeneous spaces and representation
theory of Lie groups, Okayama–Kyoto (1997) (T. Kobayashi et.al., eds.), vol. 26, Math. Soc.
Japan, Tokyo, 2000, pp. 99-127.
[51] T. Kobayashi, Theory of discrete decomposable branching laws of unitary representations ofsemisimple Lie groups and its applications, (in Japanese), Proceedings of the annual meetingof Mathematical Society of Japan, (Gakushuin University, March 25–28, 1999) (1999), Math.
Soc. Japan, 1–19.
[52] T. Kobayashi and T. Oda, Vanishing theorem of modular symbols on locally symmetric spaces,
Comment. Math. Helvetici 73 (1998), 45–70.
[53] T. Kobayashi and B. Ørsted, Conformal geometry and branching laws for unitary representations attached to minimal nilpotent orbits, C. R. Acad. Sci. Paris Sér. I Math. 326 (1998),
925–930.
[54] B. Kostant, The vanishing scalar curvature and the minimal unitary representation ofSO(4, 4), Operator Algebras, Unitary Representations, Enveloping Algebras, and Invariant
Theory (Connes et al, eds.), vol. 92, Birkhäuser, 1990, pp. 85–124.

BRANCHING PROBLEMS OF UNITARY REPRESENTATIONS

37[55] R. Langlands, On the classification of irreducible representations of real algebraic groups,
mimeographed notes (1973); finally published in Representation Theory and Harmonic Analysis on Semisimple Lie Groups (P. J. Sally and D. A. Vogan, eds.), Math. Surveys andMonographs, vol. 31, Amer. Math. Soc., 1989, pp. 101–170.
[56] J-S. Li, On the discrete series of generalized Stiefel manifolds, Trans. A. M. S. 340 (1993),
753–766.
[57] J-S. Li, Two reductive dual pairs in groups of type E, Manuscripta Math. 91 (1996), 163–177.
[58] G. W. Mackey, Induced representations of locally compact groups I, Annals of Math. 55(1952), 101–139.
[59] G. Margulis, Existence of compact quotients of homogeneous spaces, measurably proper actions, and decay of matrix coefficients, Bul. Soc. math. France 125 (1997), 1–10.
[60] T. Matsuki, Double coset decompositions of algebraic groups arising from two involutions I,
J. Algebra 175 (1995), 865–925.
[61] Y. Matsushima and S. Murakami, On vector bundle valued harmonic forms and automorphic
forms on symmetric spaces, Annals of Math. 78 (1963), 365-416.
[62] C. C. Moore, Ergodicity of flows on homogeneous spaces, Amer. J. Math. 88 (1966), 154-178.
[63] Y. A. Neretin and G. I. Ol’shanskii, Boundary values of holomorphic functions, special unitary representations of O(p, q), and their limits, Zapiski Nauchnykh Seminarov POMI 223(1995), 9–91; English transl. J. Math. Sci. 87 (1997), 3983–4035.
[64] T. Ochiai, On the spectrum of the Laplacian on Riemannian manifolds (On the work ofToshikazu Sunada, winner of the 15th Mathematical Society of Japan Prize), Sugaku 39(1987), Math. Soc. Japan, 237–242.
[65] H. Oh and D. Witte, Compact Clifford-Klein forms of homogeneous spaces of SO(2, n),Geom. Dedicata 89 (2002), 25–57.
[66] T. Oda, Real analysis for automorphic forms, Sugaku 50 (1998), Math. Soc. Japan, 350–357.
[67] G. Ólafsson and B. Ørsted, The holomorphic discrete series of an affine symmetric space, I,
J. Funct. Anal. 81 (1988), 126–159.
[68] T. Oshima, Harmonic analysis on semisimple symmetric spaces, Sugaku 37 (1985), Math.
Soc. Japan, 97–112.
[69] R. S. Palais, On the existence of slices for actions of noncompact Lie groups, Annals of Math.
73 (1961), 295–323.
[70] H. Schlichtkrull, The Langlands parameters of Flensted-Jensen’s discrete series for semisimple symmetric spaces, J. Funct. Anal. 50 (1983), 133–150.
[71] W. Schmid, Die Randwerte holomorphe Funktionen auf Hermetisch symmetrischen Raumen,
Invent. Math. 9 (1969-70), 61–80.
[72] A. Sitaram, Some remarks on measures on noncompact semisimple Lie groups, Pacific J.
Math. 110 (1984), 429–434.
[73] Y. L. Tong and S. P. Wang, Geometric realization of discrete series for semisimple symmetricspaces, Invent. Math. 96 (1989), 425–458.
[74] D. A. Vogan, Jr., The algebraic structure of the representations of semisimple Lie group I,
Annals of Math. 109 (1979), 1–60.
[75] D. A. Vogan, Jr., Representations of Real Reductive Lie Groups, Prog. Math. vol., 15,
Birkhäuser, 1981.
[76] D. A. Vogan, Jr., Unitarizability of certain series of representations, Annals of Math. (1984),
141–187.
[77] D. A. Vogan, Jr., Associated varieties and unipotent representations, Harmonic Analysis on
Reductive Lie Groups, Progr. Math. vol., 101, vol. 101, Birkhäuser, 1991, pp. 315–388.
[78] N. Wallach, Real Reductive Groups I, II, Pure and Appl. Math., vol. 132, Academic Press,
1988, 1992.
[79] J. Xie, Restriction of discrete series of SU (2, 1) to S(U (1) × U (1, 1)), J. Funct. Anal. 122(1994), 478–518, Ph.D dissertation, Rutgers University.
[80] R. J. Zimmer, Discrete groups and non-Riemannian homogeneous spaces, J. Amer. Math.
Soc. 7 (1994), 159–168.
References added in Translation[81] B. Gross and N. Wallach, Restriction of small discrete series representations to symmetricsubgroups, Proc. Sympos. Pure Math. 68 (2000), A.M.S., 255–272.

38

TOSHIYUKI KOBAYASHI[82] J-S. Huang, P. Pandžić, and G. Savin, New dual pair correspondences, Duke Math. 82 (1996),
447–471.
[83] T. Kobayashi, Conformal geometry and analysis on minimal representations, Rendicontidel Circolo Matematico di Palermo, Serie II, Suppl. 71 (2003), 15–40, Lecture Notes of theWinter School 2002 on Geometry and Physics, Czech Republic.
[84] T. Kobayashi, Discontinuous groups for non-Riemannian homogeneous spaces, Mathematics
Unlimited —2001 and Beyond (B. Engquist and W. Schmid, eds.), Springer-Verlag, 2000,
pp. 723-728.
[85] T. Kobayashi, Branching problems of unitary representations, Proc. of the I.C.M. 2002, at
Beijing, vol. 2 (2002), Higher Ed. Press, Beijing, 615–627.
[86] T. Kobayashi, Admissible restrictions of unitary representations of reductive Lie groups –discrete decomposability and symplectic geometry, in preparation.
[87] T. Kobayashi, Lectures on restrictions of unitary representations of real reductive groups,
Lie Theory II, Lie Algebras and Representations, Progr. Math., Birkhäuser (to appear).
[88] T. Kobayashi and B. Ørsted, Analysis on the minimal representation of O(p,q), I, Adv.
Math. 180 (2003), 486–512;, Part II, ibid. 180 (2003), 513–550; Part III, ibid. 180 (2003),
551–595.
[89] S-T. Lee and H-Y. Loke, Degenerate principal series of U(p,q) and Spin(p,q), Compositio
Math. 132 (2002), 311–348.
[90] H-Y. Loke, Restrictions of quaternionic representations, J. Funct. Anal. 172 (2000), 377–403.
[91] H-Y. Loke, Howe quotients of unitary characters and unitary lowest weight modules, preprint.
[92] B. Ørsted and J. Vargas, Restriction of square integrable representations: discrete spectrum,
Duke Math. (to appear).
[93] S. A. Salamanca-Riba and D. A. Vogan, Jr., On the classification of unitary representationsof reductive Lie groups, Ann. of Math. 148 (1998), 1067–1133.
[94] W. Schmid and K. Vilonen, Characteristic cycles and wave front cycles of representations of
reductive Lie groups, Ann. of Math. 151 (2000), 1071–1118.
RIMS, Kyoto University, Sakyo-ku, Kyoto, 606-8502, Japan
E-mail address: toshi@kurims.kyoto-u.ac.jp