A CUSPIDALITY CRITERION FOR THE EXTERIOR SQUARETRANSFER OF CUSP FORMS ON GL(4)MAHDI ASGARI

AND

A. RAGHURAM

Dedicated to Freydoon Shahidi on the occasion of his sixtieth birthday
Abstract. For a cuspidal automorphic representation Π of GL(4, A), H. Kim provedthat the exterior square transfer ∧2 Π is nearly an isobaric automorphic representation ofGL(6, A). In this paper we characterize those representations Π for which ∧2 Π is cuspidal.

1. Introduction and statement of the main theorem
Let F be a number field whose adèle ring we denote by AF . Let G1 and G2 be twoconnected reductive linear algebraic groups over F , with G2 quasi-split over F , and let
L G and L G be the corresponding L-groups. Given an L-homomorphism r : L G → L G ,1
2
1
2
Langlands principle of functoriality predicts the existence of a transfer Π 7→ r(Π) of the Lpacket of an automorphic representation Π of G1 (AF ) to an L-packet r(Π) of automorphic
representations of G2 (AF ). Now assume that G2 is a general linear group. We note that an
L-packet for a general linear group is a singleton set. For applications of functoriality one
needs to understand the image and fibers of the correspondence Π 7→ r(Π). In particular, itis necessary to understand what conditions on Π ensure that the transfer r(Π) is cuspidal.
The main aim of this paper is to describe a cuspidality criterion for the transfer of
automorphic representations from GL(4, AF ) to GL(6, AF ) corresponding to the exteriorsquare map ∧2 : GL(4, C) → GL(6, C). Langlands functoriality in this case is a deep
theorem due to H. Kim [13].
Let Π = ⊗v Πv and Σ = ⊗v Σv be irreducible isobaric automorphic representations ofGL(4, AF ) and GL(6, AF ), respectively. Assume that S is a finite set of places of F , including all the archimedean ones, outside of which both of the representations are unramified.
We say Σ is an exterior square transfer of Π if for all v 6∈ S we have Σv = ∧2 (Πv ), i.e., thesemi-simple conjugacy class in GL(6, C) determining Σv is generated by the image under∧2 of the semi-simple conjugacy class in GL(4, C) determining Πv . By strong multiplicity
one theorem (see Theorem 2.1 below) such a Σ would be unique. We will denote it by ∧2 Π.
The existence of ∧2 Π was established by H. Kim [13, Theorem A]. Furthermore, he showedthat if Πv is not supercuspidal for the places v dividing 2 or 3, then the local component
Πv and (∧2 Π)v are compatible via the local Langlands correspondence for GL(4, Fv ) andGL(6, Fv ). The assumption at v|2, 3 was made because of complications posed by supercuspidal representations, especially of GL(4, Fv ). In any event, it has no bearing on our
result as we do not need the fact that the local components of ∧2 Π and Π are compatible
Date: 9 September 2008.
2000 Mathematics Subject Classification. 11F70 (11R42, 22E55, 11F80).
1

2

M. ASGARI & A. RAGHURAM

via the local Langlands correspondence at all places. We now state the main theorem ofthis article.
Theorem 1.1. Let F be a number field and let Π be a cuspidal automorphic representationof GL(4, AF ). The following are equivalent:(i) ∧2 Π is not cuspidal.
(ii) Π is one of the following:(a) Π = π1  π2 , the transfer from GL(2, AF ) × GL(2, AF ) to GL(4, AF ) via theautomorphic tensor product . (This may also be viewed as the transfer fromsplit GSpin(4) to GL(4).)(b) Π = As(π), the Asai transfer of a dihedral cuspidal automorphic representation
π of GL(2, AE ) where E/F is a quadratic extension. (This may also be viewed
as the transfer to GL(4) from the quasi-split non-split GSpin∗ (4) over F which
splits over E.)(c) Π is the functorial transfer of a cuspidal representation π of GSp(4, AF ) associated with the natural embedding of the dual group GSp(4, C) into GL(4, C).
The representation π may be taken to be globally generic.
(d) Π = IFE (π), the automorphic induction of a cuspidal automorphic representation
π of GL(2, AE ), where E/F is a quadratic extension.
(iii) Π satisfies one of the following:(α) Π ∼= Π∨ ⊗ χ for some Hecke character χ of F , and Π is not the Asai transfer
of a nondihedral cuspidal representation.
(β) Π ∼= Π ⊗ χ for a nontrivial Hecke character χ of F .
We observe that the groups in (ii)(a)–(ii)(d) are some of the twisted endoscopic groups
for GL(4) × GL(1) which have the property that the image under ∧2 of the connectedcomponent of their dual groups are contained in proper Levi subgroups of GL(6, C). Recall
from [2, §3.7] that the exterior square transfer we consider here is a special case of the moregeneral transfer from GSpin(2n) to GL(2n). One would expect that the above theorem
admits a generalization to that setting through the theory of twisted endoscopy.
We now briefly sketch the proof of the theorem. It is easy to verify that (ii) implies
both (i) and (iii). In Section 3 we explicitly write down the isobaric decomposition of ∧2 Πfor each of the cases (ii)(a)–(ii)(d). In order to check that two isobaric representations
are isomorphic we repeatedly use a strong multiplicity one theorem, due to Jacquet andShalika, recalled in Section 2.1.
The proof of (i) =⇒ (iii), described in Section 4, uses some details from the Langlands–
Shahidi machinery. We show in Proposition 4.1 that if Π is not essentially self-dual, then∧2 Π cannot have a degree 1 or degree 3 isobaric summand. In Proposition 4.2 we verify
that if Π does not admit a nontrivial self-twist, then ∧2 Π cannot have a degree 2 isobaric
summand. For a cuspidal representation σ of GL(m, AF ) consider the Langlands L-function
L(s, Π×σ, ∧2 ⊗ρm ), where ρm is the standard representation of GL(m, C). These L-functions
appear in the Langlands–Shahidi machinery for a particular choice of Levi subgroup when
the ambient group is GSpin(2m + 6). We use Kim [13, §3] to show that, under the above
mentioned hypothesis on Π, these (partial) L-functions are holomorphic at s = 1. TheCUSPIDALITY CRITERION

3

implication then follows from a well-known result of Jacquet, Piatetski-Shapiro, and Shalika recalled in Section 2.1. We summarize some of the preliminaries we need from theLanglands–Shahidi machinery in Section 2.3.
Finally, the proof of (iii) =⇒ (ii) is given in Section 5. It relies on the so-called ‘descent
theory’ for classical groups. The version of descent theory for GSpin groups that we needhas now been announced by J. Hundley and E. Sayag [9]. As a general reference for descent
theory we refer to Soudry’s exposition [31].
In Sections 6.1 and 6.2 we present a few examples, some only conjectural, illustrating
the above theorem. In Section 6.3 we comment on possible intersections among the casesin (ii). In Section 6.4 we ask whether it is possible to see the cuspidality criterion fromthe ‘Galois side’. The question can be made precise based on the philosophy that there
is a correspondence between automorphic representations π of GL(n, AF ) and `-adic ndimensional representations σ of the absolute Galois group of F , or n-dimensional complex
representations of the conjectural Langlands group LF . Let us denote this correspondenceby π 7→ σ(π). Part of this philosophy is that π is supposed to be cuspidal if and only if
σ(π) is irreducible. We refer to Ramakrishnan [27] for the state of the art on this issue. Inview of the above theorem one can ask the following question. Let σ be a four-dimensional
irreducible Galois representation; what condition on σ will ensure that ∧2 σ is irreducible?
Upon posing this question in a talk at the Oklahoma Joint Automorphic Forms Seminar,
A. Kable came up with a very elegant theorem which reflects the equivalence of (i) and(iii) in Theorem 1.1. We are grateful to him for allowing us to include his theorem and its
proof in Section 6.4. Recall that in (iii)(α) of Theorem 1.1 above, we had to exclude the
Asai transfer of a nondihedral cuspidal representation if Π is essentially self-dual. On the
Galois side, this is reflected in the fact that if a four-dimensional irreducible representation
σ is essentially self-dual of orthogonal type, then for ∧2 σ to be reducible the image of σshould lie in the connected component of the identity in the algebraic group GO(4); see
Theorem 6.5.
Cuspidality criteria are important not only for their intrinsic value in helping us better
understand a given instance of functoriality but also because they have important arithmetic
applications. D. Ramakrishnan and S. Wang [29] proved a cuspidality criterion for thetransfer from GL(2) × GL(3) to GL(6) and used it to construct new cuspidal cohomology
classes for GL(6). We refer to [22] for a brief survey of cohomological applications ofLanglands functoriality. H. Kim and F. Shahidi [18] proved a cuspidality criterion for thesymmetric fourth transfer from GL(2) to GL(5), which has been used in the study of special
values of symmetric power L-functions by the second author and F. Shahidi [23]. Such a
potential arithmetic application was indeed our original motivation to seek a cuspidality
criterion for the exterior square transfer.
Acknowledgements: It is our pleasure to thank H. Kim, A. Kable, J.-P. Labesse, F. Shahidi,
and R. Zierau for helpful discussions and comments. We also thank the referee for helpfulcomments. The first author is partially supported by an Alexander von Humboldt Research
Fellowship. The second author is partially supported by an ASR +1 grant from the Collegeof Arts and Science of Oklahoma State University.

4

M. ASGARI & A. RAGHURAM

2. Some preliminaries
In this section we collect some results that we repeatedly use in later sections. To begin, we recall a theorem due to Jacquet and Shalika concerning strong multiplicity one
for isobaric automorphic representations. Then we recall an analytic criterion in terms of
Rankin–Selberg L-functions, due to Jacquet, Piatetski-Shapiro and Shalika, that characterizes when two cuspidal automorphic representations are equivalent. Next, we note thatthe natural transfer of automorphic representations of a quasi-split non-split general spingroup GSpin∗ (4) to GL(4) is in fact the Asai transfer. Finally, we recall some details fromthe Langlands–Shahidi machinery that will be of use to us, particularly when the ambientgroup is GSpin(m) with m = 8, 10 or 12.
2.1. Some results of Jacquet, Piatetski-Shapiro, and Shalika. The following strong
multiplicity one theorem for isobaric representations is due to Jacquet and Shalika [11, 12].
Theorem 2.1. Let π1 and π2 be isobaric automorphic representations of GL(n, AF ). Let
S be a finite set of places of F , containing the archimedean places, such that both π1 andπ2 are unramified outside S. If π1,v ∼/ S, then π1 ∼= π2,v for all v ∈= π2 .
Another useful technical tool for us is the following theorem, due to Jacquet, PiatetskiShapiro and Shalika [10], concerning Rankin–Selberg L-functions.
Theorem 2.2. Let π1 and π2 be cuspidal automorphic representations of GL(n1 , AF ) andGL(n2 , AF ), respectively. Let S be a finite set of places containing the archimedean places of
F and the ramified places of π1 and π2 . The partial Rankin–Selberg L-function LS (s, π1 ×π2 )is holomorphic at s = 1 unless n1 = n2 and π2 ∼= π1∨ , and in which case it has a simple poleat s = 1.
2.2. The Asai transfer and the quasi-split non-split GSpin∗ (4). Let E/F be a quadratic extension of number fields and let Γ = ΓF denote the absolute Galois group of F .
In this section we let G denote the group GSpin∗ (4), a quasi-split non-split linear algebraic
group over F , which is isomorphic to the split GSpin(4) over E. The L-group of G can be
written as L G = GSO(4, C) o Γ, where the Galois action, which factors through Gal(E/F ),is described below. We note that GSO(4, C) denotes the special orthogonal similitude group;
one can identify it as a quotient of GL(2, C) × GL(2, C) given by
GSO(4, C) = β(GL(2, C) × GL(2, C)),
where β is the map on the right of the exact sequenceβ1 −→ C∗ −→ GL(2, C) × GL(2, C) −→ GO(4, C).
For details see [26, §2]. Furthermore, the Γ-action on GSO(4, C) is as follows. If γ ∈ Γ andg = β(g1 , g2 ) with gi ∈ GL(2, C), then(β(g1 , g2 ) if γ|E = 1,γ·g =β(g2 , g1 ) if γ|E 6= 1.
We also need to recall the Asai transfer. Consider the group H = ResE/F GL(2) as agroup over F . Its L-group is given byL

H = (GL(2, C) × GL(2, C)) o Γ,

CUSPIDALITY CRITERION

5

with the Galois action given by((g1 , g2 ) if γ|E = 1,
γ · (g1 , g2 ) =(g2 , g1 ) if γ|E 6= 1.

(2.3)

Let W be a 2-dimensional C-vector space, and let V = W ⊗ W . After fixing a basis for
W , we identify GL(W ) with GL(2, C). Consider the map(2.4)
As : (GL(W ) × GL(W )) o Γ −→ GL(V ) ∼= GL(4, C)
given, on pure tensors, by(
g1 ξ1 ⊗ g2 ξ2
As(g1 , g2 ; γ)(ξ1 ⊗ ξ2 ) =g1 ξ2 ⊗ g2 ξ1

if γ|E = 1,
if γ|E 6= 1,

for all ξi ∈ W and all gi ∈ GL(W ). It is straightforward to check that this map is indeed ahomomorphism. It is called the Asai (or ‘twisted tensor’) homomorphism. (Alternatively,
one could take the map satisfying As(g1 , g2 ; γ)(ξ1 ⊗ ξ2 ) = −g1 ξ2 ⊗ g2 ξ1 when γ|E 6= 1. This
choice would lead to a quadratic twist of the above map. )Further, let ι : L G −→ GL(V ) ∼= GL(4, C) be the map defined via(
g1 ξ1 ⊗ g2 ξ2 if γ|E = 1,
ι(β(g1 , g2 ); γ)(ξ1 ⊗ ξ2 ) =g1 ξ2 ⊗ g2 ξ1 if γ|E 6= 1.
Again it is straightforward to check that this map is an L-homomorphism. It is now clearthat ι ◦ (β, id) = As. In other words, the following diagram commutes:
GL(2, C) × GL(2, C) o Γ

SSS
SSS
SSS
S
As SSSS)(β,id)/ GSO(4, C) o Γ
nnn
nnn
n
n
n ι
wnnn

GL(4, C)
Assume that π is a cuspidal automorphic representation of GL(2, AE ) and let Π = As(π)
be its Asai transfer to GL(4, AF ). (See Krishnamurthy [19] or Ramakrishnan [26].) Then
Π = ι(β(π)), where β(π) denotes the transfer of π to the group GSpin∗ (4, AF ), and ι(β(π))
denotes the transfer of β(π) from GSpin∗ (4, AF ) to GL(4, AF ). The transfer corresponding
to β exists for formal reasons and the existence of the transfer corresponding to ι (for genericrepresentations) is part of a joint work of the first author with F. Shahidi [2, 3].
2.3. The Langlands–Shahidi L-functions. Let P = M N be a maximal proper parabolic
subgroup of a connected reductive quasi-split linear algebraic group G, where M denotes a
Levi subgroup and N denotes the unipotent radical of P . Let σ be a generic automorphic
representation of M (AF ). Let r denote the adjoint action of the complex Langlands dualc on the Lie algebra of the dual of N . Write r = r1 ⊕ · · · ⊕ rm , where the ri ’sgroup Mdenote the irreducible constituents of r and the ordering is according to the eigenvalue ofthe adjoint action as in, for example, [30, p.278]. The Langlands–Shahidi method thenconstructs the L-functions L(s, σ, ri ) for 1 ≤ i ≤ m.
We need the following cases of the Langlands–Shahidi method. Let G = GSpin(2n + 6)
with n = 1, 2, 3, and consider a maximal parabolic subgroup of G with Levi subgroup6

M. ASGARI & A. RAGHURAM

M = GL(n) × GSpin(6). (One could also work with split spin groups as in [13, §3]; however,
we find it more convenient to work with the similitude version of the groups.) The algebraic
group GSpin(6) is isomorphic to a quotient of GL(1) × Spin(6) by a central subgroupA = {1, (−1, c)}, where c is the nontrivial element in the center of Spin(6) of order 2; see [2,Proposition 2.2]. The algebraic group GL(4) is isomorphic to a quotient of GL(1) × SL(4)by a cyclic central subgroup B of order 4. We identify Spin(6) with SL(4) such that B
contains A. This way we get a natural map, defined over F , from GSpin(6) to GL(4), which
in turn induces a map(2.5)

f : M −→ GL(n) × GL(4).

Let Π be an irreducible cuspidal representation of GL(4, AF ) and let σ be an irreducible
cuspidal automorphic representation of GL(n, AF ), n = 1, 2, 3. Choose any irreducible
constituent Σ of σ ⊗ Π|f (M (AF )) and let Σ also denote the corresponding representation of
M (AF ). The Langlands–Shahidi method then gives(2.6)

L(s, Σ, r1 ) = L(s, σ ⊗ Π, ρn ⊗ ∧2 ρ4 ),where ρk denotes the standard representation of GL(k, C) and the L-function on the righthand side is a Langlands L-function; see [13, §3]. We record a general fact that we needfrom the Langlands–Shahidi method.
Proposition 2.7. Let wG and wM denote the longest elements of the Weyl group of G and
M , respectively. Let w0 = wG wM . If w0 (Σ) ∼6 Σ, then L(s, Σ, r1 ) is entire.
=
Proof. This is a standard fact in the Langlands–Shahidi method. For example, see the proofof [13, Proposition 3.4].
In order to apply the above proposition one needs to know the action of w0 on a representation of M (AF ).
Proposition 2.8. Let G = GSpin(2n + 6) with n a positive integer and let σ ⊗ Π be a
representation of M (AF ) as above. Moreover, let w0 be as above and denote its image
under the map (2.5) by w0 again. Then we have(σ ∨ ⊗ (Π∨ ⊗ ωσ ) if n is odd,
w0 (σ ⊗ Π) =σ ∨ ⊗ (Π ⊗ ωσ )if n is even.
Here, ωσ denotes the central character of σ.
Proof. Recall that the nontrivial automorphism of the Dynkin diagram of type Am corresponds to an outer automorphism of GL(m + 1) and it conjugates an irreducible representation to its dual representation. The proof of the proposition will follow from a descriptionof how w0 acts on the root system of type Dr .
We use the Bourbaki notation for the simple roots:
α1 = 1 − 2 , · · · , αr−2 = r−2 − r−1 , αr−1 = r−1 − r , αr = r−1 + r .
The Weyl group of G is isomorphic with {±1}r−1 o Sr which we identify with a subgroupof signed r × r permutation matrices acting on the space Rr = R1 ⊕ · · · ⊕ Rr . With this

CUSPIDALITY CRITERION

7

identification observe that

wG =


−Ir





if r is even,


−Ir−1




!
1

if r is odd.

Let r = n+3 and let M be a maximal Levi subgroup of type An−1 ×A3 in G = GSpin(2r).
The simple roots in the A3 -factor are αr−1 , αr−2 , αr . The proposition follows by observingthat
w0 (αr−1 ) = wG wM (αr−1 ) = wG (−αr )(r−1 + r = αr
= wG (−r−1 − r ) =r−1 − r = αr−1

if r is even,
if r is odd,

w0 (αr ) = wG wM (αr ) = wG (−αr−1 )(r−1 − r = αr−1
= wG (−r−1 + r ) =r−1 + r = αr

if r is even,
if r is odd,

and

while
w0 (αr−2 ) = wG wM (αr−2 ) = wG (−αr−2 ) = wG (−r−2 − r−1 ) = αr−2
in either case. Moreover, for 1 ≤ j ≤ n − 1 we havew0 (αj ) = wG wM (αj ) = wG (−αn−j ) = wG (−n−j + n−j+1 ) = αn−j .
This means that w0 induces the nontrivial automorphism of the Dynkin diagram of theAn−1 -factor of M , and on the A3 -factor it induces the nontrivial automorphism of the
Dynkin diagram if and only if r is even.
Let m = m(g, h) be an arbitrary element in the Levi subgroup M identified with GL(n)×GSpin(6), in G = GSpin(2n + 6), where g ∈ GL(n) and h ∈ GSpin(6), and let ν = ν(m)denote its similitude character value. Thenw0 m(g, h)w0−1 = m(t g−1

ν(m), h∗ ),

where
(
∗

h =

t h−1

h

if r is even,
if r is odd.

We conclude thatw0 (σ ⊗ Π)(m(g, h)) = (σ ⊗ Π)(m(t g−1

ν(m), h∗ ))= σ(t g −1 )Π(h∗ )ωσ (ν(m))= σ ∨ (g)Π∗ (h)ωσ (ν(h))
= σ ∨ ⊗ (Π∗ ⊗ ωσ ) (m(g, h)),8

M. ASGARI & A. RAGHURAM

where

(
Π∨
Π∗ =
Π

if r is even,
if r is odd.

Note that r = n + 3 is even if and only if n is odd. This completes the proof.



3. The proof of (ii)⇒(i)
We verify that for each of (ii)(a) through (ii)(d) the exterior square transfer ∧2 Π is not
cuspidal. Indeed, it is not difficult to write down the isobaric decomposition for ∧2 Π ineach case.
3.1. (ii)(a)⇒(i).
Proposition 3.1. Let π1 and π2 be cuspidal automorphic representations of GL(2, AF ).
Let π1  π2 be the transfer to an automorphic representation of GL(4, AF ), whose existence
was established in [24]. For brevity, we let Π = π1  π2 and ω = ωπ1 ωπ2 . We have(a) ∧2 (π1  π2 ) = Sym2 (π1 ) ⊗ ωπ2  Sym2 (π2 ) ⊗ ωπ1 .
(b) Assuming Langlands functoriality one should expect
Sym2 (π1  π2 ) = Sym2 (π1 )  Sym2 (π2 )  ωπ1 ωπ2 .
(c) The partial L-function LS (s, ∧2 (Π) ⊗ ω −1 ) is entire while the partial L-function
LS (s, Π, Sym2 ⊗ ω −1 ) has a pole at s = 1.
Proof. The proof of (a) and (b), using Theorem 2.1, is an easy calculation using Satakeparameters on both sides. More precisely, for a finite place v at which both π1 and π2
are unramified we let π1,v and π2,v have Frobenius-Hecke eigenvalues t1 = diag(a1 , b1 ) and
t2 = diag(a2 , b2 ), respectively. Then
∧2 (t1 ⊗ t2 ) = diag(a21 , a1 b1 , b21 ) · a2 b2  diag(a22 , a2 b2 , b22 ) · a1 b1
and
Sym2 (t1 ⊗ t2 ) = diag(a21 , a1 b1 , b21 ) ⊗ diag(a22 , a2 b2 , b22 )  (a1 b1 · a2 b2 ) .
Part (a) has also been observed by others; see [19, (7.27)] and [28, (2.6)], for example.
For (b) to make sense one has to assume the symmetric square transfer from GL(4) toGL(10) and the automorphic tensor product from GL(3) × GL(3) to GL(9), both particularinstances of functoriality.
To prove (c), observe that Π∨ ∼= Π ⊗ ω −1 , which implies
LS (s, Π × Π∨ ) = LS (s, ∧2 (Π) ⊗ ω −1 )LS (s, Π, Sym2 ⊗ ω −1 ),where S is a finite set of places including all the archimedean ones such that Π is unramified
outside of S. From (a) we have ∧2 (Π)⊗ω −1 = Ad(π1 )Ad(π2 ). (Here Ad(πi ) = Sym2 (πi )⊗ωπ−1.) If πi is not dihedral, then Ad(πi ) is cuspidal (by Gelbart-Jacquet [7]) and hence its
ipartial L-function is entire. If πi is dihedral, say πi = IEF (χ), then it is easy to see that
Ad(πi ) = ωE/F  IEF (χ0 χ−1 ), where ωE/F is the quadratic character of F associated to E
by class field theory, and χ0 is the nontrivial Gal(E/F )-conjugate of χ. Since πi is cuspidal,
the inducing character χ is Galois regular, i.e., χ0 6= χ, or equivalently χ0 χ−1 is a nontrivial
character, whence LS (s, Ad(πi )) = LS (s, ωE/F )LS (s, IEF (χ0 χ−1 )) is entire. (In particular, itCUSPIDALITY CRITERION

9

does not have a pole at s = 1.) Therefore LS (s, ∧2 (Π)⊗ω −1 ) = LS (s, Ad(π1 ))LS (s, Ad(π2 ))does not have a pole at s = 1. However, LS (s, Π × Π∨ ) has a pole at s = 1, which implies
that LS (s, Π, Sym2 ⊗ ω −1 ) has a pole at s = 1.
Note that (c), unlike (b), is unconditional and does not depend on assuming unproveninstances of functoriality.

3.2. (ii)(b)⇒(i).
Proposition 3.2. Let E/F be a quadratic extension. Let π be a cuspidal automorphic
representation of GL(2, AE ) and let Π = As(π) be its Asai transfer. Assume that Π is a
cuspidal automorphic representation of GL(4, AF ). Then ∧2 Π is cuspidal if and only if π
is not dihedral.
Proof. The proof depends on the following identity20∧2 (As(π)) = IE
F (Sym π ⊗ ωπ ),where 0 means the nontrivial Gal(E/F )-conjugate. (See [19, §7].) To begin, assume that π is
not dihedral. By [26, Theorem 1.4] we know that As(π) is cuspidal if and only if π 0 ∼6 π⊗µ=for any µ. If ∧2 Π is not cuspidal, then0
Sym2 π ⊗ ωπ0 ∼= Sym2 π ⊗ ωπ0 .
∼ Sym2 π ⊗ ω 0 , i.e., Ad(π) =∼ Ad(π 0 ). This, in turn, implies
This implies that Sym2 π 0 ⊗ ωπ =πthat π 0 ∼= π ⊗ µ (by [24, Theorem 4.1.2]), contradicting the fact that there is no such twist.
Hence ∧2 Π is cuspidal.
Next, assume that π is dihedral. In this case Sym2 (π) is not cuspidal and therefore,E
IF (Sym2 π ⊗ ωπ0 ) cannot possibly be cuspidal.
3.3. (ii)(c)⇒(i).
Proposition 3.3. Let Π be a cuspidal automorphic representation of GL(4, AF ) and assume
that Π is a transfer from a cuspidal (generic) automorphic representation π of GSp(4, AF ).
Then∧2 Π = r̃5 (π)  ωπ ,where r̃5 is a degree 5 representation of GSp(4, C) defined below. In particular, ∧2 Π is not
cuspidal.
Proof. We use Kim [14, p. 2793]. As observed there, one has∧2ιSp(4, C) ,→ GL(4, C) −→ GL(6, C)and ∧2 ◦ ι = r5 ⊕ 11 decomposes into a direct sum of the trivial representation and afive-dimensional representation r5 . Similarly,ι̃∧2

GSp(4, C) ,→ GL(4, C) −→ GL(6, C)and ∧2 ◦ ι̃ = r̃5 ⊕ ν, where ν is the similitude character of GSp(4, C) and r̃5 is a fivedimensional representation of GSp(4, C). This implies the desired equality of automorphic
representations.
10

M. ASGARI & A. RAGHURAM

Remark 3.4. Embedded in the above proof is the assertion that a cuspidal (generic) automorphic representation π of GSp(4, AF ) admits a transfer to an automorphic representationr̃5 (π) of GL5 (AF ) corresponding to the representation r̃5 . This depends on the generic transfer from GSp(4) to GL(4) (see Asgari-Shahidi [4]), and the exterior square transfer from
GL(4) to GL(6) due to Kim [13].
3.4. (ii)(d)⇒(i).
Proposition 3.5. Let π be a cuspidal automorphic representation of GL(2, AE ), where
E/F is a quadratic extension, and let Π = IE
F (π) be the automorphic induction of π to an
automorphic representation of GL(4, AF ). Then ∧2 Π is not cuspidal.
Proof. It is known thatF
∧2 (IEF (π)) = As(π) ⊗ ωE/F  IE (ωπ ),where ωE/F is the quadratic Hecke character of F associated to E/F by class field theory.
See, for example, Kim [16, §3].

4. The proof of (i)⇒(iii)
It is equivalent to prove that if Π is a cuspidal automorphic representation of GL(4, AF )which neither has a nontrivial self-twist nor is essentially self-dual, then ∧2 (Π) is cuspidal.
Observe that if an isobaric automorphic representation ρ of GL(6, AF ) is not cuspidal,
then it must have an isobaric summand of degree 1, 2, or 3, i.e., there exists a cuspidal
representation σ of GL(n, AF ), with 1 ≤ n ≤ 3, such that LS (s, ρ × σ) has a pole at s = 1.
Again, S denotes a finite set of places of F , including all the archimedean ones, such that
all the representations involved are unramified at places outside S. Now with Π as above,the cuspidality of ∧2 Π follows from the following two propositions.
Proposition 4.1. If Π ∼6 Π∨ ⊗ χ for all χ, then LS (s, Π ⊗ σ, ∧2 ⊗ ρ2 ) is holomorphic at=
s = 1 for every cuspidal representation σ of GL(n, AF ), n = 1, 3.
Proof. Let Σ be as in (2.5). By Proposition 2.7, it is enough to show that w0 (Σ) ∼6 Σ. If=
we have w0 (Σ) ∼= Σ, then w0 (σ ⊗ Π) ∼= σ ⊗ Π. On the other hand, by Proposition 2.8 wehave w0 (σ ⊗ Π) ∼= σ ∨ ⊗ (Π∨ ⊗ ωσ ). In particular, we must have Π ∼= Π∨ ⊗ ωσ contradicting
the hypothesis.
S
2
Proposition 4.2. If Π 6∼= Π⊗χ for all nontrivial χ, then L (s, Π⊗σ, ∧ ⊗ρ2 ) is holomorphic
at s = 1 for every cuspidal representation σ of GL(2, AF ).
Proof. The same argument as in the above proof works as long as ωσ 6= 11 because by
Proposition 2.8 we have w0 (σ ⊗ Π) ∼6 σ ⊗ Π. This means that if σ is a
= σ ∨ ⊗ (Π ⊗ ωσ ) ∼=cuspidal representation of GL(2, AF ) with nontrivial central character, then σ cannot occur
as an isobaric summand of ∧2 (Π).
Now suppose that σ is a cuspidal representation of GL(2, AF ) with trivial central character and that σ is an isobaric summand of ∧2 (Π). Then the representation σ ⊗ θ2 occursin ∧2 (Π ⊗ θ) for any Hecke character θ. Note that Π ⊗ θ also satisfies the hypothesis thatit has no nontrivial self-twists. Choose θ such thatωσ⊗θ2 = ωσ θ4 = θ4 6= 11

CUSPIDALITY CRITERION

to get a contradiction.

11
5. The proof of (iii)⇒(ii)

We prove that if Π satisfies (iii)(β), then it is of the form (ii)(d), and if it satisfies(iii)(α), then it is one of (ii)(a)–(c).
5.1. (iii)(β) =⇒ (ii)(d). Assume that
(5.1)

Π∼
=Π⊗χ

for some nontrivial χ. Taking central characters we have χ4 = 11. If χ2 6= 11, then we
may replace χ with χ2 in (5.1), which means we may assume that the character χ in(5.1) is quadratic. We want to show that Π is induced from a quadratic extension. If Π
is a representation of GL(2), then the analogous statement is a well-known result due toLabesse-Langlands [20]. In our case it follows from the work of Arthur-Clozel [1] and someL-function arguments as we explain below.
Lemma 5.2. Let Π be a cuspidal representation of GL(2n, AF ) satisfying Π ∼= Π ⊗ χ forF
a nontrivial quadratic character χ. Then Π = IE (π), where E/F is the quadratic extension
associated with χ and π is a cuspidal representation of GL(n, AE ).
Proof. We first claim that the base change ΠE is not cuspidal. To see this, assume that it
is cuspidal. For a finite set S of places of F and the corresponding set T of places of E
lying above those in S, as before, we have
(5.3)

S
∨
S
∨
LT (s, ΠE × Π∨E ) = L (s, Π × Π )L (s, Π × Π ⊗ χ)

= LS (s, Π × Π∨ )2 .
For sufficiently large S, the left hand side of (5.3) has a simple pole at s = 1 while the righthand side has a double pole at s = 1. This contradiction shows that ΠE is not cuspidal.
This means that ΠE = π1 π2 , where πi are cuspidal representations of GL(n, AE ). Further,
π1 6∼= π2 , because if they are equivalent, thenT
∨ 4
LS (s, Π × Π∨ )2 = LT (s, ΠE × Π∨E ) = L (s, π1 × π1 ) ,but LS (s, Π × Π∨ )2 has a double pole and LT (s, π1 × π1∨ )4 has a pole of order 4 at s = 1.
Next, we claim that Π ∼= IFE (π1 ). To show this it is enough to prove that the partial
L-function LS (s, IFE (π1 ) × Π∨ ) has a simple pole at s = 1. This follows fromLS (s, IFE (π1 ) × Π∨ ) = LS (s, IFE (π1 × Π∨E ))= LT (s, π1 × Π∨E)
= LT (s, π1 × π1∨ )LT (s, π1 × π2∨ ).
Since π2 ∼6 π1 we know that LT (s, π1 × π1∨ )LT (s, π1 × π2∨ ) has a simple pole at s = 1.
=



12

M. ASGARI & A. RAGHURAM

5.2. (iii)(α) =⇒ (ii)(a)–(c). Now assume that
(5.4)

Π∼
= Π∨ ⊗ χ

for some χ. For a finite set S of places of F , as before, we have
LS (s, Π × Π∨ ) = LS (s, Π × (Π ⊗ χ−1 ))= LS (s, (Π × Π) ⊗ χ−1 )= LS (s, Π, ∧2 ⊗ χ−1 )LS (s, Π, Sym2 ⊗ χ−1 ).
The last two L-functions are the standard twisted exterior square and twisted symmetricsquare L-functions of Π. If S is a sufficiently large set, then LS (s, Π × Π∨ ) has a simple
pole at s = 1. Therefore one and exactly one of the partial twisted exterior or symmetricsquare L-functions has a simple pole at s = 1.
First, assume that LS (s, Π, ∧2 ⊗ χ−1 ) has a pole at s = 1. Then there exists a cuspidal
representation π, which may be taken to be globally generic, of GSp(4, AF ) such that Π isthe functorial transfer of Π, i.e., Π is a representation as in (ii)(c). This result has been
known for a long time and, we believe, is originally due to Jacquet, Piatetski-Shapiro, andShalika. See Gan-Takeda [6] for a proof. It would also follow from the more general methodof “descent” as we explain below.
Next, assume that LS (s, Π, Sym2 ⊗ χ−1 ) has a pole at s = 1. Taking central characters−1 4
χ . In other words, µ = ωΠ χ−2 is a quadratic character. If µ isin (5.4) we have ωΠ = ωΠtrivial, then Π is a transfer from a cuspidal representation π of GSpin(4), a split connected
reductive group of type D4 whose derived group is Spin(4). If µ is nontrivial, then Π isa functorial transfer from the quasi-split non-split group GSpin∗ (4) associated with thequadratic extension E/F attached to µ. These facts can be proved using the “descent”
method of Ginzburg-Rallis-Soudry. If χ is trivial, then Π would be a transfer from a special
orthogonal or symplectic group. We refer to Ginzburg-Rallis-Soudry [8, Theorem A] andSoudry [31, Theorem 4 and 12] for the proofs for classical groups, and Hundley-Sayag [9]for the case of GSpin groups.
With the above notation, if µ is trivial, then Π is as in (ii)(a), and if it is nontrivial then
Π is as in (ii)(b).

6. Examples and Complements
In this section we give a few examples of our main result. In some of them the proposed
representation Π of GL(4, AF ) is not yet proved to be automorphic, but it is conjecturally so.
We also comment on possible intersection among the four cases in part (ii) of Theorem 1.1.
Finally, we present a theorem due to A. Kable on when the exterior square of an irreducible
four-dimensional representation is reducible.

CUSPIDALITY CRITERION

13

6.1. K. Martin’s G192 . The matrices

−1

−1
a=

1



,



−1

1
b=

−1



,


1




1
,



d=
1

−1


c=
−i

1




−1
−1

−i




1

in GL(4, C) generate a group G192 of order 192. Let ρ be the four-dimensional representation of the group G192 given by inclusion. Then ρ is an irreducible representation. In [21]K. Martin showed that ρ is modular, i.e., there exists a (cuspidal) automorphic representation Π(ρ) that corresponds to ρ.
Example 6.1. Let Π1 = Π(ρ). Then Π1 is a cuspidal automorphic representation ofGL(4, AF ). Moreover, it is neither essentially self-dual nor does it have a nontrivial selftwist. It is not on the list of possibilities of (ii). Furthermore, ∧2 Π1 is cuspidal. In otherwords, Π1 is an example of a cuspidal representation which does not satisfy any of (i)–(iii)of Theorem 1.1.
Proof. First we check that ∧2 (ρ) is an irreducible representation. To see this consider the
standard basis he1 , e2 , e3 , e4 i for C4 and fix the ordered basis hw1 , w2 , . . . , w6 i of C6 = ∧2 C4
given by
w1 = e1 ⊗ e2 − e2 ⊗ e1 , w2 = e1 ⊗ e3 − e3 ⊗ e1 , w3 = e1 ⊗ e4 − e4 ⊗ e1 ,w4 = e2 ⊗ e3 − e3 ⊗ e2 , w5 = e2 ⊗ e4 − e4 ⊗ e2 , w6 = e3 ⊗ e4 − e4 ⊗ e3 .
Let A, B, C, D be the images of a, b, c, d under ∧2 , respectively. Then, with respect to theabove basis, we have



−1


1




−1
, B = 


−1




1


1
 −1


−1
A=

−1


−1
1

 −i


−i
C=

i


i
1

−1








,




1


D=
 1



1





,



−1





−1
.


−1
1

14

M. ASGARI & A. RAGHURAM

It is easy to check that if a 6 × 6 matrix X commutes with A, B, and C, then it has to beof the form


a
b


c




e
f
.
X=


−f e



d 
−b
a
Further, if XD = DX, then a = c = d = e and b = f = 0, i.e., X is a scalar matrix. Therefore, HomG192 (∧2 ρ, ∧2 ρ) = C and, by Schur’s lemma, the representation ∧2 ρ is irreducible.
This implies that ∧2 Π1 is cuspidal. (This is because if a complex Galois representation σ
is modular, i.e., corresponds to an automorphic representation π = π(σ), then σ is irreducible if and only if π is cuspidal. This fact follows from L(s, σ ⊗ σ ∨ ) = L(s, π × π ∨ ); see
Ramakrishnan [27, Introduction].) Martin observes that ρ, and hence Π1 , is not essentiallyself-dual. Clearly Π1 is not on the list of possibilities in (ii) of the main theorem because
if it were, then ∧2 Π1 would not be cuspidal (see Section 3). Hence Π1 does not satisfy anyof the equivalent statements of Theorem 1.1.
In [21] Martin considers a four-dimensional irreducible representation ρ of the absolute
Galois group of Q whose image in PGL(4, C), denoted Ḡ, is an extension of A4 by V4 . In thissituation Ḡ is either V4 o A4 or V4 · A4 . In the former case ρ is of GO(4)-type and Π = Π(ρ)
is a transfer from GL(2, AF ) × GL(2, AF ), which is contained in our (ii)(a). The example
of G192 is an instance of the latter situation. In either case, Π(ρ) may also be thought ofas being obtained by automorphic induction across a non-normal quartic extension with noquadratic subextension.
˜
6.2. The standard representation of S5 . Consider a tower of number fields J/J/E/F.
˜
Here, E/F and J/J
are quadratic extensions, J/E is an A5 -extension, J/F is an S5 ˜ is an SL(2, F5 )-extension, and J/F
˜ is Galois. (Recall that A5 ∼extension, J/E= PSL(2, F5 ).)In what follows we identify A5 , S5 and SL(2, F5 ) with these Galois groups.
Let σ be the standard four-dimensional irreducible representation of S5 ,σ : S5 −→ GL(4, C).
Here are some properties of σ:(1) ∧2 σ is a six-dimensional irreducible representation of S5 (see [5, §3.2]).
(2) σ|A5 is irreducible (because σ 6∼
= σ ⊗ , where  is the sign character of S5 ).
22(3) ∧ (σ|A5 ) = (∧ σ)|A5 is reducible (because ∧2 (σ) ∼= ∧2 (σ) ⊗ ), and its irreducible
constituents are both of degree 3.
(4) σ is self-dual (because the character of σ has integer values).
(5) σ|A5 = ρ1 ⊗ ρ2 , where ρ1 and ρ2 are the two-dimensional irreducible representationsof SL(2, F5 ) (see [15, Lemma 5.1]).
We need some details about the ρi . Let ρ̃ be the unique (up to twists) cuspidal representationof GL(2, F5 ) whose restriction to SL(2, F5 ) is reducible. In this case, ρ̃|SL(2,F5 ) = ρ1 ⊕ ρ2 . If
g ∈ GL(2, F5 ) − Z(F5 )SL(2, F5 ), then ρg1 = ρ2 . Here Z is the center of GL(2). Conjugating
SL(2, F5 ) by such an element g induces the nontrivial outer automorphism of SL(2, F5 )

CUSPIDALITY CRITERION

15

because if it were an inner automorphism, then we would have ρ1 ∼= ρ2 , which contradicts
the fact that the restriction from GL2 to SL2 is multiplicity free.
Let ρ denote either ρ1 or ρ2 . In constructing examples (to illustrate our main theorem),
we make the following assumption: ρ is modular, i.e., there exists a cuspidal automorphic
representation π(ρ) of GL(2, AE ) with ρ ↔ π(ρ). In this situation, it is expected [17] that
there exists an automorphic representation π(σ) of GL(4, AF ) with π(σ) ↔ σ and π(σ) is
the Asai transfer of π(ρ), i.e., π(σ) = As (π(ρ)).
Example 6.2. Let Π2 = π(σ) = As(π(ρ)). Then Π2 is a cuspidal automorphic representation of GL(4, AF ) which is self-dual. However, it is the Asai transfer of a nondihedral
cuspidal representation. Moreover, it has no nontrivial self-twists and it is not on the listof possibilities in (ii). Furthermore, its exterior square transfer ∧2 Π2 is cuspidal. In otherwords, Π2 is an example of a cuspidal representation which does not satisfy any of (i)–(iii)of Theorem 1.1.
Proof. Since Π2 = π(σ) and σ is irreducible, we conclude that Π2 is cuspidal. (Cuspidalityof Π2 may also be seen by appealing to the cuspidality criterion for the Asai transfer due
to Ramakrishnan [26, Theorem 1.4].)
Next, we note that Π2 is self-dual and is the Asai transfer of a cuspidal representation,
namely π(ρ). Note that π(ρ) is not dihedral as ρ is not induced from a character of an index
two subgroup because there is no such subgroup in SL(2, F5 ). Also, Π2 has no nontrivial
self-twists because σ has no nontrivial self-twists.
Finally, note that ∧2 Π2 is cuspidal since ∧2 Π2 = ∧2 π(σ) = π(∧2 σ) and ∧2 σ is an irreducible representation implying that π(∧2 σ) is cuspidal. (See, for example, Ramakrishnan[27, Introduction].)
Example 6.3. Let Π3 = (Π2 )E be the base change of Π2 to an automorphic representationof GL(4, AE ). Then Π3 is a cuspidal automorphic representation of GL(4, AE ) which is selfdual and is not the Asai transfer of a nondihedral representation. It is contained in (ii)(a)and its exterior square transfer ∧2 Π3 is not cuspidal. In other words, Π3 is an example ofa cuspidal representation which satisfies (i)–(iii) of Theorem 1.1.
Proof. To see cuspidality of Π3 , as well as the fact that it is contained in (ii)(a), note that
Π3 = (Π2 )E = π(σ)E = π(σ|A5 ) = π(ρ1 ⊗ ρ2 ) = π(ρ1 )  π(ρ2 ).
Neither ρi is monomial since SL(2, F5 ) does not have an index two subgroup. Applying
the cuspidality criterion for π(ρ1 )  π(ρ2 ) due to Ramakrishnan [25, Theorem 11.1], we
see that Π3 is not cuspidal if and only if π(ρ1 ) ∼= π(ρ2 ) ⊗ µ for some Hecke character µof E. On the other hand, π(ρ2 ) ⊗ µ = π(ρ2 ⊗ µ), where we identify the Hecke characterµ with a character of the absolute Galois group of E via global class field theory. Hence,
we have π(ρ1 ) = π(ρ2 ⊗ µ). This implies that ρ1 ∼= ρ2 ⊗ µ (since, for any two Galois
representations τ1 and τ2 , one has π(τ1 ) ∼= π(τ2 ) if and only if τ1 ∼= τ2 ; one can see this byconsidering the equality LS (s, π(τ1 ) × π(τ2 )∨ ) = LS (s, τ1 ⊗ τ2∨ )). Therefore, µ is a character˜of Gal(J/E)= SL(2, F5 ), a perfect group, hence µ is trivial. Whence ρ1 = ρ2 , which
contradicts the fact that they are inequivalent as was observed earlier.
Next, observe that ∧2 Π3 is not cuspidal because∧2 Π3 = ∧2 (π(ρ1 )  π(ρ2 )) = (Sym2 (π(ρ1 )) ⊗ ωπ(ρ2 ) ) ⊕ (Sym2 (π(ρ2 )) ⊗ ωπ(ρ1 ) ),

16

M. ASGARI & A. RAGHURAM

which is of isobaric type (3, 3). (See Proposition 3.1.)
Finally, we observe that Π3 is self-dual because σ, and hence σ|A5 , is self-dual and that
Π3 could not be an Asai transfer of a nondihedral representation because if it were, then
∧2 (Π3 ) would be cuspidal by Proposition 3.2.

6.3. On possible intersections between representations in (ii). The purpose of thissubsection is to show that the cases (ii)(a) through (ii)(d) are not mutually exclusive.
Example 6.4. Let π = IFE (χ) be a cuspidal automorphic representation of GL(2, AF ) which
is automorphically induced from a Hecke character χ of E, where E/F is a quadratic
extension. Let τ be a nondihedral cuspidal automorphic representation of GL(2, AF ). Let
Π4 = π  τ . Then Π4 is a representation that is common to (ii)(a), (ii)(c) and (ii)(d).
Proof. From Ramakrishnan’s cuspidality criterion [25, Theorem 11.1] we know that Π4 is a
cuspidal representation of GL(4, AF ). By construction, Π4 is in (ii)(a).
We observe that
Π4 = IFE (χ)  τ = IFE (χ ⊗ τE ).
Since the induced representation IFE (χ ⊗ τE ) is cuspidal, the inducing representation χ ⊗ τE
is, a fortiori, cuspidal. Hence Π4 is in (ii)(d).
Now we claim that Π is also a transfer from a (generic) cuspidal representation ofGSp(4, AF ). To see this we recall the following well known identities:
Sym2 (IFE (χ)) = IFE (χ2 )  χ|A× ,F∧2 (IFE (χ)) = χ|A× · ωE/F ,F

where ωE/F is the quadratic Hecke character of F associated to E/F by class field theory.
In particular, the central character of π is given by ωπ = χ|A× · ωE/F . From Proposition 3.1
F
we have∧2 (π  τ ) = Sym2 (π) ⊗ ωτ  Sym2 (τ ) ⊗ ωπ .
For brevity write ω = ωπ ωτ . We deduce that
∧2 (Π4 ) = IFE (χ2 ) ⊗ ωτ  ωωE/F  Sym2 (τ ) ⊗ ωπ .
Hence, the partial L-function LS (s, Π, ∧2 ⊗ (ωωE/F )−1 ) has a pole at s = 1. Applying a
recent result of Gan and Takeda [6] we conclude that Π is a transfer from GSp(4), i.e., Π isin (ii)(c).

6.4. A calculation on the Galois side. As mentioned in the introduction, one may ask
for an irreducibility criterion on the Galois side, i.e., for `-adic Galois representations orcomplex representations of the Langlands group LF , which reflects the cuspidality criterion
one is looking for. In this section we present such a theorem due to A. Kable. We are
grateful to him for the permission to include this material here. Theorem 6.5 below is theanalogue of the equivalence of (i) and (iii) in Theorem 1.1. We begin by reviewing somepreliminaries.
Let k be an algebraically closed field whose characteristic is not two and let V be afour-dimensional k-vector space. Fix a nonzero element η ∈ ∧4 V . There is a nondegenerate
symmetric bilinear form B on ∧2 V defined by ω1 ∧ ω2 = B(ω1 , ω2 )η for all ω1 , ω2 ∈ ∧2 V .

CUSPIDALITY CRITERION

17

The bilinear space (∧2 V, B) is isomorphic to the orthogonal sum of three hyperbolic planes.
Let GO(B) be the group of similitudes of B, λ : GO(B) → k × the similitude character, andGSO(B) the subgroup of proper similitudes. This subgroup consists of those T ∈ GO(B)such that det(T ) = λ(T )3 , and it coincides with the connected component of the identity
in the algebraic group GO(B). (In the literature, the group GSO(B) is also denoted by
SGO(B) or GO+ (B).) We use similar notation also for the similitude groups of forms on
V itself. Let ρ : GL(V ) → GL(∧2 V ) be the homomorphism ρ(S) = ∧2 S.
Let G be a group and let σ be an irreducible representation of G on V . Recall that σ isessentially self-dual if there is a character χ of G such that σ ∨ ∼= χ ⊗ σ. In this case, χ−1 isa subrepresentation of σ ⊗ σ. We say that σ has symplectic type if χ−1 occurs in ∧2 σ andorthogonal type if χ−1 occurs in Sym2 σ. If σ is essentially self-dual of orthogonal type, thenthere is a nonzero symmetric bilinear form C on V such that G acts on V by similitudesof C. The kernel of C is a G-invariant proper subspace of V and hence trivial. Thus, C
is nondegenerate and σ(G) ⊂ GO(C). If σ(G) ⊂ GSO(C), then we say that σ is of proper
orthogonal type; otherwise, we say that σ is of improper orthogonal type. Finally, we saythat σ has a nontrivial quadratic self-twist if there is a nontrivial {±1}-valued character χof G such that σ ∼= χ ⊗ σ.
Theorem 6.5 (A. Kable). Let σ be an irreducible 4-dimensional representation of a group
G over an algebraically closed field whose characteristic is not two. Then the following two
conditions on σ are equivalent:(1) ∧2 σ is reducible.
(2) σ satisfies at least one of the following:(a) is essentially self-dual of symplectic type,(b) has a nontrivial quadratic self-twist, or(c) is essentially self-dual of proper orthogonal type.
Toward the proof of the above theorem, we begin with a lemma.
Lemma 6.6. The bilinear space (∧2 V, B) has the following properties.
(1) The image of ρ is GSO(B).
(2) An isotropic line in ∧2 V has the form ∧2 Q, where Q < V is a uniquely determined
2-dimensional subspace of V .
(3) An isotropic 3-space in ∧2 V either has the form L ∧ V , where L < V is a uniquelydetermined line, or the form ∧2 U , where U < V is a uniquely determined 3-space.
(4) Let W < ∧2 V be a 3-space on which B is nondegenerate. Then there is a nondegenerate symmetric bilinear form C on V such that{g ∈ GL(V ) | ρ(g)(W ) = W } = GSO(C).
The form C is determined by W up to scalars. Every nondegenerate symmetricbilinear form on V occurs in this way for a suitable choice of W .
Proof. We omit the proofs of (1), (2), and (3), as they are easy exercises, and briefly sketch
the proof of (4). Let Q be the quadric hypersurface in P(∧2 V ) consisting of null vectors
for B. By (2), we may identify Q with the Grassmannian of lines in P(V ). Let W be a
3-dimensional subspace of ∧2 V on which B is nondegenerate, and Y = P(W ) ∩ Q be the18

M. ASGARI & A. RAGHURAM

smooth plane conic defined by B|W . Let T be the subvariety of P(V ) obtained by taking
the union of the lines in P(V ) corresponding to points of Y . It is easily verified that T isa smooth quadric hypersurface, and so there is a nondegenerate symmetric bilinear form C
on V , unique up to scalars, such that T has equation C(v, v) = 0. An element g ∈ GL(V )preserves T together with the ruling T → Y sending a line in T to the corresponding pointof Y if and only if g ∈ GSO(C). From the construction, the set of g with this property is
the same as the set of all g such that ρ(g)(W ) = W . The last claim in (4) follows from thefact that GL(V ) acts transitively on the set of all nondegenerate symmetric bilinear forms
on V .

Proposition 6.7. The representation (σ, V ) is essentially self-dual of symplectic type if
and only if ∧2 V contains a G-invariant line.
Proof. This follows immediately from the definitions.



Lemma 6.8. There is no G-invariant isotropic 3-space in ∧2 V .
Proof. If there is such a 3-space, then, by Lemma 6.6, it is of the form L ∧ V or of the form
∧2 U . Note that G-invariance of the 3-space combined with the uniqueness statements inLemma 6.6 imply that L or U is also G-invariant, which contradicts irreducibility of σ. The proof of the following proposition would be substantially simpler if the action of Gon ∧2 V were completely reducible. However, in the current generality, this need not be
true.
Proposition 6.9. Suppose that σ is not essentially self-dual of symplectic type. Then σ
has a nontrivial quadratic self-twist if and only if ∧2 V contains a G-invariant 2-space.
Proof. Suppose first that σ has a nontrivial quadratic self-twist, say by the character χ.
Let H be the kernel of χ and recall that, by Clifford theory, σ|H is the sum of two 2dimensional subrepresentations. Let W < V be the H-invariant 2-space on which one ofthese subrepresentations is realized. Then it is easy to see that the G-translates of ∧2 W
span a G-invariant 2-space in ∧2 V . (The reader should compare this with the proof of
Proposition 3.5. Indeed, the 2-dimensional G-invariant subspace is the induction to G ofthe determinant character of the representation of H on W . Recall that if σ is a Galois
representation that corresponds to an automorphic representation π, then the determinantcharacter of σ corresponds to the central character of π.)
Now suppose that ∧2 V contains a G-invariant 2-space P . The kernel of B|P is G-invariant
and thus is either {0} or P , for the first hypothesis implies that there can be no G-invariant
line in ∧2 V . Suppose that the kernel is P . Then the 4-space P ⊥ contains P and the form B
and the action of G pass down to P ⊥ /P . Suppose that B has a nontrivial kernel in P ⊥ /P .
This kernel cannot be all of P ⊥ /P , for then P ⊥ would be an isotropic 4-space in ∧2 V .
Thus the kernel must be a line in P ⊥ /P and this kernel is necessarily G-invariant. The
preimage of this line in P ⊥ is an isotropic G-invariant 3-space in ∧2 V , contrary to Lemma
6.8. We conclude that (P ⊥ /P, B) is a nondegenerate quadratic 2-space. Such a space isisomorphic to a hyperbolic plane and hence contains exactly two isotropic lines. The actionof G on P ⊥ /P is by similitudes, hence it permutes these lines. Taking the preimage in P ⊥ ,we obtain two isotropic 3-spaces Λ1 and Λ2 in ∧2 V that are permuted by G. By LemmaCUSPIDALITY CRITERION

19

6.8, these isotropic G-spaces cannot be fixed by G and we conclude that the stabilizer of
each is a subgroup H of index two in G. By repeating the argument of Lemma 6.8 with
H in place of G, we conclude that there is either a line L < V or a 3-space U < V that isH-invariant. By replacing σ by σ ∨ if necessary, we may assume that the former possibility
holds. Let g0 ∈ G − H. Then the 2-space L + σ(g0 )L is easily seen to be G-invariant,contrary to the irreducibility of σ and σ ∨ . This contradiction finally allows us to conclude
that the restriction of B to P is nondegenerate.
We now repeat the argument of the previous paragraph with the space P in place of thespace P ⊥ /P . It yields an index two subgroup H of G and two isotropic lines in ∧2 V thatare fixed by H. By Lemma 6.6, each of these lines has the form ∧2 Q with Q < V a 2-space.
By the uniqueness assertion from Lemma 6.6, each of these 2-spaces is H-invariant. It now
follows from Clifford theory that if χ is the nontrivial {±1}-valued character on G whose
kernel is H, then χ ⊗ σ ∼= σ. Thus σ has a nontrivial quadratic self-twist, as required. 
Proposition 6.10. Suppose that σ is neither essentially self-dual of symplectic type nor
has a nontrivial quadratic self-twist. Then σ is essentially self-dual of proper orthogonal
type if and only if ∧2 V contains a G-invariant 3-space.
Proof. Let W < ∧2 V be a G-invariant 3-space. By Lemma 6.8, W cannot be isotropic. The
group G acts on W by similitudes and so the kernel of B|W is G-invariant. We have justobserved that this kernel cannot be W and, by the hypotheses and the preceding results, itcannot be of dimension 1 or 2. Thus the restriction of B to W is nondegenerate. It follows
from Lemma 6.6 that there is a nondegenerate symmetric bilinear form C on V such that
σ(G) ⊂ GSO(C). This implies that σ is essentially self-dual of proper orthogonal type.
Now suppose that σ is essentially self-dual of proper orthogonal type, so that there is a
nondegenerate bilinear form C on V such that σ(G) ⊂ GSO(C). By Lemma 6.6, there is a
3-space W ⊂ ∧2 V such that ρ(GSO(C)) preserves W . In particular, W is G-invariant, andthe reverse implication is proved.

Proof of Theorem 6.5. The representation ∧2 σ is essentially self-dual. Thus if it has any
proper nonzero G-invariant subspace, it necessarily has such a subspace of dimension at
most 3. The proof follows from Propositions 6.7, 6.9 and 6.10.

6.5. Exception in (iii)(α). Using Theorem 6.5 it is possible to explain the seemingly
strange exception in (iii)(α) of Theorem 1.1. For this we first set up some notation.
Let G be a group and let H be a subgroup of index two in G. If (τ, W ) is a 2-dimensional
representation of H, then AsG/H (τ ), the Asai lift of τ , which is a 4-dimensional representation of G, is defined as follows. Fix g ∈ G − H. Define the representation (τ 0 , W ) of Hvia
τ 0 (h) = τ (ghg −1 ).
We have a homomorphism
τ × τ 0 : H −→ GL(W ) × GL(W ) ,→ (GL(W ) × GL(W )) o (G/H).
Here the action of G/H on (GL(W ) × GL(W )) is as in (2.3), i.e, via switching the two
factors. We can extend the map τ × τ 0 from H to a map τ̃ from G to (GL(W ) × GL(W )) o20

M. ASGARI & A. RAGHURAM(G/H) by setting τ̃ (g) = (1W , τ (g 2 ), γ), where γ denotes the nontrivial element of G/H. In
other words((τ (h), τ (ghg −1 ), 1) if x = h ∈ H,
τ̃ (x) =(τ (h), τ (ghg), γ)
if x = hg ∈ Hg.
It is easy to check that τ̃ is a homomorphism. We define AsG/H (τ ) = As ◦ τ̃ , where themap “As” is as in (2.4). This gives a representation of G on the space V = W ⊗ W . Tosummarize, we have the following commutative diagram:τ ×τ 0/ (GL(W ) × GL(W )) o (G/H)
ffff2
τ̃ffffffff
f
f
As
f
fffff

 ffffffffff/ GL(V )Gf

H

AsG/H (τ )Proposition 6.11. Assume that τ is an irreducible 2-dimensional representation of H andσ = AsG/H (τ ) is irreducible. Then τ has a nontrivial self-twist if and only if σ has anontrivial self-twist.
Proof. First, assume that τ has a nontrivial self-twist, i.e., there is a nontrivial quadratic
character χ of H and a nonzero T ∈ HomH (τ, τ ⊗χ). Set K = ker(χ). Since σ is irreducible,
by [26, Theorem 1.4] we know that K is not normal in G. Hence gKg −1 6= K. Let χ∗ be thecharacter of G obtained via composing χ with the transfer homomorphism from Gab to H ab .
Observe that χ∗ |H is nontrivial because otherwise we would have χ∗ (h) = χ(h)χ(ghg −1 ) = 1
for all h ∈ H, which implies that K = gKg −1 . Therefore, χ∗ is nontrivial. It is now easy to
see that T ⊗ T is a nonzero element of HomG (σ, σ ⊗ χ∗ ), i.e., σ has a nontrivial self-twist.
Next assume that σ has a nontrivial self-twist. This implies that σ ∼= IG
M (ρ) where M isa subgroup of index 2 in G and ρ is a 2-dimensional representation of M . Hence, we have
AsG/H (τ ) ∼= IG (ρ).
M

Take ∧2 of both sides and restrict back to H. The semisimplification of the left hand sidethen gives
ResH (∧2 (AsG/H (τ )))ss ∼= ∧2 (ResH (AsG/H (τ ))ss∼
= ∧2 (τ ⊗ τ 0 )ss∼= (Sym2 (τ ) ⊗ det(τ 0 ) ⊕ (Sym2 (τ 0 ) ⊗ det(τ ) ,i.e., a direct sum of two 3-dimensional representations of H. However, the right hand side,by [16, §3], gives
G∼
ResH (∧2 (IGM (ρ)))ss = ResH (AsG/M (ρ) ⊗ ωG/M ) ⊕ ResH (IM (det(ρ))),i.e., a direct sum of a 4-dimensional and a 2-dimensional representation. Therefore, at least
one of the two 3-dimensional representations on the left hand side should be reducible. Thisimplies that Sym2 (τ ) should be a reducible representation. Replacing τ by τ ∨ if necessary,
we may assume that there is a character χ occurring as a quotient of Sym2 (τ ). Hence,τ∼= τ∨ ⊗ χ ∼= (τ ⊗ det(τ )−1 ) ⊗ χ ∼= τ ⊗ (χ det(τ )−1 ),CUSPIDALITY CRITERION

21

i.e., τ has a self-twist by χ det(τ )−1 . We claim that χ det(τ )−1 is nontrivial, for otherwise
we would have2 ≤ dim HomH (τ ⊗ τ, det(τ )) = dim HomH (τ, τ ∨ ⊗ det(τ ))= dim HomH (τ, τ ) = 1
which is a contradiction.



The above proposition explains the strange exception in (iii)(α) of Theorem 1.1. Assume
that (τ, W ) is the parameter of a cuspidal representation π of GL(2) over a quadratic extension E/F of number fields. Then σ = As(τ ) is the parameter of Π = AsE/F (π). Assume
that Π is cuspidal, i.e., σ is irreducible. Since τ is 2-dimensional, there is a symplectic form
S on W which τ preserves up to similitudes. It is easy to see that σ preserves S ⊗ S on
W ⊗ W up to similitudes. In fact, σ is an essentially self-dual representation of improper
orthogonal type. Further, by the above proposition, we see that τ is dihedral if and only if σ
has a nontrivial (quadratic) self-twist. By Theorem 6.5, one concludes that ∧2 σ is reducible
if and only if τ is dihedral.
References[1] Arthur J. and Clozel, L. Simple algebras, base change, and the advanced theory of the trace formula.
Annals of Mathematics Studies 120 (1989), Princeton University Press, Princeton, NJ, 1989.
[2] Asgari, M. and Shahidi, F. Generic transfer for general spin groups. Duke Math. J., 132 (2006), no 1,
137–190.
[3] Asgari, M. and Shahidi, F. Generic transfer for general spin groups II. In Preparation.
[4] Asgari, M. and Shahidi, F. Generic Transfer from GSp(4) to GL(4). Compos. Math., 142 (2006), no.
3, 541–550.
[5] Fulton, W. and Harris, J. Representation theory. A first course. Graduate Texts in Mathematics, 129.
Readings in Mathematics. Springer-Verlag, New York, 1991.
[6] Gan, W. and Takeda, S. The local Langlands conjecture for GSp(4). Preprint 2007. Available at
http://arxiv.org/abs/0706.0952.
[7] Gelbart, S. and Jacquet, H. A relation between automorphic representations of GL(2) and GL(3). Ann.
Sci. École Norm. Sup. (4) 11 (1978), 471–542.
[8] Ginzburg, D. and Rallis, S. and Soudry, D. Generic Automorphic Forms on SO(2n + 1): Functorial
Lift to GL(2n), Endoscopy, and Base Change. Internat. Math. Res. Notices, 2001, no. 14, 729-764.
[9] Hundley, J. and Sayag, E. Descent Construction For Gspin Groups: Main results and applications,
Preprint, 2008. Available at http://arxiv.org/abs/0808.2269.
[10] Jacquet, H., Piatetski-Shapiro, I. and Shalika, J. Rankin–Selberg convolutions, Amer. J. Math., 105(1983), no. 2, 367–464.
[11] Jacquet, H. and Shalika, J. On Euler products and the classification of automorphic forms, I. Amer.
J. Math., 103 (1981), no. 4, 499-558.
[12] Jacquet, H. and Shalika, J. On Euler products and the classification of automorphic forms, II. Amer.
J. Math., 103 (1981), no. 4, 777-815.
[13] Kim, H. Functoriality for the exterior square of GL4 and the symmetric fourth of GL2 . With appendix
1 by Dinakar Ramakrishnan and appendix 2 by Kim and Peter Sarnak. J. Amer. Math. Soc. 16 (2003),
no. 1, 139–183.
[14] Kim, H. Applications of Langlands’ functorial lift of odd orthogonal groups. Trans. Amer. Math. Soc.
354 (2002), no. 7, 2775–2796.
[15] Kim, H. An example of non-normal quintic automorphic induction and modularity of symmetric powers
of cusp forms of icosahedral type. Invent. Math. 156 (2004), no. 3, 495–502.

22

M. ASGARI & A. RAGHURAM[16] Kim, H. An application of exterior square functoriality of GL4 ; Asai lift. Number theory, 197–202,
CRM Proc. Lecture Notes, 36, Amer. Math. Soc., Providence, RI, 2004.
[17] Kim, H. Private communication.
[18] Kim, H., and Shahidi, F. Cuspidality of symmetric powers with applications. Duke Math. J. 112 (2002),
no. 1, 177–197.
[19] Krishnamurthy, M. The Asai transfer to GL4 via the Langlands–Shahidi method. Int. Math. Res. Not.
2003, no. 41, 2221–2254.
[20] Labesse, J.-P. and Langlands, R. L-indistinguishability for SL(2). Canad. J. Math., 31 (1979) 726–785.
[21] Martin, K. Modularity of hypertetrahedral representations. C. R. Math. Acad. Sci. Paris 339 (2004),
no. 2, 99–102.
[22] Raghuram, A. and Shahidi, F. Functoriality and special values of L-functions, To appear in the proceedings of the AIMS workshop on Eisenstein series and Applications, (2005).
[23] Raghuram, A. and Shahidi, F. On certain period relations for cusp forms on GL(n), Preprint 2007.
Available at http://arxiv.org/abs/0707.1708[24] Ramakrishnan, D. Modularity of the Rankin–Selberg L-series, and multiplicity one for SL(2). Ann. of
Math. (2) 152 (2000), no. 1, 45–111.
[25] Ramakrishnan, D. Algebraic cycles on Hilbert modular fourfolds and poles of L-functions. Algebraic
groups and arithmetic, 221–274, Tata Inst. Fund. Res., Mumbai, 2004.
[26] Ramakrishnan, D. Modularity of solvable Artin representations of GO(4)-type. Int. Math. Res. Not.
2002, no. 1, 1–54.
[27] Ramakrishnan, D. Irreducibility and cuspidality. To appear in Representation Theory and Automorphic
Forms, Progress in Math., Springer-Birkhäuser, (2006).
[28] Ramakrishnan, D. and Shahidi, F. Siegel modular forms of genus 2 attached to elliptic curves. Math.
Res. Lett. 14 (2007), no. 2, 315–332.
[29] Ramakrishnan, D. and Wang, S. A cuspidality criterion for the functorial product on GL(2) × GL(3)with a cohomological application. Int. Math. Res. Not. 2004, no. 27, 1355–1394.
[30] Shahidi, F. A proof of Langlands conjecture on Plancherel measures; Complementary series for p-adic
groups. Ann. of Math., 132, 273–330 (1990).
[31] Soudry, D. On Langlands functoriality from classical groups to GLn . Formes automorphes. I. AstérisqueNo. 298 (2005), 335–390.
Department of Mathematics, Oklahoma State University, Stillwater, OK 74078, USA
E-mail address: asgari@math.okstate.edu, araghur@math.okstate.edu