# Denominator formulas for Lie superalgebras
(extended abstract)

Victor Kac<fnref n="1" /> and Pierluigi Möseneder Frajria<fnref n="2" /> and Paolo Papi<fnref n="3" />

**Abstract.** We provide formulas for the Weyl-Kac denominator and superdenominator of a basic classical Lie superalgebra for a distinguished set of positive roots.

**Résumé.** Nous donnons les formules pour les dénominateurs et super-dénominateurs de Weyl-Kac d'une superalgèbre de Lie basique classique pour un ensemble distingué de racines positives.

**Keywords:** Lie superalgebra, denominator identity, dual pair

## 1 Introduction

The Weyl denominator identity

$$ \prod_{\alpha \in \Delta^+} (1 - e^{-\alpha}) = \sum_{w \in W} \operatorname{sgn}(w) e^{w(\rho) - \rho} \qquad (1.1) $$

is one of the most intriguing combinatorial identities in the character ring of a complex finite dimensional simple Lie algebra. It admits far reaching generalizations to the Kac-Moody setting, where it provides a proof for the Macdonald’s identities (including, as easiest cases, the Jacobi triple and quintuple product identities). Its role in representation theory is well-understood, since the inverse of the l.h.s of (1.1) is the character of the Verma module $M(0)$ with highest weight 0.

The goal of the present paper is to provide an expression of the character $M(0)$ in the case of a basic classical Lie superalgebra; the analog of the l.h.s of (1.1) is the Weyl-Kac denominator [6]

$$ R = \frac{\prod_{\alpha \in \Delta_0^+} (1 - e^{-\alpha})}{\prod_{\alpha \in \Delta_1^+} (1 + e^{-\alpha})}. \qquad (1.2) $$

Here and in the remaining part of the Introduction we refer the reader to Section 2 for undefined notation. Generalizations of formulas for $R$ to affine superalgebras and their connection with number theory and the theory of special functions are thoroughly discussed in [7]. The striking differences which make the super case very different from the purely even one are the following. First, it is no more true that the sets

<footnote n="1">¹Department of Mathematics, Rm 2-178, MIT, 77 Mass. Ave, Cambridge, MA 02139;</footnote>

<footnote n="2">²Politecnico di Milano, Polo regionale di Como, Via Valleggio 11, 22100 Como, ITALY;</footnote>

<footnote n="3">³Dipartimento di Matematica, Sapienza Università di Roma, P.le A. Moro 2, 00185, Roma, ITALY.</footnote>
of positive roots are conjugate under the Weyl group (to get transitivity on the set of set of positive roots one has to consider Serganova's odd reflections, which however play no role in this paper). In particular, the denominator identity looks very different according to the chosen set of positive roots. Moreover the restriction of the supersymmetric nondegenerate invariant bilinear form to the real span of roots may be indefinite, hence isotropic sets of roots appear naturally. Indeed, one defines the *defect* $d$ of $\mathfrak{g}$ (notation def $\mathfrak{g}$) as the dimension of a maximal isotropic subspace of $\sum_{\alpha \in \Delta} \mathbb{R}\alpha$. It is shown in [7] that $d$ equals the cardinality of a *maximal isotropic subset of $\Delta^+$* (a subset $S \subset \Delta^+$ is isotropic if it is formed by linearly independent pairwise orthogonal isotropic roots).

**Definition 1.1** We call a set of positive roots distinguished if the corresponding set of simple roots has exactly one odd root.

Distinguished sets of positive roots exist for any basic classical Lie superalgebra; they are implicitly classified in [5]. The main result of the paper is the following theorem.

**Theorem 1.1** Let $\mathfrak{g} = \mathfrak{g}_0 \oplus \mathfrak{g}_1$ be a basic classical Lie superalgebra of defect $d$, where $\mathfrak{g} = A(d-1, d-1)$ is replaced by $gl(d, d)$. Then, for any distinguished set of positive roots, we have

$$e^{\rho} R = \frac{1}{C} \sum_{w \in W} \operatorname{sgn}(w) w \frac{e^{\rho}}{(1 + e^{-\gamma_1})(1 - e^{-\gamma_1 - \gamma_2}) \cdots (1 + (-1)^{n+1} e^{-\gamma_1 - \gamma_2 - \cdots - \gamma_d})}, \quad (1.3)$$

$$e^{\rho} \tilde{R} = \frac{1}{C} \sum_{w \in W} \operatorname{sgn}'(w) w \frac{e^{\rho}}{(1 - e^{-\gamma_1})(1 - e^{-\gamma_1 - \gamma_2}) \cdots (1 - e^{-\gamma_1 - \gamma_2 - \cdots - \gamma_d})} \quad (1.4)$$

where $W$ is the Weyl group of $\mathfrak{g}$, $\{\gamma_1, \dots, \gamma_d\}$ is an explicitly defined maximal isotropic subset of $\Delta^+$ and $C$ is the following constant:

$$C = \begin{cases} 1 & \text{if } \mathfrak{g} = A(n, m), \\ 2^{\min\{m,n\}} & \text{if } \mathfrak{g} = B(m, n), \\ 2^n & \text{if } \mathfrak{g} = D(m, n), m > n, \\ 2^{m-1} & \text{if } \mathfrak{g} = D(m, n), n \ge m, \\ 2 & \text{if } \mathfrak{g} = D(2, 1, \alpha), F(4), G(3). \end{cases} \quad (1.5)$$

A suitable modification of the previous statement holds for $\mathfrak{g}$ of type $A(d-1, d-1)$ too: see Remark 3.1. The elements $\gamma_i$ are defined in (3.11), (4.9) for types $A, B$, respectively.

Theorem 1.1 has been proved by Kac and Wakimoto in the defect 1 case [7] (see Theorem 3.1 below), so we are reduced to discuss the cases in which the defect of $\mathfrak{g}$ is greater than 1. Hence we have to deal with superalgebras of type $A(m, n)$, $B(m, n)$, $D(m, n)$. Our approach to these cases is based on the analysis of the $\mathfrak{g}_0$-module structure of the oscillator representation of the Weyl algebra $W(\mathfrak{g}_1)$ of $\mathfrak{g}_1$. This is done in Sections (3) and (4) relying on methods coming from the theory of Lie groups. More precisely we use Howe theory of dual pairs, and results of Kashiwara-Vergne and Li-Paul-Than-Zhu which provide explicitly the Theta correspondence. The key result in this respect is Theorem 4.1. Proofs are only outlined and sometimes skipped (mainly in the case of purely representation-theoretical results). We work out in some detail the case $A(m, n)$, where a simpler treatment using Cauchy formulas in place of Howe duality is available: see Section 3.1. We also explain our general approach in type $B(m, n)$, providing a complete
proof when $m \ge n$ and presenting an example to give the flavour of the general case when $m < n$. Type D is not treated here at all. Finally, in Section 5 we reformulate our main Theorem in a form which leads to a general conjecture for the expression of $R$, $\tilde{R}$ for any set of positive roots, involving certain maximal isotropic subsets $S$ of positive roots. A purely combinatorial proof of this conjecture for some special choices of $S$ will appear in a forthcoming publication, publication, where we actually derive the theta correspondence of [8] and [9] from the denominator identity.

## 2 Setup

In this Subsection we collect some notation and definitions which will be constantly used throughout the paper. Let $\mathfrak{g}$ be a basic classical Lie superalgebra. This means that $\mathfrak{g} = \mathfrak{g}_0 \oplus \mathfrak{g}_1$ is a finite dimensional simple Lie superalgebra such that $\mathfrak{g}_0$ is a reductive Lie algebra and that $\mathfrak{g}$ admits a nondegenerate invariant supersymmetric bilinear form $(\cdot, \cdot)$ [5].

Recall that for a Lie superalgebra $\mathfrak{g}$ the Casimir operator is defined as $\Omega_{\mathfrak{g}} = \sum_i x^i x_i$ if $\{x_i\}$ is a basis of $\mathfrak{g}$ and $\{x^i\}$ its dual basis w.r.t. $(\cdot, \cdot)$ (see [5, pag. 85]). Then $\Omega_{\mathfrak{g}}$ acts on $\mathfrak{g}$ as $2gI_{\mathfrak{g}}$, where $g$ is a constant. Choose a Cartan subalgebra $\mathfrak{h} \subset \mathfrak{g}_0$, and let $\Delta, \Delta_0, \Delta_1$ be the set of roots, even roots, odd roots, respectively. Let $W \subset GL(\mathfrak{h}^*)$ be the group generated by the reflections w.r.t. even roots. Choose a set of positive roots $\Delta^+ \subset \Delta$ and set $\Delta_i^+ = \Delta_i \cap \Delta^+$, $i = 0, 1$. Set also, as usual, for $i = 0, 1$, $\rho_i = \frac{1}{2} \sum_{\alpha \in \Delta_i^+} \alpha$, $\rho = \rho_0 - \rho_1$. Assume that $g \ne 0$. Then set $\Delta_0^\sharp = \{\alpha \in \Delta_0 | g \cdot (\alpha, \alpha) > 0\}$ and let $W^\sharp$ be the subgroup of $W$ generated by the reflections in roots from $\Delta_0^\sharp$. We refer to [7, Remark 1.1, b)] for the definition of $W^\sharp$ when $g = 0$. Set

$$ \bar{\Delta}_0 = \{\alpha \in \Delta_0 | \frac{1}{2}\alpha \notin \Delta\}, \quad \bar{\Delta}_1 = \{\alpha \in \Delta_1 | (\alpha, \alpha) = 0\}. \qquad (2.1) $$

Finally, for $w \in W$, set

$$ \operatorname{sgn}(w) = (-1)^{\ell(w)}, \quad \operatorname{sgn}'(w) = (-1)^m \qquad (2.2) $$

where $\ell$ is the usual length function on $W$ and $m$ is the number of reflections from $\bar{\Delta}_0^+$ occurring in an expression of $w$.

Beyond the Weyl denominator $R$ defined in (1.2) it will be very important for us the Weyl-Kac superdenominator, defined as

$$ \tilde{R} = \frac{\prod_{\alpha \in \Delta_0^+} (1 - e^{-\alpha})}{\prod_{\alpha \in \Delta_1^+} (1 - e^{-\alpha})}. \qquad (2.3) $$

As a notational convention, we denote by $L^X(\mu)$ the irreducible highest weight module of highest weight $\mu$ for a Lie algebra of type $X$.

## 3 Denominator formulas for distinguished set of positive roots

Kac and Wakimoto provided an expression for $R$, $\tilde{R}$ for certain systems of positive roots.
**Theorem 3.1** Let $\mathfrak{g}$ be a classical Lie superalgebras and let $\Delta^+$ be any set of positive roots such that a maximal isotropic subset $S$ of $\Delta^+$ is contained in the set of simple roots $\Pi$ corresponding to $\Delta^+$. Then

$$e^{\rho}R = \sum_{w \in W^{\mathfrak{z}}} \operatorname{sgn}(w) w \frac{e^{\rho}}{\prod_{\beta \in S} (1 + e^{-\beta})}, \qquad (3.1)$$

$$e^{\rho}\check{R} = \sum_{w \in W^{\mathfrak{z}}} \operatorname{sgn}'(w) w \frac{e^{\rho}}{\prod_{\beta \in S} (1 - e^{-\beta})}. \qquad (3.2)$$

This result has been stated in [7], and fully proved if $|S| = 1$ by using representation theoretical methods. A complete combinatorial proof has recently been obtained by Gorelik [3]. Note that a distinguished set of positive roots verifies the hypothesis of Theorem 3.1 if and only if $\mathrm{def} \, \mathfrak{g} = 1$ (i.e., $|S| = 1$).

The choice of a set of positive roots $\Delta^+$ determines a polarization $\mathfrak{g}_1 = \mathfrak{g}_1^+ + \mathfrak{g}_1^-$, where $\mathfrak{g}_1^\pm = \bigoplus_{\alpha \in \Delta_1^\pm} \mathfrak{g}_\alpha$.

Hence we can consider the Weyl algebra $W(\mathfrak{g}_1)$ of $(\mathfrak{g}_1, (\cdot, \cdot)|_{\mathfrak{g}_1})$ and construct the $W(\mathfrak{g}_1)$-module

$$M^{\Delta^{+}}(\mathfrak{g}_1) = W(\mathfrak{g}_1)/W(\mathfrak{g}_1)\mathfrak{g}_1^{+}, \qquad (3.3)$$

with action by left multiplication. The module $M^{\Delta^{+}}(\mathfrak{g}_1)$ is also a $\mathrm{sp}(\mathfrak{g}_1, (\cdot, \cdot))$-module with $T \in \mathrm{sp}(\mathfrak{g}_1, (\cdot, \cdot))$ acting by left multiplication by

$$\theta(T) = -\frac{1}{2} \sum_{i=1}^{\dim \mathfrak{g}_1} T(x_i)x^i, \qquad (3.4)$$

where $\{x_i\}$ is any basis of $\mathfrak{g}_1$ and $\{x^i\}$ is its dual basis w.r.t. $(, \cdot)$. It is easy to check that, in $W(\mathfrak{g}_1)$, relation

$$[\theta(T), x] = T(x) \qquad (3.5)$$

holds for any $x \in \mathfrak{g}_1$. This implies that we have a $\mathfrak{h}$-module isomorphism

$$M^{\Delta^{+}}(\mathfrak{g}_1) \cong S(\mathfrak{g}_1^{-}) \otimes \mathbb{C}_{-\rho_1} \qquad (3.6)$$

where $\rho_1$ is the half sum of positive odd roots and $S(\mathfrak{g}_1^{-})$ is the symmetric algebra of $\mathfrak{g}_1^{-}$. Hence its $\mathfrak{h}$-character is given by

$$chM^{\Delta^{+}}(\mathfrak{g}_1) = \frac{e^{-\rho_1}}{\prod_{\alpha \in \Delta_1^{+}} (1 - e^{-\alpha})}. \qquad (3.7)$$

The key of our approach to the denominator formula is the following observation: since $\mathrm{ad}(\mathfrak{g}_0) \subset \mathrm{sp}(\mathfrak{g}_1, (\cdot, \cdot))$, we obtain an action of $\mathfrak{g}_0$ on $M^{\Delta^{+}}(\mathfrak{g}_1)$. Upon multiplication by $e^{\rho_0} \prod_{\alpha \in \Delta_0^{+}} (1 - e^{-\alpha})$ the r.h.s. of (3.7) becomes $e^{\rho} \check{R}$ and equating it with the $\mathfrak{g}_0$-character of $M^{\Delta^{+}}(\mathfrak{g}_1)$ one obtains our formula.

Our approach to the calculation of the $\mathfrak{g}_0$-character of $M^{\Delta^{+}}(\mathfrak{g}_1)$ is outlined in Section 4. Next we deal the special case of type I Lie superalgebras (cf. [5]).
## 3.1 Type I superalgebras

The key of our approach is the following fact, which is easily proved.

**Lemma 3.2** Let $\mathfrak{g}$ be a type I basic classical Lie superalgebra and let $\Delta^+$ be a distinguished set of positive roots. Then $\mathfrak{g}_1^+$ and $\mathfrak{g}_1^-$ are $\mathfrak{g}_0$-modules.

**Corollary 3.3** For type I superalgebras, we have

$$ M^{\Delta^{+}}(\mathfrak{g}_1) \cong S(\mathfrak{g}_1^{-}) \otimes \mathbb{C}_{-\rho_1} \qquad (3.8) $$

as $\mathfrak{g}_0$-modules.

The previous corollary reduces the problem of computing the $\mathfrak{g}_0$-character of $M^{\Delta^+}(\mathfrak{g}_1)$ to the calculation of the $\mathfrak{g}_0$-character of $S(\mathfrak{g}_1^-)$. This latter character is well-known: for a uniform approach one might e.g. refer to the work of Schmid [10].

We start discussing the denominator formula in type $A(m, n)$, $m \neq n$. Introduce the following notation: $\mathfrak{h}$ is the set of diagonal matrices in $gl(m+1|n+1)$ with zero supertrace, $\{\epsilon_i\}$ is the standard basis of $(\mathbb{C}^{m+n+2})^*$ and $\delta_i = \epsilon_{m+i+1}$, $1 \le i \le n+1$.

It follows from the analysis made in [5, 2.5.4] that in this case there are two distinguished sets of positive roots up to W-action: if we fix $\Delta_0^+ = \{\epsilon_i - \epsilon_j \mid 1 \le i < j \le m+1\} \cup \{\delta_i - \delta_j \mid 1 \le i < j \le n+1\}$, and we set $\Delta_1^+ = \{\epsilon_i - \delta_j \mid 1 \le i \le m+1, 1 \le j \le n+1\}$, then the only distinguished sets of positive roots containing $\Delta_0^+$ are $\Delta_0^+ \cup \Delta_1^+$ and $\Delta_0^+ \cup -\Delta_1^+$. The arguments which follow clearly hold for both systems, hence we deal only with $\Delta_0^+ \cup \Delta_1^+$ which we denote by $\Delta_A^+$ (or just by $\Delta^+$). Its corresponding set of simple roots is $\Pi = \{\epsilon_1 - \epsilon_2, \epsilon_2 - \epsilon_3, \dots, \epsilon_{m+1} - \delta_1, \delta_1 - \delta_2, \dots, \delta_n - \delta_{n+1}\}$.

By Corollary 3.3 we have to calculate the $\mathfrak{g}_0$-character of $S(\mathfrak{g}_1^-)$. Note that, according to our identifications, the action of $\mathfrak{g}_0$ on $\mathfrak{g}_1^-$ is the natural action of $\{(A, B) \in gl(n+1) \times gl(m+1) \mid tr(A)+tr(B) = 0\}$ on $(\mathbb{C}^{n+1})^* \otimes \mathbb{C}^{m+1}$. Assume $m > n$. Cauchy formulas in our setting give

$$ \mathrm{ch}(S(\mathfrak{g}_1^-)) = \mathrm{ch}(S((\mathbb{C}^{n+1})^* \otimes \mathbb{C}^{m+1})) = \sum_{\lambda} L^{A_m}(\tau(\lambda))L^{A_n}(\lambda) \qquad (3.9) $$

where for $\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_{n+1}$

$$ \lambda = \sum_{i=1}^{n+1} \lambda_i \delta_i, \quad \tau(\lambda) = -w_0\left(\sum_{i=1}^{n+1} \lambda_i \epsilon_i\right). \qquad (3.10) $$

and $w_0$ is the longest element in the symmetric group $W(A_m)$. Set

$$ \gamma_1 = \epsilon_{m+1} - \delta_1, \quad \gamma_2 = \epsilon_m - \delta_2, \dots, \gamma_{n+1} = \epsilon_{m-n+1} - \delta_{n+1}. \qquad (3.11) $$

Then (3.8) and (3.9) imply

$$ S(\mathfrak{g}_1^-) \otimes \mathbb{C}_{-\rho_1} = \bigoplus_{s_1 \ge s_2 \ge \dots \ge s_{n+1}} L^{A_m \times A_n} (-\rho_1 - s_1 \gamma_1 - \dots - s_{n+1} \gamma_{n+1}). \qquad (3.12) $$
Denote by $\lambda_{s_1, \dots, s_{n+1}}$ the $\mathfrak{g}_0$-dominant weight appearing in the r.h.s. of the above expression. Taking the $\mathfrak{g}_0$-supercharacter of both sides of (3.12) and using the Weyl character formula, we have

$$ \frac{e^{-\rho_1}}{\prod_{\beta \in \Delta_1^+} (1 + e^{-\beta})} = \sum_{s_1 \ge s_2 \ge \dots \ge s_{n+1}} (-1)^{s_1+s_2+\dots+s_{n+1}} \mathrm{chL}^{A_m \times A_n}(\lambda_{s_1, \dots, s_{n+1}}) = \quad (3.13) $$

$$ \sum_{s_1 \ge s_2 \ge \dots \ge s_{n+1}} (-1)^{s_1+s_2+\dots+s_{n+1}} \sum_{w \in W} \operatorname{sgn}(w) \frac{e^{w(\lambda_{s_1, \dots, s_{n+1}} + \rho_0)}}{e^{\rho_0} \prod_{\beta \in \Delta_0^+} (1 - e^{-\alpha})} $$

Then, multiplying the first and last member of the equalities in (3.13) by $e^{\rho_0} \prod_{\beta \in \Delta_0^+} (1 - e^{-\beta})$, we obtain

$$ e^{\rho} R = \sum_{s_1 \ge s_2 \ge \dots \ge s_{n+1}} (-1)^{s_1+s_2+\dots+s_{n+1}} \sum_{w \in W} \operatorname{sgn}(w) e^{w(\rho - s_1\gamma_1 - \dots - s_{n+1}\gamma_{n+1})} $$

Hence we have proven formula (3.14) below, which is an instance of (1.3). Deriving the companion formula (3.15) is even easier: start from

$$ \frac{e^{-\rho_1}}{\prod_{\beta \in \Delta_1^+} (1 - e^{-\beta})} = \sum_{s_1 \ge s_2 \ge \dots \ge s_{n+1}} \mathrm{chL}^{A_m \times A_n}(\lambda_{s_1, \dots, s_{n+1}}) $$

and proceed as above. So we have proved the following proposition.

**Proposition 3.4** Let $\mathfrak{g}$ be a Lie superalgebra of type $A(m, n)$, $m > n$. Then for a distinguished set of positive roots we have:

$$ e^{\rho} R = \sum_{w \in W} \operatorname{sgn}(w) w \frac{e^{\rho}}{(1 + e^{-\gamma_1})(1 - e^{-\gamma_1 - \gamma_2}) \cdots (1 + (-1)^{n+1} e^{-\gamma_1 - \gamma_2 - \cdots - \gamma_{n+1}})}, \quad (3.14) $$

$$ e^{\rho} \tilde{R} = \sum_{w \in W} \operatorname{sgn}(w) w \frac{e^{\rho}}{(1 - e^{-\gamma_1})(1 - e^{-\gamma_1 - \gamma_2}) \cdots (1 - e^{-\gamma_1 - \gamma_2 - \cdots - \gamma_{n+1}})}. \quad (3.15) $$

**Remark 3.1** The above formulas hold clearly in $gl(n + 1, n + 1)$, but do not restrict to $sl(n + 1, n + 1)$, since the last factor in the r.h.s. of (3.15) has a pole. Note that this factor is $W$-invariant, hence can be taken out of the sum. Since the left hand side restricts to $sl(n + 1, n + 1)$, the sum

$$ \sum_{w \in W} \operatorname{sgn}(w) w \frac{e^{\rho}}{(1 - e^{-\gamma_1})(1 - e^{-\gamma_1 - \gamma_2}) \cdots (1 - e^{-\gamma_1 - \gamma_2 - \cdots - \gamma_n})} $$

is divisible by $1 - e^{-\gamma_1 - \gamma_2 - \cdots - \gamma_{n+1}}$. After simplifying, we may restrict to the Cartan subalgebra of $A(n, n)$ getting a superdenominator formula in this type too.

**Remark 3.2** The above reasoning works also in type C. There are two distinguished sets of positive roots (cf. [5, 2.5.4]), one being the opposite of the other. Using Corollary 3.3 and a theorem of Schmid [10] in place of Cauchy formulas we get (1.3) and (1.4) in this case.
## 4 The $\mathfrak{g}_0$-character of $M^{\Delta^+}(\mathfrak{g}_1)$ via compact dual pairs

We start discussing the possible distinguished root systems up to W-equivalence for type II Lie superalgebras of defect greater than 1, following [5].

In type $B(m, n)$ there is a unique distinguished set of positive roots $\Delta_B^+$, which, with notation as in [5], can be described as follows. We have, for $1 \le i \ne j \le m$, $1 \le k \ne l \le n$,

$$ \Delta_0^+ = \{\epsilon_i \pm \epsilon_j, \epsilon_i, \delta_k \pm \delta_l, 2\delta_k\}, \quad \Delta_1^+ = \{\delta_k \pm \epsilon_i, \delta_k\}, \qquad (4.1) $$

$$ \bar{\Delta}_0^+ = \{\epsilon_i \pm \epsilon_j, \epsilon_i, \delta_k \pm \delta_l\}, \quad \bar{\Delta}_1^+ = \{\delta_k \pm \epsilon_i\}, \qquad (4.2) $$

$$ \Pi = \{\delta_1 - \delta_2, \dots, \delta_n - \epsilon_1, \epsilon_1 - \epsilon_2, \dots, \epsilon_{m-1} - \epsilon_m, \epsilon_m\}, \qquad (4.3) $$

$$ 2\rho_1 = (2m + 1)(\delta_1 + \dots + \delta_n). \qquad (4.4) $$

Note that $\pm\{\epsilon_i \pm \epsilon_j, \epsilon_i\}$ is a root system of type $B_m$ (which will be denoted by $\Delta(B_m)$), that $\pm\{\delta_k \pm \delta_l, 2\delta_k\}$ is a root system of type $C_n$ (which will be denoted by $\Delta(C_n)$) and that $\pm\{\delta_k - \delta_l \mid 1 \le k \ne l \le n\}$ is a root system of type $A_{n-1}$ (which will be denoted by $\Delta(A_{n-1})$).

In type $D(m, n)$ there are three distinguished sets of positive roots $\Delta_{D1}^+$, $\Delta_{D2}^+$, $\Delta_{D2'}^+$. The corresponding sets of simple roots are

$$ \Pi_1 = \{\delta_1 - \delta_2, \dots, \delta_n - \epsilon_1, \epsilon_1 - \epsilon_2, \dots, \epsilon_{m-1} - \epsilon_m, \epsilon_{m-1} + \epsilon_m\}, $$

$$ \Pi_2 = \{\epsilon_1 - \epsilon_2, \dots, \epsilon_{m-1} - \epsilon_m, \epsilon_m - \delta_1, \delta_1 - \delta_2, \dots, \delta_{n-1} - \delta_n, 2\delta_n\}, $$

$$ \Pi_2' = \{\epsilon_1 - \epsilon_2, \dots, \epsilon_{m-1} + \epsilon_m, -\epsilon_m - \delta_1, \delta_1 - \delta_2, \dots, \delta_{n-1} - \delta_n, 2\delta_n\}. $$

**Theorem 4.1** The character of $M^{\Delta^+}(\mathfrak{g}_1)$ as a $\mathfrak{g}_0$-module is afforded by the Theta correspondence for the compact dual pairs $(G_1, G_2)$ as in the following table

$$ \begin{array}{c|c} \Delta^{+} & (G_1, G_2) \\ \hline \Delta_{B}^{+} & (O(2m+1), Sp(2n, \mathbb{R})) \\ \Delta_{A}^{+} & (U(m), U(n)) \\ \Delta_{D1}^{+} & (O(2m), Sp(2n, \mathbb{R})) \\ \Delta_{D2}^{+} & (Sp(m), O^{*}(2n)) \\ \Delta_{D2'}^{+} & (Sp(m), O^{*}(2n)) \end{array} \qquad (4.5) $$

For a quick review of the Theta correspondence see e.g. [1]. The explicit Theta correspondence is provided in [8] for the first, second and third dual pairs and in [9] for the fourth and fifth.

### 4.1 $B(m, n)$, $m \ge n$.

Theorem 7.2 of [8] and (3.7) give

$$ \mathrm{ch} M^{\Delta^{+}}(\mathfrak{g}_1) = \frac{e^{-\rho_1}}{\prod_{\beta \in \Delta_1^+} (1 - e^{-\beta})} = \qquad (4.6) $$

$$ \sum_{a_1 \ge a_2 \ge \dots \ge a_n \ge 0} \mathrm{ch} L^{C_n} \left(-\left(a_n + m + \frac{1}{2}\right)\delta_1 - \dots - \left(a_1 + m + \frac{1}{2}\right)\delta_n\right) \mathrm{ch} L^{B_m} \left(a_1 \epsilon_1 + \dots + a_n \epsilon_n\right). $$
By [4, Theorem 9.2 a)], $L^{C_n}(-(a_n + m + \frac{1}{2})\delta_1 - \dots - (a_1 + m + \frac{1}{2})\delta_n)$ is an irreducible parabolic Verma module w.r.t. $\Delta(A_{n-1})$. To prove irreducibility we have to show that if $\lambda = -(a_n + m + \frac{1}{2})\delta_1 - \dots - (a_1 + m + \frac{1}{2})\delta_n$ then

$$ (\lambda + \rho^{C_n}, \psi) \notin \mathbb{Z}^{>0} \quad \forall \psi \in \Delta^+(C_n) \setminus \Delta^+(A_{n-1}). $$

Since $\lambda + \rho^{C_n} = (n - a_n - m - \frac{1}{2})\delta_1 + (n - a_{n-1} - m - \frac{3}{2})\delta_2 + \dots + (-a_1 - m - \frac{1}{2})\delta_n$, the condition $m \ge n$ implies that all coefficients of the $\delta_i$, $\delta_i + \delta_j$ are not positive integers and the claim follows. Therefore the character is given by

$$ \begin{aligned} & chL^{C_n}(-(a_n + m + \frac{1}{2})\delta_1 - \dots - (a_1 + m + \frac{1}{2})\delta_n) \\ &= \frac{chL^{A_{n-1}}(-(a_n + m + \frac{1}{2})\delta_1 - \dots - (a_1 + m + \frac{1}{2})\delta_n)}{\prod_{1 \le k, l \le n} (1 - e^{-(\delta_k + \delta_l)})} \\ &= \frac{\sum_{w \in W(A_{n-1})} \operatorname{sgn}(w) \operatorname{we}^{\rho^{A_{n-1}} - (a_n + m + \frac{1}{2})\delta_1 - \dots - (a_1 + m + \frac{1}{2})\delta_n}}{\prod_{1 \le k, l \le n} (1 - e^{-(\delta_k + \delta_l)}) \cdot \prod_{1 \le k < l \le n} (1 - e^{-(\delta_k - \delta_l)})} \end{aligned} \quad (4.7) $$

where the second equality has been obtained using the Weyl character formula. Again Weyl formula allows us to make explicit the character of $L^{B_m}(a_1\epsilon_1 + \dots + a_n\epsilon_n)$:

$$ chL^{B_m}(a_1\epsilon_1 + \dots + a_n\epsilon_n) = \frac{\sum_{w \in W(B_m)} \operatorname{sgn}(w) \operatorname{we}^{\rho^{B_m} + a_1\epsilon_1 + \dots + a_n\epsilon_n}}{\prod_{1 \le i < j \le m} (1 - e^{-(\epsilon_i - \epsilon_j)})(1 - e^{-(\epsilon_i + \epsilon_j)}) \prod_{i=1}^{m} (1 - e^{-\epsilon_i})} \quad (4.8) $$

Set now

$$ \gamma_1 = \delta_n - \epsilon_1, \quad \gamma_2 = \delta_{n-1} - \epsilon_2, \dots, \gamma_n = \delta_1 - \epsilon_n. \quad (4.9) $$

Combining (4.6), (4.7), (4.8), (4.4) we obtain

**Proposition 4.2** If $\gamma_1, \dots, \gamma_n$ are defined by (4.9), we have

$$ e^{\rho} R = \sum_{w \in W(A_{n-1}) \times W(B_m)} \operatorname{sgn}(w) w \frac{e^{\rho}}{(1 + e^{-\gamma_1})(1 - e^{-(\gamma_1 + \gamma_2)}) \dots (1 + (-1)^{n+1} e^{-(\gamma_1 + \dots + \gamma_n)})}, \quad (4.10) $$

$$ e^{\rho} \check{R} = \sum_{w \in W(A_{n-1}) \times W(B_m)} \operatorname{sgn}(w) w \frac{e^{\rho}}{(1 - e^{-\gamma_1})(1 - e^{-(\gamma_1 + \gamma_2)}) \dots (1 - e^{-(\gamma_1 + \dots + \gamma_n)})}. \quad (4.11) $$

**Remark 4.1** We want to prove that (4.11) coincides with (1.4). Recall that $e^{\rho} \check{R}$ is such that $w(e^{\rho} \check{R}) = \operatorname{sgn}'(w)e^{\rho} \check{R}$. Take $g \in \Gamma = W(C_n)/W(A_{n-1})$, i.e., a sign change on the $\delta_i$, and compute:

$$ \sum_{g \in \Gamma} \operatorname{sgn}'(g) g (e^{\rho} \check{R}) = 2^n e^{\rho} \check{R}. $$

On the other hand, note that $\Gamma W(A_{n-1}) = W(C_n)$, therefore if we apply $\sum_{g \in \Gamma} \operatorname{sgn}'(g)g$ we get the (suitably signed) summation over the full Weyl group $W$, and (4.11) becomes (1.4).
## 4.2 $B(2, 4)$.

This is a defect 2 case. By Kashiwara-Vergne theorem,

$$ M^{\Delta^{+}}(\mathfrak{g}_1) = \sum_{a_1 \ge a_2 \ge 0} L(-\frac{5}{2}\delta_1 - \frac{5}{2}\delta_2 - (\frac{5}{2} + a_2)\delta_3 - (\frac{5}{2} + a_1)\delta_4) \otimes L(a_1\epsilon_1 + a_2\epsilon_2) + \\ \sum_{a_1 \ge a_2 \ge 1} L(-\frac{5}{2}\delta_1 - \frac{7}{2}\delta_2 - (\frac{5}{2} + a_2)\delta_3 - (\frac{5}{2} + a_1)\delta_3) \otimes L(a_1\epsilon_1 + a_2\epsilon_2) \quad (4.12) $$

A computation with Kazhdan-Lusztig polynomials shows that we can write the sp(8, $\mathbb{C}$)-modules appearing in terms of the $\Delta(A_3)$-parabolic Verma modules whose highest weights shifted by $\rho^{C_4}$ are

$$ \begin{aligned} & \frac{3}{2}\delta_1 + \frac{1}{2}\delta_2 - (\frac{1}{2} + a_2)\delta_3 - (\frac{3}{2} + a_1)\delta_4, & -\frac{1}{2}\delta_1 - \frac{3}{2}\delta_2 - (\frac{1}{2} + a_2)\delta_3 - (\frac{3}{2} + a_1)\delta_4, & (4.13) \\ & \frac{3}{2}\delta_1 - \frac{1}{2}\delta_2 - (\frac{1}{2} + a_2)\delta_3 - (\frac{3}{2} + a_1)\delta_4, & \frac{1}{2}\delta_1 - \frac{3}{2}\delta_2 - (\frac{1}{2} + a_2)\delta_3 - (\frac{3}{2} + a_1)\delta_4. & \end{aligned} $$

Hence we have

$$ e^{\rho} \tilde{R} = \sum_{w \in W(A_3)} \sum_{u \in A} \sum_{v \in W(B_2)} \operatorname{sgn}(w) \operatorname{sgn}(v) w u v \frac{e^{\rho}}{(1 - e^{-\delta_3 + \epsilon_1})(1 - e^{-\delta_3 - \delta_4 + \epsilon_1 + \epsilon_2})} \quad (4.14) $$

where $A$ is a set of coset representatives related to the list (4.13). Now argue as in Remark 4.1. Take $g \in \Gamma = W(C_4)/W(A_3)$. On the one hand $\sum_{g \in \Gamma} \operatorname{sgn}'(g)g(e^{\rho} \tilde{R}) = 16e^{\rho} \tilde{R}$. On the other hand, note that $\Gamma W(A_3) = \Gamma W(A_3)A = W(C_4)$, therefore if we apply $\sum_{g \in \Gamma} \operatorname{sgn}'(g)g$ to the r.h.s. of (4.14) we get four times the r.h.s. of (4.14). So

$$ e^{\rho} \tilde{R} = \frac{1}{4} \sum_{w \in W} \operatorname{sgn}'(w) w \frac{e^{\rho}}{(1 - e^{-\delta_3 + \epsilon_1})(1 - e^{-\delta_3 - \delta_4 + \epsilon_1 + \epsilon_2})} $$

proving (1.4) in this case. In the general case, the calculation of the KL-polynomials is replaced by the use of a result of Enright on the u-homology of unitary highest weight modules (cf. [2]).

# 5 Final remarks

We would like to rephrase our main theorem in a form which seems most suitable for a generalization. We need to single out a special maximal isotropic subset $S$ of positive roots. Fix a distinguished set of positive roots $\Delta^+$. Construct $S = S_1 \cup \dots \cup S_m = \{\gamma_1, \dots, \gamma_d\}$, $d = \text{def } \mathfrak{g}$, as follows: $S_1$ is an isotropic subset having maximal cardinality in the set of simple roots, and inductively $S_i$ is such a subset in the set of indecomposable roots of $S_{i-1} \setminus S_{i-1}$. Define

$$ \gamma_i^{<} = \{\beta \in S_i, \beta \le \gamma_i\}, \quad \langle \gamma_i \rangle = \sum_{\beta \in \gamma_i^{<}} \beta, \quad \operatorname{sgn}(\gamma_i) = (-1)^{|\gamma_i^{<}|+1} \quad (5.1) $$

where as usual $\alpha \le \beta$ if $\beta - \alpha$ is a sum of positive roots. This procedure determines uniquely $S$ once $\Delta^+$ is fixed (up to a mild exception in type $D$) and gives rise to the set $\{\gamma_1, \dots, \gamma_d\}$ of Theorem 1.1.
**Theorem 5.1** Let $\mathfrak{g} = \mathfrak{g}_0 \oplus \mathfrak{g}_1$ be a basic classical Lie superalgebra of defect $d$, where $\mathfrak{g} = A(d-1, d-1)$ is replaced by $\mathrm{gl}(d, d)$. Then, for any distinguished set of positive roots, if $S$ is as above, we have

$$e^{\rho}R = \frac{1}{C} \sum_{w \in W} \operatorname{sgn}(w) w \frac{e^{\rho}}{\prod_{i=1}^{d} (1 + \operatorname{sgn}(\gamma_i) e^{-\langle \gamma_i \rangle})}, \qquad (5.2)$$

$$e^{\rho} \tilde{R} = \frac{1}{C} \sum_{w \in W} \operatorname{sgn}'(w) w \frac{e^{\rho}}{\prod_{i=1}^{d} (1 - e^{-\langle \gamma_i \rangle})}, \qquad (5.3)$$

where $C$ is defined in (1.5).

We would like to remark that the above statement holds true in the hypothesis of Kac-Wakimoto-Gorelik theorem (in which case $e^{-\langle \gamma_i \rangle} = e^{-\gamma_i}$ and $C = |W/W^\#|$).

Denote by $Q$, $Q_0$ the lattices generated by all roots and even roots, respectively. Set

$$\varepsilon(\eta) = \begin{cases} 1 & \text{if } \eta \in Q_0 \\ -1 & \text{if } \eta \in Q \setminus Q_0, \quad \|\eta\| = \sum_{\beta \in \gamma} \varepsilon(\gamma - \beta)\beta. \end{cases}$$

Note that for the $\gamma_i$ appearing in (5.2), (5.3) the equality $\langle \gamma_i \rangle = \|\gamma_i\|$ holds. We modify the construction of $S$ as follows: $S_1$ is an isotropic subset having maximal cardinality in a maximal subdiagram of type $A$ of odd cardinality having only odd simple roots, and inductively $S_i$ is such a subset in the set of indecomposable roots of $S_{i-1}^{\perp} \setminus S_{i-1}$. This time the choice of $S$ is not unique.

**Conjecture 5.2** Let $\mathfrak{g}$ be a basic classical Lie superalgebra of defect $d$, where $\mathfrak{g} = A(d-1, d-1)$ is replaced by $\mathrm{gl}(d, d)$, and $\Delta^+$ any set of positive roots. Let $S$ be any maximal isotropic subset of $\Delta^+$ built up as above. Then

$$e^{\rho}R = \frac{1}{K} \sum_{w \in W} \operatorname{sgn}(w) w \frac{e^{\rho}}{\prod_{i=1}^{d} (1 + \operatorname{sgn}(\gamma_i) e^{-\|\gamma_i\|})},$$

$$e^{\rho} \tilde{R} = \frac{1}{K} \sum_{w \in W} \operatorname{sgn}'(w) w \frac{e^{\rho}}{\prod_{i=1}^{d} (1 - e^{-\|\gamma_i\|})},$$

where

$$K = \frac{C d!}{\prod_{\gamma \in S} \frac{ht(\gamma)+1}{2}},$$

$C$ is defined in (1.5) and $ht(\gamma)$ denotes the height of the root $\gamma$ w.r.t. to the simple roots corresponding to $\Delta^+$. Moreover, there exists a choice of $S$ for which $\|\gamma_i\|$ is a linear combination with non negative coefficients of positive roots for any $i = 1, \dots, d$.

Acknowledgement

We would like to thank Corrado De Concini for useful conversations.
References

[1] J. Adams, *The Theta correspondence over $\mathbb{R}$*, Harmonic analysis, group representations, automorphic forms and invariant theory, Lecture Notes Series, Institut für Mathematische Sciences, National University of Singapore, Vol. 12, p. 1–37

[2] T. Enright, *Analogues of Kostant's $u$-cohomology formulas for unitary highest weight modules*. J. Reine Angew. Math. 392 (1988), 27–36.

[3] M. Gorelik, *Weyl denominator identity for finite-dimensional Lie superalgebras*, arXiv:0905.1181

[4] J. Humphreys, *Representations of semisimple Lie algebras in the BGG category O*. Graduate Studies in Mathematics, 94. American Mathematical Society, Providence, RI, 2008.

[5] V. G. Kac, *Lie superalgebras*, Advances in Math. **26** (1977), no. 1, 8–96.

[6] V. G. Kac, *Representations of classical Lie superalgebras*. Differential geometrical methods in mathematical physics, II (Proc. Conf., Univ. Bonn, Bonn, 1977), pp. 597–626, Lecture Notes in Math., 676, Springer, Berlin, 1978.

[7] V. G. Kac, W. Wakimoto, *Integrable highest weight modules over affine superalgebras and number theory*. Lie theory and geometry, 415–456, Progr. Math., 123, Birkhäuser Boston, 1994.

[8] M. Kashiwara, M. Vergne, *On the Segal-Shale-Weil representations and harmonic polynomials*. In-vent. Math. **44** (1978), no. 1, 1–47.

[9] J.-S. Li, A. Paul, E.-C. Tan, C.-B. Zhu, *The explicit duality correspondence of $(Sp(p, q), O^*(2n))$*. J. Funct. Anal. 200 (2003), no. 1, 71–100.

[10] W. Schmid, *Die Randwerte holomorpher Funktionen auf hermiteschymmetrischen Räumen*, Invent. Math. **9** (1969/70), 61–80.