# Deformations of Weyl's Denominator Formula: Six Conjectures and One Result

Angèle M. Hamel <fnref n="1" />* and Ronald C. King <fnref n="2" />†

**Abstract.** We introduce a series of conjectured identities that deform Weyl's denominator formula and generalize Tokuyama's formula to other root systems. These conjectures generalize a number of well-known results due to Okada. We also prove a related result for $B'_n$ that generalizes a theorem of Simpson.

**Résumé.**

Nous proposons une série de conjectures qui sont des déformations de la formule dénominateur de Weyl et qui généralisent la formule de Tokuyama à d'autres systèmes de racines. Ces résultats sont des généralisations de théorèmes bien connus dus à Okada. Nous donnons aussi la preuve d'un résultat pour $B'_n$ qui est une généralisation d'un théorème de Simpson.

**Keywords:** Weyl's denominator formula, alternating sign matrices, lattice paths

## 1 Introduction

Weyl's denominator formula is a fundamental and well-studied identity in algebra; additionally, it is valued for its connections to combinatorics and analytic number theory. One of the most interesting generalizations of this identity is due to Tokuyama [16] who obtained a deformation involving a single free parameter $t$. Deformations of Weyl's denominator formula for the root systems of type $B_n$, $C_n$ (two cases) and $D_n$ have been discovered by Okada [11], and for $B_n$ a further deformation was obtained by Simpson [13]. For $A_n$ Tokuyama also established a remarkable identity [16] expressing a certain combinatorial sum as a product of his deformed Weyl denominator and an undeformed Weyl character of an irreducible representation of $A_n$ known as a Schur function. It is thus natural to ask for versions of Tokuyama's identity in the case of other root systems. Some cases are already known, and previous work includes [1], [2], [3], [5], [6] and [15].

The impetus for our tackling this problem comes from the 2010 BIRS workshop "Whittaker Functions, Crystal Bases, and Quantum Groups" at which Ben Brubaker asked the first author whether we could discover the deformation of the $B_n$ Weyl denominator formula, along the same lines as our previous work [6]

<footnote n="1">¹Department of Physics and Computer Science, Wilfrid Laurier University, Waterloo, Ontario, N2L 3C5, Canada</footnote>

<footnote n="2">²School of Mathematics, University of Southampton, Southampton, SO17 1BJ, United Kingdom</footnote>

<footnote>*Email: ahamel@wlu.ca.</footnote>

<footnote>†Email: r.c.king@soton.ac.uk.</footnote>
on symplectic shifted tableaux. From this query we first developed a number of conjectures for cases that we refer to as $B_n$, $B'_n$, $C_n$, $C'_n$, $D_n$, and $D'_n$ (the six conjectures) that generalise to a multi-parameter setting the one-parameter results of Okada and Simpson, and have proved a version of Tokuyama's identity for $B'_n$ (the one result). This more general result whose proof we discuss here encompasses the three $B_n$, $D_n$ and $B'_n$ conjectures that are themselves generalisations of Okada's Theorems 2.1 and 4.1 [11] and Simpson's Theorem 1 [13]. The $B_n$ case of our general result also includes as a special case the result of Tabony [15]. The paper is organized as follows: Section 2 covers definitions and previous results, Section 3 introduces the conjectures, and, finally, Section 4 gives the result and specializations.

# 2 Background

## 2.1 Half Turn Alternating Sign Matrices

An alternating sign matrix (ASM) is a square matrix with entries 0, 1, and -1 such that each row or column sum is 1, while each partial row or column sum is either 1 or 0. The setting for all these identities is a particular type of alternating sign matrix, described either as an ASM invariant under 180 degree rotation (Okada [11]) or as a half-turn ASM (Robbins [12], Kuperberg [9], Tabony [15]). We consider three different types of half-turn ASM. The variation centres around the presence or absence or content of a middle row and column. Since there is rotational symmetry, from one half of the ASM we can reconstruct the entire matrix immediately if $m$ is even, while for $m$ odd we also need the ASM conditions themselves to construct the central column. We will often use the right half of the matrix, with the turn on the left boundary, so row $i$ continues as row $m + 1 - i$. We also use $\bar{1}$ to denote $-1$.

The various types of half-turn ASM are as follows: **Type $B_n$**: Even-sided half-turn ASM (Okada [11], Tabony [15]) and **Type $B'_n$**: Odd-sided half-turn ASM—special central column (Simpson [13]); **Type $C_n$**: Odd-sided half-turn ASM (Okada [11]) and **Type $C'_n$**: Odd-sided half-turn ASM—special central row (Okada [11]); **Type $D_n$**: Even-sided half-turn ASM—special pair of central columns (Okada [11]) and **Type $D'_n$**: Odd-sided half-turn ASM—special central row. Diagrams below:

$$ \begin{array}{ccc} B_n & C_n & D_n \\ \begin{bmatrix} 0 & 1 & 0 & 0 \\ 1 & \bar{1} & 1 & 0 \\ 0 & 1 & \bar{1} & 1 \\ 0 & 0 & 1 & 0 \end{bmatrix} & \begin{bmatrix} 0 & 0 & 0 & 1 & 0 \\ 0 & 0 & 1 & 0 & 0 \\ 1 & 0 & \bar{1} & 0 & 1 \\ 0 & 0 & 1 & 0 & 0 \\ 0 & 1 & 0 & 0 & 0 \end{bmatrix} & \begin{bmatrix} 0 & \mathbf{0} & 1 & 0 \\ 1 & \mathbf{0} & 0 & 0 \\ 0 & 0 & \mathbf{0} & 1 \\ 0 & 1 & \mathbf{0} & 0 \end{bmatrix} \\ \hline \end{array} \qquad (1) $$

$$ \begin{array}{ccc} B'_n & C'_n & D'_n \\ \begin{bmatrix} 0 & 1 & \mathbf{0} & 0 & 0 \\ 0 & 0 & \mathbf{0} & 1 & 0 \\ 1 & \bar{1} & \mathbf{1} & \bar{1} & 1 \\ 0 & 1 & \mathbf{0} & 0 & 0 \\ 0 & 0 & \mathbf{0} & 1 & 0 \end{bmatrix} & \begin{bmatrix} 0 & 0 & 1 & 0 & 0 \\ 1 & 0 & \bar{1} & 1 & 0 \\ \mathbf{0} & \mathbf{0} & \mathbf{1} & \mathbf{0} & \mathbf{0} \\ 0 & 1 & \bar{1} & 0 & 1 \\ 0 & 0 & 1 & 0 & 0 \end{bmatrix} & \begin{bmatrix} \mathbf{0} & 0 & 1 & 0 & \mathbf{0} \\ \mathbf{0} & 1 & 0 & 0 & \mathbf{0} \\ \mathbf{1} & \mathbf{0} & \bar{1} & \mathbf{0} & \mathbf{1} \\ \mathbf{0} & 0 & 0 & 1 & \mathbf{0} \\ \mathbf{0} & 0 & 1 & 0 & \mathbf{0} \end{bmatrix} \end{array} $$

## 2.2 Compass Point Matrices

The combinatorial objects we deal with can also be described differently; in fact, they can be interpreted at least three different ways: as compass point matrices (CPM), as half-turn ASM, or as shifted tableaux.
**Theorem 4 (Reformulation of Okada's $C_n$ identity [11])**

$$ \prod_{i=1}^{n} (1 - tx_i)(1 + t^2 x_i) \prod_{1 \le i < j \le n} (1 - t^2 x_i x_j)(1 - t^2 x_i x_j^{-1}) = \sum_{C \equiv A \in \mathcal{C}_n} \prod_{i=1}^{n} x_i^{-i} \prod_{i=1}^{2n+1} \prod_{j=n+1}^{2n+1} \mathrm{wgt}(c_{ij}) $$

**Theorem 5 (Reformulation of Okada's $C'_n$ identity [11])**

$$ \prod_{i=1}^{n} (1 + tx_i^2) \prod_{1 \le i < j \le n} (1 + tx_i x_j)(1 + tx_i x_j^{-1}) = \sum_{C \equiv A \in \mathcal{D}_n} \prod_{i=1}^{n} x_i^{-i} \prod_{i=1}^{2n+1} \prod_{j=n+1}^{2n+1} \mathrm{wgt}(c_{ij}) \quad \text{with weights:} $$

<table><thead><tr><th>Entry</th><th>$i < n + 1$</th><th>$i = n + 1$</th><th>$i < n + 1$</th><th>$i = n + 1$</th><th>$i > n + 1$</th></tr><tr><th>at $(i, j)$</th><th>$j = n + 1$</th><th>$j = n + 1$</th><th>$j > n + 1$</th><th>$j > n + 1$</th><th>$j > n + 1$</th></tr></thead><tbody><tr><td>WE</td><td>$x_i$</td><td>1</td><td>$t^{-1/2} x_i$</td><td></td><td>$x_{2n+2-i}$</td></tr><tr><td>NS</td><td>$(1 + t)$</td><td></td><td>$t^{1/2} (1 + t)$</td><td></td><td>$(1 + t)$</td></tr><tr><td>NE</td><td>$t^{1/2}$</td><td></td><td>$t^{1/2}$</td><td>$t^{1/2}$</td><td>$t^{1/2} x_{2n+2-i}$</td></tr><tr><td>SE</td><td>1</td><td></td><td>1</td><td>1</td><td>$x_{2n+2-i}$</td></tr><tr><td>NW</td><td>$x_i$</td><td></td><td>$x_i$</td><td></td><td>1</td></tr><tr><td>SW</td><td>$t^{1/2} x_i$</td><td></td><td>$t^{1/2} x_i$</td><td></td><td>$t^{1/2}$</td></tr></tbody></table>

**Theorem 6 (Reformulation of Simpson's $B'_n$ identity [13])**

$$ \prod_{i=1}^{n} (1 + tx_i) \prod_{1 \le i < j \le n} (1 + tx_i x_j)(1 + tx_i x_j^{-1}) = \sum_{C \equiv A \in \mathcal{B}'_n} \prod_{i=1}^{n} x_i^{-i} \prod_{i=1}^{2n+1} \prod_{j=n+1}^{2n+1} \mathrm{wgt}(c_{ij}) $$

A further multi-parameter extension of these identities takes the following form:

**Theorem 7 (Tabony's Generalisation of Okada's $B_n$ identity [15])**

$$ \prod_{i=1}^{n} (1 - t_i x_i) \prod_{1 \le i < j \le n} (1 - t_i t_j x_i x_j)(1 - t_i t_j x_i x_j^{-1}) = \sum_{C \equiv A \in \mathcal{B}_n} \prod_{i=1}^{n} x_i^{-i} \prod_{i=1}^{2n} \prod_{j=n+1}^{2n} \mathrm{wgt}(c_{ij}) $$

## 3 Six Conjectures

The third kind of object that indexes these identities is the set of primed shifted tableaux (PSTs). For fixed $n \in \mathbb{N}$ let $\lambda = (\lambda_1, \lambda_2, \dots, \lambda_n)$ be a strict partition with $n$ distinct parts. Then PSTs of shape $\lambda$ are arrays of boxes with rows of lengths $\lambda_i$ left adjusted to a diagonal line. Their boxes are filled with entries from the set $1' < 1 < 2' < 2 < 3' < 3 < \dots < n' < n < 0' < 0 < \bar{n}' < \bar{n} < \bar{n} - 1' < \bar{n} - 1 < \dots < \bar{1}' < \bar{1}$ with these rules: 1) entries weakly increase across rows and down columns, 2) entries strictly increase down left-to-right diagonals, 3) at most one $k'$ and at most one $\bar{k}'$ appears in each row, 4) at most one $k$ and at most one $\bar{k}$ appears in each column, 5) exactly one of $\{k, \bar{k}\}$ appears on the main diagonal for all $1 \le k \le n$. We use the notation $P \sim A$ to indicate a PST is associated with an ASM.
<figure><img src="image_1.png" /><figcaption>**Fig. 1:** A PST and its associated half-turn ASM and CPM. Only the right hand portion of the ASM and CPM are represented, as the left hand portion is deducible by symmetry and the central column by a row sum argument.</figcaption></figure>

There are a six cases to consider for our conjectures, each associated with a type of λ-HTASM described in section 2.1. We denote these ASMs by $\mathcal{A}(X_n)$ with $X_n = B_n, B'_n, C_n, C'_n, D_n$ and $D'_n$. To each $A \in \mathcal{A}(X_n)$ we associate a corresponding CPM $C$ and $2^m$ PSTs $P$, where the multiplicity $2^m$ is generally, but not always, the number of entries $-1$ in $A$. Define $\mathcal{P}$ as the set of all partitions and $\mathcal{C}$ as the set of all partitions with arm lengths exceeding leg lengths by 1 in Frobenius notation. For $\lambda = \mu + \delta$ where $\delta = (n, n-1, \dots, 1)$ is a staircase partition and $\mu$ is a partition of length $\ell(\lambda) \le n$, the sets of PSTs of shapes $\lambda$ and $\delta$ are signified by $\mathcal{P}_{(X_n)}^{\lambda}$ and $\mathcal{P}_{(X_n)}^{\delta}$. We let $\mathbf{tx} = (t_1x_1, t_2x_2, \dots, t_nx_n)$ and $\mathbf{t}\bar{x} = (t_1/x_1, t_2/x_2, \dots, t_n/x_n)$.

With this notation we have

$$ \sum_{P \in \mathcal{P}_{(X_n)}^{\lambda}} \mathrm{wtg}(P) = \sum_{P \in \mathcal{P}_{(X_n)}^{\delta}} \mathrm{wtg}(P) \Delta_{X_n}^{\mu}(\mathbf{x}; \mathbf{t}) \quad \text{where} \quad \sum_{P \in \mathcal{P}_{(X_n)}^{\delta}} \mathrm{wtg}(P) = \Phi_{X_n}(\mathbf{x}; \mathbf{t}), \quad (4) $$

with $\Phi_{X_n}(\mathbf{x}; \mathbf{t})$ a type of deformation of the factorised form of a Weyl denominator formula with the exponents of $\mathbf{x}$ defining the roots and $\mathbf{t}$ a sequence of deformation parameters, while $\Delta_{X_n}^{\mu}(\mathbf{x}; \mathbf{t})$ is a type of deformation of a Weyl character of an orthogonal or symplectic algebra. The various $\Delta_{X_n}$ and $\Phi_{X_n}$ are tabulated in Figure 2. Here we have adopted the notation $\bar{z} = z^{-1}$ for any $z$.

# 4 One Result

Related to the six conjectures given in Section 3 is the following result that takes the form of a Tokuyama type identity for the root system of type $B_n$:

**Theorem 8** Let $\mathbf{sx} = (s_1x_1, s_2x_2, \dots, s_nx_n)$ and $\mathbf{t}\bar{x} = (t_1x_1^{-1}, t_2x_2^{-1}, \dots, t_nx_n^{-1})$. Let $\lambda = \mu + \delta$ with $\mu \in \mathcal{P}$ of length $\ell(\mu) \le n$ and $\delta = (n, n-1, \dots, 1)$. Then we have

$$ \sum_{P \sim A \in B_n^{\lambda}} \mathrm{wtg}(P) = \sum_{P \sim A \in B_n^{\delta}} \mathrm{wtg}(P) \sum_{\gamma \in \mathcal{C}} s_{\mu/\gamma}(t_0, \mathbf{sx}, \mathbf{t}\bar{x}). \quad (5) $$
<table><caption>Fig. 2: Form of the six conjectures</caption><thead><tr><th>X<sub>n</sub></th><th>Type of half-turn ASM</th></tr></thead><tbody><tr><td>B<sub>n</sub></td><td>Even sided: side 2n<br>Φ<sub>X<sub>n</sub></sub> = &prod;<sub>i=1</sub><sup>n</sup>(1 - t<sub>i</sub>x<sub>i</sub>) &prod;<sub>1 &le; j &lt; k &le; n</sub>(1 - t<sub>j</sub>t<sub>k</sub>x<sub>j</sub>x<sub>k</sub>)(1 - t<sub>j</sub>t<sub>k</sub>x<sub>j</sub>x<sub>k</sub><sup>-1</sup>)<br>Δ<sub>X<sub>n</sub></sub> = (√-1)<sup>|μ|</sup> &sum;<sub>γ &isin; C</sub>(-1)<sup>|γ|/2</sup> s<sub>μ/γ</sub>(1, tx, t<span style="text-decoration: overline">x</span>)</td></tr><tr><td>B'<sub>n</sub></td><td>Odd sided: side 2n + 1; central column (0<sup>n</sup>, 1, 0<sup>n</sup>)<br>Φ<sub>X<sub>n</sub></sub> = &prod;<sub>i=1</sub><sup>n</sup>(1 + t<sub>0</sub>t<sub>i</sub>x<sub>i</sub>) &prod;<sub>1 &le; j &lt; k &le; n</sub>(1 + t<sub>j</sub>t<sub>k</sub>x<sub>j</sub>x<sub>k</sub>)(1 + t<sub>j</sub>t<sub>k</sub>x<sub>j</sub>x<sub>k</sub><sup>-1</sup>)<br>Δ<sub>X<sub>n</sub></sub> = &sum;<sub>γ &isin; C</sub> s<sub>μ/γ</sub>(t<sub>0</sub>, tx, t<span style="text-decoration: overline">x</span>)</td></tr><tr><td>C<sub>n</sub></td><td>Odd sided: side 2n + 1<br>Φ<sub>X<sub>n</sub></sub> = &prod;<sub>i=1</sub><sup>n</sup>(1 - t<sub>i</sub>x<sub>i</sub>)(1 - t<sub>0</sub>t<sub>i</sub>x<sub>i</sub>) &prod;<sub>1 &le; j &lt; k &le; n</sub>(1 + t<sub>j</sub>t<sub>k</sub>x<sub>j</sub>x<sub>k</sub>)(1 + t<sub>j</sub>t<sub>k</sub>x<sub>j</sub>x<sub>k</sub><sup>-1</sup>)<br>Δ<sub>X<sub>n</sub></sub> = (√-1)<sup>|μ|</sup> &sum;<sub>γ &isin; C</sub>(-1)<sup>|γ|/2</sup> s<sub>μ/γ</sub>(1, t<sub>0</sub>, tx, t<span style="text-decoration: overline">x</span>)</td></tr><tr><td>C'<sub>n</sub></td><td>Odd sided: side 2n + 1; central row (0<sup>n</sup>, 1, 0<sup>n</sup>)<br>Φ<sub>X<sub>n</sub></sub> = &prod;<sub>i=1</sub><sup>n</sup>(1 + t<sub>i</sub><sup>2</sup>x<sub>i</sub><sup>2</sup>) &prod;<sub>1 &le; j &lt; k &le; n</sub>(1 + t<sub>j</sub>t<sub>k</sub>x<sub>j</sub>x<sub>k</sub>)(1 + t<sub>j</sub>t<sub>k</sub>x<sub>j</sub>x<sub>k</sub><sup>-1</sup>)<br>Δ<sub>X<sub>n</sub></sub> = &sum;<sub>α &isin; A</sub> s<sub>μ/α</sub>(tx, t<span style="text-decoration: overline">x</span>)</td></tr><tr><td>D<sub>n</sub></td><td>Even sided: side 2n, pair central columns (0<sup>n</sup>, 1, 0<sup>n-1</sup>) and (0<sup>n-1</sup>, 1, 0<sup>n</sup>)<br>Φ<sub>X<sub>n</sub></sub> = &prod;<sub>1 &le; j &lt; k &le; n</sub>(1 + t<sub>j</sub>t<sub>k</sub>x<sub>j</sub>x<sub>k</sub>)(1 + t<sub>j</sub>t<sub>k</sub>x<sub>j</sub>x<sub>k</sub><sup>-1</sup>)<br>Δ<sub>X<sub>n</sub></sub> = &sum;<sub>γ &isin; C</sub> s<sub>μ/γ</sub>(tx, t<span style="text-decoration: overline">x</span>)</td></tr><tr><td>D'<sub>n</sub></td><td>Odd sided: side 2n + 1, central row (1, 0<sup>n-1</sup>, -1, 0<sup>n-1</sup>, 1)<br>Φ<sub>X<sub>n</sub></sub> = &prod;<sub>1 &le; j &lt; k &le; n</sub>(1 + t<sub>j</sub>t<sub>k</sub>x<sub>j</sub>x<sub>k</sub>)(1 + t<sub>j</sub>t<sub>k</sub>x<sub>j</sub>x<sub>k</sub><sup>-1</sup>)<br>Δ<sub>X<sub>n</sub></sub> = &sum;<sub>γ &isin; C</sub> s<sub>λ/γ</sub>(tx, t<span style="text-decoration: overline">x</span>) with μ = (μ<sub>1</sub>, λ)</td></tr></tbody></table>
with

$$ \sum_{P \sim A \in B'_n^{\delta}} \operatorname{wtg}(P) = \prod_{i=1}^{n} (1 + t_0 s_i x_i) \prod_{1 \le i < j \le n} (1 + s_i s_j x_i x_j) (1 + s_i t_j x_i \bar{x}_j). \quad (6) $$

where the assignment of weights to $P$ is given below in (8).

Consider the odd-sided half-turn alternating sign matrices $A$ with special central column $(0^n 10^n)$ that are associated with the $B'_n$ case introduced by Simpson and that appear in (1). Brubaker and Schultz [4] have defined a universal weighting in this case for the compass points matrix. We reinterpret this universal weighting in a form more amenable to primed shifted tableaux as follows:

$$ \begin{array}{c|lll} \text{Entry} & 1 \le i \le n & i = n \\ \text{at } (i, j) & n + 2 \le j \le 2n + 1 & n + 2 \le j \le 2n + 1 \\ \hline \text{WE} & 1 & 1 \\ \text{NS} & (s_i + t_i)x_i & (t_0 + \bar{t}_0) \\ \text{NE} & 1 & 1 \\ \text{SE} & 1 & 1 \\ \text{NW} & \bar{t}_i x_i & \bar{t}_0 \\ \text{SW} & s_i x_i & t_0 \\ \end{array} \qquad \begin{array}{l} 1 \\ (s_{2n+2-i} + t_{2n+2-i})\bar{x}_{2n+2-i} \\ 1 \\ 1 \\ \bar{s}_{2n+2-i}\bar{x}_{2n+2-i} \\ t_{2n+2-i}\bar{x}_{2n+2-i} \\ \end{array} \qquad (7) $$

together with an additional factor of $t_0^{L_{n+1}} \prod_{i=1}^n (t_i \bar{x}_i)^{L_i} (s_i x_i)^{L_{2n+2-i}}$, where $L_i$ is the number of entries $WE, NW, SW$ above the entry $c_{i,n+1}$ of $C$ for $i = 1, 2, \dots, 2n+1$. However, $L_{2n+2-i} = n-i+L_{i+1}$ and the additional factor can then be written in the form $\prod_{i=1}^n (s_i x_i)^{n-i} (s_i t_i)^{L_i} (t_0 s_i x_i)^{L_{i+1}-L_i}$, where $L_{i+1}-L_i$ is 1 if $c_{i,n+1} \in \{WE, NW, SW\}$, and 0 otherwise.

The above weighting of compass point matrices $C$ leads directly to the following weighting of primed shifted tableaux $P$, together with additional factors $\prod_{i=1}^n (s_i x_i)^{n-i} \prod_{j=1}^n (s_j t_j)^{L_j}$.

$$ \begin{array}{c|lllll} \text{Diagonal entries} & k \mapsto t_0 s_k x_k & \bar{k} \mapsto 1 & & & \\ \text{Off-diagonal entries} & k \mapsto s_k x_k & k' \mapsto \bar{t}_k x_k & \bar{k} \mapsto t_k \bar{x}_k & \bar{k}' \mapsto \bar{s}_k \bar{x}_k & 0 \mapsto t_0 & 0' \mapsto \bar{t}_0 \\ \end{array} \qquad (8) $$

There also exist primed orderly tableaux $D \in \mathcal{D}^\delta$ that are of shifted shape $\delta$ and are filled with entries in accordance with the two rules: (1) that the entry at each main diagonal position $(i, i)$ is either $i$ or $\bar{i}$; and (2) that the entry at each off-diagonal position $(i, j)$ with $i < j$ is either $i, \bar{i}, j'$ or $\bar{j}'$. They may be assigned the following weights for all possible entries in position $(i, j)$ in $D$:

$$ \begin{array}{c|lllll} \text{Diagonal entries in position } (i, i) & i \mapsto t_0 s_i x_i & \bar{i} \mapsto 1 & & & \\ \text{Off-diagonal entries in position } (i, j) & i \mapsto (s_j t_j) s_i x_i & \bar{i} \mapsto \bar{s}_i \bar{x}_i & \bar{j}' \mapsto s_j x_j & \bar{j}' \mapsto t_j \bar{x}_j & \\ \end{array} \qquad (9) $$

together with the an overall factor of $\prod_i (s_i x_i)^{n-i}$ previously encountered in connection with $\operatorname{wtg}(P)$. This weighting offers an alternative combinatorial realisation of the right hand side of (6) since it implies that $\sum_{D \in \mathcal{D}^\delta} \operatorname{wtg}(D) = \prod_{i=1}^n (s_i x_i)^{n-i} \prod_{i=1}^n (1 + t_0 s_i x_i) \prod_{1 \le i < j \le n} (s_i s_j t_j x_i + \bar{s}_i \bar{x}_i + s_j x_j + t_j \bar{x}_j)$ which can be rewritten in the required form $\prod_{i=1}^n (1 + t_0 s_i x_i) \prod_{1 \le i < j \le n} (1 + s_i s_j x_i x_j) (1 + s_i t_j x_i \bar{x}_j)$.
## 4.1 Proof of Theorem 8.

The proof of Theorem 8 relies on lattice paths and follows the approach of Okada [10] (which contains one of the early proofs of Tokuyama's result [16]). Note that we could have also followed a different path, so to speak, making use of Theorem 4.2, equation (32), of Ishikawa and Wakayama [8] instead. Their theorem is a more general case of Stembridge's result [14]; however, their theorem holds in the case when the starting and ending points are not $D$-compatible (as defined in [14]), which is the situation we would construct here. The connection to [8] and [14] implies there would be Pfaffian results as well, and these will be the subject of some of our future work.

<figure><img src="image_2.png" /><figcaption>Fig. 3: Example of the lattice paths for the given tableau, in the case $n = 3$, $\delta = (3, 2, 1)$ and $\mu = (5, 4, 2)$.</figcaption></figure>

We will apply Okada's theorem to the situation with starting points $u_k = (-n+k, 0)$ and ending points $v_k = (\mu_k + n - k + 1, \bar{0})$ for $k = 1, 2, \dots, n$, and the following edge weights on curved, horizontal, and diagonal edges (vertical edges weighted 1):

1) For a curved edge from $(-n+k, 0)$ to $(0, k)$ the weight is $t_0 s_k x_k \prod_{i=k+1}^{n} s_i t_i$, and from $(-n+k, 0)$ to $(0, \bar{k})$, the weight is 1.

2) For a horizontal edge for unbarred nonzero $i$ from $(j, i)$ to $(j+1, i)$, the weight is $s_i x_i$. For a horizontal edge for barred nonzero $i$ from $(j, \bar{i})$ to $(j+1, \bar{i})$, the weight is $t_i \bar{x}_i$. For a horizontal edge for $0$ from $(j, 0)$ to $(j+1, 0)$, the weight is $t_0$.

3) For a diagonal edge for unbarred nonzero $i$ from $(j, i-1)$ to $(j+1, i)$ the weight is $\bar{t}_i x_i$. For a diagonal edge for barred nonzero $i$ from $(j, i+1)$ to $((j, \bar{i})$ the weight is $\bar{s}_i \bar{x}_i$. For a diagonal edge for $0$ from $(j, n)$ to $(j+1, 0)$ the weight is $\bar{t}_0$.

Then we assert that with this weighting, the set of non-intersecting lattice paths from $u$ to $v$ gives the generating function for the required sum of weights of primed shifted tableaux. In particular, the row starting with $k$ or $\bar{k}$ corresponds to the lattice path starting at $(-n+k, 0)$. The generating function for the weights of all possible rows starting with $k$ in the tableaux can be expressed as

$$ f_{k,l}(q_l) := \frac{t_0 s_k x_k q_l}{1 - s_k x_k q_l} \frac{1 + \bar{t}_0 q_l}{1 - t_0 q_l} \prod_{i=k+1}^{n} s_i t_i \frac{1 + \bar{t}_i x_i q_l}{1 - s_i x_i q_l} \prod_{i=1}^{n} \frac{1 + \bar{s}_i \bar{x}_i q_l}{1 - t_i \bar{x}_i q_l} \quad (11) $$

**Justification:** The first (curved) step is at height $k$ and has weight $t_0 s_k x_k \prod_{i=k+1}^{n} s_i t_i$. After that
we cannot have any diagonal steps at height $k$ but we can have horizontal steps. These are counted by $(1-s_k x_k q_l)^{-1}$. Then we can have the remaining steps from any height bigger than $k$. We can have at most one of these as a diagonal step, and these are counted by $(1 + \bar{t}_i x_i q_l)$ for unbarred entries, by $(1 + \bar{t}_0 q_l)$ for zero entries, and by $(1 + \bar{s}_i \bar{x}_i q_l)$ for barred entries, or we can have many horizontal steps, and these are counted by $(1 - s_i x_i q_l)^{-1}$ for unbarred entries, by $(1 - t_0 q_l)$ for zero entries, and by $(1 - t_i \bar{x}_i q_l)^{-1}$ for barred entries.

A similar argument implies that the generating function for the weights of all possible rows starting with $\bar{k}$ in the tableaux can be expressed as

$$ g_{k,l}(q_l) := \frac{q_l}{1 - t_k \bar{x}_k q_l} \prod_{i=1}^{k-1} \frac{1 + \bar{s}_i \bar{x}_i q_l}{1 - t_i \bar{x}_i q_l}. \qquad (12) $$

In the expansion of each of these generating functions $f_{k,l}(q_l)$ and $g_{k,l}(q_l)$ the parameter $q_l$ carries the total number of steps to the right in each lattice path, and thereby its exponent gives the number of the column in which each lattice path terminates. In dealing with lattice paths associated with primed shifted tableaux of shape $\lambda$ it is convenient to use the notation whereby $[q_l^{\lambda_l}]$ picks out the coefficient of $q_l^{\lambda_l}$ in the expansion of these generating functions and $[q^{\lambda}]$ the coefficient of $q_1^{\lambda_1} q_2^{\lambda_2} \cdots q_n^{\lambda_n}$. Putting all this information together, and for simplicity writing $P \in \mathcal{P}^{\lambda}(n)$ for $P \sim A \in \mathcal{B}'_{n}^{\lambda}$, we arrive at the following:

**Theorem 9** For all $n \in \mathbb{N}$ let $\lambda = \mu + \delta$ where $\delta = (n, n-1, \dots, 1)$ with $\mu$ a partition of length $\ell(\mu) \le n$. Then

$$ \sum_{P \in \mathcal{P}^{\lambda}(n)} \mathrm{wgt}(P) = [q^{\lambda}] \det(h_{k,l}(q_l)) \quad \text{where } h_{k,l}(q_l) = f_{k,l}(q_l) + (-1)^{n-k} g_{k,l}(q_l). $$

**Proof:** Following Okada we refer to the sequence of entries $\mathbf{d} = (d_1, d_2, \dots, d_n)$ on the main diagonal of $P$ as its profile, and note that there are $2^n$ distinct profiles $\mathbf{d}$ of those $P$ of shape $\lambda$, where each such profile $\mathbf{d}$ contains either $k$ or $\bar{k}$ but not both. From the argument in [10], the generating function for all non-intersecting paths for a particular profile $\mathbf{d}$ is $\det(f_{kl}(q_l))$ if $k \in \mathbf{d}$ and $\det(g_{kl}(q_l))$ if $\bar{k} \in \mathbf{d}$.

Let $\pi$ be the permutation mapping $\mathbf{d}$ to $\mathbf{d}^{\pi}$ where $\mathbf{d}^{\pi} = k$ or $\bar{k}$. For each allowed profile $\mathbf{d}$ the permutation $\pi$ is a product of cycles having parity $(-1)^{n-k}$ moving $\bar{k}$ to position $k$. Summing over all distinct profiles $\mathbf{d}$ and permuting rows and combining determinants one arrives at $\sum_{\mathbf{d}} \det(\tilde{h}_{kl}(q_l)) = \det(f_{kl}(q_l) + (-1)^{n-k} g_{kl}(q_l))$. The proof is completed by restricting attention to those PST of shape $\lambda$ in which case one picks out the coefficient of $q_l^{\lambda_l}$ from each term in the $l$th column. Since $q_l$ only appears in this column, the operator $[q^{\lambda}]$ may be taken out of the determinant, yielding the result. $\square$

Now it remains to expand the determinant on the right hand side of Theorem 9.

**Lemma 10** For all $n \in \mathbb{N}$, let $\mathbf{z} = (s_1 x_1, \dots, s_n x_n, t_0, t_1 \bar{x}_1, \dots, t_n \bar{x}_n)$ and $\mathbf{q} = (q_1, q_2, \dots, q_n)$. Then

$$ \det(h_{k,l}(q_l)) = D(\mathbf{z}) K(\mathbf{z}, \mathbf{q}) C(\mathbf{q}) $$

where

$$ K(\mathbf{z}, \mathbf{q}) = \frac{1}{\prod_{j=1}^{n} ((1 - q_j t_0) \prod_{i=1}^{n} (1 - q_j s_i x_i)(1 - q_j t_i \bar{x}_i))}, $$

$$ C(\mathbf{q}) = \prod_{i=1}^{n} q_i \prod_{1 \le i < j \le n} (q_i - q_j) \prod_{1 \le i \le j \le n} (1 + q_i q_j), $$
and $D(\mathbf{z})$ is some multinomial in the components of $\mathbf{z}$ independent of $\mathbf{q}$.

**Proof:** First remove the factor $1/((1-t_0q_j) \prod_{i=1}^n (1-s_ix_iq_j)(1-t_i\bar{x}_i\bar{q}_j))$ from each element in the $j$th column. This gives $\det(h_{k,l}(q_l)) = K(\mathbf{z}, \mathbf{q}) \det(\tilde{h}_{k,l}(q_l))$ where $\tilde{h}_{k,l}(q_l) = \tilde{f}_{k,l}(q_l) + \tilde{g}_{k,l}(q_l)$. with

$$ 
\begin{aligned}
\tilde{f}_{k,l}(q_l) &= (t_0 s_k x_k q_l)(1 + \bar{t}_0 q_l) \prod_{i=1}^{k-1} (1 - s_i x_i q_l) \prod_{\substack{i=k+1 \\ k-1}}^{n} s_i t_i (1 + \bar{t}_i x_i q_l) \prod_{i=1}^{n} (1 + \bar{s}_i \bar{x}_i q_l), \\
\tilde{g}_{k,l}(q_l) &= (-1)^{n-k} q_l (1 - t_0 q_l) \prod_{i=1}^{n} (1 - s_i x_i q_l) \prod_{i=1}^{n} (1 + \bar{s}_i \bar{x}_i q_l) \prod_{i=k+1}^{n} (1 - t_i \bar{x}_i q_l).
\end{aligned}
\quad (13) $$

This implies that $\tilde{h}_{k,l}(q_l)$ contains $q_l$ as a common factor and is a polynomial in $q_l$ of degree $2n+1$. It follows that $\det(\tilde{h}_{k,l}(q_l))$ contains $\prod_{i=1}^n q_i$ as a factor and is also polynomial in $q_l$ for each $l = 1, 2, \dots, n$. Moreover, for each pair $(i, j)$ with $1 \le i < j \le n$, if $q_i = q_j$ then the $i$th and $j$th columns of the matrix $(\tilde{h}_{k,l}(q_l))$ are identical and $\det(\tilde{h}_{k,l}(q_l)) = 0$. It follows that the expansion of $\det(\tilde{h}_{k,l}(q_l))$ must contain the factor $a_\delta(\mathbf{q}) = \prod_{i=1}^n q_i \prod_{1 \le i < j \le n} (q_i - q_j)$.

To see that $\det(\tilde{h}_{k,l}(q_l))$ also contains factors $(1+q_l^2)$ note that $\tilde{h}_{k,l}(q_l)$ can be rewritten as

$$ \tilde{h}_{k,l}(q_l) = \left( t_0 q_l^{n+1} \prod_{1=1}^{k-1} \bar{s}_i \bar{x}_i \prod_{i=k+1}^{n} t_i \bar{x}_i \right) \left( \tilde{d}_{k,l}(q_l) - \tilde{d}_{k,l}(-\bar{q}_l) \right) \quad (14) $$

where $\tilde{d}_{k,l} = (1 + \bar{t}_0 q_l)(1 + s_k x_k \bar{q}_l) \prod_{i=1}^{k-1} (1 - s_i x_i q_l)(1 + s_i x_i \bar{q}_l) \prod_{i=k+1}^{n} (1 + s_i x_i \bar{q}_l)(1 + \bar{t}_i x_i q_l)$.

It follows that $\tilde{h}_{k,l}(q_l) = 0$ if $q_l = -\bar{q}_l$. Since $\tilde{h}_{k,l}(q_l)$ is a polynomial in $q_l$ it follows that $\tilde{h}_{k,l}(q_l)$ contains a factor $(1+q_l^2)$ for all $k=1, 2, \dots, n$.

In addition, it follows from (14) that for any pair $(i, j)$ with $1 \le i < j \le n$ the $i$th and $j$th columns of $\tilde{h}_{k,l}(q_l)$ will be proportional to one another if $q_i = q_j$ or $q_i = -\bar{q}_j$. In such a case $\det(\tilde{h}_{k,l}(q_l)) = 0$. Hence $\det(\tilde{h}_{k,l}(q_l))$ must not only contain the factor $q_i - q_j$ found previously, but also the factor $q_i + \bar{q}_j$. The polynomial nature of $\det(\tilde{h}_{k,l}(q_l))$ then implies that it contains a factor $(1+q_i q_j)$. To summarise, we have shown that the expansion of $\det(\tilde{h}_{k,l}(q_l))$ must contain the factor $C(\mathbf{q}) = \prod_{i=1}^n q_i \prod_{1 \le i < j \le n} (q_i - q_j) \prod_{1 \le i \le j \le n} (1+q_i q_j)$. However, like $\det(\tilde{h}_{k,l}(q_l))$, this is of degree $2n+1$ in $q_l$ for each $l=1, 2, \dots, n$. It follows that the quotient $\det(\tilde{h}_{k,l}(q_l))/C(\mathbf{q})$ is independent of $q$ that is to say $\det(\tilde{h}_{k,l}(q_l)) = C(\mathbf{q}) D(\mathbf{z})$ for some $D(\mathbf{z})$ independent of $\mathbf{q}$. $\square$

Lemma 10 enables us to establish:

**Theorem 11** For all $n \in \mathbb{N}$ let $\lambda = \mu + \delta$ where $\delta = (n, n-1, \dots, 1)$ with $\mu$ a partition of length $\ell(\mu) \le n$. Then

$$ \sum_{P \in \mathcal{P}^\lambda(n)} \mathrm{wtg}(P) = \sum_{P \in \mathcal{P}^\delta(n)} \mathrm{wtg}(P) \sum_{\gamma \in \mathcal{C}} s_{\mu/\gamma}(\mathbf{z}) $$
**Proof:** Our previous results taken together imply that

$$
\begin{align*}
\sum_{P \in \mathcal{P}^{\lambda}(n)} \operatorname{wtg}(P) &= [\mathbf{q}^{\lambda}] \det(h_{k,l}(q_l)) = [q^{\lambda}] K(\mathbf{z}, \mathbf{q}) C(\mathbf{q}) D(\mathbf{z}) \\
&= D(\mathbf{z}) [q^{\lambda}] \sum_{\nu \in \mathcal{P}} s_{\nu}(\mathbf{z}) s_{\nu}(\mathbf{q}) a_{\delta}(q) \sum_{\gamma \in \mathcal{C}} s_{\gamma}(\mathbf{q}) = D(\mathbf{z}) [q^{\lambda}] \sum_{\mu \in \mathcal{P}} \sum_{\gamma \in \mathcal{C}} s_{\mu/\gamma}(\mathbf{z}) a_{\delta}(\mathbf{q}) s_{\mu}(\mathbf{q}) \\
&= D(\mathbf{z}) \sum_{\mu \in \mathcal{P}} \sum_{\gamma \in \mathcal{C}} s_{\mu/\gamma}(\mathbf{z}) [q^{\lambda}] a_{\mu+\delta}(\mathbf{q}) = D(\mathbf{z}) \sum_{\gamma \in \mathcal{C}} s_{\mu/\gamma}(\mathbf{z}).
\end{align*}
$$

where use has been made of the following facts about Schur functions: (i) $K(\mathbf{z}, \mathbf{q}) = \sum_{\nu \in \mathcal{P}} s_{\nu}(\mathbf{z}) s_{\nu}(\mathbf{q})$;
(ii) $C(\mathbf{q}) = \sum_{\gamma \in \mathcal{C}} a_{\delta}(\mathbf{q}) s_{\gamma}(\mathbf{q})$; (iii) $s_{\nu}(\mathbf{q}) s_{\gamma}(\mathbf{q}) = \sum_{\lambda \in \mathcal{P}} c_{\nu\gamma}^{\lambda} s_{\lambda}(\mathbf{q})$; (iv) $\sum_{\lambda \in \mathcal{P}} c_{\nu\gamma}^{\lambda} s_{\nu}(\mathbf{z}) = s_{\lambda/\gamma}(\mathbf{z})$;
(v) $a_{\delta}(\mathbf{q}) s_{\lambda}(\mathbf{q}) = a_{\lambda+\delta}(\mathbf{q})$; (vi) $[q^{\lambda}] a_{\mu+\delta}(\mathbf{q}) = \delta_{\lambda, \mu+\delta}$. Then setting $\mu = 0$ gives $\sum_{P \in \mathcal{P}^{\delta}(n)} \operatorname{wtg}(P) = D(\mathbf{z})$, thereby completing the proof. $\square$

Since $\mathbf{z} = (t_0, \mathbf{s}\mathbf{x}, \mathbf{t}\bar{\mathbf{x}})$, this theorem immediately yields the first part (5) of our main result Theorem 8.
To complete the story it only remains to verify that $D(\mathbf{z})$ is given by the right hand side of (6). Although
we have not been able to derive a combinatorial proof of this through a bijective map from primed shifted
tableaux to orderly shifted tableaux, we offer an algebraic proof in [7] and a second independent proof is
to be found in [4], thereby completing the proof of Theorem 8. This then implies the following

**Theorem 12** For all $n \in \mathbb{N}$ and any strict partition $\lambda = \mu + \delta$

$$
\sum_{P \in \mathcal{P}^{\lambda}(n)} \operatorname{wtg}(P) = \sum_{D \in \mathcal{D}^{\delta}} \operatorname{wtg}(D) \sum_{\gamma \in \mathcal{C}} s_{\mu/\gamma}(\mathbf{z})
$$

where the first summation on the right is over all primed orderly tableaux of shape $\delta$ weighted as in (9).

## 4.2 Specializations of Theorem 8

<table>
  <tbody>
    <tr>
      <td>Case &mu; &ne; 0</td>
      <td>
        s<sub>i</sub> &mapsto; t<sub>i</sub> (Fig. 2, B'<sub>n</sub> conjecture)<br/>
        s<sub>i</sub> &mapsto; &radic;-1t<sub>i</sub>, t<sub>0</sub> &mapsto; &radic;-1, t<sub>i</sub> &mapsto; &radic;-1t<sub>i</sub> (Fig. 2, B<sub>n</sub> conjecture)<br/>
        s<sub>i</sub> &mapsto; t<sub>i</sub>, t<sub>0</sub> &mapsto; 0 (Fig. 2, D<sub>n</sub> conjecture)<br/>
        s<sub>i</sub> &mapsto; &radic;-1t, t<sub>0</sub> &mapsto; &radic;-1, t<sub>i</sub> &mapsto; &radic;-1t (Tabony's extended B<sub>n</sub> identity, [15])
      </td>
    </tr>
    <tr>
      <td>Case &mu; = 0</td>
      <td>
        s<sub>i</sub> &mapsto; &radic;-1t<sub>i</sub>, t<sub>0</sub> &mapsto; &radic;-1, t<sub>i</sub> &mapsto; &radic;-1t<sub>i</sub> (Theorem 7, Tabony's B<sub>n</sub> identity)<br/>
        s<sub>i</sub> &mapsto; t, t<sub>0</sub> &mapsto; 1, t<sub>i</sub> &mapsto; t (Theorem 6, Simpson's B'<sub>n</sub> identity, )<br/>
        s<sub>i</sub> &mapsto; &radic;-1t, t<sub>0</sub> &mapsto; &radic;-1, t<sub>i</sub> &mapsto; &radic;-1t (Theorem 1, Okada's B<sub>n</sub> identity, )<br/>
        s<sub>i</sub> &mapsto; t, t<sub>0</sub> &mapsto; 0, t<sub>i</sub> &mapsto; t (Theorem 2, Okada's D<sub>n</sub> identity,)
      </td>
    </tr>
  </tbody>
</table>

Acknowledgements

We are grateful to the organizers of the Banff International Research Station for Mathematical Innovation and Discovery (BIRS) 2010 5-day workshop, “Whittaker Functions, Crystal Bases, and Quantum Groups” where the first author (AMH) first learned of this problem. She also acknowledges the support of a Discovery Grant from the Natural Sciences and Engineering Research Council of Canada (NSERC). The second author (RCK) is grateful for the hospitality extended to him while visiting Wilfrid Laurier University, and for the financial support making such visits possible.
References

[1] B. Brubaker, D. Bump, S. Friedberg, Schur polynomials and the Yang-Baxter equation, *Comm. in Math. Physics* **308** (2011), 1563–1571.

[2] B. Brubaker, D. Bump, G. Chinta, S. Friedberg, P.E. Gunnells, Metaplectic ice, in *Multiple Dirichlet Series, L-Functions, and Automorphic Forms*, D. Bump, S. Friedberg, and D. Goldfield, eds., *Progress in Math. 300* Birkhäuser Boston, 2012, 65–92.

[3] B. Brubaker, D. Bump, G. Chinta, P.E. Gunnells, Metaplectic Whittaker functions and crystals of type B, in *Multiple Dirichlet Series, L-Functions, and Automorphic Forms*, D. Bump, S. Friedberg, and D. Goldfield, eds., *Progress in Math. 300* Birkhäuser Boston, 2012, 93–118.

[4] B. Brubaker, A. Schultz, Deformations of the Weyl character formula for classical groups and the six-vertex model, arXiv:1402.2339v1[math.CO], 2014.

[5] G. Chinta, P.E. Gunnells, Littlemann patterns and Weyl group multiple Dirichlet series of type D, in *Multiple Dirichlet Series, L-Functions, and Automorphic Forms*, D. Bump, S. Friedberg, and D. Goldfield, eds., *Progress in Math. 300* Birkhäuser Boston, 2012, 119–130.

[6] A.M. Hamel, R.C. King, Symplectic shifted tableaux and deformations of Weyl's denominator formula for $sp(2n)$, *J. Algebraic Comb.* **16** (2002), 269–300.

[7] A.M. Hamel, R.C. King, Half-turn symmetric alternating sign matrices and Tokuyama type factorisation for orthogonal group characters, arXiv:1402.6773v1[math.CO], 2014.

[8] M. Ishikawa, M. Wakayama, Applications of minor summation formula III, Plücker relations, lattice paths and Pfaffian identities, *J. of Combinatorial Theory A* **113** (2006), 113–155.

[9] G. Kuperberg, Symmetry classes of alternating-sign matrices under one roof, *Ann. Math.* **156** (2002), 835–866.

[10] S. Okada, Partially strict shifted plane partitions, *J. Combin. Th. A* **53** (1990), 143–156.

[11] S. Okada, Alternating sign matrices and some deformations of Weyl's denominator formula, *J. Alg. Combin.* **2** (1993), 155–176.

[12] D.P. Robbins, Symmetry classes of alternating sign matrices, arXiv: CO/008045 v1 5 Aug 2000.

[13] T. Simpson, Another deformation of Weyl's denominator formula, *J. Combin. Th. A* **77** (1997), 349–356.

[14] J.R. Stembridge, Nonintersecting paths, pfaffians, and plane partitions, *Adv. in Math.* **83** (1990), 96–131.

[15] S.J. Tabony, *Deformations of characters, metaplectic Whittaker functions, and the Yang-Baxter equation*, PhD Thesis, Massachusetts Institute of Technology, USA, 2011.

[16] T. Tokuyama, A generating function of strict Gelfand patterns and some formulas on characters of general linear groups, *J. Math. Soc. Japan* **40** (1988), 671–685.