# Root-theoretic Young Diagrams, Schubert Calculus and Adjoint Varieties

Dominic Searles†and Alexander Yong‡

Department of Mathematics, University of Illinois at Urbana-Champaign, Urbana, IL 61801, USA

**Abstract.** Root-theoretic Young diagrams are a conceptual framework to discuss existence of a root-system uniform and manifestly nonnegative combinatorial rule for Schubert calculus. Our main results use them to obtain formulas for (co)adjoint varieties of classical Lie type. This case is the simplest after the previously solved (co)minuscule family. Yet our formulas possess both uniform and non-uniform features.

**Résumé.** Les diagrammes de Young racine-théoriques forment un cadre conceptuel qui permet de discuter l'existence de règles de calcul de Schubert explicitement non-négatives et uniformes sur les systèmes de racines. Notre principal résultat est leur utilisation pour obtenir des formules pour les variétés (co)adjointes de types classiques. C'est le cas le plus simple après celui la famille (co)minuscule, déja résolue. Nos formules possèdent toutefois des propriétés uniformes et non-uniformes.

**Keywords:** Root-theoretic Young diagrams, Schubert calculus, Adjoint varieties

## Overview

This extended abstract concerns the following question:

Does there exist a root-system uniform and manifestly nonnegative combinatorial rule for Schubert calculus?

We elaborate on this problem and suggest an approach to it. Let $G$ be a complex reductive Lie group. Fix a choice $B$ of a Borel subgroup and maximal torus $T$, and let $W$ be its Weyl group: $W \cong N_G(T)/T$. Write $\Phi = \Phi^+ \cup \Phi^-$ to be the partition of roots into positives and negatives, and let $\Delta$ be the base of simple roots. Let $\Omega_G = (\Phi^+, <)$ denote the canonical poset structure on $\Phi^+$. Suppose $\Delta_P = \{\beta(P)_1, \dots, \beta(P)_k\} \subseteq \Delta$ identifies the parabolic subgroup $P$, and set $W_P := W_{\Delta_P}$ as the associated parabolic subgroup of $W$. Consider the subposet

$$ \Lambda_{G/P} = \{\alpha \in \Phi^+ : \beta_i(P) < \alpha \text{ for some } i\} \subseteq \Omega_G. $$

<footnote>†Supported by NSF grant DMS 0901331 and a grant from UIUC's Campus Research Board</footnote>

<footnote>‡Supported by NSF grants DMS 0901331 and DMS 1201595; this work was partially completed while a Beckman fellow at UIUC's Center for Advanced Study and while supported by a Helen Corley Petit endowment; thanks goes to Nicolas Perrin, Kevin Purbhoo and Hugh Thomas for helpful discussions, and to Philippe Di Francesco for a translation of the abstract</footnote>
The **Schubert varieties** in $G/P$ are $X_{wW_P} = \overline{B_wP/P}$ where $wW_P \in W/W_P$. Suppose $w$ is the minimal length coset representative of $wW_P$; $w$'s inversion set $\bar{\lambda}$ sits inside $\Lambda_{G/P}$. Let us write $X_{\bar{\lambda}} := X_{wW_P}$. Call $\bar{\lambda}$ a **root-theoretic Young diagram** (RYD). Let $\mathbb{Y}_{G/P}$ be the set of RYDs for $G/P$.

The cohomology ring $H^*(G/P, \mathbb{Q})$ has a $\mathbb{Z}$-additive basis of **Schubert classes** $\sigma_{\bar{\lambda}}$. Let $C_{\bar{\lambda},\bar{\mu}}^{\bar{\nu}}(G/P)$ denote the Schubert structure constants for $G/P$, i.e.,

$$ \sigma_{\bar{\lambda}} \cdot \sigma_{\bar{\mu}} = \sum_{\bar{\nu}} C_{\bar{\lambda},\bar{\mu}}^{\bar{\nu}}(G/P) \sigma_{\bar{\nu}}. $$

When $G/P$ is the Grassmannian $Gr_k(\mathbb{C}^n)$, $C_{\bar{\lambda},\bar{\mu}}^{\bar{\nu}} := C_{\bar{\lambda},\bar{\mu}}^{\bar{\nu}}(Gr_k(\mathbb{C}^n))$ is computed by the *Littlewood-Richardson rule*.

Ideally, there is a generalization that computes any $C_{\bar{\lambda},\bar{\mu}}^{\bar{\nu}}(G/P)$ in a cancellation-free fashion, but only in terms of the associated root datum (cancellative formulas are known, see, e.g., [Kn03]). Actually, often the main question is phrased presuming the existence of a rule. However, in that case, what is the qualitative nature of such a putative rule? Is it too much to expect a “counting rule” like the *Littelmann path model*? Should one instead search for a “patchwork” of counting rules and nonnegative recursions through different $G/P$'s for varying $G$'s? How can one classify special cases? Why are some special cases of the problem seemingly harder than others? Finally, if one believes that such a rule does not exist, what are concrete and/or falsifiable reasons for that belief?

Our thesis is that RYDs provide a simple but uniform combinatorial perspective to discuss such questions mathematically, make precise comparisons, and to measure progress towards a rule (uniform or otherwise).

For instance, from this perspective, Grassmannians are special because they sit in the family of $G/P$'s for which the above root-system setup is especially graphical:

(I) $\Lambda_{G/P}$ is a planar poset;

(II) the RYDs are lower order ideals (and in fact classical Young diagrams, thus explaining our nomenclature);

(III) Bruhat order is containment of RYDs.

These properties also hold for all *cominuscule* $G/P$'s. Together with earlier work of R. Proctor [Pr06], they help demonstrate existence for (co)minuscule Schubert calculus [ThYo09].

Right now, using RYDs is the only known way to solve the existence problem for (co)minuscule $G/P$. Conversely, it is only for (co)minuscule $G/P$'s that there is a uniform rule. Given this condition, it is therefore sensible to use RYDs to study other families.

We assert the key next case is the family of (co)adjoint $G/P$'s. One reason is that this family extends the (co)minuscule $G/P$'s, see, e.g., [LaMuSe79]. However, in terms of RYDs, it is important that for (co)adjoint varieties, none of the properties (I), (II) or (III) hold in general. Also important is that the failures of these properties are *quantifiably mild* (see Fact 0.1 below).

We obtain positive Schubert calculus rules in the classical (co)adjoint types. These rules have significant (but far from complete) uniformity. Our rules are sufficiently simple to admit nonzeroness criteria extending a simple case of the Horn inequalities, and also to completely classify what numbers occur as structure constants.
Additional complexity of OG(2, 2n) comes from the nonplanarity of Λ<sub>OG(2,2n)</sub>. To our best knowl-
edge, we give the first complete formula for any G/P with such nonplanarity — and what we find is
that it has “patchwork” features for which we have no broad explanation. It first separates out the cases
covered by the Pieri rule of [BuKrTa09]. Perhaps surprisingly, it is these “Pieri cases” that would bring
unappetizing complications to our rule. Yet, even after removing these cases, the rule exhibits a depen-
dency on the parity of n. This is traceable to the fact that Λ<sub>Q<sup>2n-4</sup></sub> is a subposet of Λ<sub>OG(2,2n)</sub> and that the
even-dimensional quadric Q<sup>2n-4</sup> has this dependency as well [ThYo09].

It seems plausible to us that the patchwork features of *OG*(2, 2*n*) are unavoidable if maintaining uni-
formity with the other (co)adjoint and (co)minuscule varieties. That is, we would infer that our results in
this special case challenge the existence of a root-system uniform “counting” rule. Now, there are specific
reasons to doubt this interpretation. First, in [ChPe11], RYDs are used to generalize [ThYo09]. Their
extension uniformly covers a subset of the Schubert problems in each of the (co)adjoint varieties — but
precisely those that are most “cominuscule-like”. Second, the “flattening trick” used for the *OG*(2, 2*n*)
problem is patently non-uniform. However, this step is what allows us to make comparisons with the other
(co)adjoint formulas. Third there are alternative and known uniform models such as “chains in Bruhat
order”, see, e.g., [BerSot98]. However, we reiterate that these alternative approaches are not known to
resolve the (co)minuscule case, which from our perspective is the simpler problem.

Definition of (co)adjoint varieties

The following definition is standard. Fix a representation $\rho : G \to GL(V)$ for some finite dimensional complex vector space $V$. The group $G$ acts on $\mathbb{P}(V)$ through the projection $\pi : V \setminus \{0\} \to \mathbb{P}(V)$. If $\vec{v}$ is a highest weight vector of $\rho$, then $\pi(G \cdot v) \subseteq \mathbb{P}(V)$ is a homogeneous projective variety, see, e.g., [FuHa04, Section 23.3]. This variety is **adjoint** if $\rho$ is the adjoint representation of $G$. Adjoint varieties have a root-system theoretic classification, see, e.g., [ChPe11] and the references therein. Then a variety is **coadjoint** if it is adjoint for the dual root system.

Call the highest root of Λ<sub>G/P</sub> the **adjoint root**. If λ̄ uses it we say λ̄ is **on** and we write λ̄ = ⟨λ|•⟩;
otherwise we say λ̄ is **off** and we write λ̄ = ⟨λ|○⟩, where λ̄ comprises the roots of Λ<sub>G/P</sub> \ {adjoint root}
used by λ̄. We state some facts that are easily checked for the setting of our theorems; cf., [ChPe11,
Section 2].

**Fact 0.1** If *G/P* is *adjoint* then:

(i) $|\Lambda_{G/P}|$ is odd

(ii) If $\bar{\lambda} = \langle \lambda | \circ \rangle$ then $|\bar{\lambda}| < \frac{1}{2} |\Lambda_{G/P}|$

(iii) If $\bar{\lambda} = \langle \lambda | \bullet \rangle$ then $|\bar{\lambda}| > \frac{1}{2} |\Lambda_{G/P}|$

(iv) $\lambda$ is a lower order ideal in the poset $\Lambda_{G/P} \setminus \{\text{adjoint root}\}$

For example, point (iv) explains in what sense the failure of (II) above is “mild”.

Warmup with the “(line,hyperplane)” flag variety $Fl_{1,n-1;n}$

We begin with a simple case of the adjoint varieties, $G/P = Fl_{1,n-1;n}$. This is the two step partial flag variety $\{\langle 0 \rangle \subset F_1 \subset F_{n-1} \subset \mathbb{C}^n\}$ where $F_1$ and $F_{n-1}$ have dimensions 1 and $n-1$ respectively. It
has dimension $|\Lambda_{G/P}| = 2n - 3$. All two-step flag manifolds have been solved, in a different way, by I. Coskun [Co09]. However, our approach will naturally extend to other (co)adjoint cases.

<figure><img src="image_1.png" /></figure>

We denote the shapes $\bar{\lambda}$ by $\langle \lambda_1, \lambda_2 | \bullet / \circ \rangle$, where $0 \le \lambda_1, \lambda_2 \le \frac{|\Lambda_{G/P}|-1}{2}$ and $\bullet / \circ$ indicates if $\bar{\lambda}$ is on or off.

We will need some reusable definitions. For any $\nu = (\nu_1, \nu_2) \in \mathbb{Z}^2$ let $\nu^* = (\nu_1 - 1, \nu_2)$ and $\nu_* = (\nu_1, \nu_2 - 1)$. Fix $\bar{\lambda}$ and $\bar{\mu}$ and define

$$ A_{\bar{\lambda},\bar{\mu}}(\nu) = \begin{cases} 0 & \text{if } \bar{\lambda} \text{ and } \bar{\mu} \text{ are on} \\ \sigma_{\langle \nu | \bullet \rangle} & \text{if exactly one of } \bar{\lambda} \text{ or } \bar{\mu} \text{ is on} \\ \sigma_{\langle \nu | \circ \rangle} & \text{if } |\bar{\lambda}| + |\bar{\mu}| < \frac{1}{2}|\Lambda_{G/P}| \\ \sigma_{\langle \nu^* | \bullet \rangle} + \sigma_{\langle \nu_* | \bullet \rangle} & \text{otherwise.} \end{cases} $$

Set $\sigma_{\langle \nu | \bullet / \circ \rangle}, \sigma_{\langle \nu^* | \bullet \rangle}$ or $\sigma_{\langle \nu_* | \bullet \rangle}$ to be zero if $\nu, \nu^*$ or $\nu_*$ are not in $[0, \frac{|\Lambda_{G/P}|-1}{2}] \times [0, \frac{|\Lambda_{G/P}|-1}{2}]$.

The “otherwise” case of the definition of $A_{\bar{\lambda},\bar{\mu}}(\nu)$ is what we call “adjoint jumping”: a nonadjoint root from $\nu$ has “jumped” to become the adjoint root. Understanding how this occurs in each type is a key idea needed in the (co)adjoint cases. This reflects the additional complexity coming from the failure of (II).

**Proposition 0.2** $\sigma_{\bar{\lambda}} \cdot \sigma_{\bar{\mu}} = A_{\bar{\lambda},\bar{\mu}}(\lambda + \mu) \in H^*(Fl_{1,n-1;n}, \mathbb{Q})$.

**Example 0.3** For $n = 5$, the rule gives $\sigma_{\langle 2,0 | \circ \rangle} \cdot \sigma_{\langle 1,2 | \circ \rangle} = A_{\langle 2,0 | \circ \rangle, \langle 1,2 | \circ \rangle}(3,2) = \sigma_{\langle 2,2 | \bullet \rangle} + \sigma_{\langle 3,1 | \bullet \rangle}$. Pictorially:

<figure><img src="image_2.png" /></figure>

**Corollary 0.4** $\{C_{\bar{\lambda},\bar{\mu}}^{\bar{\nu}}(Fl_{1,n-1;n})\} = \{0,1\}$.

In the style of Horn type theorems (see, e.g., [Fu00b]), we give a polytopal characterization of when $C_{\bar{\lambda},\bar{\mu}}^{\bar{\nu}}(Fl_{1,n-1;n}) \neq 0$. Identify

$$ \bar{\lambda} = \langle \lambda_1, \lambda_2 | \circ \rangle \text{ with } (\lambda_1, \lambda_2, 0) \in \mathbb{Z}^3 \text{ and } \bar{\lambda} = \langle \lambda_1, \lambda_2 | \bullet \rangle \text{ with } (\lambda_1, \lambda_2, 1) \in \mathbb{Z}^3. \quad (1) $$
**Corollary 0.5** Assume $\lambda = (\lambda_1, \lambda_2)$, $\mu = (\mu_1, \mu_2)$, $\nu = (\nu_1, \nu_2) \in \mathbb{Z}^2 \cap [0, \frac{|\Lambda_{G/P}|-1}{2}] \times [0, \frac{|\Lambda_{G/P}|-1}{2}]$ and $\lambda_3, \mu_3, \nu_3 \in \{0, 1\}$. Then $C_{\bar{\lambda}, \bar{\mu}}^{\bar{\nu}}(Fl_{1,n-1;n}) \neq 0$ if and only if:

$$ \begin{aligned} |\bar{\nu}| &= |\bar{\lambda}| + |\bar{\mu}| \\ \nu_1 &\le \lambda_1 + \mu_1 \\ \nu_2 &\le \lambda_2 + \mu_2 \\ \lambda_3 + \mu_3 &\le \nu_3 \end{aligned} $$

**Main theorem for odd orthogonal Grassmannians OG(2, 2n + 1) and Lagrangian Grassmannians LG(2, 2n)**

For the type $B_n$ root system, the adjoint variety $G/P = OG(2, 2n + 1)$ is the space of isotropic 2-planes with respect to a non-degenerate symmetric bilinear form on $\mathbb{C}^{2n+1}$. It has dimension $|\Lambda_{G/P}| = 4n - 5$.

<figure><img src="image_3.png" /></figure>

$\Lambda_{OG(2,2n+1)}, \Omega_{SO_{2n+1}}$ and a shape (for $n=4$)

The coadjoint partner to $OG(2, 2n + 1)$ in the $C_n$ root system is the variety $LG(2, 2n)$ of isotropic 2-planes with respect to a non-degenerate skew-symmetric bilinear form on $\mathbb{C}^{2n}$. Currently, we study the coadjoint variety with RYDs for its adjoint partner. This is analogous to [ThYo09]. We denote the shapes $\bar{\lambda}$ by $\langle \lambda | \bullet | \circ \rangle$, where $\lambda$ is a partition in $2 \times (\frac{|\Lambda_{G/P}|-1}{2})$.

Say $\sigma_{\langle \nu | \bullet | \circ \rangle}, \sigma_{\langle \nu^* | \bullet \rangle}$ or $\sigma_{\langle \nu_* | \bullet \rangle}$ is zero if $\nu, \nu^*$ or $\nu_*$ is not a partition in $2 \times (\frac{|\Lambda_{G/P}|-1}{2})$. Define $\mathrm{sh}(\bar{\nu})$ to be the number of short roots used by $\bar{\nu}$. The short roots of $\Lambda_{OG(2,2n+1)}$ consist of the middle pair of the nonadjoint roots.

**Theorem 0.6**

$$ \sigma_{\bar{\lambda}} \cdot \sigma_{\bar{\mu}} = \sum_{\nu \subseteq (\frac{|\Lambda_{G/P}|-1}{2}, \frac{|\Lambda_{G/P}|-1}{2})} C_{\lambda, \mu}^{\nu} A_{\bar{\lambda}, \bar{\mu}}(\nu) \in H^*(LG(2, 2n), \mathbb{Q}). $$

In $H^*(OG(2, 2n + 1))$, multiply each coefficient by $2^{\mathrm{sh}(\bar{\nu})-\mathrm{sh}(\bar{\lambda})-\mathrm{sh}(\bar{\mu})}$.

**Example 0.7** In $H^*(LG(2, 8), \mathbb{Q})$, we compute $\sigma_{\langle 3,1|\circ \rangle} \cdot \sigma_{\langle 3,2|\circ \rangle} = 2\sigma_{\langle 5,3|\bullet \rangle} + \sigma_{\langle 4,4|\bullet \rangle}$:

<figure><img src="image_4.png" /></figure>
Similarly, in $H^*(OG(2, 9), \mathbb{Q})$, we compute $\sigma_{\langle 2,1|\circ \rangle} \cdot \sigma_{\langle 3,2|\circ \rangle} = \sigma_{\langle 5,2|\bullet \rangle} + 4\sigma_{\langle 4,3|\bullet \rangle}$. $\square$

**Corollary 0.8** $\{C_{\bar{\nu},\bar{\mu}}^{\bar{\nu}}(LG(2,2n))\} = \{0,1,2\}$ and $\{C_{\bar{\nu},\bar{\mu}}^{\bar{\nu}}(OG(2,2n+1))\} = \{0,1,2,4,8\}$.

As with the case of $Fl_{1,n-1;n}$, one can describe the nonzero structure constants in terms of a polytope. To do so we make the identification (1).

**Corollary 0.9** Assume $\lambda = (\lambda_1, \lambda_2)$, $\mu = (\mu_1, \mu_2)$, $\nu = (\nu_1, \nu_2) \subset 2 \times (\frac{|\Lambda_{G/P}|-1}{2})$ are partitions and $\lambda_3, \mu_3, \nu_3 \in \{0,1\}$. Then $C_{\bar{\nu},\bar{\mu}}^{\bar{\nu}}(LG(2,2n)) \neq 0$ and $C_{\bar{\nu},\bar{\mu}}^{\bar{\nu}}(OG(2,2n+1)) \neq 0$ if and only if:

$$ \begin{aligned} |\bar{\nu}| &= |\bar{\lambda}| + |\bar{\mu}| \\ \nu_1 &\le \lambda_1 + \mu_1 \\ \nu_2 &\le \lambda_1 + \mu_2 \\ \nu_2 &\le \lambda_2 + \mu_1 \\ \lambda_3 + \mu_3 &\le \nu_3 \end{aligned} \qquad (2) $$

**Main theorem for even orthogonal Grassmannians OG(2, 2n)**

The adjoint variety $G/P = OG(2, 2n)$ is the space of isotropic 2-planes with respect to a non-degenerate symmetric bilinear form on $\mathbb{C}^{2n}$. It has dimension $|\Lambda_{G/P}| = 4n - 7$.

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

$\Lambda_{OG(2,2n)}, \Omega_{SO_{2n}}(\mathbb{C})$ and a shape (for $n=5$)

Here $\Lambda_{G/P}$ is not planar. Consider:

<figure><img src="image_7.png" /></figure>

Describe a shape $\bar{\lambda} = \langle \lambda|\bullet/\circ \rangle$ in $\Lambda_{G/P}$ as a triple $\langle \lambda^{(1)}, \lambda^{(2)}|\bullet/\circ \rangle$, where $\lambda^{(1)}$ (respectively, $\lambda^{(2)}$) is the Young diagram, in French notation, for the “bottom” (respectively, “top”) $2 \times (\frac{|\Lambda_{G/P}|-1}{4})$ rectangle. For example,

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

Define $\pi(\lambda) = \lambda^{(1)} + \lambda^{(2)} := (\lambda_1, \lambda_2)$; the result is a partition inside the $2 \times (\frac{|\Lambda_{G/P}|-1}{2})$ rectangle.
Consider an auxiliary poset $\Lambda'_{OG(2,2n)}$, a "planarization" of $\Lambda_{OG(2,2n)}$:

<figure><img src="image_10.png" /></figure>

In the above figure, we have marked the roots of the "top layer" for emphasis.

Shapes of $\Lambda'_{OG(2,2n)}$ are $\bar{\kappa} = \langle \kappa | \bullet / \circ \rangle$ where $\kappa$ is a partition contained in a $2 \times \left( \frac{|\Lambda_{G/P}|-1}{2} \right)$ rectangle and $\bullet / \circ$ indicates use of the adjoint root in $\Lambda'_{OG(2,2n)}$. Extend $\pi$ to a map $\Pi : \mathbb{Y}_{OG(2,2n)} \to \mathbb{Y}'_{OG(2,2n)}$ by defining $\Pi(\bar{\lambda}) = \langle \pi(\lambda) | \bullet \rangle$ if $\bar{\lambda}$ is on, and $\Pi(\bar{\lambda}) = \langle \pi(\lambda) | \circ \rangle$ otherwise.

For $\bar{\kappa} \in \mathbb{Y}'_{OG(2,2n)}$, let fsh($\bar{\kappa}$) be the number of **fake short roots** used by $\bar{\kappa}$, i.e., the number of roots in the $(n-2)$-th column used by $\bar{\kappa}$. The one exception is that we need fsh($\langle n-2, n-2 | \circ \rangle$) = 1. For $\bar{\nu} \in \mathbb{Y}_{OG(2,2n)}$, let fsh($\bar{\nu}$) denote fsh($\Pi(\bar{\nu})$).

The map $\Pi$ is either 1 : 1 or 2 : 1. In the former case, we identify $\bar{\kappa}$ and $\Pi^{-1}(\bar{\kappa})$. In the latter case, $\Pi^{-1}(\bar{\kappa}) = \{\bar{\kappa}^{\uparrow}, \bar{\kappa}^{\downarrow}\}$ and we call $\bar{\kappa}$ **ambiguous**. Call $\bar{\kappa}^{\uparrow}$ and $\bar{\kappa}^{\downarrow}$ **charged**. If $\bar{\kappa}$ is on (respectively, off), let $\bar{\kappa}^{\downarrow}$ be the shape such that the second part (respectively, first part) of the Young diagram $(\pi^{-1}(\kappa))^{(2)}$ is zero; let $\bar{\kappa}^{\uparrow}$ be the other one. Thus in Example 0.11 below, $\bar{\lambda}$ is up and $\bar{\mu}$ is down. Two charged shapes $\bar{\lambda}$ and $\bar{\mu}$ match if their arrows match and are **opposite** otherwise. Let

$$ \eta_{\bar{\lambda}, \bar{\mu}} = \begin{cases} 2 & \text{if } \bar{\lambda}, \bar{\mu} \text{ are charged and match and } n \text{ is even;} \\ 2 & \text{if } \bar{\lambda}, \bar{\mu} \text{ are charged and opposite and } n \text{ is odd;} \\ 1 & \text{if } \bar{\lambda} \text{ or } \bar{\mu} \text{ are not charged;} \\ 0 & \text{otherwise} \end{cases} $$

Say $\sigma_{\langle \nu | \bullet / \circ \rangle}$, $\sigma_{\langle \nu^* | \bullet \rangle}$ or $\sigma_{\langle \nu_* | \bullet \rangle}$ is zero if $\nu$, $\nu^*$ or $\nu_*$ is not a partition in $2 \times \left( \frac{|\Lambda_{G/P}|-1}{2} \right)$.

**Theorem 0.10** If either $\pi(\lambda)$ or $\pi(\mu)$ equals $(j, 0)$ (for some $0 \le j \le \frac{|\Lambda_{G/P}|-1}{2}$) then the Schubert expansion of $\sigma_{\bar{\lambda}} \cdot \sigma_{\bar{\mu}} \in H^*(OG(2, 2n), \mathbb{Q})$ is obtained by the Pieri rule of [BuKrTa09]. Otherwise, compute

$$ \sigma_{\Pi(\bar{\lambda})} \cdot \sigma_{\Pi(\bar{\mu})} = \sum_{\nu \subseteq \left( \frac{|\Lambda_{G/P}|+1}{2}, \frac{|\Lambda_{G/P}|-1}{2} \right)} C_{\pi(\lambda), \pi(\mu)}^{\nu} A_{\bar{\lambda}, \bar{\mu}}(\nu). \quad (3) $$

(i) Replace any term $\sigma_{\bar{\kappa}}$ that has $\kappa_1 = \frac{|\Lambda_{G/P}|-1}{2}$ by $\eta_{\bar{\lambda}, \bar{\mu}} \sigma_{\bar{\kappa}}$

(ii) Next, replace each $\sigma_{\bar{\kappa}}$ by $2^{\text{fsh}(\bar{\kappa}) - \text{fsh}(\bar{\lambda}) - \text{fsh}(\bar{\mu})} \sigma_{\bar{\kappa}}$

(iii) Finally, for any ambiguous $\bar{\kappa}$ replace $\sigma_{\bar{\kappa}}$ by $\frac{1}{2}(\sigma_{\bar{\kappa}^{\uparrow}} + \sigma_{\bar{\kappa}^{\downarrow}})$

The result is a provably integral, and manifestly nonnegative, Schubert basis expansion, which equals $\sigma_{\bar{\lambda}} \cdot \sigma_{\bar{\mu}} \in H^*(OG(2, 2n), \mathbb{Q})$.
Integrality is not manifest because of (ii) and (iii); however, it is easy to prove. Rule (i) extends a parity dependency for even-dimensional quadrics, described in [ThYo09]. The point is that the “double tailed diamond” which is $\Lambda_{\mathbb{Q}^{2n-4}}$ sits as a “side” of $\Lambda_{OG(2,2n)}$. Rule (ii) is analogous to our rule for $OG(2, 2n + 1)$. Rule (iii) describes how to “disambiguate”.

**Example 0.11** We wish to compute $\sigma_{\bar{\lambda}} \cdot \sigma_{\bar{\mu}} \in H^*(OG(2, 12))$ where:

<figure><img src="image_11.png" /></figure>

Both of these shapes are charged. Here $\pi(\lambda) = (4, 1)$ and $\pi(\mu) = (4, 2)$.

The $\nu \subseteq (\frac{|\Lambda_{G/P}|+1}{2}, \frac{|\Lambda_{G/P}|-1}{2}) = (9, 8)$ such that $C_{\pi(\lambda), \pi(\mu)}^{\nu} = 1$ are $(8, 3), (7, 4)$ and $(6, 5)$. All other $\nu$ have $C_{\pi(\lambda), \pi(\mu)}^{\nu} = 0$. Thus we compute

$$ \begin{aligned} \sigma_{\Pi(\bar{\lambda})} \cdot \sigma_{\Pi(\bar{\mu})} &= A_{\bar{\lambda}, \bar{\mu}}(8,3) + A_{\bar{\lambda}, \bar{\mu}}(7,4) + A_{\bar{\lambda}, \bar{\mu}}(6,5) \\ &= (\langle 7,3|\cdot \rangle + \langle 8,2|\cdot \rangle) + (\langle 6,4|\cdot \rangle + \langle 7,3|\cdot \rangle) + (\langle 5,5|\cdot \rangle + \langle 6,4|\cdot \rangle) \\ &= \langle 8,2|\cdot \rangle + 2\langle 7,3|\cdot \rangle + 2\langle 6,4|\cdot \rangle + \langle 5,5|\cdot \rangle \\ \mapsto & 0\langle 8,2|\cdot \rangle + 2\langle 7,3|\cdot \rangle + 2\langle 6,4|\cdot \rangle + \langle 5,5|\cdot \rangle \quad (\text{by (i) and } \eta_{\bar{\lambda},\bar{\mu}} = 0) \\ \mapsto & \langle 7,3|\cdot \rangle + 2\langle 6,4|\cdot \rangle + \langle 5,5|\cdot \rangle \quad (\text{by (ii) and } \text{fsh}(\bar{\lambda}) = \text{fsh}(\bar{\mu}) = 1) \end{aligned} $$

Finally, (iii) applies to the ambiguous shape $\langle 6, 4|\cdot \rangle$. Hence we conclude:

$$ \sigma_{\bar{\lambda}} \cdot \sigma_{\bar{\mu}} = \langle 7,3|\cdot \rangle + (\langle 6,4|\cdot \rangle^{\uparrow} + \langle 6,4|\cdot \rangle^{\downarrow}) + \langle 5,5|\cdot \rangle. $$

Each step is nonnegative and integral, in agreement with our theorem. $\square$

**Corollary 0.12** $\{C_{\bar{\lambda},\bar{\mu}}^{\bar{\nu}}(OG(2, 2n))\} = \{0, 1, 2, 4, 8\}$.

We make the following identifications; cf. (1):

$$ \Pi(\bar{\lambda}) = \langle \lambda_1, \lambda_2 | \circ \rangle \text{ with } (\lambda_1, \lambda_2, 0) \in \mathbb{Z}^3 \text{ and } \Pi(\bar{\lambda}) = \langle \lambda_1, \lambda_2 | \bullet \rangle \text{ with } (\lambda_1, \lambda_2, 1) \in \mathbb{Z}^3. $$

As with the other cases, we can give a criterion for nonzeroness:

**Corollary 0.13** If either $\pi(\lambda)$ or $\pi(\mu)$ equals $(j, 0)$ (for some $0 \le j \le \frac{|\Lambda_{G/P}|-1}{2}$) then nonzeroness of $C_{\bar{\lambda},\bar{\mu}}^{\bar{\nu}}(OG(2, 2n))$ is determined by the Pieri rule of [BuKrTa09].

If $\nu_1 = \frac{|\Lambda_{G/P}|-1}{2}$ then $C_{\bar{\lambda},\bar{\mu}}^{\bar{\nu}} \neq 0$ if and only if $\eta_{\bar{\lambda},\bar{\mu}} \neq 0$ and the inequalities (2) hold.

Otherwise, assume $(\lambda_1, \lambda_2), (\mu_1, \mu_2), (\nu_1, \nu_2) \subset 2 \times (\frac{|\Lambda_{G/P}|-1}{2})$ are partitions and $\lambda_3, \mu_3, \nu_3 \in \{0, 1\}$. Then $C_{\bar{\lambda},\bar{\mu}}^{\bar{\nu}}(OG(2, 2n)) \neq 0$ if and only if the inequalities (2) hold.
Concluding remarks

In [SeYo13+] we prove that our rules define an associative ring and agree with known Pieri rules [PrRa96, BuKrTa09]. This implies the claimed results. The remaining cases of classical type are easy.

We will also computationally analyze (co)adjoint Schubert calculus in all the exceptional Lie types. This is achieved by using the cohomology ring presentation of [ChPe11]. In this manner, we can extend the shortroots relationship between cominuscule/adjoint varieties with their minuscule/coadjoint “partners”, give information about which numbers appear, and describe which structure constants are nonzero.

In [Se13+], RYDs are used to give a new rule for the Belkale-Kumar coefficients in type A (after [KnPu11]). This rule explains why these Schubert problems are special, from the perspective of RYDs.

RYDs can also be applied to the study of Kazhdan-Lusztig polynomials; see [WoYo13+].

The “adjoint jumping” that is encoded by “otherwise” case of the operator $\bar{A}_{\bar{\lambda},\bar{\mu}}$ appears more generally. A small example: in type $A_{n-1}$ we may take $G/P = Fl_{2,4;5}$. The parabolic subgroup is associated to roots 2 and 4 of the Dynkin diagram of $A_4$. Thus, $\Lambda_{Fl_{2,4;5}}$ is the overlay of $\Lambda_{Gr_2(\mathbb{C}^5)}$ and $\Lambda_{Gr_4(\mathbb{C}^5)}$. This is depicted below, where one naturally splits $\Lambda_{Fl_{2,4;5}}$ into the three regions “L”, “R” and “T”.

<figure><img src="image_12.png" /></figure>

Schubert classes in $H^*(Fl_{2,4;5}, \mathbb{Q})$ are indexed by inversion sets $\bar{\lambda}$ consisting of (ordinary) Young diagrams sitting in each region. However, not all such collections of Young diagrams are inversion sets for some permutation. In this example, consider a root $\beta$ in the top region, and look at the “hook” of roots $\alpha \in \Lambda_{Fl_{2,4;5}}$ that appear diagonally southwest and southeast of $\beta$ and in region $L$ or $R$ (but not $T$). A condition just like the one for adjoint varieties appears: $\beta$ must be used if strictly more than half of these roots $\alpha$ are used by $\bar{\lambda}$. Also $\beta$ cannot be used if strictly less than half of the roots $\alpha$ are used.

**Example 0.14** $\sigma_{12|45|3} \cdot \sigma_{34|12|5} = \sigma_{35|14|2} + \sigma_{34|25|1} + \sigma_{45|12|3} \in H^*(Fl_{2,4;5}, \mathbb{Q})$. Pictorially:

<figure><img src="image_13.png" /></figure>

Although $\bar{\lambda}$ and $\bar{\mu}$ use no roots in region $T$, any $\bar{\nu}$ that appears must use roots in that region. $\square$

Clearly, $\Lambda_{GL_n/P}$ is planar for any $GL_n/P$. Thus the complexity of this case comes (in part) from finding the analogue of “adjoint jumping”. Also, planarity suggests that as a whole, type A Schubert problems are easier than type D problems. If nothing else, this agrees with practical experience.
References

[BerSot98] N. Bergeron and F. Sottile, *Schubert polynomials, the Bruhat order, and the geometry of flag manifolds*, Duke Math. J. Volume 95, Number 2 (1998), 373-423.

[BuKrTa09] A. Buch, A. Kresch and H. Tamvakis, *Quantum Pieri Rules for Isotropic Grassmannians*, Invent. Math. 178 (2009), 345-405.

[ChPe12] P. E. Chaput and N. Perrin, *Towards a Littlewood-Richardson rule for Kac-Moody homogeneous spaces*, J. Lie Theory (2012) **22**(1) 17-80.

[ChPe11] P. E. Chaput and N. Perrin, *On the quantum cohomology of adjoint varieties*, Proc. London Math. Soc. (2011) **103**(2), 294-330.

[Co09] I. Coskun, *A Littlewood-Richardson rule for two-step flag varieties*, Invent. Math. 176 no 2 (2009), 325-395.

[Fu00b] W. Fulton, *Eigenvalues, invariant factors, highest weights, and Schubert calculus*, Bull. Amer. Math. Soc. (N.S.) **37** (2000), no. 3, 209-249 (electronic).

[FuHa04] W. Fulton and J. Harris, *Representation Theory, a first course*, Springer-Verlag, 2004.

[Kn03] A. Knutson, A Schubert calculus recurrence from the noncomplex W-action on $G/B$, preprint 2003. arXiv:math/0306304

[KnPu11] A. Knutson and K. Purbhoo, *Product and Puzzle Formulae for $GL_n$ Belkale-Kumar Coefficients*, Electr. J. Comb. 18(1): (2011).

[LaMuSe79] V. Lakshmibai, C. Musili, C. S. Seshadri, *Geometry of $G/P$. III. Standard monomial theory for a quasi-minuscule $P$*, Proc. Indian Acad. Sci. Sect. A Math. Sci. **88**(1979), no. 3.

[PrRa96] P. Pragacz and J. Ratajski, *A Pieri-type theorem for Lagrangian and odd orthogonal Grassmannians*, J. Reine Angew. Math. 476 (1996), 143-189.

[Pr06] R. Proctor, *d-Complete posets generalize Young diagrams for the jeu de taquin property*, preprint 2006.

[Se13+] D. Searles, in preparation, 2013.

[SeYo13+] D. Searles and A. Yong, in preparation, 2013.

[ThYo09] H. Thomas and A. Yong, *A combinatorial rule for (co)minuscule Schubert calculus*, Adv. Math., **222**(2009), no. 2, 596-620.

[WoYo13+] A. Woo and A. Yong, in preparation, 2013.