# Poset Pinball, the Dimension Pair Algorithm, and TypeA Regular Nilpotent Hessenberg Varieties

Authors: Darius Bayegan; Megumi Harada
Journal: ISRN Geometry
 (2012)

---

## Abstract

We develop the theory ofposet pinball, a combinatorial game introduced by Harada-Tymoczko to study the equivariant cohomology ring of a GKM-compatible subspace X of a GKM space; Harada and Tymoczko also prove that, in certain circumstances, a successful outcome of Betti poset pinball yields a module basis for the equivariant cohomology ring of X. First we define the dimension pair algorithm, which yields a successful outcome of Betti poset pinball for any type A regular nilpotent Hessenberg and any type A nilpotent Springer variety, considered as GKM-compatible subspaces of the flag variety. The algorithm is motivated by a correspondence between Hessenberg affine cells and certain Schubert polynomials which we learned from Insko. Second, in a special case of regular nilpotent Hessenberg varieties, we prove that our pinball outcome is poset-upper-triangular, and hence the corresponding classes form a HS1*(pt)-module basis for the S1-equivariant cohomology ring of the Hessenberg variety.


---

## Body

## 1. Introduction

The purpose of this paper is to further develop the theory ofposet pinball, a combinatorial game introduced in [1] for the purpose of computing in equivariant cohomology rings (all cohomology rings in this note are with ℂ coefficients), in certain cases of typeAnilpotent Hessenberg varieties. One of the main uses of poset pinball in [1] is to construct module bases for the equivariant cohomology rings of GKM-compatible subspacesof GKM spaces [1, Definition  4.5]. In the context of this paper, the ambient GKM space is the flag variety ℱℓags(ℂn) equipped with the action of the diagonal subgroup T of U(n,ℂ), and the GKM-compatible subspaces are the nilpotent Hessenberg varieties. It is well recorded in the literature (e.g., [2] and references therein) that GKM spaces often have geometrically and/or combinatorially natural module bases for their equivariant cohomology rings; the basis of equivariant Schubert classes {σw}w∈Sn for HT*(ℱℓags(ℂn)) is a famous example. The results of this paper represent first steps towards the larger goal of using poset pinball to construct a similarly computationally effective and convenient module bases for a GKM-compatible subspace by exploiting the structure of the ambient GKM space.We briefly recall the setting of our results. LetN:ℂn→ℂn be a nilpotent operator. Let h:{1,2,…,n}→{1,2,…,n} be a function satisfying h(i)≥i for all 1≤i≤n and h(i+1)≥h(i) for all 1≤i<n. The associated Hessenberg variety Hess(N,h) is then defined as the following subvariety of ℱℓags(ℂn):(1.1)Hess(N,h):={V•=(0⊆V1⊆V2⊆⋯⊆Vn-1⊆Vn=Cn)∣NVi⊆Vh(i)∀i=1,…,n}.
Since we deal exclusively with type A in this paper, henceforth we omit this phrase from our terminology. Two special cases of Hessenberg varieties are of particular interest in this paper: when N is the principal nilpotent operator (in this case Hess(N,h) is called a regular nilpotent Hessenberg variety) and when h is the identity function h(i)=i for all 1≤i≤n (in this case Hess(N,h) is called a nilpotent Springer variety and is sometimes denoted 𝒮N). Hessenberg varieties arise in many areas of mathematics, including geometric representation theory [3–5], numerical analysis [6], mathematical physics [7, 8], combinatorics [9], and algebraic geometry [10, 11], so it is of interest to explicitly analyze their topology, for example, the structure of their (equivariant) cohomology rings. We do so through poset pinball and Schubert calculus techniques, as initiated and developed in [1, 12, 13] and briefly recalled below.The following relationship between two group actions on the nilpotent Hessenberg variety and the flag variety, respectively, allows us to use the theory of GKM-compatible subspaces and poset pinball. There is a naturalS1 subgroup of the unitary diagonal matrices T which acts on Hess(N,h) (defined precisely in Section 2). The group T, the maximal torus of U(n,ℂ), acts on ℱℓags(ℂn) in the standard fashion. It turns out that the S1-fixed points Hess(N,h)S1 are a subset of the T-fixed points ℱℓags(ℂn)T≅Sn. Moreover, the inclusion of Hess(N,h) into ℱℓags(ℂn) and the inclusion of groups S1 into T then induce a natural ring homomorphism(1.2)HT*(Flags(Cn))⟶HS1*(Hess(N,h)).
As mentioned above, it is well known in Schubert calculus that the equivariant Schubert classes {σw}w∈Sn are a computationally convenient HT*(pt)-module basis for HT*(ℱℓags(ℂn)). We refer to the images in HS1*(Hess(N,h)) of the equivariant Schubert classes {σw}w∈Sn via the projection (1.2) as Hessenberg Schubert classes. Given this setup and following [1], the game of poset pinball uses the data of the fixed points ℱℓags(ℂn)T≅Sn (considered as a partially ordered set with respect to Bruhat order) and the subset(1.3)Hess(N,h)S1⊆Flags(Cn)T≅Sn
to determine a set of rolldownsin Sn. It is shown in [1] that, under certain circumstances (one of which is discussed in more detail below), such a set of rolldowns in turn specifies a subset of the Hessenberg Schubert classes which form a HS1*(pt)-module basis of HS1*(Hess(N,h)). Thus poset pinball is an important tool for building computationally effective module bases for the equivariant cohomology of Hessenberg varieties. Indeed, the results of [13] accomplish precisely this goal—that is, of constructing a module basis via poset pinball techniques—in the special case of the Peterson variety, which is the regular nilpotent Hessenberg variety with Hessenberg function h defined by h(i)=i+1 for 1≤i≤n-1 and h(n)=n. Exploiting this explicit module basis, in [13, Theorem  6.12] the second author and Tymoczko give a manifestly positive Monk formula for the product of a degree-2 Peterson Schubert class with an arbitrary Peterson Schubert class, expressed as a HS1*(pt)-linear combination of Peterson Schubert classes. This is an example of equivariant Schubert calculus in the realm of Hessenberg varieties, and it is an open problem to generalize the results of [13] to a wider class of Hessenberg varieties.We now describe our main results. First, we explain in detail an algorithm which we dub thedimension pair algorithm and which associates to each S1-fixed point w∈Hess(N,h)S1 a permutation in Sn, which we call the rolldown ofw following terminology in [1] and denoted roℓℓ(w)∈Sn. In the special cases of regular nilpotent Hessenberg varieties and nilpotent Springer varieties, we show that the set {roℓℓ(w)}w∈Hess(N,h)S1 can be interpreted as the result of a successful game of Betti pinball (in the sense of [1]). The main motivation for our construction is that a successful outcome of Betti pinball can, under some circumstances, produce a module basis for the associated equivariant cohomology ring (cf. [1, Section  4.3]). In this sense, our algorithm represents a significant step towards the construction of module bases for the equivariant cohomology rings of general nilpotent Hessenberg varieties, thus extending the theory developed in [1, 13]. Although we formulate our algorithm in terms of dimension pairs and permissible fillings following terminology of Mbirika [14], the essential idea comes from a correspondence between Hessenberg affine cells and certain Schubert polynomials which we learned from Erik Insko.Second, for a specific case of a regular nilpotent Hessenberg variety which we call a 334-type Hessenberg variety, we prove that the set of rolldowns {roℓℓ(w)}w∈Hess(N,h)S1 obtained from the dimension pair algorithm is in fact poset-upper-triangular in the sense of [1]. As shown in [1], this is one of the possible circumstances under which we can conclude that the corresponding set of Hessenberg Schubert classes forms a module basis for the S1-equivariant cohomology ring of the variety. Thus our result gives rise to a new family of examples of Hessenberg varieties (and GKM-compatible subspaces) for which poset pinball successfully produces explicit module bases. We mention that the dimension pair algorithm also produces module bases in a special case of Springer varieties [15]. Although we do not know whether the dimension pair algorithm always succeeds in producing module bases for the S1-equivariant cohomology rings for a general nilpotent Hessenberg variety, the evidence thus far is suggestive. We leave further investigation to future work.We give a brief summary of the contents of this manuscript. In Section2 we recall some definitions and constructions necessary for later statements. In Section 3.1 we describe the dimension pair algorithm and prove that the result of the algorithm satisfies the conditions to be the outcome of a successful game of Betti poset pinball in the special cases of regular nilpotent Hessenberg varieties and nilpotent Springer varieties. We briefly review in Section 3.2 the theory developed in [1] which show that if the rolldown set obtained from a successful game of Betti poset pinball also satisfies poset-upper-triangularity conditions, then it yields a module basis in equivariant cohomology. In Sections 4 and 5 we prove that the dimension pair algorithm produces a poset-upper-triangular module basis in a special class of regular nilpotent Hessenberg varieties which we call 334-type Hessenberg varieties. We close with some open questions in Section 6.


## 2. Background

We begin with necessary definitions and terminology for what follows. In Section2.1 we recall the geometric objects and the group actions under consideration. In Section 2.2 we recall some combinatorial definitions associated to Young diagrams. We recall a bijection between Hessenberg fixed points and certain fillings of Young diagrams in Section 2.3. The discussion closely follows previous work (e.g., [1, 13] and also [16]) so we keep exposition brief.


### 2.1. Hessenberg Varieties, Highest Forms, and Fixed Points

By theflag varietywe mean the homogeneous space GL(n,ℂ)/B which is also identified with(2.1)Flags(Cn):={V•=({0}⊆V1⊆V2⊆⋯Vn-1⊆Vn=Cn)∣dim⁡C(Vi)=i}.
A Hessenberg function is a function h:{1,2,…,n}→{1,2,…,n} satisfying h(i)≥i for all 1≤i≤n and h(i+1)≥h(i) for all 1≤i<n. We frequently denote a Hessenberg function by listing its values in sequence, h=(h(1),h(2),…,h(n)=n). Let N:ℂn→ℂn be a linear operator. The Hessenberg varietyHess(N,h) is defined as the following subvariety of ℱℓags(ℂn):(2.2)Hess(N,h):={V•∈Flags(Cn)∣NVi⊆Vh(i)∀i=1,…,n}⊆Flags(Cn).
If N is nilpotent, we say Hess(N,h) is a nilpotent Hessenberg variety, and if N is the principal nilpotent operator (i.e., has one Jordan block with eigenvalue 0), then Hess(N,h) is called a regular nilpotent Hessenberg variety. If N is nilpotent and h is the identity function h(i)=i for all 1≤i≤n, then Hess(N,h) is called a nilpotent Springer variety and often denoted 𝒮N. In this manuscript we study in some detail the regular nilpotent case, and as such sometimes notate Hess(N,h) as Hess(h) when N is understood to be the standard principal nilpotent operator.SupposeN is a nilpotent matrix in standard Jordan canonical form. It turns out that for many of our statements below we must use a choice of conjugate of N which is in highest form [16, Definition  4.2]. We recall the following.Definition 2.1 (see [16, Definitions 4.1 and 4.2]).
(i) LetX be any m×n matrix. We call the entry Xik a pivot of X if Xik is nonzero and if all entries below and to its left vanish, that is, Xij=0 if j<k and Xjk=0 if j>i. Moreover, given i, define ri to be the row of Xri,i if the entry is a pivot, and 0 otherwise.
(ii) LetN be an upper-triangular nilpotent n×n matrix. Then we say N is in highest form if its pivots form a nondecreasing sequence, namely, r1≤r2≤⋯≤rn.We do not require the details of the theory of highest forms of linear operators; for the purposes of the present manuscript it suffices to remark firstly that whenN is the principal nilpotent matrix, then N is already in highest form, and secondly that any nilpotent matrix can be conjugated by an appropriate n×n permutation matrix σ so that Nhf:=σNσ-1 is in highest form. However, the following observation will be relevant in Section 2.3.Remark 2.2.
In this manuscript we always assume that our highest formNhf=σNσ-1 has been chosen in accordance to the recipe described by Tymoczko in [16, Section 4]. Since the precise method of this construction is not relevant for the rest of the present manuscript, we omit further explanation here. In the case when N is principal nilpotent, we take Nhf=N since N is already in highest form and this is the form chosen by Tymoczko in [16]. A more detailed discussion of highest forms as it pertains to poset pinball theory is in [15].For details on the following facts we refer the reader to, for example, [1, 13, 16] and references therein. Let N be an n×n nilpotent matrix in Jordan canonical form and let σ denote a permutation matrix such that Nhf:=σNσ-1 is in highest form. It is known and straightforward to show that the following S1 subgroup of U(n,ℂ) preserves Hess(N,h) for N as above and any Hessenberg function h:(2.3)S1={[tn0⋯00tn-1000⋱000t]∣t∈C,‖t‖=1}⊆Tn⊆U(n,C).
Here Tn is the standard maximal torus of U(n,ℂ) consisting of diagonal unitary matrices.This implies that the conjugate circle subgroupσS1σ-1 preserves Hess(Nhf,h). By abuse of notation we will denote both circle subgroups by S1, since it is clear by context which is meant. The S1-fixed points of Hess(N,h) and Hess(Nhf,h) are isolated and are a subset of the Tn-fixed points of ℱℓags(ℂn). Since the set of Tn-fixed points ℱℓags(ℂn)Tn may be identified with the Weyl group W=Sn, and since Hess(N,h)S1 (resp., Hess(Nhf,h)S1) is a subset of ℱℓags(ℂn)Tn, any Hessenberg fixed point may be thought of as a permutation w∈Sn.


### 2.2. Permissible Fillings, Dimension Pairs, Lists of Top Parts, and Associated Permutations

Recall that there is a bijective correspondence between the set of conjugacy classes of nilpotentn×n complex matrices N and Young diagrams (we use English notation for Young diagrams) with n boxes, given by associating to N the Young diagram λ with row lengths the sizes of the Jordan blocks of N listed in weakly decreasing order. We will use this bijection to often treat such N and λ as the same data; we sometimes denote by λN the Young diagram given as above corresponding to a nilpotent N.For more details on the following see [14].Definition 2.3.
Letλ be a Young diagram with n boxes. Let h:{1,2,…,n}→{1,2,…,n} be a Hessenberg function. A filling of λ by the alphabet {1,2,…,n} is an injective placing of the integers {1,2,…,n} into the boxes of λ. A filling of λ is called a (h,λ)-permissible filling if for every horizontal adjacency   in the filling we have k≤h(j).Remark 2.4.
In this manuscript theλ and h will frequently be understood by context. When there is no danger of confusion we simply refer to permissible fillings.Example 2.5.
Letn=5. Suppose λ=(5) and h=(3,3,4,5,5). Then  is a permissible filling, whereas  is not, since 4≰h(1).
We denote a permissible filling ofλ by T, in analogy with standard notation for Young tableaux. Next we focus attention on certain pairs of entries in a permissible filling T.Definition 2.6.
Leth:{1,2,…,n}→{1,2,…,n} be a Hessenberg function and λ a Young diagram with n boxes. A pair (a,b) is a dimension pair of an (h,λ)-permissible filling T of λ if the following conditions hold:(1)
b>a,(2)
b is either
(i)
belowa in the same column of a, or(ii)
anywhere in a column strictly to the left of the column ofa, and(3)
if there exists a box with fillingc directly adjacent to the right of a, then b≤h(c).For a dimension pair(a,b) of T, we will refer to b as the top part of the dimension pair.Example 2.7.
Letλ, h be as in Example 2.5. The dimension pairs in the permissible filling  are (1,2), (1,3), and (1,4). Note that (3,4) is not a dimension pair because 1 is directly to the right of the 3 and 4≰h(1).Given a permissible fillingT of λ, we follow [14] and denote by DPT the set of dimension pairs of T. For each integer ℓ with 2≤ℓ≤n, let (2.4)xl:=|{(a,l)∣(a,l)∈DPT}|
so xℓ is the number of times ℓ occurs as a top part in the set of dimension pairs of T. From the definitions it follows that 0≤xℓ≤ℓ-1forall2≤ℓ≤n. We call the integral vector x=(x2,x3,…,xn) the list of top parts of T.To each suchx we associate a permutation in Sn as follows. As a preliminary step, for each ℓ with 2≤ℓ≤n define(2.5)ul(x):={sl-1sl-2⋯sl-xl,ifxl>0,1,ifxl=0,
where si denotes the simple transposition (i,i+1) in Sn and 1 denotes the identity permutation. Now define the association(2.6)x⟼ω(x):=u2(x)u3(x)⋯un(x)∈Sn.
It is not difficult to see that (2.6) is a bijection between the set of integral vectors x∈ℤn-1 satisfying 0≤xℓ≤ℓ-1 for all 2≤ℓ≤n-1 and the group Sn. In fact the word given by (2.6) is a reduced word decomposition of ω(x) and the xℓ count the number of inversions in ω(x) with ℓ as the higher integer. The following simple fact will be used later.Fact 1.
Supposex=(x2,…,xn),y=(y2,…,yn)∈ℤ≥0n-1 are both lists of top parts. Suppose further that for all 2≤ℓ≤n, we have xℓ≤yℓ.Then ω(x)≤ω(y) in Bruhat order. This follows immediately from the definition  (2.6).Example 2.8.
Continuing with Examples2.5 and 2.7, for the permissible filling , the set DPT of top parts of dimension pairs is {2,3,4}, yielding the integer vector x=(1,1,1,0). The associated permutation ω(x) is then s1s2s3.Example 2.9.
Letλ,h be as in Example 2.5. The filling  is also permissible, with dimension pairs (1,2), (1,3), (1,4), (2,3). Hence x=(1,2,1,0) and the associated permutation ω(x) is s1(s2s1)s3.


### 2.3. Bijection between Fixed Points and Permissible Fillings

For nilpotent Hessenberg varieties, theS1-fixed points Hess(N,h)S1 are in bijective correspondence with the set of permissible fillings of the Young diagram λ=λN, as we now describe. We will use this correspondence in the formulation of our dimension pair algorithm.Supposeλ is a Young diagram with n boxes. We begin by defining a bijective correspondence between the set ℱiℓℓ(λ) of all fillings (not necessarily permissible) of λ with permutations in Sn. Given a filling, read the entries of the filling by reading along each column from the bottom to the top, starting with the leftmost column and proceeding to the rightmost column. The association ℱiℓℓ(λ)↔Sn is then given by interpreting the resulting word as the one-line notation of a permutation. For example, the filling(2.7)
has associated permutation 641523. It is easily seen that this is a bijective correspondence. Given a filling T of λ, we denote its associated permutation by ϕλ(T).Remark 2.10.
In the case whenN is the principal nilpotent n×n matrix, the corresponding Young diagram λ=λN=(n) has only one row, so the above correspondence simply reads off the (one row of the) filling from left to right. In this case we abuse notation and denote ϕλ-1(w) by just w. For instance, the permissible filling of λ=(5) in Example 2.9 has associated permutation 43215.
Now let(2.8)PFill(λ,h)
denote the set of (h,λ)-permissible fillings of λ. Recall that elements in Hess(N,h)S1 are viewed as permutations in Sn via the identification ℱℓags(ℂn)Tn≅Sn. The next proposition follows from the definitions and some linear algebra. It is proven and discussed in more detail in [15], where the notation used is slightly different.Proposition 2.11.
Fixn a positive integer. Let h:{1,2,…,n}→{1,2,…,n} be a Hessenberg function and λ a Young diagram with n boxes. Suppose Nhf is a nilpotent operator in highest form as chosen in [16] (cf. Remark 2.2) with λNhf=λ. Let Hess(Nhf,h) denote the associated nilpotent Hessenberg variety. Then the map from the S1-fixed points Hess(Nhf,h)S1 to the set of permissible fillings 𝒫Fiℓℓ(λ,h)(2.9)w∈Hess(h)S1⊆Sn⟼ϕλ-1(w-1)∈PFill(λ,h)
is well defined and is a bijection.Remark 2.12.
In the case whenN is the principal nilpotent n×n matrix, λ is the Young diagram with only one row. Thus the map (2.9) above simplifies to w↦w-1 where we abuse notation (cf. Remark 2.10) and denote ϕλ-1(w-1) by w-1.


## 2.1. Hessenberg Varieties, Highest Forms, and Fixed Points

By theflag varietywe mean the homogeneous space GL(n,ℂ)/B which is also identified with(2.1)Flags(Cn):={V•=({0}⊆V1⊆V2⊆⋯Vn-1⊆Vn=Cn)∣dim⁡C(Vi)=i}.
A Hessenberg function is a function h:{1,2,…,n}→{1,2,…,n} satisfying h(i)≥i for all 1≤i≤n and h(i+1)≥h(i) for all 1≤i<n. We frequently denote a Hessenberg function by listing its values in sequence, h=(h(1),h(2),…,h(n)=n). Let N:ℂn→ℂn be a linear operator. The Hessenberg varietyHess(N,h) is defined as the following subvariety of ℱℓags(ℂn):(2.2)Hess(N,h):={V•∈Flags(Cn)∣NVi⊆Vh(i)∀i=1,…,n}⊆Flags(Cn).
If N is nilpotent, we say Hess(N,h) is a nilpotent Hessenberg variety, and if N is the principal nilpotent operator (i.e., has one Jordan block with eigenvalue 0), then Hess(N,h) is called a regular nilpotent Hessenberg variety. If N is nilpotent and h is the identity function h(i)=i for all 1≤i≤n, then Hess(N,h) is called a nilpotent Springer variety and often denoted 𝒮N. In this manuscript we study in some detail the regular nilpotent case, and as such sometimes notate Hess(N,h) as Hess(h) when N is understood to be the standard principal nilpotent operator.SupposeN is a nilpotent matrix in standard Jordan canonical form. It turns out that for many of our statements below we must use a choice of conjugate of N which is in highest form [16, Definition  4.2]. We recall the following.Definition 2.1 (see [16, Definitions 4.1 and 4.2]).
(i) LetX be any m×n matrix. We call the entry Xik a pivot of X if Xik is nonzero and if all entries below and to its left vanish, that is, Xij=0 if j<k and Xjk=0 if j>i. Moreover, given i, define ri to be the row of Xri,i if the entry is a pivot, and 0 otherwise.
(ii) LetN be an upper-triangular nilpotent n×n matrix. Then we say N is in highest form if its pivots form a nondecreasing sequence, namely, r1≤r2≤⋯≤rn.We do not require the details of the theory of highest forms of linear operators; for the purposes of the present manuscript it suffices to remark firstly that whenN is the principal nilpotent matrix, then N is already in highest form, and secondly that any nilpotent matrix can be conjugated by an appropriate n×n permutation matrix σ so that Nhf:=σNσ-1 is in highest form. However, the following observation will be relevant in Section 2.3.Remark 2.2.
In this manuscript we always assume that our highest formNhf=σNσ-1 has been chosen in accordance to the recipe described by Tymoczko in [16, Section 4]. Since the precise method of this construction is not relevant for the rest of the present manuscript, we omit further explanation here. In the case when N is principal nilpotent, we take Nhf=N since N is already in highest form and this is the form chosen by Tymoczko in [16]. A more detailed discussion of highest forms as it pertains to poset pinball theory is in [15].For details on the following facts we refer the reader to, for example, [1, 13, 16] and references therein. Let N be an n×n nilpotent matrix in Jordan canonical form and let σ denote a permutation matrix such that Nhf:=σNσ-1 is in highest form. It is known and straightforward to show that the following S1 subgroup of U(n,ℂ) preserves Hess(N,h) for N as above and any Hessenberg function h:(2.3)S1={[tn0⋯00tn-1000⋱000t]∣t∈C,‖t‖=1}⊆Tn⊆U(n,C).
Here Tn is the standard maximal torus of U(n,ℂ) consisting of diagonal unitary matrices.This implies that the conjugate circle subgroupσS1σ-1 preserves Hess(Nhf,h). By abuse of notation we will denote both circle subgroups by S1, since it is clear by context which is meant. The S1-fixed points of Hess(N,h) and Hess(Nhf,h) are isolated and are a subset of the Tn-fixed points of ℱℓags(ℂn). Since the set of Tn-fixed points ℱℓags(ℂn)Tn may be identified with the Weyl group W=Sn, and since Hess(N,h)S1 (resp., Hess(Nhf,h)S1) is a subset of ℱℓags(ℂn)Tn, any Hessenberg fixed point may be thought of as a permutation w∈Sn.


## 2.2. Permissible Fillings, Dimension Pairs, Lists of Top Parts, and Associated Permutations

Recall that there is a bijective correspondence between the set of conjugacy classes of nilpotentn×n complex matrices N and Young diagrams (we use English notation for Young diagrams) with n boxes, given by associating to N the Young diagram λ with row lengths the sizes of the Jordan blocks of N listed in weakly decreasing order. We will use this bijection to often treat such N and λ as the same data; we sometimes denote by λN the Young diagram given as above corresponding to a nilpotent N.For more details on the following see [14].Definition 2.3.
Letλ be a Young diagram with n boxes. Let h:{1,2,…,n}→{1,2,…,n} be a Hessenberg function. A filling of λ by the alphabet {1,2,…,n} is an injective placing of the integers {1,2,…,n} into the boxes of λ. A filling of λ is called a (h,λ)-permissible filling if for every horizontal adjacency   in the filling we have k≤h(j).Remark 2.4.
In this manuscript theλ and h will frequently be understood by context. When there is no danger of confusion we simply refer to permissible fillings.Example 2.5.
Letn=5. Suppose λ=(5) and h=(3,3,4,5,5). Then  is a permissible filling, whereas  is not, since 4≰h(1).
We denote a permissible filling ofλ by T, in analogy with standard notation for Young tableaux. Next we focus attention on certain pairs of entries in a permissible filling T.Definition 2.6.
Leth:{1,2,…,n}→{1,2,…,n} be a Hessenberg function and λ a Young diagram with n boxes. A pair (a,b) is a dimension pair of an (h,λ)-permissible filling T of λ if the following conditions hold:(1)
b>a,(2)
b is either
(i)
belowa in the same column of a, or(ii)
anywhere in a column strictly to the left of the column ofa, and(3)
if there exists a box with fillingc directly adjacent to the right of a, then b≤h(c).For a dimension pair(a,b) of T, we will refer to b as the top part of the dimension pair.Example 2.7.
Letλ, h be as in Example 2.5. The dimension pairs in the permissible filling  are (1,2), (1,3), and (1,4). Note that (3,4) is not a dimension pair because 1 is directly to the right of the 3 and 4≰h(1).Given a permissible fillingT of λ, we follow [14] and denote by DPT the set of dimension pairs of T. For each integer ℓ with 2≤ℓ≤n, let (2.4)xl:=|{(a,l)∣(a,l)∈DPT}|
so xℓ is the number of times ℓ occurs as a top part in the set of dimension pairs of T. From the definitions it follows that 0≤xℓ≤ℓ-1forall2≤ℓ≤n. We call the integral vector x=(x2,x3,…,xn) the list of top parts of T.To each suchx we associate a permutation in Sn as follows. As a preliminary step, for each ℓ with 2≤ℓ≤n define(2.5)ul(x):={sl-1sl-2⋯sl-xl,ifxl>0,1,ifxl=0,
where si denotes the simple transposition (i,i+1) in Sn and 1 denotes the identity permutation. Now define the association(2.6)x⟼ω(x):=u2(x)u3(x)⋯un(x)∈Sn.
It is not difficult to see that (2.6) is a bijection between the set of integral vectors x∈ℤn-1 satisfying 0≤xℓ≤ℓ-1 for all 2≤ℓ≤n-1 and the group Sn. In fact the word given by (2.6) is a reduced word decomposition of ω(x) and the xℓ count the number of inversions in ω(x) with ℓ as the higher integer. The following simple fact will be used later.Fact 1.
Supposex=(x2,…,xn),y=(y2,…,yn)∈ℤ≥0n-1 are both lists of top parts. Suppose further that for all 2≤ℓ≤n, we have xℓ≤yℓ.Then ω(x)≤ω(y) in Bruhat order. This follows immediately from the definition  (2.6).Example 2.8.
Continuing with Examples2.5 and 2.7, for the permissible filling , the set DPT of top parts of dimension pairs is {2,3,4}, yielding the integer vector x=(1,1,1,0). The associated permutation ω(x) is then s1s2s3.Example 2.9.
Letλ,h be as in Example 2.5. The filling  is also permissible, with dimension pairs (1,2), (1,3), (1,4), (2,3). Hence x=(1,2,1,0) and the associated permutation ω(x) is s1(s2s1)s3.


## 2.3. Bijection between Fixed Points and Permissible Fillings

For nilpotent Hessenberg varieties, theS1-fixed points Hess(N,h)S1 are in bijective correspondence with the set of permissible fillings of the Young diagram λ=λN, as we now describe. We will use this correspondence in the formulation of our dimension pair algorithm.Supposeλ is a Young diagram with n boxes. We begin by defining a bijective correspondence between the set ℱiℓℓ(λ) of all fillings (not necessarily permissible) of λ with permutations in Sn. Given a filling, read the entries of the filling by reading along each column from the bottom to the top, starting with the leftmost column and proceeding to the rightmost column. The association ℱiℓℓ(λ)↔Sn is then given by interpreting the resulting word as the one-line notation of a permutation. For example, the filling(2.7)
has associated permutation 641523. It is easily seen that this is a bijective correspondence. Given a filling T of λ, we denote its associated permutation by ϕλ(T).Remark 2.10.
In the case whenN is the principal nilpotent n×n matrix, the corresponding Young diagram λ=λN=(n) has only one row, so the above correspondence simply reads off the (one row of the) filling from left to right. In this case we abuse notation and denote ϕλ-1(w) by just w. For instance, the permissible filling of λ=(5) in Example 2.9 has associated permutation 43215.
Now let(2.8)PFill(λ,h)
denote the set of (h,λ)-permissible fillings of λ. Recall that elements in Hess(N,h)S1 are viewed as permutations in Sn via the identification ℱℓags(ℂn)Tn≅Sn. The next proposition follows from the definitions and some linear algebra. It is proven and discussed in more detail in [15], where the notation used is slightly different.Proposition 2.11.
Fixn a positive integer. Let h:{1,2,…,n}→{1,2,…,n} be a Hessenberg function and λ a Young diagram with n boxes. Suppose Nhf is a nilpotent operator in highest form as chosen in [16] (cf. Remark 2.2) with λNhf=λ. Let Hess(Nhf,h) denote the associated nilpotent Hessenberg variety. Then the map from the S1-fixed points Hess(Nhf,h)S1 to the set of permissible fillings 𝒫Fiℓℓ(λ,h)(2.9)w∈Hess(h)S1⊆Sn⟼ϕλ-1(w-1)∈PFill(λ,h)
is well defined and is a bijection.Remark 2.12.
In the case whenN is the principal nilpotent n×n matrix, λ is the Young diagram with only one row. Thus the map (2.9) above simplifies to w↦w-1 where we abuse notation (cf. Remark 2.10) and denote ϕλ-1(w-1) by w-1.


## 3. The Dimension Pair Algorithm for Betti Poset Pinball for Nilpotent Hessenberg Varieties

In this section we first explain thedimension pair algorithm which associates to any Hessenberg fixed point a permutation in Sn. The name is due to the fact that the construction proceeds by computing dimension pairs in appropriate permissible fillings. We then interpret this algorithm as a method for choosing rolldowns associated to the Hessenberg fixed points in a game of Betti poset pinball in the sense of [1]. The algorithm makes sense for any nilpotent Hessenberg variety, so it is defined in that generality in Section 3.1. However, our proof that the algorithm produces a successful outcome of Betti poset pinball in the sense of [1] is only for the special cases of regular nilpotent Hessenberg varieties and nilpotent Springer varieties. In Section 3.2 we briefly recall the setup and necessary results of poset pinball which allow us to conclude that our poset pinball result yields an explicit module basis for equivariant cohomology.


### 3.1. The Dimension Pair Algorithm for Nilpotent Hessenberg Varieties

LetNhf be a nilpotent n×n matrix in highest form chosen as in Remark 2.2 and let λ:=λNhf. Let h:{1,2,…,n}→{1,2,…,n} be a Hessenberg function and Hess(Nhf,h) the corresponding nilpotent Hessenberg variety.The definition of the dimension pair algorithm is pure combinatorics. It produces for each Hessenberg fixed pointw∈Hess(Nhf,h)S1 an element in Sn. Following terminology of poset pinball, we denote this function by(3.1)roll:Hess(Nhf,h)S1⟶Sn.Definition 3.1 (the dimension pair algorithm).
We defineroℓℓ:Hess(Nhf,h)S1→Sn as follows.(1)
Letw∈Hess(Nhf,h)S1 and let ϕλ-1(w-1) be its corresponding permissible filling as defined in (2.9).(2)
LetDPϕλ-1(w-1) be the set of dimension pairs in the permissible filling ϕλ-1(w-1).(3)
For eachℓ with 2≤ℓ≤n, set
(3.2)xl:=|{(a,l)∣(a,l)∈DPϕλ-1(w-1)}|
as in (2.4) and define x:=(x2,…,xn).(4)
Defineroℓℓ(w):=(ω(x))-1 where ω(x) is the permutation associated to the integer vector x defined in (2.6).Example 3.2.
Letλ, h be as in Example 2.5. The permutation w=43215∈Sn is in Hess(Nhf,h)S1, as can be checked. The associated permissible filling is  .  In Example 2.9 we saw that the associated permutation is s1(s2s1)s3, so we conclude roℓℓ(w)=s3(s1s2)s1.We next show that the rolldown functionroℓℓ:Hess(h)S1→Sn defined by the dimension pair algorithm above satisfies the conditions to be a successful outcome of Betti poset pinball as in [1] in certain cases of nilpotent Hessenberg varieties. The statement of one of the conditions requires advance knowledge of the Betti numbers of nilpotent Hessenberg varieties, for which we recall the following result (reformulated in our language) from [16].Theorem 3.3 (see [16, Theorem  1.1]).
LetNhf:ℂn→ℂn be a nilpotent matrix in highest form chosen as in Remark 2.2 and let λ:=λNhf. Let h:{1,2,…,n}→{1,2,…,n} be a Hessenberg function and let Hess(Nhf,h) denote the corresponding nilpotent Hessenberg variety. There is a paving by (complex) affine cells of Hess(Nhf,h) such that(i)
the affine cells are in one-to-one correspondence withHess(Nhf,h)S1, and(ii)
the (complex) dimension of the affine cellCw corresponding to a fixed point w∈Hess(N,h)S1 is(3.3)dim⁡C(Cw)=|DPϕλ-1(w-1)|.In particular, Theorem3.3 implies that the odd Betti numbers of Hess(Nhf,h) are 0, and the 2kth even Betti number is precisely the number of fixed points w in Hess(Nhf,h)S1 such that |DPϕλ-1(w-1)|=k. Given the regular nilpotent Hessenberg variety Hess(Nhf,h), denote by bk its 2kth Betti number, that is,
(3.4)bk:=dim⁡CH2k(Hess(Nhf,h)).
We may now formulate the conditions that guarantee that roℓℓ:Hess(Nhf,h)S1→Sn is a successful outcome of Betti pinball. For more details we refer the reader to [1, Section 3]. It suffices to check the following(1)
roℓℓ:Hess(Nhf,h)S1→Sn is injective,(2)
for everyw∈Hess(Nhf,h)S1, we have roℓℓ(w)≤w in Bruhat order, and(3)
for everyk≥0, k∈ℤ, we have
(3.5)bk=|{roll(w)∣w∈Hess(Nhf,h)S1withl(roll(w))=k}|,
where ℓ(roℓℓ(w)) denotes the Bruhat length of roℓℓ(w)∈Sn.We prove each claim in turn. For the first assertion we restrict to two special cases of Hessenberg varieties.Lemma 3.4.
Suppose thatHess(Nhf,h) is either a regular nilpotent Hessenberg variety or a nilpotent Springer variety. Then the function roℓℓ:Hess(Nhf,h)S1→Sn is injective.Proof.
Since the associationx=(x2,x3,…,xn)↦ω(x) given in (2.6) is a bijection, it suffices to show that the map which sends a Hessenberg fixed point w∈Hess(h)S1 to the list of top parts x of its associated permissible filling is injective. Mbirika shows that, in the cases of regular nilpotent Hessenberg varieties and nilpotent Springer varieties, there exists an inverse to this map (Mbirika works with monomials in n-1 variables constructed from the list of top parts, but this is equivalent data) [14, Section 3.2]. The result follows.Lemma 3.5.
For everyw∈Hess(h)S1, one has roℓℓ(w)≤w in Bruhat order.Proof.
Since Bruhat order is preserved under taking inverses, it suffices to prove thatω(x) is Bruhat-less than w-1. For any permutation u∈Sn, set
(3.6)yl:={(a,l)∣(a,l)isaninversioninu}
and let y:=(y2,y3,…,yn). Then the association (2.6) applied to the vector y recovers the permutation u. By definition of ϕλ and the definition of dimension pairs, the set DPϕλ-1(w-1) is always a subset of the set of inversions of the permutation w-1. From Fact 1 it follows that the permutation ω(x) is Bruhat-less than w-1 as desired.Lemma 3.6.
LetNhf:ℂn→ℂn be a nilpotent matrix in highest form chosen as in Remark 2.2 and let λ:=λNhf. Let h:{1,2,…,n}→{1,2,…,n} be a Hessenberg function and Hess(Nhf,h) the associated nilpotent Hessenberg variety. For every k≥0, k∈ℤ, one has
(3.7)bk=|{roll(w)∣w∈Hess(h)S1withl(roll(w))=k}|,
where ℓ(roℓℓ(w)) denotes the Bruhat length of roℓℓ(w)∈Sn.Proof.
By construction,roℓℓ(w) has a reduced word decomposition consisting of precisely |DPϕλ-1(w-1)| simple transpositions. Hence its Bruhat length is |DPϕλ-1(w-1)|. By Theorem 3.3, bk is precisely the number of fixed points w with |DPϕλ-1(w-1)|=k so the result follows.The following is immediate from the above lemmas and the definition of Betti pinball given in [1, Section 3].Proposition 3.7.
Suppose thatHess(Nhf,h) is either a regular nilpotent Hessenberg variety or a nilpotent Springer variety. Then the association w↦roℓℓ(w) given by the dimension pair algorithm is a possible outcome of a successful game of Betti poset pinball played with ambient partially ordered set Sn equipped with Bruhat order, rank function ρ=ℓ:Sn→ℤ given by Bruhat length, initial subset Hess(h)S1⊆Sn, and target Betti numbers bk:=dimℂH2k(Hess(h);ℂ).Remark 3.8.
Lemmas3.5 and 3.6 hold for general nilpotent Nhf and Hessenberg functions h. Hence to prove that Proposition 3.7 holds for more general cases of nilpotent Hessenberg varieties, it suffices to check that the injectivity assertion (1) above holds. We do not know counterexamples where the injectivity fails. It would be of interest to clarify the situation for more general Nhf and h.


### 3.2. Betti Pinball, Poset-Upper-Triangularity, and Module Bases

In the context of aGKM-compatible subspace of a GKM space [1, Definition  4.5], it is explained in [1, Section 4] that the outcome of a game of poset pinball may be interpreted as specifying a set of equivariant cohomology classes which, under additional conditions, yields a module basis for the equivariant cohomology of the GKM-compatible subspace. In this paper, the GKM space is the flag variety ℱℓags(ℂn) with the standard Tn-action and the GKM-compatible subspace is Hess(Nhf,h) with the S1-action specified above. Consider the HTn*(pt)-module basis for HTn*(ℱℓags(ℂn)) given by the equivariant Schubert classes {σw}w∈Sn. The dimension pair algorithm then specifies the set(3.8){proll(w)∣w∈Hess(Nhf,h)S1}⊆HS1*(Hess(Nhf,h)),
where for any u∈Sn the class pu:=π(σu) is defined to be the image of σu under the natural projection map(3.9)π:HTn*(Flags(Cn))⟶HS1*(Hess(Nhf,h))
induced by the inclusion of groups S1↪Tn and the S1-equivariant inclusion of spaces Hess(Nhf,h)↪ℱℓags(ℂn). We refer to the images pu as Hessenberg Schubert classes.Following the methods of [1] we view HTn*(ℱℓags(ℂn)) and HS1*(Hess(Nhf,h)) as subrings of(3.10)HTn*((Flags(Cn))Tn)≅⨁w∈SnHTn*(pt)respectivelyHS1*((Hess(Nhf,h))S1)≅⨁w∈Hess(Nhf,h)S1HS1*(pt).
We denote by σw(w′), proℓℓ(w)(w′) the value of the w′th coordinate in the direct sums above, for w, w′∈Sn or w, w′∈Hess(Nhf,h)S1, respectively. If(3.11)proll(w)(w)≠0,proll(w)(w′)=0ifw≰w′
for all w, w′∈Hess(Nhf,h)S1, then the set {proℓℓ(w)∣w∈Hess(Nhf,h)S1} in HS1*(Hess(Nhf,h)) is called poset-upper-triangular (with respect to the partial order on Hess(Nhf,h)S1⊆Sn induced from Bruhat order) [1, Definition 2.3]. Finally, recall that the cohomology degree of an equivariant Schubert class σw (and hence also the corresponding Hessenberg Schubert class pw) is 2·ℓ(w).The following is immediate from [1, Proposition  4.14] and the above discussion.Proposition 3.9.
LetHess(Nhf,h) be either a regular nilpotent Hessenberg variety or a nilpotent Springer variety. Let roℓℓ:Hess(Nhf,h)S1→Sn be the dimension pair algorithm defined above. Suppose (3.11) holds for all w∈Hess(Nhf,h)S1. Then the Hessenberg Schubert classes {proℓℓ(w)∣w∈Hess(Nhf,h)S1} form a HS1*(pt)-module basis for the S1-equivariant cohomology ring HS1*(Hess(Nhf,h)).Therefore, in order to prove that the Hessenberg Schubert classes above form a module basis as desired, it suffices to show that they satisfy the upper-triangularity conditions (3.11) for all w, w′∈Hess(Nhf,h)S1. The proof of this assertion, for a special class of regular nilpotent Hessenberg varieties closely related to Peterson varieties, is the content of Sections 4 and 5.We close the section with a brief discussion of matchings. Following [1, Section 4.3], define(3.12)deg⁡Hess(Nhf,h)(w):=dim⁡C(Cw)
to be the (complex) dimension of the affine cell Cw containing the fixed point w in Tymoczko’s paving by affines of Hess(Nhf,h) in Theorem 3.3. Then from the discussion above we know(3.13)deg⁡Hess(Nhf,h)(w)=|DPϕλ-1(w-1)|=l(roll(w)),
and since the cohomology degree of proℓℓ(w) is 2·ℓ(roℓℓ(w)), we see that the association w↦roℓℓ(w) from Hess(Nhf,h)S1→Sn is also amatchingin the sense of [1] with respect to degHess(Nhf,h) and rank function ρ on Sn given by Bruhat length. Thus the fact that the {proℓℓ(w)∣w∈Hess(Nhf,h)S1} form a module basis can also be deduced from [1, Theorem  4.18].


## 3.1. The Dimension Pair Algorithm for Nilpotent Hessenberg Varieties

LetNhf be a nilpotent n×n matrix in highest form chosen as in Remark 2.2 and let λ:=λNhf. Let h:{1,2,…,n}→{1,2,…,n} be a Hessenberg function and Hess(Nhf,h) the corresponding nilpotent Hessenberg variety.The definition of the dimension pair algorithm is pure combinatorics. It produces for each Hessenberg fixed pointw∈Hess(Nhf,h)S1 an element in Sn. Following terminology of poset pinball, we denote this function by(3.1)roll:Hess(Nhf,h)S1⟶Sn.Definition 3.1 (the dimension pair algorithm).
We defineroℓℓ:Hess(Nhf,h)S1→Sn as follows.(1)
Letw∈Hess(Nhf,h)S1 and let ϕλ-1(w-1) be its corresponding permissible filling as defined in (2.9).(2)
LetDPϕλ-1(w-1) be the set of dimension pairs in the permissible filling ϕλ-1(w-1).(3)
For eachℓ with 2≤ℓ≤n, set
(3.2)xl:=|{(a,l)∣(a,l)∈DPϕλ-1(w-1)}|
as in (2.4) and define x:=(x2,…,xn).(4)
Defineroℓℓ(w):=(ω(x))-1 where ω(x) is the permutation associated to the integer vector x defined in (2.6).Example 3.2.
Letλ, h be as in Example 2.5. The permutation w=43215∈Sn is in Hess(Nhf,h)S1, as can be checked. The associated permissible filling is  .  In Example 2.9 we saw that the associated permutation is s1(s2s1)s3, so we conclude roℓℓ(w)=s3(s1s2)s1.We next show that the rolldown functionroℓℓ:Hess(h)S1→Sn defined by the dimension pair algorithm above satisfies the conditions to be a successful outcome of Betti poset pinball as in [1] in certain cases of nilpotent Hessenberg varieties. The statement of one of the conditions requires advance knowledge of the Betti numbers of nilpotent Hessenberg varieties, for which we recall the following result (reformulated in our language) from [16].Theorem 3.3 (see [16, Theorem  1.1]).
LetNhf:ℂn→ℂn be a nilpotent matrix in highest form chosen as in Remark 2.2 and let λ:=λNhf. Let h:{1,2,…,n}→{1,2,…,n} be a Hessenberg function and let Hess(Nhf,h) denote the corresponding nilpotent Hessenberg variety. There is a paving by (complex) affine cells of Hess(Nhf,h) such that(i)
the affine cells are in one-to-one correspondence withHess(Nhf,h)S1, and(ii)
the (complex) dimension of the affine cellCw corresponding to a fixed point w∈Hess(N,h)S1 is(3.3)dim⁡C(Cw)=|DPϕλ-1(w-1)|.In particular, Theorem3.3 implies that the odd Betti numbers of Hess(Nhf,h) are 0, and the 2kth even Betti number is precisely the number of fixed points w in Hess(Nhf,h)S1 such that |DPϕλ-1(w-1)|=k. Given the regular nilpotent Hessenberg variety Hess(Nhf,h), denote by bk its 2kth Betti number, that is,
(3.4)bk:=dim⁡CH2k(Hess(Nhf,h)).
We may now formulate the conditions that guarantee that roℓℓ:Hess(Nhf,h)S1→Sn is a successful outcome of Betti pinball. For more details we refer the reader to [1, Section 3]. It suffices to check the following(1)
roℓℓ:Hess(Nhf,h)S1→Sn is injective,(2)
for everyw∈Hess(Nhf,h)S1, we have roℓℓ(w)≤w in Bruhat order, and(3)
for everyk≥0, k∈ℤ, we have
(3.5)bk=|{roll(w)∣w∈Hess(Nhf,h)S1withl(roll(w))=k}|,
where ℓ(roℓℓ(w)) denotes the Bruhat length of roℓℓ(w)∈Sn.We prove each claim in turn. For the first assertion we restrict to two special cases of Hessenberg varieties.Lemma 3.4.
Suppose thatHess(Nhf,h) is either a regular nilpotent Hessenberg variety or a nilpotent Springer variety. Then the function roℓℓ:Hess(Nhf,h)S1→Sn is injective.Proof.
Since the associationx=(x2,x3,…,xn)↦ω(x) given in (2.6) is a bijection, it suffices to show that the map which sends a Hessenberg fixed point w∈Hess(h)S1 to the list of top parts x of its associated permissible filling is injective. Mbirika shows that, in the cases of regular nilpotent Hessenberg varieties and nilpotent Springer varieties, there exists an inverse to this map (Mbirika works with monomials in n-1 variables constructed from the list of top parts, but this is equivalent data) [14, Section 3.2]. The result follows.Lemma 3.5.
For everyw∈Hess(h)S1, one has roℓℓ(w)≤w in Bruhat order.Proof.
Since Bruhat order is preserved under taking inverses, it suffices to prove thatω(x) is Bruhat-less than w-1. For any permutation u∈Sn, set
(3.6)yl:={(a,l)∣(a,l)isaninversioninu}
and let y:=(y2,y3,…,yn). Then the association (2.6) applied to the vector y recovers the permutation u. By definition of ϕλ and the definition of dimension pairs, the set DPϕλ-1(w-1) is always a subset of the set of inversions of the permutation w-1. From Fact 1 it follows that the permutation ω(x) is Bruhat-less than w-1 as desired.Lemma 3.6.
LetNhf:ℂn→ℂn be a nilpotent matrix in highest form chosen as in Remark 2.2 and let λ:=λNhf. Let h:{1,2,…,n}→{1,2,…,n} be a Hessenberg function and Hess(Nhf,h) the associated nilpotent Hessenberg variety. For every k≥0, k∈ℤ, one has
(3.7)bk=|{roll(w)∣w∈Hess(h)S1withl(roll(w))=k}|,
where ℓ(roℓℓ(w)) denotes the Bruhat length of roℓℓ(w)∈Sn.Proof.
By construction,roℓℓ(w) has a reduced word decomposition consisting of precisely |DPϕλ-1(w-1)| simple transpositions. Hence its Bruhat length is |DPϕλ-1(w-1)|. By Theorem 3.3, bk is precisely the number of fixed points w with |DPϕλ-1(w-1)|=k so the result follows.The following is immediate from the above lemmas and the definition of Betti pinball given in [1, Section 3].Proposition 3.7.
Suppose thatHess(Nhf,h) is either a regular nilpotent Hessenberg variety or a nilpotent Springer variety. Then the association w↦roℓℓ(w) given by the dimension pair algorithm is a possible outcome of a successful game of Betti poset pinball played with ambient partially ordered set Sn equipped with Bruhat order, rank function ρ=ℓ:Sn→ℤ given by Bruhat length, initial subset Hess(h)S1⊆Sn, and target Betti numbers bk:=dimℂH2k(Hess(h);ℂ).Remark 3.8.
Lemmas3.5 and 3.6 hold for general nilpotent Nhf and Hessenberg functions h. Hence to prove that Proposition 3.7 holds for more general cases of nilpotent Hessenberg varieties, it suffices to check that the injectivity assertion (1) above holds. We do not know counterexamples where the injectivity fails. It would be of interest to clarify the situation for more general Nhf and h.


## 3.2. Betti Pinball, Poset-Upper-Triangularity, and Module Bases

In the context of aGKM-compatible subspace of a GKM space [1, Definition  4.5], it is explained in [1, Section 4] that the outcome of a game of poset pinball may be interpreted as specifying a set of equivariant cohomology classes which, under additional conditions, yields a module basis for the equivariant cohomology of the GKM-compatible subspace. In this paper, the GKM space is the flag variety ℱℓags(ℂn) with the standard Tn-action and the GKM-compatible subspace is Hess(Nhf,h) with the S1-action specified above. Consider the HTn*(pt)-module basis for HTn*(ℱℓags(ℂn)) given by the equivariant Schubert classes {σw}w∈Sn. The dimension pair algorithm then specifies the set(3.8){proll(w)∣w∈Hess(Nhf,h)S1}⊆HS1*(Hess(Nhf,h)),
where for any u∈Sn the class pu:=π(σu) is defined to be the image of σu under the natural projection map(3.9)π:HTn*(Flags(Cn))⟶HS1*(Hess(Nhf,h))
induced by the inclusion of groups S1↪Tn and the S1-equivariant inclusion of spaces Hess(Nhf,h)↪ℱℓags(ℂn). We refer to the images pu as Hessenberg Schubert classes.Following the methods of [1] we view HTn*(ℱℓags(ℂn)) and HS1*(Hess(Nhf,h)) as subrings of(3.10)HTn*((Flags(Cn))Tn)≅⨁w∈SnHTn*(pt)respectivelyHS1*((Hess(Nhf,h))S1)≅⨁w∈Hess(Nhf,h)S1HS1*(pt).
We denote by σw(w′), proℓℓ(w)(w′) the value of the w′th coordinate in the direct sums above, for w, w′∈Sn or w, w′∈Hess(Nhf,h)S1, respectively. If(3.11)proll(w)(w)≠0,proll(w)(w′)=0ifw≰w′
for all w, w′∈Hess(Nhf,h)S1, then the set {proℓℓ(w)∣w∈Hess(Nhf,h)S1} in HS1*(Hess(Nhf,h)) is called poset-upper-triangular (with respect to the partial order on Hess(Nhf,h)S1⊆Sn induced from Bruhat order) [1, Definition 2.3]. Finally, recall that the cohomology degree of an equivariant Schubert class σw (and hence also the corresponding Hessenberg Schubert class pw) is 2·ℓ(w).The following is immediate from [1, Proposition  4.14] and the above discussion.Proposition 3.9.
LetHess(Nhf,h) be either a regular nilpotent Hessenberg variety or a nilpotent Springer variety. Let roℓℓ:Hess(Nhf,h)S1→Sn be the dimension pair algorithm defined above. Suppose (3.11) holds for all w∈Hess(Nhf,h)S1. Then the Hessenberg Schubert classes {proℓℓ(w)∣w∈Hess(Nhf,h)S1} form a HS1*(pt)-module basis for the S1-equivariant cohomology ring HS1*(Hess(Nhf,h)).Therefore, in order to prove that the Hessenberg Schubert classes above form a module basis as desired, it suffices to show that they satisfy the upper-triangularity conditions (3.11) for all w, w′∈Hess(Nhf,h)S1. The proof of this assertion, for a special class of regular nilpotent Hessenberg varieties closely related to Peterson varieties, is the content of Sections 4 and 5.We close the section with a brief discussion of matchings. Following [1, Section 4.3], define(3.12)deg⁡Hess(Nhf,h)(w):=dim⁡C(Cw)
to be the (complex) dimension of the affine cell Cw containing the fixed point w in Tymoczko’s paving by affines of Hess(Nhf,h) in Theorem 3.3. Then from the discussion above we know(3.13)deg⁡Hess(Nhf,h)(w)=|DPϕλ-1(w-1)|=l(roll(w)),
and since the cohomology degree of proℓℓ(w) is 2·ℓ(roℓℓ(w)), we see that the association w↦roℓℓ(w) from Hess(Nhf,h)S1→Sn is also amatchingin the sense of [1] with respect to degHess(Nhf,h) and rank function ρ on Sn given by Bruhat length. Thus the fact that the {proℓℓ(w)∣w∈Hess(Nhf,h)S1} form a module basis can also be deduced from [1, Theorem  4.18].


## 4. Poset-Upper-Triangularity of Rolldown Classes for 334-Type Hessenberg Varieties

In this section and in Section5 we analyze in detail the dimension pair algorithm in the case of a Hessenberg variety which is closely related to the Peterson variety and in particular prove that the algorithm produces a poset-upper-triangular module basis for its S1-equivariant cohomology ring. Here and below the nilpotent operator N under consideration is always the principal nilpotent, so we omit the N from the notation and write Hess(h). Similarly the corresponding Young diagram is always λ=(n) so we omit the λ from notation and write 𝒫Fiℓℓ(h) instead of 𝒫Fiℓℓ(λ,h).We fix for this discussion the Hessenberg function given by(4.1)h(1)=h(2)=3,h(i)=i+1for3≤i≤n-1,h(n)=n.
The only difference between this function h and the Hessenberg function for the Peterson variety studied in [13] is that the value of h(1) is 3 instead of 2. In this sense this h is “close’’ to the Peterson case. Thus it is natural that much of our analysis follows that for Peterson varieties in [13], although it is still necessary to introduce new ideas and terminology to handle the Hessenberg fixed points in Hess(h)S1 which do not arise in the Peterson case.The Hessenberg functionh in (4.1) is trivial if n=3 since in that case h(1)=h(2)=h(3)=3 which implies that the corresponding Hessenberv variety Hess(h) is equal to the full flag variety ℱℓags(ℂ3). Hence we assume n≥4 throughout. Under this assumption and following the notation introduced in Section 2, the Hessenberg function is of the form h=(3,3,4,…). As such, for the purposes of this manuscript, we refer to this family of regular nilpotent Hessenberg varieties as 334-type Hessenberg varieties.Our main result is the following theorem.Theorem 4.1.
Letn≥4 and let Hess(h) be the 334-type Hessenberg variety in ℱℓags(ℂn). Let roℓℓ:Hess(h)S1→Sn be the dimension pair algorithm defined in Section 3. Then
(4.2)proll(w)(w)≠0,proll(w)(w′)=0ifw≰w′
for all w, w′∈Hess(h)S1. In particular the Hessenberg Schubert classes {proℓℓ(w)∣w∈Hess(h)S1} form a HS1*(pt)-module basis for the S1-equivariant cohomology ring HS1*(Hess(h)).For ease of exposition, and because the arguments required are of a somewhat different nature, we prove Theorem4.1 by proving the two assertions in (4.2) separately, as follows.Proposition 4.2.
Letn, h, Hess(h) and roℓℓ be as above. Then,
(4.3)proll(w)(w)≠0
for all w∈Hess(h)S1.Proposition 4.3.
Letn, h, Hess(h) and roℓℓ be as above. Then,
(4.4)proll(w)(w′)=0ifw≰w′
for all w, w′∈Hess(h)S1.The proof of Proposition4.2 is the content of Section 5. The main result of the present section is the upper-triangularity property asserted in Proposition 4.3. Its proof requires a number of preliminary results. We first begin by reformulating the problem in terms of Bruhat relations among the fixed points.Lemma 4.4.
Letn, h, Hess(h) and roℓℓ be as above. If for all w, w′∈Hess(h)S1 one has
(4.5)roll(w)≤w′⟺w≤w′
in Bruhat order, then the Hessenberg Schubert classes {proℓℓ(w)∣w∈Hess(h)S1} satisfy (4.4).Proof.
Recall that the equivariant Schubert classes are poset-upper-triangular with respect to Bruhat order onSn. In particular, for all w, w′∈Sn we have σw(w′)=0 if w′≱w. Since the Hessenberg Schubert classes are images of the Schubert classes and the diagram
(4.6)HTn*(Flags(Cn))HTn*((Flags(Cn))Tn)≅⨁w∈WHTn*(pt)HS1*(Hess(h))HS1*((Hess(h))S1)≅⨁w∈Hess(h)S1HS1*(pt)
commutes, it follows that if for all w, w′∈Hess(h)S1, we have
(4.7)roll(w)≤w′⟺w≤w′
in Bruhat order, then (4.4) follows.The rest of this section is devoted to the proof of (4.5), which by Lemma 4.4 then proves Proposition 4.3.


### 4.1. Fixed Points and Associated Subsets for the 334-Type Hessenberg Variety

In this section we enumerate the fixed points in the 334-type Hessenberg variety and also associate to each fixed point inHess(h)S1 a subset of {1,2,…,n-1}. As we show below, the set of fixed points in the Peterson variety is a subset of the fixed points of the 334-type Hessenberg variety, so the main task is to describe the new fixed points which arise in the 334-type case. We begin with a general observation.Lemma 4.5.
Letn∈ℕ and let h,h′:{1,2,…,n}→{1,2,…,n} be two Hessenberg functions. If h(i)≥h′(i) for all i, 1≤i≤n, then
(4.8)Hess(h′)⊆Hess(h).
The inclusion Hess(h′)↪Hess(h) is S1-equivariant and in particular Hess(h′)S1⊆Hess(h)S1 and 𝒫Fiℓℓ(h′)⊆𝒫Fiℓℓ(h).Proof.
LetV•=(Vi) denote an element in ℱℓags(ℂn). By definition the regular nilpotent Hessenberg variety Hess(h′) associated to h′ is
(4.9)Hess(h′):={V•∈Flags(Cn)∣NVi⊆Vh′(i),∀1≤i≤n},
where N is the principal nilpotent operator. Since Vi⊆Vi+1 for all 1≤i≤n-1 by definition of flags and Vn=ℂn for all flags, if h′(i)≤h(i) for all i, then NVi⊆Vh′(i) automatically implies NVi⊆Vh(i). We conclude Hess(h′)⊆Hess(h). The S1-equivariance of the inclusion Hess(h′)↪Hess(h) follows from the definition of the S1-action of (2.3).Applying Lemma4.5 to the Hessenberg function(4.10)h′(i)=i+1for1≤i≤n-1,h′(n)=n
corresponding to the Peterson variety Hess(h′) and h the 334-type Hessenberg function (4.1), we conclude that all fixed points in Hess(h′)S1 also arise as fixed points in Hess(h)S1. We refer to the elements of Hess(h′)S1 (viewed as elements of Hess(h)S1) as Peterson-type fixed points.It therefore remains to describe Hess(h)S1∖Hess(h′)S1. It turns out to be convenient to do this by first describing 𝒫Fiℓℓ(h)∖𝒫Fiℓℓ(h′).We first introduce some terminology. Given a permutationw=(w(1)w(2)⋯w(n)) in one-line notation and some i, ℓ, we say that the entries {w(i),w(i+1),…,w(i+ℓ)} form a decreasing staircase, or simply a staircase, if w(j+1)=w(j)-1 for all i≤j<i+ℓ. For example, for w=4327516, the segment 432 is a staircase, but 751, though the entries decrease, is not. We will say that a consecutive series of staircases is an increasing sequence of staircases (or simply increasing staircases) if each entry in a given staircase is smaller than any entry in any following staircase (reading from left to right). For instance, w=654987321 is a sequence of staircases 654, 987, and 321, but is not an increasing sequence of staircases since the entries 4,5,6 are not smaller than the entries in the later staircase 321. However, w=321654987 is an increasing sequence of (three) staircases 321, 654, and 987.It is shown in [13] that the S1-fixed points of the Peterson variety Hess(h′) consist precisely of those permutations w∈Sn such that the one-line notation of w is an increasing sequence of staircases. Since such w are equal to their own inverses, the permissible fillings 𝒫Fiℓℓ(h′) corresponding to Hess(h′) are precisely those which are increasing sequences of staircases (cf. Remark 2.12). We now describe the permissible fillings 𝒫Fiℓℓ(h) which are not Peterson-type fillings. We use the language of h-tableau trees introduced by Mbirika; see [14, Section 3.1] for definitions. Recall from Remark 2.10 that we identify permissible fillings with permutations in Sn via one-line notation.Lemma 4.6.
Letn≥4 and let Hess(h) be the 334-type Hessenberg variety in ℱℓags(ℂn). Let w∈𝒫Fiℓℓ(h) be a permissible filling for Hess(h) which is not of Peterson type, that is, w∈𝒫Fiℓℓ(h)∖𝒫Fiℓℓ(h′). Then precisely one of the following hold.(i)
The one-line notation ofw is of the form
(4.11)w′w′312w′′,
where w′ is a (possibly empty) staircase such that w′3 is also a staircase, and w′′ is an increasing sequence of staircases. We refer to these as 312-type permissible fillings.(ii)
The one-line notation ofw is of the form
(4.12)2w′31w′′,
where w′ is a (possibly empty) staircase such that w′3 is also a staircase, and w′′ is an increasing sequence of staircases. We refer to these as 231-type permissible fillings.
Moreover, any filling satisfying either of the above conditions appears in𝒫Fiℓℓ(h)∖𝒫Fiℓℓ(h′).Proof of Lemma4.6.
For any Hessenberg functionh:{1,2,…,n}→{1,2,…,n}, Mbirika shows in [14, Section 3.2] that the Level n fillings in an h-tableau tree are precisely the permissible fillings with respect to h. For the Peterson Hessenberg function in (4.10) Mbirika’s corresponding h-tableau tree has the property that for every k with 1≤k≤n-1 and every vertex at Level k, there are precisely 2 edges going down from that vertex to a Level k+1 vertex (this is because the corresponding degree tupleβ [14, Definition  3.1.1] has βi=2 for all 1≤i≤n-1). In the case of the 334-type Hessenberg function, by definition the h-tableau tree also has precisely 2 edges going down from every vertex at Level k for all k≠2, 1≤k≤n-1. However, at Level 2, each vertex has not 2 but 3 edges pointing down to a vertex at Level 3.
From [14, Section 3] (cf. in particular [14, Definition 3.1.9]) it can be seen that for the case of the Peterson Hessenberg function, the corresponding h-tableau tree at Level 2 has vertices •21• and 1•2•, whereas for the 334-type Hessenberg function, the Level 2 vertices have the form •2•1• and •1•2•. Here the bullets indicate the locations of the h-permissible positions available for the placement of the next index 3, in the sense of [14, Section 3] (cf. in particular [14, Lemma  3.1.8]). In particular, since we saw above that the edges going down from Level 3 onwards are identical in both the Peterson and 334-type Hessenberg case, it follows that the branches of the tree emanating downwards from the two Level 3 vertices 321•, 213• (coming from •2•1•) and the two vertices 1•32•, 12•3• (coming from •1•2•) are identical to the corresponding branches in the h-tableau tree for the Peterson Hessenberg function. Hence all permissible fillings at the final Level n of these branches are of Peterson type. In contrast, the branches emanating from 2•31• and •312• do not appear in the Peterson h-tableau tree, and none of the fillings appearing at Level n in these branches can be Peterson permissible fillings since a 3 appears directly before a 1. Hence it is precisely these branches which account for the permissible fillings which are not of Peterson type. As noted above, the rest of the branch only has 2 edges going down from each vertex with h-permissible positions determined exactly as in the Peterson case. In particular, except for the exceptional 3 appearing directly to the left of a 1, the fillings must consist of decreasing staircases and all possible arrangements of decreasing staircases do appear. The result follows.Example 4.7.
Supposen=8. Then w=54312876 is an example of a 312-type permissible filling where w′=54 and w′′=876. An example of a 231-type permissible filling is w=25431876 where w′=54 and w′′=876. Neither of these are permissible with respect to the Peterson Hessenberg function h′ since a 3 appears directly to the left of a 1. Nevertheless, both of these fillings are closely related to the Peterson-type permissible filling w=54321876; this relationship is closely analyzed and used below.We now give explicit descriptions of the corresponding non-Peterson-type elements inHess(h)S1, obtained by taking inverses of the permissible fillings described in Lemma 4.6.Definition 4.8.
Letw∈Hess(h)S1. We say w is a 312-type (resp.,231-type) fixed point if its inverse w-1 is a permissible filling of 312-type (resp., 231-type).
As observed above, since Peterson-type permissible fillings are equal to their own inverses, in that case there is no distinction between the fillings and their associated fixed points. For the 312 and 231 types, however, this is not the case. We record the following. The proof is a straightforward computation and is left to the reader.Lemma 4.9.
Letw be a 312-type (resp., 231-type) permissible filling. Let a2 be the integer such that a2+1 is the first entry (resp., second entry) in the one-line notation of w. Let w-1 be the corresponding 312-type (resp., 231 type) fixed point. Then,(i)
the one-line notation ofw-1 is the same as that of w for all ℓth entries with ℓ>a2+1,(ii)
ifw is 312-type, then the first a2+1 entries of the one-line notation of w-1 are
(4.13)a2a2+1a2-1a2-2⋯21,(iii)
ifw is 231-type, then the first a2+1 entries of the one-line notation of w-1 are
(4.14)a2+11a2a2-1⋯32.In the case of the Peterson variety, there is a convenient bijective correspondence between the set ofS1-fixed points of the Peterson variety and subsets 𝒜 of {1,2,…,n-1} given as follows [13, Section 2.3]. Let w be a Peterson-type fixed point. Then the corresponding subset is(4.15)A:={i:1≤i≤n-1,w(i)=w(i+1)+1}⊆{1,2,…,n-1}.
In the case of the 334-type Hessenber variety, it is also useful to assign a subset of {1,2,…,n-1} to each fixed point as follows.Definition 4.10.
Letw∈Hess(h)S1. The associated subset of{1,2,…,n}corresponding tow, notated 𝒜(w), is defined as follows.(i)
Supposew is of Peterson type. Then 𝒜(w) is defined to be the set 𝒜 in (4.15).(ii)
Supposew is 312-type. Consider the permutation w′:=ws1 (i.e., swap the a2 and the a2+1 in the one-line notation (4.13)). This is a fixed point of Peterson type. Define 𝒜(w):=𝒜(w′).(iii)
Supposew is 231-type. Consider the permutation
(4.16)w′=ws2s3⋯sa2
(i.e., move the 1 to the right of the 2 in the one-line notation (4.14)). This is a fixed point of Peterson type.   Define 𝒜(w):=𝒜(w′).Example 4.11.
Supposen=8.(i)
Supposew is the Peterson-type fixed point w=54321876. Then 𝒜(w)={1,2,3,4}∪{6,7}. This agrees with the association w↦𝒜(w) used in [13].(ii)
Supposew is the 312-type fixed point w=34217658 (corresponding to the 312-type permissible filling 43127658). Then w′=ws1=43217658 and 𝒜(w):=𝒜(w′)={1,2,3}∪{5,6}.(iii)
Supposew is the 231-type fixed point w=51432768 (corresponding to the 231-type permissible filling 25431768). Then w′=54321768 and 𝒜(w):=𝒜(w′)={1,2,3,4}∪{6}.Remark 4.12.
The three fixed pointsw=54321876, w=45321876, and w=51432876, which are, respectively, of Peterson type, 312 type, and 231-type, all have the same associated subset 𝒜(w)={1,2,3,4}∪{6,7}.It is useful to observe that the 312-type and 231-type fixed points have associated subsets that always contain 1 and 2.Lemma 4.13.
Letw be a 334-type Hessenberg fixed point. Suppose further that w is not of Peterson type. Then {1,2}⊆𝒜(w).Proof.
From the explicit descriptions of the one-line notation of the 312 type (resp., 231-type) fixed points given above, we know that the initial segmenta2a2+1⋯21 (resp, a2+11a2⋯32) in the one-line notation is such that a2≥2. From Definition 4.10 it follows that the first decreasing staircase of the associated Peterson-type fixed point ws1 (resp., ws2s3⋯sa2) is of length at least 3. In particular, the first staircase starts with an integer k which is ≥ 3. The result follows.As noted in Remark4.12, the association w↦𝒜(w) given in Definition 4.10 is not one-to-one and hence in particular not a bijective correspondence. This makes our analysis more complicated than in [13], but the notion is still useful for our arguments below.


### 4.2. Reduced Word Decompositions for 334-Type Fixed Points and Rolldowns

In this section we fix particular choices of reduced word decompositions for the fixed points inHess(h)S1 which we use in our arguments below. We also compute, and fix choices of reduced words for, the rolldowns roℓℓ(w) of the fixed points.The associationw↦𝒜(w) of the previous section allows us to describe these reduced word decompositions in relation to that of the Peterson-type fixed points. Let a be a positive integer and k a nonnegative integer. Recall that a reduced word decomposition of the maximal element (the full inversion) in the subgroup S{a,a+1,…,a+k+1}⊆Sn is given by(4.17)sa(sa+1sa)(sa+2sa+1sa)⋯(sa+ksa+k-1⋯sa+1sa).
For the purposes of this manuscript, we call this the standard reduced word (decomposition) for the maximal element. (This is different from the choice of reduced word decomposition used in [13, Section 2.3].) We denote a consecutive set of integers {a,a+1,…,a+k} for a positive and k a nonnegative integer by [a,a+k]. We say that [a,a+k] is a maximal consecutive substring of 𝒜 if [a,a+k]⊆𝒜 and neither a-1 nor a+k+1 are in 𝒜. It is straightforward that any subset 𝒜 of {1,2,…,n-1} uniquely decomposes into a disjoint union of maximal consecutive substrings(4.18)A=[a1,a2]∪[a3,a4]∪⋯∪[am-1,am].
For instance, for 𝒜={1,2,3,5,6,9,10,11}, the decomposition is 𝒜=[1,3]∪[5,6]∪[9,11]. For any [a,b], denote by w[a,b] the full inversion in the subgroup S[a,b+1]. Then it follows from Definition 4.10 (see also [13, Section 2.3]) that the Peterson-type fixed point associated to 𝒜, which we denote by w𝒜, is the product(4.19)wA:=w[a1,a2]w[a3,a4]w[a5,a6]⋯w[am-1,am].
We fix a choice of reduced word decomposition of w𝒜 given by taking the product of the standard reduced words (4.17) for each of the full inversions w[aj,aj+1] appearing in (4.19). For the purposes of this manuscript we call this the standard reduced word decomposition of a Peterson-type fixed pointw𝒜.Example 4.14.
Letn=7 and let w=4321765 be a Peterson-type fixed point. Then the two decreasing staircases are 4321 and 765, the associated subset 𝒜(w) is {1,2,3}∪{5,6} with maximal consecutive strings [1,3]:={1,2,3} and [5,6]:={5,6}. The standard reduced word decomposition of w is
(4.20)w{1,2,3}∪{5,6}=w[1,3]w[5,6]=s1(s2s1)(s3s2s1)s5(s6s5).We now fix a reduced word decomposition of the non-Peterson-type fixed points.Lemma 4.15.
Letw∈Hess(h)S1 be a fixed point which is not of Peterson type and let 𝒜(w)=[a1,a2]∪[a3,a4]∪⋯∪[am-1,am] be the associated subset with its decomposition into maximal consecutive substrings.(i)
Ifw is 312-type, then a reduced word decomposition for w is given by
(4.21)s1(s2s1)⋯(sa2sa2-1⋯s3s2)w[a3,a4]⋯w[am-1,am].(ii)
Ifw is 231-type, then a reduced word decomposition for w is given by
(4.22)s2(s3s2)⋯(sa2-1sa2-2⋯s3s2)(sa2sa2-1⋯s2s1)w[a3,a4]⋯w[am-1,am],
where the w[aℓ,aℓ+1] in the above expressions are assumed to be given the reduced word decomposition described in (4.17).Proof.
For the first assertion, observe that the explicit description of the one-line notation 312-type fixed points in (4.13) implies that w has precisely 1 fewer inversion than w𝒜(w). An explicit computation shows that the given word (4.21) is equal to w, so it is a word decomposition of w with exactly as many simple transpositions as the Bruhat length of w. In particular it must be reduced. A similar argument proves the second assertion.Example 4.16.
Supposen=7. Suppose w=3421765 is a 312-type fixed point. Then the reduced word decomposition of w given in Lemma 4.15 is
(4.23)w=s1(s2s1)(s3s2)s5(s6s5).
Similarly suppose w=4132765 is a 231-type fixed point. Then the reduced word decomposition of w given in Lemma 4.15 is
(4.24)w=s2(s3s2s1)s5(s6s5).Henceforth, we always use the reduced words given above.Next we explicitly describe the rolldownsroℓℓ(w) associated to each w in Hess(h)S1 by the dimension pair algorithm. We begin with the Peterson-type fixed points. It turns out there are two important subcases of Peterson-type fixed points.Definition 4.17.
We say that a Peterson-type fixed pointwcontains the string321 (or simply contains321) if, in the one-line notation of w, the string 321 appears (equivalently, if {1,2}⊆𝒜(w)). We say wdoes not contain the string321 (or simply does not contain321) otherwise.Remark 4.18.
Note that Definition4.17 is different from the standard notion of pattern-containing or pattern-avoiding permutations since here we require the one-line notation of w to contain the string 321 exactly.Given a subset𝒜={j1<j2<⋯<jk}⊆{1,2,…,n-1} and corresponding Peterson-type fixed point w𝒜, we call the permutation(4.25)sjksjk-1⋯sj2sj1∈Sn
the Peterson case rolldown ofw𝒜. Note that the word (4.25) is in fact a reduced word decomposition of this permutation; we always use this choice of reduced word. The terminology is motivated by the fact that (4.25) is the (inverse of the) permutation given in [13, Definition  4.1]. (The fact that it is the inverse of the permutation used in [13] does not affect the theory very much, as is explained in [13, Proposition  5.16]).Lemma 4.19.
Letn≥4 and let Hess(h) be the 334-type Hessenberg variety in ℱℓags(ℂn). Let w be a Peterson-type fixed point and let 𝒜(w)={j1<j2<⋯<jk} be its associated subset.(i)
Supposew does not contain 321. Then roℓℓ(w) is the Peterson case rolldown of w𝒜(w).(ii)
Supposew does contain 321, that is, 𝒜(w)={j1<j2<⋯<jk} for k≥2 and j1=1 and j2=2. Then roℓℓ(w) is(4.26)roll(w)=sjksjk-1⋯sj3s1s2s1.
In particular, if a Peterson-type fixed point w contains 321, then its rolldown roℓℓ(w) is Bruhat-greater and has Bruhat length 1 greater than the Peterson case rolldown of w.Proof.
Ifw contains a 321, then by Definition 2.6, the pairs (1,3),(2,3), and (1,2) are all dimension pairs in w. Hence 3 appears precisely twice as a top part of a dimension pair and 2 appears precisely once. Thus by constructing the dimension pair algorithm the permutation ω(x) begins with the word s1(s2s1). With respect to all other indices j∈𝒜(w), the 334-type Hessenberg function is identical to the Peterson Hessenberg function and hence, for each such j, the index j+1 appears precisely once as a top part of a dimension pair of w and thus contributes precisely one sj to ω(x). Taking the inverse yields (4.26) as desired.
Ifw does not contain 321, then 3 appears at most once as the top part of a dimension pair in w, and again for all other indices the computations are identical to the Peterson case as above. Hence roℓℓ(w) is identical to the Peterson case rolldown. This completes the proof.Next, we give an explicit description, along with a choice of reduced word decomposition, of the rolldowns corresponding to the non-Peterson-type fixed points.Lemma 4.20.
Letw∈Hess(h)S1 and suppose that w is not of Peterson type. Let 𝒜(w)={j1=1<j2=2<j3<⋯<jk} for some k≥2.(1)
Ifw is of 312-type, then the dimension pair algorithm associates to w the permutation
(4.27)roll(w)=sjksjk-1⋯sj4sj3s1s2.(2)
Ifw is 231-type, then the dimension pair algorithm associates to w the permutation(4.28)roll(w)=sjksjk-1⋯sj4sj3s2s1.Proof.
Supposew is a 312-type fixed point so ϕλ-1(w-1) is a 312-type permissible filling. By definition of dimension pairs, 2 does not appear as the top part of any dimension pair (since it appears to the right of a 1). Also by definition, 3 appears as a top part of the two dimension pairs (1,3) and (2,3). The form of the 312-type permissible fillings described in Lemma 4.6 and the definition of 𝒜(w) imply that the other dimension pairs are precisely the pairs (j,j+1) for j∈𝒜(w) (for j≠1,2), from which it follows that ω(x)=s2s1sj3sj4⋯sjk-1sjk. Taking inverses yields (4.27). The proof of the second assertion is similar.Example 4.21.
(i) Supposew=54321876. This is of Peterson type. Then roℓℓ(w)=(ω(x))-1=s7s6s4s3s1s2s1.(ii)
Supposew=45321876. This is 312-type. Then roℓℓ(w)=(ω(x))-1=s7s6s4s3s1s2.(iii)
Supposew=51432876. This is 231-type. Then roℓℓ(w)=(ω(x))-1=s7s6s4s3s2s1.We conclude the section with a computation of the one-line notation of the rolldowns for different types; we leave proofs to the reader.Lemma 4.22.
Letw be a 334-type Hessenberg fixed point and let 𝒜(w)=[a1,a2]∪⋯∪[am-1,am] be its associated subset with its decomposition into maximal consecutive substrings. Suppose w is of Peterson type that contains 321, 312-type, or of 231 type. Then a1=1, a2≥2 and the first a2+1 entires of the one-line notation of roℓℓ(w) are
(4.29)a2+12134⋯a2
for w of Peterson type that contains 321,
(4.30)2a2+1134⋯a2
for w312-type, and
(4.31)a2+1123⋯a2
for w231-type.


### 4.3. Bruhat Order Relations

In this section we analyze the properties of the associationw↦𝒜(w) with respect to comparisons in Bruhat order.The first two lemmas are straightforward and proofs are left to the reader.Lemma 4.23.
Let𝒜⊆{1,2,…,n-1} and let w𝒜 be the Peterson-type filling associated to 𝒜. Then w𝒜 is maximal in the subgroup S𝒜 of Sn generated by the simple transpositions {si}i∈𝒜. In particular, w𝒜 is Bruhat-bigger than any permutation w∈S𝒜.Lemma 4.24.
Letw∈Hess(h)S1. Suppose w is not of Peterson type. Then w is Bruhat-less than the Peterson type fixed point w𝒜(w) corresponding to 𝒜(w).We also observe that a Bruhat relationw<w′ implies a containment relation of the associated subsets.Lemma 4.25.
Letw, w′∈Hess(h)S1 and let 𝒜(w), 𝒜(w′) be the respective associated subsets. Let si be a simple transposition. Then,(1)
si<w if and only if i∈𝒜(w),(2)
si<roℓℓ(w) if and only if i∈𝒜(w),(3)
ifw≤w′ or roℓℓ(w)≤w′, then 𝒜(w)⊆𝒜(w′).Proof.
Bruhat order is independent of choice of reduced word decomposition forw. Therefore, a simple transposition si is less than w in Bruhat order if and only if si appears in a (and hence any) reduced word decomposition of w. In particular, to prove the first claim it suffices to observe that by the definitions of 𝒜(w), the index i appears in 𝒜(w) precisely when si appears in the choice of reduced word for w given above. A similar argument using the explicit reduced words given for roℓℓ(w) in Lemmas 4.19 and 4.20 proves the second claim. The last claim follows from the first two.We have just seen thatw≤w′ implies 𝒜(w)⊆𝒜(w′). In the case of the Peterson variety Hess(h′) these Bruhat relations are precisely encoded by the partial ordering given by containment of the 𝒜(w); specifically, by Lemma 4.23, w𝒜≤wℬ if and only if 𝒜⊆ℬ. In our 334-type Hessenberg case this is no longer true although the sets 𝒜(w) do still encode the Bruhat data. The precise statements occupy the next several lemmas.We take a moment to recall thetableau criterion for determining Bruhat order in the Weyl group Sn (see, e.g., [17]) which will be useful in the discussion below. For w∈Sn, denote by DR(w) the descent set of w, namely,(4.32)DR(w):={i∣w(i)>w(i+1),1≤i≤n-1}.
For example, for w=368475912 the descent set is DR(w)={3,5,7}.Theorem 4.26 (the tableau criterion [17, Theorem  2.6.3]).
Forw, v∈Sn, let wi,k be the ith element in the increasing rearrangement of w(1),w(2),…,w(k), and similarly for vi,k. Then w≤v in Bruhat order if and only if
(4.33)wi,k≤vi,k,∀k∈DR(w),1≤i≤k.For example, supposew=368475912 and v=694287531. Since DR(w)={3,5,7}, we examine the three increasing rearrangements of initial segments of w and v of lengths 3, 5, and 7, respectively, which we may organize into Young tableaux:
(4.34)Comparing corresponding entries, there are two violations of the tableau condition of the proposition(3>2) in the upper-left corner, so we conclude that w≮v.Now we observe that some Bruhat relations never arise.Lemma 4.27.
Letw, w′∈Hess(h)S1. Let 𝒜(w)=[a1,a2]∪⋯∪[am-1,am] be the associated subset of w with its decomposition into maximal consecutive substrings. Suppose one of the following conditions hold:(1)
w′ is of Peterson type that does not contain 321 while w is not,(2)
w′ is 231-type while w is either of Peterson type that contains 321 or is 312-type. Then w≮w′ and roℓℓ(w)≮w′.Proof.
Ifw′ is of Peterson type that does not contain 321, then {1,2}⊈𝒜(w′) by definition of the associated subsets. All other types (Peterson type that contains 321, or 312-type, or 231-type) have associated subsets containing {1,2} by Lemma 4.13 and by definition of 𝒜(w). The claim (1) now follows from Lemma 4.25.
Next supposew′ is 231-type and w is of Peterson type that contains 321. Then the first two entries of the one-line notation of w must be both strictly greater than 1, and 2∈DR(w). Similarly if w is a 312-type fixed point, then a2≥2. From (4.13) it follows that the first two entries in the one-line notation of w are also strictly greater than 1, and 2∈DR(w). On the other hand, the one-line notation for a 231-type fixed point, in (4.14) has a 1 in the second entry. By the tableau criterion, if w<w′, then, since 2∈DR(w) in both cases under consideration, we must have that one of the first two entries of w is equal to 1, but we have just seen that is impossible. Hence w≮w′. The assertion that roℓℓ(w)≮w′ follows by a similar argument using (4.29), (4.30), and (4.37).For the next lemma and below, we say two fixed points areof the same typeif both are Peterson-type, or both are 312-type, or both are 231-type.Lemma 4.28.
Letw, w′∈Hess(h)S1. Suppose one of the following conditions hold:(i)
w and w′ are of the same type, or(ii)
w is of Peterson type and does not contain 321, and w′ is either 312-type or 231-type, or(iii)
w is either 312-type or 231-type, and w′ is of Peterson type.Then,
(4.35)w<w′iffA(w)⊆A(w′).Proof.
Since the lemma above shows thatw<w′ implies 𝒜(w)⊆𝒜(w′), for all cases it suffices to show the reverse implication. First suppose w and w′ are of the same type and 𝒜(w)⊆𝒜(w′). An examination of the reduced word decompositions of the 334-type fillings given in the above discussion and an argument similar to that in [13] implies w<w′. Now suppose w is of Peterson type and does not contain 321 and w′ is either of 312 type or 231-type. Then since {1,2}⊈𝒜(w), either 1∉𝒜(w) or 2∉𝒜(w). From the explicit reduced word decompositions of 312 or 231-type fixed points chosen above it can be seen that w′ is Bruhat-greater than both w𝒜(w′)∖{1} and w𝒜(w′)∖{2}. The claim now follows from Lemma 4.23. Finally suppose w is either 312-type or 231-type and w′ is of Peterson type. Since 𝒜(w)⊆𝒜(w′) we know from Lemma 4.23 that w𝒜(w)<w𝒜(w′)=w′. Lemma 4.24 shows that w<w𝒜(w) so the result follows.The next step is to show that Bruhat relations between certain Hessenberg fixed points are connected to lengths of initial maximal consecutive substrings in the associated subsets. We need some notation. Let𝒜⊆{1,2,…,n-1}. Recall we denote by w𝒜 the Peterson-type fixed point associated to 𝒜. For the purposes of this discussion we let u𝒜 (resp., v𝒜) denote the 312-type (resp., 231-type) fixed point with associated subset 𝒜. Thus for 𝒜=[1,a] for some a with 2≤a≤n-1, we have(4.36)u[1,a]=(a+1a⋯312a+2a+3⋯n)-1=aa+1a-1a-2⋯21a+2a+3⋯n,(4.37)v[1,a]=(2a+1a⋯431a+2a+3⋯n)-1=a+11aa-1⋯32a+2a+3⋯n
in one-line notation. For general subsets(4.38)A=[a1,a2]∪[a3,a4]∪⋯∪[am-1,am],
with a1=1 and a2≥2, the definitions 312-type and 231-type fixed points imply that(4.39)uA=u[a1,a2]w[a3,a4]⋯w[am-1,am],(4.40)vA=v[a1,a2]w[a3,a4]⋯w[am-1,am].Lemma 4.29.
Let𝒜, ℬ be subsets of {1,2,…,n-1} and let
(4.41)A=[a1,a2]∪[a3,a4]∪⋯∪[am-1,am],B=[b1,b2]∪[b3,b4]∪⋯∪[bm-1,bm]
be the respective decompositions into maximal consecutive substrings. Assume both 𝒜 and ℬ contain {1,2}. Let w𝒜 (resp., v𝒜) be the Peterson-type (resp., 231-type) fixed point corresponding to 𝒜 and let uℬ be the 312-type fixed point corresponding to ℬ. Then
(4.42)wA<uB(resp.,vA<uB)iffA⊆B,b2≥a2+1.Proof.
We begin by recalling two basic observations about Bruhat order inSn. Both follow straightforwardly from its definition in terms of reduced word decompositions. Suppose w,w′∈Sn and assume that w and w′ do not share any simple transpositions in their reduced word decompositions, that is, si<w implies si≮w′ and vice versa. Then firstly, w·w′<w′′ for w′′∈Sn if and only if both w<w′′ and w′<w′′. Secondly, w<w′·w′′ if and only if w<w′′.
Recall thatw𝒜 can be written as
(4.43)wA=w[a1,a2]⋅w[a3,a4]⋯w[am-1,am].
Moreover, each factor appearing in the decomposition (4.43) (resp. (4.40) and (4.39)) for w𝒜 (resp., v𝒜 and uℬ) has the property that it does not share any simple transpositions with any other factor appearing in the decomposition.
Now supposev𝒜 (resp., w𝒜) is Bruhat-less than uℬ. Then we know from Lemma 4.25 that 𝒜⊆ℬ so it suffices to prove b2≥a2+1. From Lemma 4.9 and the definition of the Peterson type fixed points we know that the one-line notation for v𝒜 (resp., w𝒜) has first a2+1 entries
(4.44)a2+11a2a2-1⋯32
(resp., a2+1a2a2-1⋯321) while the one-line notation of uℬ has first b2+1 entries given by
(4.45)b2b2+1b2-1⋯321.
In particular 1∈DR(v𝒜) and also 1∈DR(w𝒜). By the tableau criterion, this implies that the first entry of the one-line notation of v𝒜 and w𝒜 must be less than or equal to the first entry of that of uℬ. Hence a2+1≤b2 as desired.
Conversely suppose𝒜⊆ℬ and b2≥a2+1. Then an examination of the one-line notation of v[a1,a2] (resp., w[a1,a2]) compared to that of u[b1,b2] and another application of the tableau criterion implies that v[a1,a2]<u[b1,b2] and w[a1,a2]<u[b1,b2]. In particular v[a1,a2] and w[a1,a2] are also Bruhat-less than uℬ. Moreover, since 𝒜⊆ℬ it follows that [a3,a4]∪⋯∪[am-1,am]⊆ℬ so Lemma 4.23 implies w′:=w[a3,a4]⋯w[am-1,am]<wℬ=uℬ·s1, where the last equality follows from Lemma 4.9. Since s1 does not appear in any factor of w′ the general fact above implies w′<uℬ. Finally since neither w[a1,a2] nor v[a1,a2] share any simple transpositions with w′, the other general fact above yields v𝒜<uℬ, w𝒜<uℬ as desired.


### 4.4. Proof of Proposition4.3

We may now prove the upper-triangular vanishing property of 334-type Hessenberg Schubert classes.Proof of Proposition4.3.
Letw,w′∈Hess(h)S1 and let
(4.46)A(w)=[a1,a2]∪[a3,a4]∪⋯∪[am-1,am],A(w′)=[a1′,a2′]∪[a3′,a4′]∪⋯∪[ar-1′,ar′]
be the respective associated subsets decomposed into maximal consecutive substrings. By Lemmas 3.5 and 4.4 it suffices to prove that if roℓℓ(w)≤w′, then w≤w′. So suppose roℓℓ(w)≤w′. By Lemma 4.25 this implies 𝒜(w)⊆𝒜(w′). By Lemma 4.28 we can conclude w≤w′ if one of the following holds:(i)
w and w′ are of the same type, or(ii)
w is of Peterson type that does not contain 321, and w′ is either 312-type or 231-type, or(iii)
w is either 312-type or 231-type, and w′ is of Peterson type.Now suppose one of the following holds:(i)
w′ is of Peterson type that does not contain 321 and w is not,(ii)
w′ is 231-type and w is of Peterson type that contains 321, or(iii)
w′ is 231-type and w is 312-type.In these cases, Lemma 4.27 implies that roℓℓ(w)≮w′ so there is nothing to prove.
It remains to discuss the cases when(i)
w is of Peterson type that contains 321 and w′ is 312-type, or(ii)
w is 231-type and w′ is 312-type.By Lemma 4.29 it suffices to show that a2′≥a2+1. Suppose w is of Peterson type that contains 321. In particular a2≥2. From (4.29) we know that 1∈DR(roℓℓ(w)) and the first entry in the one-line notation of roℓℓ(w) is a2+1. On the other hand (4.36) implies the one-line notation of w′ begins with a2′. So if roℓℓ(w)<w′, then the tableau criterion implies a2′≥a2+1 as desired. Now suppose w is of 231-type. Again a2≥2 and from (4.31) we know roℓℓ(w) has 1∈DR(roℓℓ(w)) and a2+1 as its first entry. By the same argument, a2′≥a2+1 as desired. The result follows.


## 4.1. Fixed Points and Associated Subsets for the 334-Type Hessenberg Variety

In this section we enumerate the fixed points in the 334-type Hessenberg variety and also associate to each fixed point inHess(h)S1 a subset of {1,2,…,n-1}. As we show below, the set of fixed points in the Peterson variety is a subset of the fixed points of the 334-type Hessenberg variety, so the main task is to describe the new fixed points which arise in the 334-type case. We begin with a general observation.Lemma 4.5.
Letn∈ℕ and let h,h′:{1,2,…,n}→{1,2,…,n} be two Hessenberg functions. If h(i)≥h′(i) for all i, 1≤i≤n, then
(4.8)Hess(h′)⊆Hess(h).
The inclusion Hess(h′)↪Hess(h) is S1-equivariant and in particular Hess(h′)S1⊆Hess(h)S1 and 𝒫Fiℓℓ(h′)⊆𝒫Fiℓℓ(h).Proof.
LetV•=(Vi) denote an element in ℱℓags(ℂn). By definition the regular nilpotent Hessenberg variety Hess(h′) associated to h′ is
(4.9)Hess(h′):={V•∈Flags(Cn)∣NVi⊆Vh′(i),∀1≤i≤n},
where N is the principal nilpotent operator. Since Vi⊆Vi+1 for all 1≤i≤n-1 by definition of flags and Vn=ℂn for all flags, if h′(i)≤h(i) for all i, then NVi⊆Vh′(i) automatically implies NVi⊆Vh(i). We conclude Hess(h′)⊆Hess(h). The S1-equivariance of the inclusion Hess(h′)↪Hess(h) follows from the definition of the S1-action of (2.3).Applying Lemma4.5 to the Hessenberg function(4.10)h′(i)=i+1for1≤i≤n-1,h′(n)=n
corresponding to the Peterson variety Hess(h′) and h the 334-type Hessenberg function (4.1), we conclude that all fixed points in Hess(h′)S1 also arise as fixed points in Hess(h)S1. We refer to the elements of Hess(h′)S1 (viewed as elements of Hess(h)S1) as Peterson-type fixed points.It therefore remains to describe Hess(h)S1∖Hess(h′)S1. It turns out to be convenient to do this by first describing 𝒫Fiℓℓ(h)∖𝒫Fiℓℓ(h′).We first introduce some terminology. Given a permutationw=(w(1)w(2)⋯w(n)) in one-line notation and some i, ℓ, we say that the entries {w(i),w(i+1),…,w(i+ℓ)} form a decreasing staircase, or simply a staircase, if w(j+1)=w(j)-1 for all i≤j<i+ℓ. For example, for w=4327516, the segment 432 is a staircase, but 751, though the entries decrease, is not. We will say that a consecutive series of staircases is an increasing sequence of staircases (or simply increasing staircases) if each entry in a given staircase is smaller than any entry in any following staircase (reading from left to right). For instance, w=654987321 is a sequence of staircases 654, 987, and 321, but is not an increasing sequence of staircases since the entries 4,5,6 are not smaller than the entries in the later staircase 321. However, w=321654987 is an increasing sequence of (three) staircases 321, 654, and 987.It is shown in [13] that the S1-fixed points of the Peterson variety Hess(h′) consist precisely of those permutations w∈Sn such that the one-line notation of w is an increasing sequence of staircases. Since such w are equal to their own inverses, the permissible fillings 𝒫Fiℓℓ(h′) corresponding to Hess(h′) are precisely those which are increasing sequences of staircases (cf. Remark 2.12). We now describe the permissible fillings 𝒫Fiℓℓ(h) which are not Peterson-type fillings. We use the language of h-tableau trees introduced by Mbirika; see [14, Section 3.1] for definitions. Recall from Remark 2.10 that we identify permissible fillings with permutations in Sn via one-line notation.Lemma 4.6.
Letn≥4 and let Hess(h) be the 334-type Hessenberg variety in ℱℓags(ℂn). Let w∈𝒫Fiℓℓ(h) be a permissible filling for Hess(h) which is not of Peterson type, that is, w∈𝒫Fiℓℓ(h)∖𝒫Fiℓℓ(h′). Then precisely one of the following hold.(i)
The one-line notation ofw is of the form
(4.11)w′w′312w′′,
where w′ is a (possibly empty) staircase such that w′3 is also a staircase, and w′′ is an increasing sequence of staircases. We refer to these as 312-type permissible fillings.(ii)
The one-line notation ofw is of the form
(4.12)2w′31w′′,
where w′ is a (possibly empty) staircase such that w′3 is also a staircase, and w′′ is an increasing sequence of staircases. We refer to these as 231-type permissible fillings.
Moreover, any filling satisfying either of the above conditions appears in𝒫Fiℓℓ(h)∖𝒫Fiℓℓ(h′).Proof of Lemma4.6.
For any Hessenberg functionh:{1,2,…,n}→{1,2,…,n}, Mbirika shows in [14, Section 3.2] that the Level n fillings in an h-tableau tree are precisely the permissible fillings with respect to h. For the Peterson Hessenberg function in (4.10) Mbirika’s corresponding h-tableau tree has the property that for every k with 1≤k≤n-1 and every vertex at Level k, there are precisely 2 edges going down from that vertex to a Level k+1 vertex (this is because the corresponding degree tupleβ [14, Definition  3.1.1] has βi=2 for all 1≤i≤n-1). In the case of the 334-type Hessenberg function, by definition the h-tableau tree also has precisely 2 edges going down from every vertex at Level k for all k≠2, 1≤k≤n-1. However, at Level 2, each vertex has not 2 but 3 edges pointing down to a vertex at Level 3.
From [14, Section 3] (cf. in particular [14, Definition 3.1.9]) it can be seen that for the case of the Peterson Hessenberg function, the corresponding h-tableau tree at Level 2 has vertices •21• and 1•2•, whereas for the 334-type Hessenberg function, the Level 2 vertices have the form •2•1• and •1•2•. Here the bullets indicate the locations of the h-permissible positions available for the placement of the next index 3, in the sense of [14, Section 3] (cf. in particular [14, Lemma  3.1.8]). In particular, since we saw above that the edges going down from Level 3 onwards are identical in both the Peterson and 334-type Hessenberg case, it follows that the branches of the tree emanating downwards from the two Level 3 vertices 321•, 213• (coming from •2•1•) and the two vertices 1•32•, 12•3• (coming from •1•2•) are identical to the corresponding branches in the h-tableau tree for the Peterson Hessenberg function. Hence all permissible fillings at the final Level n of these branches are of Peterson type. In contrast, the branches emanating from 2•31• and •312• do not appear in the Peterson h-tableau tree, and none of the fillings appearing at Level n in these branches can be Peterson permissible fillings since a 3 appears directly before a 1. Hence it is precisely these branches which account for the permissible fillings which are not of Peterson type. As noted above, the rest of the branch only has 2 edges going down from each vertex with h-permissible positions determined exactly as in the Peterson case. In particular, except for the exceptional 3 appearing directly to the left of a 1, the fillings must consist of decreasing staircases and all possible arrangements of decreasing staircases do appear. The result follows.Example 4.7.
Supposen=8. Then w=54312876 is an example of a 312-type permissible filling where w′=54 and w′′=876. An example of a 231-type permissible filling is w=25431876 where w′=54 and w′′=876. Neither of these are permissible with respect to the Peterson Hessenberg function h′ since a 3 appears directly to the left of a 1. Nevertheless, both of these fillings are closely related to the Peterson-type permissible filling w=54321876; this relationship is closely analyzed and used below.We now give explicit descriptions of the corresponding non-Peterson-type elements inHess(h)S1, obtained by taking inverses of the permissible fillings described in Lemma 4.6.Definition 4.8.
Letw∈Hess(h)S1. We say w is a 312-type (resp.,231-type) fixed point if its inverse w-1 is a permissible filling of 312-type (resp., 231-type).
As observed above, since Peterson-type permissible fillings are equal to their own inverses, in that case there is no distinction between the fillings and their associated fixed points. For the 312 and 231 types, however, this is not the case. We record the following. The proof is a straightforward computation and is left to the reader.Lemma 4.9.
Letw be a 312-type (resp., 231-type) permissible filling. Let a2 be the integer such that a2+1 is the first entry (resp., second entry) in the one-line notation of w. Let w-1 be the corresponding 312-type (resp., 231 type) fixed point. Then,(i)
the one-line notation ofw-1 is the same as that of w for all ℓth entries with ℓ>a2+1,(ii)
ifw is 312-type, then the first a2+1 entries of the one-line notation of w-1 are
(4.13)a2a2+1a2-1a2-2⋯21,(iii)
ifw is 231-type, then the first a2+1 entries of the one-line notation of w-1 are
(4.14)a2+11a2a2-1⋯32.In the case of the Peterson variety, there is a convenient bijective correspondence between the set ofS1-fixed points of the Peterson variety and subsets 𝒜 of {1,2,…,n-1} given as follows [13, Section 2.3]. Let w be a Peterson-type fixed point. Then the corresponding subset is(4.15)A:={i:1≤i≤n-1,w(i)=w(i+1)+1}⊆{1,2,…,n-1}.
In the case of the 334-type Hessenber variety, it is also useful to assign a subset of {1,2,…,n-1} to each fixed point as follows.Definition 4.10.
Letw∈Hess(h)S1. The associated subset of{1,2,…,n}corresponding tow, notated 𝒜(w), is defined as follows.(i)
Supposew is of Peterson type. Then 𝒜(w) is defined to be the set 𝒜 in (4.15).(ii)
Supposew is 312-type. Consider the permutation w′:=ws1 (i.e., swap the a2 and the a2+1 in the one-line notation (4.13)). This is a fixed point of Peterson type. Define 𝒜(w):=𝒜(w′).(iii)
Supposew is 231-type. Consider the permutation
(4.16)w′=ws2s3⋯sa2
(i.e., move the 1 to the right of the 2 in the one-line notation (4.14)). This is a fixed point of Peterson type.   Define 𝒜(w):=𝒜(w′).Example 4.11.
Supposen=8.(i)
Supposew is the Peterson-type fixed point w=54321876. Then 𝒜(w)={1,2,3,4}∪{6,7}. This agrees with the association w↦𝒜(w) used in [13].(ii)
Supposew is the 312-type fixed point w=34217658 (corresponding to the 312-type permissible filling 43127658). Then w′=ws1=43217658 and 𝒜(w):=𝒜(w′)={1,2,3}∪{5,6}.(iii)
Supposew is the 231-type fixed point w=51432768 (corresponding to the 231-type permissible filling 25431768). Then w′=54321768 and 𝒜(w):=𝒜(w′)={1,2,3,4}∪{6}.Remark 4.12.
The three fixed pointsw=54321876, w=45321876, and w=51432876, which are, respectively, of Peterson type, 312 type, and 231-type, all have the same associated subset 𝒜(w)={1,2,3,4}∪{6,7}.It is useful to observe that the 312-type and 231-type fixed points have associated subsets that always contain 1 and 2.Lemma 4.13.
Letw be a 334-type Hessenberg fixed point. Suppose further that w is not of Peterson type. Then {1,2}⊆𝒜(w).Proof.
From the explicit descriptions of the one-line notation of the 312 type (resp., 231-type) fixed points given above, we know that the initial segmenta2a2+1⋯21 (resp, a2+11a2⋯32) in the one-line notation is such that a2≥2. From Definition 4.10 it follows that the first decreasing staircase of the associated Peterson-type fixed point ws1 (resp., ws2s3⋯sa2) is of length at least 3. In particular, the first staircase starts with an integer k which is ≥ 3. The result follows.As noted in Remark4.12, the association w↦𝒜(w) given in Definition 4.10 is not one-to-one and hence in particular not a bijective correspondence. This makes our analysis more complicated than in [13], but the notion is still useful for our arguments below.


## 4.2. Reduced Word Decompositions for 334-Type Fixed Points and Rolldowns

In this section we fix particular choices of reduced word decompositions for the fixed points inHess(h)S1 which we use in our arguments below. We also compute, and fix choices of reduced words for, the rolldowns roℓℓ(w) of the fixed points.The associationw↦𝒜(w) of the previous section allows us to describe these reduced word decompositions in relation to that of the Peterson-type fixed points. Let a be a positive integer and k a nonnegative integer. Recall that a reduced word decomposition of the maximal element (the full inversion) in the subgroup S{a,a+1,…,a+k+1}⊆Sn is given by(4.17)sa(sa+1sa)(sa+2sa+1sa)⋯(sa+ksa+k-1⋯sa+1sa).
For the purposes of this manuscript, we call this the standard reduced word (decomposition) for the maximal element. (This is different from the choice of reduced word decomposition used in [13, Section 2.3].) We denote a consecutive set of integers {a,a+1,…,a+k} for a positive and k a nonnegative integer by [a,a+k]. We say that [a,a+k] is a maximal consecutive substring of 𝒜 if [a,a+k]⊆𝒜 and neither a-1 nor a+k+1 are in 𝒜. It is straightforward that any subset 𝒜 of {1,2,…,n-1} uniquely decomposes into a disjoint union of maximal consecutive substrings(4.18)A=[a1,a2]∪[a3,a4]∪⋯∪[am-1,am].
For instance, for 𝒜={1,2,3,5,6,9,10,11}, the decomposition is 𝒜=[1,3]∪[5,6]∪[9,11]. For any [a,b], denote by w[a,b] the full inversion in the subgroup S[a,b+1]. Then it follows from Definition 4.10 (see also [13, Section 2.3]) that the Peterson-type fixed point associated to 𝒜, which we denote by w𝒜, is the product(4.19)wA:=w[a1,a2]w[a3,a4]w[a5,a6]⋯w[am-1,am].
We fix a choice of reduced word decomposition of w𝒜 given by taking the product of the standard reduced words (4.17) for each of the full inversions w[aj,aj+1] appearing in (4.19). For the purposes of this manuscript we call this the standard reduced word decomposition of a Peterson-type fixed pointw𝒜.Example 4.14.
Letn=7 and let w=4321765 be a Peterson-type fixed point. Then the two decreasing staircases are 4321 and 765, the associated subset 𝒜(w) is {1,2,3}∪{5,6} with maximal consecutive strings [1,3]:={1,2,3} and [5,6]:={5,6}. The standard reduced word decomposition of w is
(4.20)w{1,2,3}∪{5,6}=w[1,3]w[5,6]=s1(s2s1)(s3s2s1)s5(s6s5).We now fix a reduced word decomposition of the non-Peterson-type fixed points.Lemma 4.15.
Letw∈Hess(h)S1 be a fixed point which is not of Peterson type and let 𝒜(w)=[a1,a2]∪[a3,a4]∪⋯∪[am-1,am] be the associated subset with its decomposition into maximal consecutive substrings.(i)
Ifw is 312-type, then a reduced word decomposition for w is given by
(4.21)s1(s2s1)⋯(sa2sa2-1⋯s3s2)w[a3,a4]⋯w[am-1,am].(ii)
Ifw is 231-type, then a reduced word decomposition for w is given by
(4.22)s2(s3s2)⋯(sa2-1sa2-2⋯s3s2)(sa2sa2-1⋯s2s1)w[a3,a4]⋯w[am-1,am],
where the w[aℓ,aℓ+1] in the above expressions are assumed to be given the reduced word decomposition described in (4.17).Proof.
For the first assertion, observe that the explicit description of the one-line notation 312-type fixed points in (4.13) implies that w has precisely 1 fewer inversion than w𝒜(w). An explicit computation shows that the given word (4.21) is equal to w, so it is a word decomposition of w with exactly as many simple transpositions as the Bruhat length of w. In particular it must be reduced. A similar argument proves the second assertion.Example 4.16.
Supposen=7. Suppose w=3421765 is a 312-type fixed point. Then the reduced word decomposition of w given in Lemma 4.15 is
(4.23)w=s1(s2s1)(s3s2)s5(s6s5).
Similarly suppose w=4132765 is a 231-type fixed point. Then the reduced word decomposition of w given in Lemma 4.15 is
(4.24)w=s2(s3s2s1)s5(s6s5).Henceforth, we always use the reduced words given above.Next we explicitly describe the rolldownsroℓℓ(w) associated to each w in Hess(h)S1 by the dimension pair algorithm. We begin with the Peterson-type fixed points. It turns out there are two important subcases of Peterson-type fixed points.Definition 4.17.
We say that a Peterson-type fixed pointwcontains the string321 (or simply contains321) if, in the one-line notation of w, the string 321 appears (equivalently, if {1,2}⊆𝒜(w)). We say wdoes not contain the string321 (or simply does not contain321) otherwise.Remark 4.18.
Note that Definition4.17 is different from the standard notion of pattern-containing or pattern-avoiding permutations since here we require the one-line notation of w to contain the string 321 exactly.Given a subset𝒜={j1<j2<⋯<jk}⊆{1,2,…,n-1} and corresponding Peterson-type fixed point w𝒜, we call the permutation(4.25)sjksjk-1⋯sj2sj1∈Sn
the Peterson case rolldown ofw𝒜. Note that the word (4.25) is in fact a reduced word decomposition of this permutation; we always use this choice of reduced word. The terminology is motivated by the fact that (4.25) is the (inverse of the) permutation given in [13, Definition  4.1]. (The fact that it is the inverse of the permutation used in [13] does not affect the theory very much, as is explained in [13, Proposition  5.16]).Lemma 4.19.
Letn≥4 and let Hess(h) be the 334-type Hessenberg variety in ℱℓags(ℂn). Let w be a Peterson-type fixed point and let 𝒜(w)={j1<j2<⋯<jk} be its associated subset.(i)
Supposew does not contain 321. Then roℓℓ(w) is the Peterson case rolldown of w𝒜(w).(ii)
Supposew does contain 321, that is, 𝒜(w)={j1<j2<⋯<jk} for k≥2 and j1=1 and j2=2. Then roℓℓ(w) is(4.26)roll(w)=sjksjk-1⋯sj3s1s2s1.
In particular, if a Peterson-type fixed point w contains 321, then its rolldown roℓℓ(w) is Bruhat-greater and has Bruhat length 1 greater than the Peterson case rolldown of w.Proof.
Ifw contains a 321, then by Definition 2.6, the pairs (1,3),(2,3), and (1,2) are all dimension pairs in w. Hence 3 appears precisely twice as a top part of a dimension pair and 2 appears precisely once. Thus by constructing the dimension pair algorithm the permutation ω(x) begins with the word s1(s2s1). With respect to all other indices j∈𝒜(w), the 334-type Hessenberg function is identical to the Peterson Hessenberg function and hence, for each such j, the index j+1 appears precisely once as a top part of a dimension pair of w and thus contributes precisely one sj to ω(x). Taking the inverse yields (4.26) as desired.
Ifw does not contain 321, then 3 appears at most once as the top part of a dimension pair in w, and again for all other indices the computations are identical to the Peterson case as above. Hence roℓℓ(w) is identical to the Peterson case rolldown. This completes the proof.Next, we give an explicit description, along with a choice of reduced word decomposition, of the rolldowns corresponding to the non-Peterson-type fixed points.Lemma 4.20.
Letw∈Hess(h)S1 and suppose that w is not of Peterson type. Let 𝒜(w)={j1=1<j2=2<j3<⋯<jk} for some k≥2.(1)
Ifw is of 312-type, then the dimension pair algorithm associates to w the permutation
(4.27)roll(w)=sjksjk-1⋯sj4sj3s1s2.(2)
Ifw is 231-type, then the dimension pair algorithm associates to w the permutation(4.28)roll(w)=sjksjk-1⋯sj4sj3s2s1.Proof.
Supposew is a 312-type fixed point so ϕλ-1(w-1) is a 312-type permissible filling. By definition of dimension pairs, 2 does not appear as the top part of any dimension pair (since it appears to the right of a 1). Also by definition, 3 appears as a top part of the two dimension pairs (1,3) and (2,3). The form of the 312-type permissible fillings described in Lemma 4.6 and the definition of 𝒜(w) imply that the other dimension pairs are precisely the pairs (j,j+1) for j∈𝒜(w) (for j≠1,2), from which it follows that ω(x)=s2s1sj3sj4⋯sjk-1sjk. Taking inverses yields (4.27). The proof of the second assertion is similar.Example 4.21.
(i) Supposew=54321876. This is of Peterson type. Then roℓℓ(w)=(ω(x))-1=s7s6s4s3s1s2s1.(ii)
Supposew=45321876. This is 312-type. Then roℓℓ(w)=(ω(x))-1=s7s6s4s3s1s2.(iii)
Supposew=51432876. This is 231-type. Then roℓℓ(w)=(ω(x))-1=s7s6s4s3s2s1.We conclude the section with a computation of the one-line notation of the rolldowns for different types; we leave proofs to the reader.Lemma 4.22.
Letw be a 334-type Hessenberg fixed point and let 𝒜(w)=[a1,a2]∪⋯∪[am-1,am] be its associated subset with its decomposition into maximal consecutive substrings. Suppose w is of Peterson type that contains 321, 312-type, or of 231 type. Then a1=1, a2≥2 and the first a2+1 entires of the one-line notation of roℓℓ(w) are
(4.29)a2+12134⋯a2
for w of Peterson type that contains 321,
(4.30)2a2+1134⋯a2
for w312-type, and
(4.31)a2+1123⋯a2
for w231-type.


## 4.3. Bruhat Order Relations

In this section we analyze the properties of the associationw↦𝒜(w) with respect to comparisons in Bruhat order.The first two lemmas are straightforward and proofs are left to the reader.Lemma 4.23.
Let𝒜⊆{1,2,…,n-1} and let w𝒜 be the Peterson-type filling associated to 𝒜. Then w𝒜 is maximal in the subgroup S𝒜 of Sn generated by the simple transpositions {si}i∈𝒜. In particular, w𝒜 is Bruhat-bigger than any permutation w∈S𝒜.Lemma 4.24.
Letw∈Hess(h)S1. Suppose w is not of Peterson type. Then w is Bruhat-less than the Peterson type fixed point w𝒜(w) corresponding to 𝒜(w).We also observe that a Bruhat relationw<w′ implies a containment relation of the associated subsets.Lemma 4.25.
Letw, w′∈Hess(h)S1 and let 𝒜(w), 𝒜(w′) be the respective associated subsets. Let si be a simple transposition. Then,(1)
si<w if and only if i∈𝒜(w),(2)
si<roℓℓ(w) if and only if i∈𝒜(w),(3)
ifw≤w′ or roℓℓ(w)≤w′, then 𝒜(w)⊆𝒜(w′).Proof.
Bruhat order is independent of choice of reduced word decomposition forw. Therefore, a simple transposition si is less than w in Bruhat order if and only if si appears in a (and hence any) reduced word decomposition of w. In particular, to prove the first claim it suffices to observe that by the definitions of 𝒜(w), the index i appears in 𝒜(w) precisely when si appears in the choice of reduced word for w given above. A similar argument using the explicit reduced words given for roℓℓ(w) in Lemmas 4.19 and 4.20 proves the second claim. The last claim follows from the first two.We have just seen thatw≤w′ implies 𝒜(w)⊆𝒜(w′). In the case of the Peterson variety Hess(h′) these Bruhat relations are precisely encoded by the partial ordering given by containment of the 𝒜(w); specifically, by Lemma 4.23, w𝒜≤wℬ if and only if 𝒜⊆ℬ. In our 334-type Hessenberg case this is no longer true although the sets 𝒜(w) do still encode the Bruhat data. The precise statements occupy the next several lemmas.We take a moment to recall thetableau criterion for determining Bruhat order in the Weyl group Sn (see, e.g., [17]) which will be useful in the discussion below. For w∈Sn, denote by DR(w) the descent set of w, namely,(4.32)DR(w):={i∣w(i)>w(i+1),1≤i≤n-1}.
For example, for w=368475912 the descent set is DR(w)={3,5,7}.Theorem 4.26 (the tableau criterion [17, Theorem  2.6.3]).
Forw, v∈Sn, let wi,k be the ith element in the increasing rearrangement of w(1),w(2),…,w(k), and similarly for vi,k. Then w≤v in Bruhat order if and only if
(4.33)wi,k≤vi,k,∀k∈DR(w),1≤i≤k.For example, supposew=368475912 and v=694287531. Since DR(w)={3,5,7}, we examine the three increasing rearrangements of initial segments of w and v of lengths 3, 5, and 7, respectively, which we may organize into Young tableaux:
(4.34)Comparing corresponding entries, there are two violations of the tableau condition of the proposition(3>2) in the upper-left corner, so we conclude that w≮v.Now we observe that some Bruhat relations never arise.Lemma 4.27.
Letw, w′∈Hess(h)S1. Let 𝒜(w)=[a1,a2]∪⋯∪[am-1,am] be the associated subset of w with its decomposition into maximal consecutive substrings. Suppose one of the following conditions hold:(1)
w′ is of Peterson type that does not contain 321 while w is not,(2)
w′ is 231-type while w is either of Peterson type that contains 321 or is 312-type. Then w≮w′ and roℓℓ(w)≮w′.Proof.
Ifw′ is of Peterson type that does not contain 321, then {1,2}⊈𝒜(w′) by definition of the associated subsets. All other types (Peterson type that contains 321, or 312-type, or 231-type) have associated subsets containing {1,2} by Lemma 4.13 and by definition of 𝒜(w). The claim (1) now follows from Lemma 4.25.
Next supposew′ is 231-type and w is of Peterson type that contains 321. Then the first two entries of the one-line notation of w must be both strictly greater than 1, and 2∈DR(w). Similarly if w is a 312-type fixed point, then a2≥2. From (4.13) it follows that the first two entries in the one-line notation of w are also strictly greater than 1, and 2∈DR(w). On the other hand, the one-line notation for a 231-type fixed point, in (4.14) has a 1 in the second entry. By the tableau criterion, if w<w′, then, since 2∈DR(w) in both cases under consideration, we must have that one of the first two entries of w is equal to 1, but we have just seen that is impossible. Hence w≮w′. The assertion that roℓℓ(w)≮w′ follows by a similar argument using (4.29), (4.30), and (4.37).For the next lemma and below, we say two fixed points areof the same typeif both are Peterson-type, or both are 312-type, or both are 231-type.Lemma 4.28.
Letw, w′∈Hess(h)S1. Suppose one of the following conditions hold:(i)
w and w′ are of the same type, or(ii)
w is of Peterson type and does not contain 321, and w′ is either 312-type or 231-type, or(iii)
w is either 312-type or 231-type, and w′ is of Peterson type.Then,
(4.35)w<w′iffA(w)⊆A(w′).Proof.
Since the lemma above shows thatw<w′ implies 𝒜(w)⊆𝒜(w′), for all cases it suffices to show the reverse implication. First suppose w and w′ are of the same type and 𝒜(w)⊆𝒜(w′). An examination of the reduced word decompositions of the 334-type fillings given in the above discussion and an argument similar to that in [13] implies w<w′. Now suppose w is of Peterson type and does not contain 321 and w′ is either of 312 type or 231-type. Then since {1,2}⊈𝒜(w), either 1∉𝒜(w) or 2∉𝒜(w). From the explicit reduced word decompositions of 312 or 231-type fixed points chosen above it can be seen that w′ is Bruhat-greater than both w𝒜(w′)∖{1} and w𝒜(w′)∖{2}. The claim now follows from Lemma 4.23. Finally suppose w is either 312-type or 231-type and w′ is of Peterson type. Since 𝒜(w)⊆𝒜(w′) we know from Lemma 4.23 that w𝒜(w)<w𝒜(w′)=w′. Lemma 4.24 shows that w<w𝒜(w) so the result follows.The next step is to show that Bruhat relations between certain Hessenberg fixed points are connected to lengths of initial maximal consecutive substrings in the associated subsets. We need some notation. Let𝒜⊆{1,2,…,n-1}. Recall we denote by w𝒜 the Peterson-type fixed point associated to 𝒜. For the purposes of this discussion we let u𝒜 (resp., v𝒜) denote the 312-type (resp., 231-type) fixed point with associated subset 𝒜. Thus for 𝒜=[1,a] for some a with 2≤a≤n-1, we have(4.36)u[1,a]=(a+1a⋯312a+2a+3⋯n)-1=aa+1a-1a-2⋯21a+2a+3⋯n,(4.37)v[1,a]=(2a+1a⋯431a+2a+3⋯n)-1=a+11aa-1⋯32a+2a+3⋯n
in one-line notation. For general subsets(4.38)A=[a1,a2]∪[a3,a4]∪⋯∪[am-1,am],
with a1=1 and a2≥2, the definitions 312-type and 231-type fixed points imply that(4.39)uA=u[a1,a2]w[a3,a4]⋯w[am-1,am],(4.40)vA=v[a1,a2]w[a3,a4]⋯w[am-1,am].Lemma 4.29.
Let𝒜, ℬ be subsets of {1,2,…,n-1} and let
(4.41)A=[a1,a2]∪[a3,a4]∪⋯∪[am-1,am],B=[b1,b2]∪[b3,b4]∪⋯∪[bm-1,bm]
be the respective decompositions into maximal consecutive substrings. Assume both 𝒜 and ℬ contain {1,2}. Let w𝒜 (resp., v𝒜) be the Peterson-type (resp., 231-type) fixed point corresponding to 𝒜 and let uℬ be the 312-type fixed point corresponding to ℬ. Then
(4.42)wA<uB(resp.,vA<uB)iffA⊆B,b2≥a2+1.Proof.
We begin by recalling two basic observations about Bruhat order inSn. Both follow straightforwardly from its definition in terms of reduced word decompositions. Suppose w,w′∈Sn and assume that w and w′ do not share any simple transpositions in their reduced word decompositions, that is, si<w implies si≮w′ and vice versa. Then firstly, w·w′<w′′ for w′′∈Sn if and only if both w<w′′ and w′<w′′. Secondly, w<w′·w′′ if and only if w<w′′.
Recall thatw𝒜 can be written as
(4.43)wA=w[a1,a2]⋅w[a3,a4]⋯w[am-1,am].
Moreover, each factor appearing in the decomposition (4.43) (resp. (4.40) and (4.39)) for w𝒜 (resp., v𝒜 and uℬ) has the property that it does not share any simple transpositions with any other factor appearing in the decomposition.
Now supposev𝒜 (resp., w𝒜) is Bruhat-less than uℬ. Then we know from Lemma 4.25 that 𝒜⊆ℬ so it suffices to prove b2≥a2+1. From Lemma 4.9 and the definition of the Peterson type fixed points we know that the one-line notation for v𝒜 (resp., w𝒜) has first a2+1 entries
(4.44)a2+11a2a2-1⋯32
(resp., a2+1a2a2-1⋯321) while the one-line notation of uℬ has first b2+1 entries given by
(4.45)b2b2+1b2-1⋯321.
In particular 1∈DR(v𝒜) and also 1∈DR(w𝒜). By the tableau criterion, this implies that the first entry of the one-line notation of v𝒜 and w𝒜 must be less than or equal to the first entry of that of uℬ. Hence a2+1≤b2 as desired.
Conversely suppose𝒜⊆ℬ and b2≥a2+1. Then an examination of the one-line notation of v[a1,a2] (resp., w[a1,a2]) compared to that of u[b1,b2] and another application of the tableau criterion implies that v[a1,a2]<u[b1,b2] and w[a1,a2]<u[b1,b2]. In particular v[a1,a2] and w[a1,a2] are also Bruhat-less than uℬ. Moreover, since 𝒜⊆ℬ it follows that [a3,a4]∪⋯∪[am-1,am]⊆ℬ so Lemma 4.23 implies w′:=w[a3,a4]⋯w[am-1,am]<wℬ=uℬ·s1, where the last equality follows from Lemma 4.9. Since s1 does not appear in any factor of w′ the general fact above implies w′<uℬ. Finally since neither w[a1,a2] nor v[a1,a2] share any simple transpositions with w′, the other general fact above yields v𝒜<uℬ, w𝒜<uℬ as desired.


## 4.4. Proof of Proposition4.3

We may now prove the upper-triangular vanishing property of 334-type Hessenberg Schubert classes.Proof of Proposition4.3.
Letw,w′∈Hess(h)S1 and let
(4.46)A(w)=[a1,a2]∪[a3,a4]∪⋯∪[am-1,am],A(w′)=[a1′,a2′]∪[a3′,a4′]∪⋯∪[ar-1′,ar′]
be the respective associated subsets decomposed into maximal consecutive substrings. By Lemmas 3.5 and 4.4 it suffices to prove that if roℓℓ(w)≤w′, then w≤w′. So suppose roℓℓ(w)≤w′. By Lemma 4.25 this implies 𝒜(w)⊆𝒜(w′). By Lemma 4.28 we can conclude w≤w′ if one of the following holds:(i)
w and w′ are of the same type, or(ii)
w is of Peterson type that does not contain 321, and w′ is either 312-type or 231-type, or(iii)
w is either 312-type or 231-type, and w′ is of Peterson type.Now suppose one of the following holds:(i)
w′ is of Peterson type that does not contain 321 and w is not,(ii)
w′ is 231-type and w is of Peterson type that contains 321, or(iii)
w′ is 231-type and w is 312-type.In these cases, Lemma 4.27 implies that roℓℓ(w)≮w′ so there is nothing to prove.
It remains to discuss the cases when(i)
w is of Peterson type that contains 321 and w′ is 312-type, or(ii)
w is 231-type and w′ is 312-type.By Lemma 4.29 it suffices to show that a2′≥a2+1. Suppose w is of Peterson type that contains 321. In particular a2≥2. From (4.29) we know that 1∈DR(roℓℓ(w)) and the first entry in the one-line notation of roℓℓ(w) is a2+1. On the other hand (4.36) implies the one-line notation of w′ begins with a2′. So if roℓℓ(w)<w′, then the tableau criterion implies a2′≥a2+1 as desired. Now suppose w is of 231-type. Again a2≥2 and from (4.31) we know roℓℓ(w) has 1∈DR(roℓℓ(w)) and a2+1 as its first entry. By the same argument, a2′≥a2+1 as desired. The result follows.


## 5. Combinatorial Formulae for Restrictions to Fixed Points of 334-Type Hessenberg Schubert Classes

Our goal in this section is to give a combinatorial formula forproℓℓ(w)(w) from which it follows as a corollary that it is nonzero. This proves Proposition 4.2 and hence Theorem 4.1. Although not strictly necessary for the proof of Proposition 4.2, we choose to prove the explicit formula (Proposition 5.8 below) since such a formula is a first step towards a derivation of a Monk formula for 334-type Hessenberg varieties and because it conveys a flavor of the combinatorics embedded in the GKM theory of Hessenberg varieties which are larger than the Peterson varieties in [13]. Many of our computations are analogues of those in [13, Section 5]. Our main tool is Billey’s formula. We briefly recall some definitions and results (see also discussion in [13, Section 4]).Definition 5.1 (see [13, Definition  4.7]).
Given a permutationw∈Sn, an index j∈{1,2,…,ℓ(w)}, and a choice of reduced word decomposition b=(b1,b2,…,bℓ(w)) (corresponding to the word w=sb1sb2⋯sbℓ(w)) for w, define
(5.1)r(j,b):=sb1sb2⋯sbj-1(tbj-tbj+1).
From the definition it follows thatr(j,b) is an element of HT*(pt)≅Sym(t*)≅ℂ[t1,t2,…,tn] of the form tℓ-tk for some ℓ, k. These elements r(j,b) are the building blocks of Billey’s formula [18, Theorem  4] which computes the restrictions σv(w) of equivariant Schubert classes σv at arbitrary permutations w in Sn.Theorem 5.2 (Billey’s formula, [18, Theorem  4]).
Letw∈Sn. Fix a reduced word decomposition w=sb1sb2⋯sbℓ(w) and let b=(b1,b2,…,bℓ(w)) be the sequence of its indices. Let v∈Sn. Then the restriction σv(w) of the Schubert class σv at the T-fixed point w is given by
(5.2)σv(w)=∑r(j1,b)r(j2,b)⋯r(jl(v),b),
where the sum is taken over subwords sbj1sbj2⋯sbjℓ(v) of b that are reduced words for v.We record the following fact, used in the proof below, which follows straightforwardly from the Billey formula.Fact 2.
Supposev,w∈Sn with v≤w in Bruhat order. Suppose there exists a decomposition w=w′·w′′forw′,w′′∈Sn where v≤w′ and, for all simple transpositions si such that si<v,we have si≰w′′.Then σv(w)=σv(w′).Following terminology in [13], we refer to an individual summand of the expression in the right hand side of (5.2), corresponding to a single reduced subword v=sbj1sbj2⋯sbjℓ(v) of w, as a summand in Billey’s formula. In order to derive formulas for pv(w) where pv is a Hessenberg Schubert class, we use the linear projection πS1:t*→Lie(S1)* dual to the inclusion of our circle subgroup S1 into T given by (2.3). More specifically, since the diagram (4.6) commutes, we have(5.3)pv(w)=∑πS1(r(j1,b))πS1(r(j2,b))⋯πS1(r(jl(v),b)).
We refer to the right hand side of the above equality as Billey’s formula forpv(w). Recall πS1(tℓ-tk+1)=(k+1-ℓ)t for a positive root tℓ-tk+1 [13, Section 5].We also use the following.Definition 5.3 (see [13, Definition 5.4]).
Fix𝒜⊆{1,2,…,n-1}. Define ℋ𝒜:𝒜→𝒜 by
(5.4)HA(j)=themaximalelementinthemaximalconsecutivesubstringofAcontainingj.Definition 5.4 (see [13, Definition 5.5]).
Fix𝒜⊆{1,2,…,n-1}. Define 𝒯𝒜:𝒜→𝒜 by
(5.5)TA(j)=theminimalelementinthemaximalconsecutivesubstringofAcontainingj.We proceed to some preliminary computations. Letb=(b1,…,bℓ(w)) be a reduced word decomposition w=sb1sb2⋯sbℓ(w) of w and let i be an index appearing in b, that is, bℓ=i for some 1≤ℓ≤ℓ(w). Our first computation, Lemma 5.5, gives an expression for πS1(r(ℓ,b)) which shows in particular that the value of πS1(r(ℓ,b)) depends only on the value of the index bℓ=i and not on its location ℓ in the word b. Note that if v=si, then the summands in Billey’s formula for pv(w)=psi(w) are precisely equal to r(ℓ,b) for each ℓ such that bℓ=i. Thus an equivalent formulation of the claim is that the summands in Billey’s formula for psi(w) are all equal. This is analogous to a result in the Peterson case [13, Lemma  5.2] except that, in our situation, the form of the formulas depend on the index i as well as on the type of the fixed point w in question.Lemma 5.5.
Letw∈Hess(h)S1 and let b=(b1,…,bℓ(w)) be the reduced word decomposition of w chosen in Section 4. Let 𝒜(w)=[a1,a2]∪[a3,a4]∪⋯∪[am-1,am] be the associated subset of w decomposed into maximal consecutive substrings. Let i∈{1,2,…,n-1}.(1)
Ifi∉𝒜(w), then each summand in Billey’s formula for psi(w) is 0. In particular, psi(w)=0.(2)
Supposei∈𝒜(w) and suppose one of the following conditions hold:
(a)
w is of Peterson type, or(b)
w is 312-type, or(c)
w is 231-type and i∉[a1,a2]. Then each summand in Billey’s formula for psi(w) is equal to
(5.6)(i-TA(w)(i)+1)t.(3)
Supposew is 231-type and i=1. Then each summand in Billey’s formula for psi(w) is equal to
(5.7)a2t=HA(w)(1)t.(4)
Supposew if 231-type and i∈[2,a2]. Then each summand in Billey’s formula for psi(w) is equal to
(5.8)(i-TA(w)(i))t=(i-1)t.Proof.
Ifi does not occur in 𝒜(w), then each summand is 0 by Billey’s formula for σsi(w), since si≮w and thus never appears in the reduced word decomposition of w. For the next claim, the fact that each summand is equal to (i-𝒯𝒜(w)(i)+1)t for the listed cases follows from examination of the chosen reduced word decompositions of w and an argument identical to that in [13]. Thus it remains to check the cases in which the summand differs from the case of Peterson varieties. First suppose w is 231-type and that i=1. From the choice of explicit reduced word decomposition for such w given in (4.22) and Billey’s formula, it follows that each summand in Billey’s formula for σs1(w) is equal to
(5.9)r2(r3r2)⋯(ra2-1ra2-2⋯r3r2)(ra2ra2-1⋯r2(t1-t2))=r2(r3r2)⋯(ra2-1ra2-2⋯r3r2)(t1-ta2+1)=t1-ta2+1
since the reflection rj switches tj and tj+1. Hence we have ps1(w)=πS1(t1-ta2+1)=(a2+1-1)t=a2t=ℋ𝒜(w)(1)t. Now suppose w is 231-type and i∈[2,a2]. The factor in the reduced word decomposition (4.22) corresponding to [1=a1,a2] is equal to w[2,a2]·s1. By Fact 2, for i>1 the presence of the extra s1 does not affect the Billey computation, so each summand is equal to that for the Peterson type fixed point w[2,a2] and hence is equal to
(5.10)(i-TA(w)(i))=(i-1)t,
as desired.Our next lemma concerns the summands in Billey’s formula forproℓℓ(w)(w) for w∈Hess(h)S1.Lemma 5.6.
Letw∈Hess(h)S1.(i)
Supposew is 312-type or w is of Peterson type that does not contain 321. Then each summand for Billey’s formula for proℓℓ(w)(w) is equal to
(5.11)(∏i∈A(w)(i-TA(w)(i)+1))⋅t|A(w)|.(ii)
Supposew is 231-type. Then each summand for Billey’s formula for proℓℓ(w)(w) is equal to
(5.12)HA(w)(1)⋅(∏i=2HA(w)(1)(i-1))⋅(∏i∈A(w)∖[TA(w)(1),HA(w)(1)](i-TA(w)(i)+1))t|A(w)|.(iii)
Supposew is of Peterson type that contains 321. Then each summand for Billey’s formula for proℓℓ(w)(w) is equal to
(5.13)(∏i∈A(w)(i-TA(w)(i)+1))⋅t|A(w)|+1.Proof.
Before considering the separate cases we make a general observation. By Lemma5.5 and the discussion before Lemma 5.5 we know that the summands in Billey’s formula for psi(w) for i∈𝒜(w) are exactly the terms πS1(r(ℓ,b)) for ℓ such that bℓ=i. Suppose in addition that w∈Hess(h)S1 is such that roℓℓ(w) contains at most one simple transposition si for each i∈{1,2,…,n-1}, that is, roℓℓ(w)=si1si2⋯siℓ(roℓℓ(w)) is a reduced word for roℓℓ(w) where all ik are distinct for 1≤k≤ℓ(roℓℓ(w)). This implies that any subword of a reduced word decomposition b of w which is a reduced word of roℓℓ(w) also must contain precisely one sik for each 1≤k≤ℓ(roℓℓ(w)). From Billey’s formula (5.3) for proℓℓ(w)(w) we know that a summand is of the form
(5.14)πS1(r(j1,b))πS1(r(j2,b))⋯πS1(r(jl(v),b)),
where sbj1sbj2⋯sbjℓ(roℓℓ(w)) is a reduced word of roℓℓ(w). Since{bj1,…,bjℓ(roℓℓ(w))}={i1,i2,…,iℓ(roℓℓ(w))} for each such summand, the quantity (5.14) is equal to
(5.15)∏k=1l(roll(w))psik(w).
We now take cases. Supposew is not a Peterson-type that contains 321. Then from the explicit descriptions of roℓℓ(w) given in Section 4 it follows that roℓℓ(w) contains in its reduced word a single si for each i∈𝒜(w). Thus we are in the situation described in the above paragraph and the claims follow from the computations given in Lemma 5.5.
Supposew is Peterson type and contains 321. Let b be the standard reduced word decomposition (cf. (4.17) and (4.19)) of w. We claim that the only reduced word decompositions of roℓℓ(w) that occur as a subword of b are those which contain two s1’s, only one s2, and precisely one sjℓ for all other jℓ. Indeed let 𝒜(w)=[a1,a2]∪[a3,a4]∪⋯∪[am-1,am] be the decomposition of 𝒜(w) into maximal consecutive substrings. Recall a2≥2 and a1=1 in this case. The rolldown roℓℓ(w) is
(5.16)(samsam-1⋯sam-1)⋯(sa4sa4-1⋯sa3)⋅(sa2sa2-1⋯s1s2s1)
and w is
(5.17)w=w[a1,a2]w[a3,a4]⋯w[am-1,am].
Let ℓ>1. There is only one reduced word decomposition of the factor saℓ+1saℓ+1-1⋯saℓ in roℓℓ(w), so it remains to analyze the subwords of w[a1,a2] which are reduced words of sa2sa2-1⋯s1s2s1. Let b denote the standard reduced word of w[a1,a2]. Note that another valid reduced word of sa2sa2-1⋯s1s2s1 is sa2sa2-1⋯s2s1s2. Since s2 does not commute with s1, the rightmost s2 in the word sa2sa2-1⋯s2s1s2 must appear to the right of the sa2; in particular, there are two s2’s to the right of the sa2 in this word. Since there is only one s2 appearing to the right of the sa2 in b, we conclude that the reduced word of sa2sa2-1⋯s1s2s1 containing two copies of s2 never appears as a subword of b. Hence the only subwords of b contributing to summands in Billey’s formula for proℓℓ(w)(w) contain two s1’s and one s2, as claimed. Now since j1=1 and j2=2 and ps1(w)=(1-𝒯𝒜(w)(1)+1)t=(1-1+1)t=t by Lemma 5.5, the claim follows.We have just seen that all summands in Billey’s formula forproℓℓ(w)(w) are equal for all fixed points w∈Hess(h)S1. In order to finish the computation we must now compute the number of summands which occur.Lemma 5.7.
Letw∈Hess(h)S1.(i)
Supposew is of Peterson type that contains 321. Then the number of summands in Billey’s formula for proℓℓ(w)(w) is ℋ𝒜(w)(1)-1.(ii)
Supposew is of Peterson type that does not contain 321. Then the number of summands in Billey’s formula for proℓℓ(w)(w) is 1.(iii)
Supposew is 312-type. Then the number of summands in Billey’s formula for proℓℓ(w)(w) is ℋ𝒜(w)(1)-1.(iv)
Supposew is 231-type. Then the number of summands in Billey’s formula for proℓℓ(w)(w) is 1.Proof.
We consider each case in turn. Supposew is of Peterson type that contains 321. Let 𝒜(w)=[a1,a2]∪[a3,a4]∪⋯∪[am-1,am] be the decomposition of 𝒜(w) into maximal consecutive substrings. Recall a2≥2 and a1=1 in this case. The rolldown roℓℓ(w) is
(5.18)(samsam-1⋯sam-1)⋯(sa4sa4-1⋯sa3)⋅(sa2sa2-1⋯s1s2s1)
and w is
(5.19)w=w[a1,a2]w[a3,a4]⋯w[am-1,am].
Let ℓ>1. As observed in the proof of Lemma 5.6, there is only one reduced word decomposition of the factor saℓ+1saℓ+1-1⋯saℓ in roℓℓ(w). Moreover, by examination it is evident that it appears only once in the standard reduced word decomposition of the corresponding w[aℓ,aℓ+1] factor in w. Hence in order to count the number of ways roℓℓ(w) appears in w it suffices to count the number of subwords of the standard reduced word decomposition b of w[a1,a2] which are reduced subwords of sa2sa2-1⋯s1s2s1. We already saw in the proof of Lemma 5.6 that the reduced word sa2sa2-1⋯s2s1s2 never appears in b. On the other hand, since s1 commutes with any sk with k≥3, another reduced word decomposition of sa2sa2-1⋯s1s2s1 is s1sa2sa2-1⋯s2s1. From examination of b it can be seen that the word s1sa2sa2-1⋯s2s1 appears as a subword in the standard reduced word of w[a1,a2] precisely a2-1=ℋ𝒜(w)(1)-1 times and that these are the only subwords of b which equal sa2sa2-1⋯s1s2s1. The claim follows.
Supposew is of Peterson type that does not contain 321. Then the rolldown roℓℓ(w) is the Peterson case rolldown so the claim follows from explicit examination of the standard reduced word of w (alternatively from [13, Fact  4.5]).
Supposew is 312-type. Then the rolldown roℓℓ(w) is of the form
(5.20)roll(w)=(samsam-1⋯sam-1)⋯(sa4sa4-1⋯sa3)⋅(sa2sa2-1⋯s1s2)
from Lemma 4.20. By an argument similar to the case of Peterson type that contains 321 it suffices to analyze only the factors in both roℓℓ(w) and w corresponding to the initial maximal consecutive substring [a1,a2]. As above we have
(5.21)sa2sa2-1⋯s1s2=s1sa2sa2-1⋯s2
and again it follows from examination of the standard reduced word of u[a1,a2] that s1sa2sa2-1⋯s2 appears precisely a2-1=ℋ𝒜(w)(1)-1 times.
Finally supposew is 231-type. Then the rolldown roℓℓ(w) coincides with the Peterson case rolldown of w𝒜(w) and the claim follows from examination of the reduced word decomposition (4.22).The following is immediate from Lemmas5.6 and 5.7.Proposition 5.8.
Letw∈Hess(h)S1.(i)
Supposew is of Peterson type that contains 321. Then
(5.22)proll(w)(w)=(HA(w)(1)-1)(∏i∈A(w)(i-TA(w)(i)+1))⋅t|A(w)|+1.(ii)
Supposew is of Peterson type that does not contain 321. Then
(5.23)proll(w)(w)=(∏i∈A(w)(i-TA(w)(i)+1))t|A(w)|.(iii)
Supposew is of type 312. Then,
(5.24)proll(w)(w)=(HA(w)(1)-1)(∏i∈A(w)(i-TA(w)(i)+1))⋅t|A(w)|.(iv)
Supposew is of type 231. Then,(5.25)proll(w)(w)=HA(w)(1)⋅(∏i=2HA(w)(1)(i-1))⋅(∏i∈A(w)∖[TA(w)(1),HA(w)(1)]×(i-TA(w)(i)+1))t|A(w)|.The proofs of the main results are now immediate.Proof of Proposition4.2.
Letw∈Hess(h)S1. From the explicit formulas given in Proposition 5.8 it follows that proℓℓ(w)(w)≠0 for all possible types of fixed points w.Proof of Theorem4.1.
Since both (4.3) and (4.4) are satisfied for all w, w′∈Hess(h)S1 by Propositions 4.3 and 4.2, respectively, the result follows.


## 6. Open Questions

This manuscript raises more questions than it answers. We close by mentioning some of them.Question 1.
Forn≥4, Theorem 4.1 shows that for the case when N:ℂn→ℂn is the principal nilpotent operator and h is the 334-type Hessenberg function, the dimension pair algorithm produces a set of Hessenberg Schubert classes {proℓℓ(w)}w∈Hess(N,h)S1 which are poset-upper-triangular and hence form a HS1*(pt)-module basis for HS1*(Hess(N,h)).(1)
What are other examples ofN and h such that the conclusion of Lemma 3.4 holds (cf. Remark 3.8)?(2)
What are other examples ofN and h for which the dimension pair algorithm produces a successful outcome of Betti poset pinball which is also poset-upper-triangular? Are there necessary and sufficient conditions on N and h that guarantee poset-upper-triangularity?(3)
What are other examples ofN and h for which the dimension pair algorithm produces a successful outcome of Betti poset pinball which corresponds to a linearly independent set of classes and hence a module basis? Are there necessary and sufficient conditions on N and h that guarantee this?Question 2.
In [13] the explicit module basis consisting of Peterson Schubert classes is used to derive a manifestly positive Monk formula in the S1-equivariant cohomology of Peterson varieties. Preliminary investigation suggests that an analogous Monk formula for the 334-type Hessenberg varieties, using the module basis of Hessenberg Schubert classes derived in this manuscript, would be computationally much more complex. Thus, we may ask the following.(1)
Does there exist a combinatorially elegant or computationally effective Monk formula for the 334-type Hessenberg varieties?(2)
Can such a Monk formula be further generalized to a larger family of regular nilpotent Hessenberg varieties? For instance, can our techniques be generalized to give new insights to the equivariant Schubert calculus of the full flag varietyℱℓags(ℂn) (which is an example of a regular nilpotent Hessenberg variety)?(3)
In [12] the Monk formula for Peterson varieties is used to derive a Giambelli formula. Does there also exist a combinatorially elegant and/or computationally effective Giambelli formula for other cases of regular nilpotent Hessenberg varieties?



---

*Source: 254235-2012-06-10.xml*