# Engel's theorem

In representation theory, a branch of mathematics, Engel's theorem states that a finite-dimensional Lie algebra ${\mathfrak {g}}$ is a nilpotent Lie algebra if and only if for each $X\in {\mathfrak {g}}$, the adjoint map
    $\operatorname {ad} (X)\colon {\mathfrak {g}}\to {\mathfrak {g}},$
given by $\operatorname {ad} (X)(Y)=[X,Y]$, is a nilpotent endomorphism on ${\mathfrak {g}}$; i.e., $\operatorname {ad} (X)^{k}=0$ for some k. It is a consequence of the theorem, also called Engel's theorem, which says that if a Lie algebra of matrices consists of nilpotent matrices, then the matrices can all be simultaneously brought to a strictly upper triangular form. Note that if we merely have a Lie algebra of matrices which is nilpotent as a Lie algebra, then this conclusion does not follow (i.e. the naïve replacement in Lie's theorem of "solvable" with "nilpotent", and "upper triangular" with "strictly upper triangular", is false; this already fails for the one-dimensional Lie subalgebra of scalar matrices).
The theorem is named after the mathematician Friedrich Engel, who sketched a proof of it in a letter to Wilhelm Killing dated 20 July 1890 (Hawkins 2000, p. 176). Engel's student K.A. Umlauf gave a complete proof in his 1891 dissertation, reprinted as (Umlauf 2010).

## Statements

Let ${\mathfrak {gl}}(V)$ be the Lie algebra of the endomorphisms of a finite-dimensional vector space V and ${\mathfrak {g}}\subset {\mathfrak {gl}}(V)$ a subalgebra. Then Engel's theorem states the following are equivalent:
* Each $X\in {\mathfrak {g}}$ is a nilpotent endomorphism on V.
* There exists a flag $V=V_{0}\supset V_{1}\supset \cdots \supset V_{n}=0,\,\operatorname {codim} V_{i}=i$ such that ${\mathfrak {g}}\cdot V_{i}\subset V_{i+1}$; i.e., the elements of ${\mathfrak {g}}$ are simultaneously strictly upper-triangulizable.
Note that no assumption on the underlying base field is required.
We note that Statement 2. for various ${\mathfrak {g}}$ and V is equivalent to the statement
* For each nonzero finite-dimensional vector space V and a subalgebra ${\mathfrak {g}}\subset {\mathfrak {gl}}(V)$, there exists a nonzero vector v in V such that $X(v)=0$ for every $X\in {\mathfrak {g}}.$
This is the form of the theorem proven in #Proof. (This statement is trivially equivalent to Statement 2 since it allows one to inductively construct a flag with the required property.)
In general, a Lie algebra ${\mathfrak {g}}$ is said to be nilpotent if the lower central series of it vanishes in a finite step; i.e., for $C^{0}{\mathfrak {g}}={\mathfrak {g}},C^{i}{\mathfrak {g}}=[{\mathfrak {g}},C^{i-1}{\mathfrak {g}}]$ = (i+1)-th power of ${\mathfrak {g}}$, there is some k such that $C^{k}{\mathfrak {g}}=0$. Then Engel's theorem implies the following theorem (also called Engel's theorem): when ${\mathfrak {g}}$ has finite dimension,
* ${\mathfrak {g}}$ is nilpotent if and only if $\operatorname {ad} (X)$ is nilpotent for each $X\in {\mathfrak {g}}$.
Indeed, if $\operatorname {ad} ({\mathfrak {g}})$ consists of nilpotent operators, then by 1. $\Leftrightarrow$ 2. applied to the algebra $\operatorname {ad} ({\mathfrak {g}})\subset {\mathfrak {gl}}({\mathfrak {g}})$, there exists a flag ${\mathfrak {g}}={\mathfrak {g}}_{0}\supset {\mathfrak {g}}_{1}\supset \cdots \supset {\mathfrak {g}}_{n}=0$ such that $[{\mathfrak {g}},{\mathfrak {g}}_{i}]\subset {\mathfrak {g}}_{i+1}$. Since $C^{i}{\mathfrak {g}}\subset {\mathfrak {g}}_{i}$, this implies ${\mathfrak {g}}$ is nilpotent. (The converse follows straightforwardly from the definition.)

## Proof

We prove the following form of the theorem: if ${\mathfrak {g}}\subset {\mathfrak {gl}}(V)$ is a Lie subalgebra such that every $X\in {\mathfrak {g}}$ is a nilpotent endomorphism and if V has positive dimension, then there exists a nonzero vector v in V such that $X(v)=0$ for each X in ${\mathfrak {g}}$.
The proof is by induction on the dimension of ${\mathfrak {g}}$ and consists of a few steps. (Note the structure of the proof is very similar to that for Lie's theorem, which concerns a solvable algebra.) The basic case is trivial and we assume the dimension of ${\mathfrak {g}}$ is positive.
Step 1: Find an ideal ${\mathfrak {h}}$ of codimension one in ${\mathfrak {g}}$.
    This is the most difficult step. Let ${\mathfrak {h}}$ be a maximal (proper) subalgebra of ${\mathfrak {g}}$, which exists by finite-dimensionality. We claim it is an ideal of codimension one. For each $X\in {\mathfrak {h}}$, it is easy to check that (1) $\operatorname {ad} (X)$ induces a linear endomorphism ${\mathfrak {g}}/{\mathfrak {h}}\to {\mathfrak {g}}/{\mathfrak {h}}$ and (2) this induced map is nilpotent (in fact, $\operatorname {ad} (X)$ is nilpotent as $X$ is nilpotent; see Jordan decomposition in Lie algebras). Thus, by inductive hypothesis applied to the Lie subalgebra of ${\mathfrak {gl}}({\mathfrak {g}}/{\mathfrak {h}})$ generated by $\operatorname {ad} ({\mathfrak {h}})$, there exists a nonzero vector v in ${\mathfrak {g}}/{\mathfrak {h}}$ such that $\operatorname {ad} (X)(v)=0$ for each $X\in {\mathfrak {h}}$. That is to say, if $v=[Y]$ for some Y in ${\mathfrak {g}}$ but not in ${\mathfrak {h}}$, then $[X,Y]=\operatorname {ad} (X)(Y)\in {\mathfrak {h}}$ for every $X\in {\mathfrak {h}}$. But then the subspace ${\mathfrak {h}}'\subset {\mathfrak {g}}$ spanned by ${\mathfrak {h}}$ and Y is a Lie subalgebra in which ${\mathfrak {h}}$ is an ideal of codimension one. Hence, by maximality, ${\mathfrak {h}}'={\mathfrak {g}}$. This proves the claim.
Step 2: Let $W=\{v\in V|X(v)=0,X\in {\mathfrak {h}}\}$. Then ${\mathfrak {g}}$ stabilizes W; i.e., $X(v)\in W$ for each $X\in {\mathfrak {g}},v\in W$.
    Indeed, for $Y$ in ${\mathfrak {g}}$ and $X$ in ${\mathfrak {h}}$, we have: $X(Y(v))=Y(X(v))+[X,Y](v)=0$ since ${\mathfrak {h}}$ is an ideal and so $[X,Y]\in {\mathfrak {h}}$. Thus, $Y(v)$ is in W.
Step 3: Finish up the proof by finding a nonzero vector that gets killed by ${\mathfrak {g}}$.
    Write ${\mathfrak {g}}={\mathfrak {h}}+L$ where L is a one-dimensional vector subspace. Let Y be a nonzero vector in L and v a nonzero vector in W. Now, $Y$ is a nilpotent endomorphism (by hypothesis) and so $Y^{k}(v)\neq 0,Y^{k+1}(v)=0$ for some k. Then $Y^{k}(v)$ is a required vector as the vector lies in W by Step 2. $\square$