# On the classification of sub-Riemannian structures on a 5D two-step nilpotent Lie group

Rory Biggs, Odirile O.K.L. Ntshudisane

► To cite this version:

Rory Biggs, Odirile O.K.L. Ntshudisane. On the classification of sub-Riemannian structures on a 5D two-step nilpotent Lie group. Communications in Mathematics, In press, Volume 32 (2024), Issue 1 (1), pp.13-22. 10.46298/cm.10550. hal-03894564v2

HAL Id: hal-03894564

https://hal.science/hal-03894564v2

Submitted on 3 Jan 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# On the classification of sub-Riemannian structures on a 5D two-step nilpotent Lie group

*Rory Biggs and Odirile Ntshudisane*

**Abstract.** We classify the left-invariant sub-Riemannian structures on the unique five-dimensional simply connected two-step nilpotent Lie group with two-dimensional commutator subgroup; this 5D group is the first two-step nilpotent Lie group beyond the three- and five-dimensional Heisenberg groups. Alongside, we also present a classification, up to automorphism, of the subspaces of the associated Lie algebra (together with a complete set of invariants).

## 1 Introduction

Invariant sub-Riemannian structures on Lie groups have proved to be a well-suited differential geometric language for the study of several physical systems as well as being a rich source of examples and counterexamples for a number of fundamental questions and conjectures in sub-Riemannian geometry (see, e.g., [3], [12], [17], [20]). Much work has been done in studying specific structures, their geodesics, and trying to classify various families of structures, for instance studying the class of structures in three dimensions (see e.g., [2], [8], [10], [11], [18], [19], [21]), in four dimensions (see e.g., [1], [4], [5], [6], [7]), or for some sufficiently regular and thus amenable families of structures like those on the $(2n + 1)$-dimensional Heisenberg groups (see, e.g., [9] and the references therein).

In this paper we consider the left-invariant sub-Riemannian structures on a five-dimensional two-step nilpotent Lie group with two-dimensional commutator subgroup, which we denote by $T$. This group is the first (lowest-dimensional) two-step nilpotent Lie group

<footnote>MSC 2020: 17B40, 53C17, 22E25.</footnote>

<footnote>Keywords: Two-step nilpotent Lie group, sub-Riemannian structure, isometries.</footnote>

<footnote>Affiliation:</footnote>

<footnote>Rory Biggs – Department of Mathematics and Applied Mathematics, University of Pretoria, 0002 Pretoria, South Africa</footnote>

<footnote>E-mail: rorybiggs@gmail.com</footnote>

<footnote>Odirile K.L. Ntshudisane – Department of Mathematics and Applied Mathematics, University of Pretoria, 0002 Pretoria, South Africa</footnote>

<footnote>E-mail: ontshudisane@yahoo.com</footnote>
beyond the three- and five-dimensional Heisenberg groups. We note that although the four-dimensional Engel group (the simply connected Lie group with Lie algebra having nonzero-commutator relations $[E_2, E_4] = E_1$, $[E_3, E_4] = E_2$) has a smaller dimension than $T$, the fact that it is a three-step nilpotent Lie group makes the sub-Riemannian structures on the Engel group arguably more complicated (cf. [1], [5], [6], [7]).

In Section 2, we give a matrix representation for $T$, determine the group of automorphisms of its Lie algebra $t$, and classify the subspaces of $t$ up to automorphism. In Section 3 we then proceed to classify the sub-Riemannian structures on $T$ up to isometry (by making use of the fact that all isometries are affine in this context [16]) and briefly describe the isotropy subgroups of identity.

## 2 The Lie group $T$

There is only one five-dimensional two-step nilpotent simply connected (real) Lie group with two-dimensional commutator subgroup (see, e.g., [22]). We denote this group $T$ and its Lie algebra $t$. The Lie group $T$ has the following matrix representation (cf. [13])

$$ T = \left\{ \begin{bmatrix} 1 & x_1 & x_4 & x_5 \\ 0 & 1 & x_2 & x_3 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{bmatrix} : x_1, x_2, x_3, x_4, x_5 \in \mathbb{R} \right\} \\ t = \left\{ \begin{bmatrix} 0 & v_1 & v_4 & v_5 \\ 0 & 0 & v_2 & v_3 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{bmatrix} = \sum_{i=1}^{5} v_i E_i : v_1, \dots, v_5 \in \mathbb{R} \right\}. $$

The non-zero Lie brackets of $t$ are given by

$$ [E_1, E_2] = E_4, \quad [E_1, E_3] = E_5. $$

The centre $\mathfrak{z} = \langle E_4, E_5 \rangle$ of $t$ coincides with its commutator subalgebra.

**Lemma 2.1.** *The group of automorphisms of $t$ is given by*

$$ \mathrm{Aut}(t) = \left\{ \begin{bmatrix} a_1 & 0 & 0 & 0 \\ a_2 & b_1 & c_1 & 0 \\ a_3 & b_2 & c_2 & 0 \\ a_4 & b_3 & c_3 & a_1 b_1 & a_1 c_1 \\ a_5 & b_4 & c_4 & a_1 b_2 & a_1 c_2 \end{bmatrix} \in \mathbb{R}^{5 \times 5} : a_1 \neq 0, b_1 c_2 - c_1 b_2 \neq 0 \right\} $$

with respect to the ordered basis $(E_1, E_2, E_3, E_4, E_5)$.

*Proof.* Suppose $\varphi \in \mathrm{Aut}(t)$. That is, $\varphi : t \to t$ is a linear isomorphism that preserves Lie brackets. Let $[\varphi_{ij}]$ be the matrix representation of $\varphi$ relative to the ordered basis $(E_1, E_2, E_3, E_4, E_5)$.
As $\varphi \cdot \mathfrak{z} = \mathfrak{z}$, we have that $\varphi_{14} = \varphi_{24} = \varphi_{34} = 0$ and $\varphi_{15} = \varphi_{25} = \varphi_{35} = 0$. As $\varphi$ preserves the Lie bracket $[E_1, E_2] = E_4$, we get $\varphi_{44} = (\varphi_{11}\varphi_{22} - \varphi_{21}\varphi_{12})$ and $\varphi_{54} = (\varphi_{11}\varphi_{32} - \varphi_{31}\varphi_{12})$. Similarly, as $\varphi$ preserves the Lie bracket $[E_1, E_3] = E_5$, we have that $\varphi_{45} = (\varphi_{11}\varphi_{23} - \varphi_{21}\varphi_{13})$ and $\varphi_{55} = (\varphi_{11}\varphi_{33} - \varphi_{31}\varphi_{13})$. We thus have that

$$ [\varphi_{ij}] = \begin{bmatrix} \varphi_{11} & \varphi_{12} & \varphi_{13} & 0 & 0 \\ \varphi_{21} & \varphi_{22} & \varphi_{23} & 0 & 0 \\ \varphi_{31} & \varphi_{32} & \varphi_{33} & 0 & 0 \\ \varphi_{41} & \varphi_{42} & \varphi_{43} & (\varphi_{11}\varphi_{22} - \varphi_{21}\varphi_{12}) & (\varphi_{11}\varphi_{23} - \varphi_{21}\varphi_{13}) \\ \varphi_{51} & \varphi_{52} & \varphi_{53} & (\varphi_{11}\varphi_{32} - \varphi_{31}\varphi_{12}) & (\varphi_{11}\varphi_{33} - \varphi_{31}\varphi_{13}) \end{bmatrix} $$

Preservation of the Lie bracket $[E_2, E_3] = 0$ gives the conditions $\varphi_{12}\varphi_{23} - \varphi_{22}\varphi_{13} = 0$ and $\varphi_{12}\varphi_{33} - \varphi_{32}\varphi_{13} = 0$. If $\varphi_{12} \neq 0$, then $\varphi_{23} = \frac{\varphi_{22}\varphi_{13}}{\varphi_{12}}$, $\varphi_{33} = \frac{\varphi_{32}\varphi_{13}}{\varphi_{12}}$ and so $\det \varphi = 0$, which is a contradiction. Thus $\varphi_{12} = 0$. Similarly, assuming that $\varphi_{13} \neq 0$ leads to a contradiction and thus $\varphi_{13} = 0$. Therefore,

$$ [\varphi_{ij}] = \begin{bmatrix} \varphi_{11} & 0 & 0 & 0 & 0 \\ \varphi_{21} & \varphi_{22} & \varphi_{23} & 0 & 0 \\ \varphi_{31} & \varphi_{32} & \varphi_{33} & 0 & 0 \\ \varphi_{41} & \varphi_{42} & \varphi_{43} & \varphi_{11}\varphi_{22} & \varphi_{11}\varphi_{23} \\ \varphi_{51} & \varphi_{52} & \varphi_{53} & \varphi_{11}\varphi_{32} & \varphi_{11}\varphi_{33} \end{bmatrix} $$

with $\varphi_{11} \neq 0$ and $\varphi_{22}\varphi_{33} - \varphi_{23}\varphi_{32} \neq 0$. It is a simple matter to show that any such map $\varphi$ is an automorphism. $\square$

## Subspace classification

Let $\mathfrak{s}$ and $\mathfrak{w}$ be two subspaces of a Lie algebra $\mathfrak{g}$. We say that $\mathfrak{s}$ and $\mathfrak{w}$ are equivalent if there exists an automorphism $\varphi \in \mathrm{Aut}(\mathfrak{g})$ such that $\varphi \cdot \mathfrak{s} = \mathfrak{w}$. The subspace $\mathfrak{s}$ is called bracket generating if the smallest subalgebra of $\mathfrak{g}$ containing $\mathfrak{s}$ is $\mathfrak{g}$ itself. If $\mathfrak{s}$ is an ideal, then it is said to be a fully characteristic ideal if $\varphi \cdot \mathfrak{s} = \mathfrak{s}$ for all $\varphi \in \mathrm{Aut}(\mathfrak{g})$.

We identify some scalar invariants for subspaces of the Lie algebra $\mathfrak{t}$. A simple invariant is the dimension of a subspace: if $\mathfrak{s}$ is equivalent to $\mathfrak{w}$, then $\dim(\mathfrak{s}) = \dim(\mathfrak{w})$. Two more invariants can be found by considering the dimension of the intersection of a given subspace with any fully characteristic ideal. Accordingly, since the centre

$$ \mathfrak{z} = \langle E_4, E_5 \rangle $$

and the subspace

$$ \mathfrak{c} = \langle E_2, E_3, E_4, E_5 \rangle $$

are both fully characteristic ideals (by Lemma 2.1 in the case of $\mathfrak{c}$), we have that

$$ \dim(\mathfrak{s} \cap \mathfrak{z}) = \dim(\mathfrak{w} \cap \mathfrak{z}) \quad \text{and} \quad \dim(\mathfrak{s} \cap \mathfrak{c}) = \dim(\mathfrak{w} \cap \mathfrak{c}). $$

whenever $\mathfrak{s}$ and $\mathfrak{w}$ are equivalent. The last scalar invariant is slightly more involved.
**Lemma 2.2.** If $\mathfrak{s}$ is equivalent to $\mathfrak{w}$, then

$$ \dim(\mathfrak{s} \cap \mathfrak{z} \cap [E_1, \mathfrak{s} \cap \mathfrak{c}]) = \dim(\mathfrak{w} \cap \mathfrak{z} \cap [E_1, \mathfrak{w} \cap \mathfrak{c}]). $$

*Proof.* Let $\varphi \in \mathrm{Aut}(\mathfrak{t})$ such that $\varphi \cdot \mathfrak{s} = \mathfrak{w}$. Then there exists $\psi \in \mathrm{Aut}(\mathfrak{t})$ such that $\psi \cdot E_1 = a_1 E_1$, $a_1 \neq 0$, $\psi|_{\mathfrak{c}} = \varphi|_{\mathfrak{c}}$, and $\psi|_{\mathfrak{z}} = \varphi|_{\mathfrak{z}}$ (see Lemma 2.1). Therefore

$$ \begin{aligned} \mathfrak{w} \cap \mathfrak{z} \cap [E_1, \mathfrak{w} \cap \mathfrak{c}] &= (\varphi \cdot \mathfrak{s}) \cap \mathfrak{z} \cap [E_1, (\varphi \cdot \mathfrak{s}) \cap \mathfrak{c}] \\ &= \varphi \cdot (\mathfrak{s} \cap \mathfrak{z}) \cap [E_1, \varphi \cdot (\mathfrak{s} \cap \mathfrak{c})] \\ &= \psi \cdot (\mathfrak{s} \cap \mathfrak{z}) \cap \left[ \frac{1}{a_1} \psi \cdot E_1, \psi \cdot (\mathfrak{s} \cap \mathfrak{c}) \right] \\ &= \psi \cdot (\mathfrak{s} \cap \mathfrak{z} \cap [E_1, \mathfrak{s} \cap \mathfrak{c}]) \end{aligned} $$

and so $\dim(\mathfrak{w} \cap \mathfrak{z} \cap [E_1, \mathfrak{w} \cap \mathfrak{c}]) = \dim(\mathfrak{s} \cap \mathfrak{z} \cap [E_1, \mathfrak{s} \cap \mathfrak{c}])$. $\square$

With these invariants at hand, we now proceed to classify the subspaces of $\mathfrak{t}$. In Table 1 we list the equivalence class representatives identified alongside their associated values for the scalar invariants.

<table><caption>Table 1: Subspace equivalence class representatives for $\mathfrak{t}$ with values for scalar invariants</caption><thead><tr><th>Subspace s</th><th>dim(s)</th><th>dim(s &cap; c)</th><th>dim(s &cap; z)</th><th>dim(s &cap; z &cap; [E<sub>1</sub>, s &cap; c])</th></tr></thead><tbody><tr><td>&lt;E<sub>1</sub>&gt;</td><td></td><td>0</td><td>0</td><td>0</td></tr><tr><td>&lt;E<sub>2</sub>&gt;</td><td>1</td><td>1</td><td>0</td><td>0</td></tr><tr><td>&lt;E<sub>4</sub>&gt;</td><td></td><td>1</td><td>1</td><td>0</td></tr><tr><td>&lt;E<sub>1</sub>, E<sub>2</sub>&gt;</td><td></td><td>1</td><td>0</td><td>0</td></tr><tr><td>&lt;E<sub>1</sub>, E<sub>4</sub>&gt;</td><td></td><td>1</td><td>1</td><td>0</td></tr><tr><td>&lt;E<sub>2</sub>, E<sub>3</sub>&gt;</td><td>2</td><td>2</td><td>0</td><td>0</td></tr><tr><td>&lt;E<sub>2</sub>, E<sub>5</sub>&gt;</td><td></td><td>2</td><td>1</td><td>0</td></tr><tr><td>&lt;E<sub>2</sub>, E<sub>4</sub>&gt;</td><td></td><td>2</td><td>1</td><td>1</td></tr><tr><td>&lt;E<sub>4</sub>, E<sub>5</sub>&gt;</td><td></td><td>2</td><td>2</td><td>0</td></tr><tr><td>&lt;E<sub>1</sub>, E<sub>2</sub>, E<sub>3</sub>&gt;</td><td></td><td>2</td><td>0</td><td>0</td></tr><tr><td>&lt;E<sub>1</sub>, E<sub>2</sub>, E<sub>5</sub>&gt;</td><td></td><td>2</td><td>1</td><td>0</td></tr><tr><td>&lt;E<sub>1</sub>, E<sub>2</sub>, E<sub>4</sub>&gt;</td><td>3</td><td>2</td><td>1</td><td>1</td></tr><tr><td>&lt;E<sub>1</sub>, E<sub>4</sub>, E<sub>5</sub>&gt;</td><td></td><td>2</td><td>2</td><td>0</td></tr><tr><td>&lt;E<sub>2</sub>, E<sub>3</sub>, E<sub>4</sub>&gt;</td><td></td><td>3</td><td>1</td><td>1</td></tr><tr><td>&lt;E<sub>2</sub>, E<sub>4</sub>, E<sub>5</sub>&gt;</td><td></td><td>3</td><td>2</td><td>1</td></tr><tr><td>&lt;E<sub>1</sub>, E<sub>2</sub>, E<sub>3</sub>, E<sub>4</sub>&gt;</td><td></td><td>3</td><td>1</td><td>1</td></tr><tr><td>&lt;E<sub>1</sub>, E<sub>2</sub>, E<sub>4</sub>, E<sub>5</sub>&gt;</td><td>4</td><td>3</td><td>2</td><td>1</td></tr><tr><td>&lt;E<sub>2</sub>, E<sub>3</sub>, E<sub>4</sub>, E<sub>5</sub>&gt;</td><td></td><td>4</td><td>2</td><td>2</td></tr><tr><td>&lt;E<sub>1</sub>, E<sub>2</sub>, E<sub>3</sub>, E<sub>4</sub>, E<sub>5</sub>&gt;</td><td>5</td><td>4</td><td>2</td><td>2</td></tr></tbody></table>
**Theorem 2.3.** Any proper subspace of the Lie algebra **t** is equivalent to exactly one of the following

$SA: \langle E_1 \rangle, \langle E_2 \rangle, \langle E_2, E_3 \rangle, \langle E_1, E_4 \rangle, \langle E_2, E_5 \rangle, \langle E_2, E_3, E_4 \rangle, \langle E_1, E_2, E_4 \rangle$

$I: \langle E_4 \rangle, \langle E_2, E_4 \rangle, \langle E_1, E_4, E_5 \rangle, \langle E_2, E_4, E_5 \rangle, \langle E_1, E_2, E_4, E_5 \rangle$

$FCI: \langle E_4, E_5 \rangle, \langle E_2, E_3, E_4, E_5 \rangle$

$Gen: \langle E_1, E_2, E_3 \rangle, \langle E_1, E_2, E_3, E_4 \rangle$

$S: \langle E_1, E_2 \rangle, \langle E_1, E_2, E_5 \rangle.$

Here, the subspaces are listed according to their class: subalgebras (SA), ideals (I), fully
characteristics ideals (FCI), bracket generating subspaces (Gen), or subspaces (S) with
none of these properties.

*Proof.* We treat a typical case for determining a class representative. Suppose $\mathfrak{s}$ is a subspace of the Lie algebra $\mathfrak{t}$ with

$$
\dim(\mathfrak{s}) = 3. \tag{1}
$$

Further, suppose

$$
\dim(\mathfrak{s} \cap \mathfrak{z}) = 1. \tag{2}
$$

Let $X \in \mathfrak{s} \cap \mathfrak{z}$, $X = x_4 E_4 + x_5 E_5$. Then

$$
\varphi =
\begin{bmatrix}
1 & 0 & 0 & 0 & 0 \\
0 & x_4 & -x_5 & 0 & 0 \\
0 & x_5 & x_4 & 0 & 0 \\
0 & 0 & 0 & x_4 & -x_5 \\
0 & 0 & 0 & x_5 & x_4
\end{bmatrix}
$$

is an automorphism such that $\varphi \cdot E_4 = X$. Thus $\mathfrak{s}$ is equivalent to a subspace $\bar{\mathfrak{s}}$ containing $E_4$.

Now, by a simple dimensionality argument, $2 \le \dim(\bar{\mathfrak{s}} \cap \mathfrak{c}) \le 3$. Let us suppose

$$
\dim(\bar{\mathfrak{s}} \cap \mathfrak{c}) = 2. \tag{3}
$$

Since $E_4 \in \bar{\mathfrak{s}} \cap \mathfrak{c}$, there exists $\bar{V}, \bar{W} \in \bar{\mathfrak{s}}$ such that $\langle \bar{W}, E_4 \rangle = \bar{\mathfrak{s}} \cap \mathfrak{c}$, $\langle \bar{V}, \bar{W}, E_4 \rangle = \bar{\mathfrak{s}}$ and $\bar{V} \notin \mathfrak{c}$. This implies that $\bar{v}_1 \neq 0$ and $\bar{w}_1 = 0$; here $\bar{V} = \sum \bar{v}_i E_i$ and $\bar{W} = \sum \bar{w}_i E_i$.

Finally, suppose that

$$
\dim(\bar{\mathfrak{s}} \cap \mathfrak{z} \cap [E_1, \bar{\mathfrak{s}} \cap \mathfrak{c}]) = 1. \quad (4)
$$

Then

$$
\begin{align*}
1 &= \dim(\langle \bar{V}, \bar{W}, E_4 \rangle \cap \mathfrak{z} \cap [E_1, \langle \bar{V}, \bar{W}, E_4 \rangle \cap \langle E_2, E_3, E_4, E_5 \rangle]) \\
&= \dim(\langle E_4 \rangle \cap [E_1, \langle \bar{W}, E_4 \rangle]) \\
&= \dim(\langle E_4 \rangle \cap [E_1, \langle \bar{W} \rangle]) \\
&= \dim(\langle E_4 \rangle \cap \langle \bar{w}_2 E_4 + \bar{w}_3 E_5 \rangle)
\end{align*}
$$
and so it follows that $\bar{w}_2 \neq 0$ and $\bar{w}_3 = 0$. Therefore

$$ \varphi' = \begin{bmatrix} \bar{v}_1 & 0 & 0 & 0 & 0 \\ \bar{v}_2 & \bar{w}_2 & 0 & 0 & 0 \\ \bar{v}_3 & 0 & 1 & 0 & 0 \\ \bar{v}_4 & \bar{w}_4 & 0 & \bar{v}_1 \bar{w}_2 & 0 \\ \bar{v}_5 & \bar{w}_5 & 0 & 0 & \bar{v}_1 \end{bmatrix} $$

is an automorphism such that $\varphi' \cdot \langle E_1, E_2, E_4 \rangle = \langle \bar{V}, \bar{W}, E_4 \rangle = \bar{s}$. Thus $\bar{s}$ (and therefore $\mathfrak{s}$) is equivalent to $\langle E_1, E_2, E_4 \rangle$.

By considering all other possible values of the invariants in (1), (2), (3), and (4) one obtains all possible class representatives. As all representatives obtained are differentiated by the set of scalar invariants (see Table 1), they are mutually non-equivalent. Standard computations determine whether each class representative is a subalgebra, ideal, fully characteristic or generating subspace. $\square$

Since the four scalar invariants identified evaluate distinctly for each equivalence class (see Table 1), these invariants form a complete set.

**Corollary 2.4.** *Two subspaces $\mathfrak{s}$ and $\mathfrak{w}$ of $\mathfrak{t}$ are equivalent if and only if*

$$ \dim(\mathfrak{s}) = \dim(\mathfrak{w}), \\ \dim(\mathfrak{s} \cap \mathfrak{z}) = \dim(\mathfrak{w} \cap \mathfrak{z}), \\ \dim(\mathfrak{s} \cap \mathfrak{c}) = \dim(\mathfrak{w} \cap \mathfrak{c}), \text{ and} \\ \dim(\mathfrak{s} \cap \mathfrak{z} \cap [E_1, \mathfrak{s} \cap \mathfrak{c}]) = \dim(\mathfrak{w} \cap \mathfrak{z} \cap [E_1, \mathfrak{w} \cap \mathfrak{c}]). $$

Here $\mathfrak{z} = \langle E_4, E_5 \rangle$ and $\mathfrak{c} = \langle E_2, E_3, E_4, E_5 \rangle$.

### 3 Sub-Riemannian structures on $\mathcal{T}$

A left-invariant sub-Riemannian structure is a triple $(\mathcal{G}, \mathcal{D}, \mathfrak{g})$ where $\mathcal{G}$ is a real, finite-dimensional, connected Lie group, $\mathcal{D}$ is a smooth bracket generating left-invariant distribution on $\mathcal{G}$, and $\mathfrak{g}$ is a left-invariant Riemannian metric on $\mathcal{D}$. Equivalently: $\mathcal{D}(\mathbf{1})$ is a bracket generating linear subspace of the Lie algebra $\mathfrak{g}$ of $\mathcal{G}$ with $\mathcal{D}(x) = d_1 L_x \cdot \mathcal{D}(\mathbf{1})$ for every $x \in \mathcal{G}$, where $L_x : \mathcal{G} \to \mathcal{G}$, $y \mapsto xy$; $\mathfrak{g}_1$ is a positive definite, symmetric bilinear form on $\mathcal{D}(\mathbf{1})$ with $\mathfrak{g}_x(d_1 L_x \cdot A, d_1 L_x \cdot B) = \mathfrak{g}_1(A, B)$ for every $A, B \in \mathcal{D}(\mathbf{1})$.

Let $(\mathcal{G}, \mathcal{D}, \mathfrak{g})$ and $(\mathcal{G}', \mathcal{D}', \mathfrak{g}')$ be two left-invariant sub-Riemannian structures. An isometry between $(\mathcal{G}, \mathcal{D}, \mathfrak{g})$ and $(\mathcal{G}', \mathcal{D}', \mathfrak{g}')$ is a diffeomorphism $\phi : \mathcal{G} \to \mathcal{G}'$ such that $\phi_* \mathcal{D} = \mathcal{D}'$ and $\mathfrak{g} = \phi^* \mathfrak{g}'$; that is,

$$ d_x \phi \cdot \mathcal{D}(x) = \mathcal{D}'(\phi(x)) \quad \text{and} \quad \mathfrak{g}_x(X, Y) = \mathfrak{g}'_{\phi(x)}(d_x \phi \cdot X, d_x \phi \cdot Y), $$

for all $x \in \mathcal{G}$ and $X, Y \in \mathcal{D}(x)$. By definition, left translations $L_x$ are isometries. Isometries preserve the Carnot–Carathéodry distance associated to the sub-Riemannian structure.
It turns out that for left-invariant sub-Riemannian structures on simply connected nilpotent Lie groups, every isometry is the composition of a left-translation and a Lie group isomorphism [16]. (Indeed in [16] this is proved more generally for nilpotent metric Lie groups.) Therefore, since all left translations are isometries, if two such structures are isometric then there exists a Lie group isomorphism between them that realizes the isometry. We note that there is a one-to-one correspondence between the Lie group automorphisms on a simply connected Lie group and the Lie algebra automorphisms on its Lie algebra (see, e.g., [14]). Consequently, we have the following simple algebraic characterization for two sub-Riemannian structures on a simply connected nilpotent Lie group G with Lie algebra g to be isometric.

**Proposition 3.1.** (cf. [7], [9]) Two left-invariant sub-Riemannian structures $(G, \mathcal{D}, g)$ and $(G, \mathcal{D}', g')$ on a simply connected nilpotent Lie group G are isometric if and only if there exists an automorphism $\psi \in \mathrm{Aut}(g)$ such that

$$ \psi \cdot \mathcal{D}(\mathbf{1}) = \mathcal{D}'(\mathbf{1}) \quad \text{and} \quad \mathbf{g}_1 = \psi^* \mathbf{g}'_1. $$

$$ \text{Here } (\psi^* \mathbf{g}'_1)(A, B) = \mathbf{g}'_1(\psi \cdot A, \psi \cdot B) \text{ for } A, B \in \mathcal{D}(\mathbf{1}). $$

Accordingly, the distribution $\mathcal{D}$ of any left-invariant sub-Riemannian structure $(T, \mathcal{D}, g)$ on $T$, is isometric to a structure with distribution at identity being one of the bracket generating subspaces listed in Theorem 2.3. All that remains to be done is to normalize the metrics $g$ by Lie algebra automorphisms using Proposition 3.1. Doing this we arrive at the following classification of left-invariant sub-Riemannian structures on $T$.

**Theorem 3.2.** Any left-invariant sub-Riemannian structure $(T, \mathcal{D}, g)$ is isometric to exactly one of the following:

$$ \begin{align*} (T, \mathcal{H}_3, \mathbf{h}^3) : & \begin{cases} \mathcal{H}_3(\mathbf{1}) = \langle E_1, E_2, E_3 \rangle \\ \mathbf{h}_1^3 = \mathrm{diag}(1, 1, 1) \end{cases} \\ (T, \mathcal{H}_4, \mathbf{h}^{4,\alpha}) : & \begin{cases} \mathcal{H}_4(\mathbf{1}) = \langle E_1, E_2, E_3, E_4 \rangle \\ \mathbf{h}_1^{4,\alpha} = \alpha \cdot \mathrm{diag}(1, 1, 1, 1), \alpha > 0 \end{cases} \\ (T, \mathcal{H}_5, \mathbf{h}^{5,(\alpha,\beta)}) : & \begin{cases} \mathcal{H}_5(\mathbf{1}) = \langle E_1, E_2, E_3, E_4, E_5 \rangle \\ \mathbf{h}_1^{5,(\alpha,\beta)} = \mathrm{diag}(1, 1, 1, \alpha, \beta), \alpha \ge \beta > 0. \end{cases} \end{align*} $$

Here the metrics are written with respect to the bases given for their respective distributions.

**Remark 3.3.** $(T, \mathcal{H}_5, \mathbf{h}^{5,(\alpha,\beta)})$ corresponds to the result in [15, Proposition 6] for the classification of invariant Riemannian structures on $T$.

*Proof.* We treat the rank 4 structures (i.e., those with $\dim \mathcal{D}(g) = 4$, $g \in T$) as a typical case. Let $(T, \mathcal{D}, g)$ be a rank 4 left-invariant sub-Riemannian structure. By Theorem 2.3 there exists $\psi_0 \in \mathrm{Aut}(t)$ such that $\psi_0 \cdot \mathcal{D}(\mathbf{1}) = \mathcal{H}_4(\mathbf{1})$. By Proposition 3.1, $(T, \mathcal{D}, g)$ is isometric to $(T, \mathcal{H}_4, \mathbf{g}^1)$ for some metric $\mathbf{g}^1$ on $\mathcal{H}_4$.
We can write $\mathbf{g}_1^1$ as a positive definite symmetric matrix with respect to the basis $(E_1, E_2, E_3, E_4)$ for $\mathcal{H}_4(\mathbf{1})$:

$$ \mathbf{g}_1^1 = \begin{bmatrix} h_1 & a_1 & a_2 & a_3 \\ a_1 & h_2 & a_4 & a_5 \\ a_2 & a_4 & h_3 & a_6 \\ a_3 & a_5 & a_6 & h_4 \end{bmatrix}. $$

Now

$$ \psi_1 = \begin{bmatrix} 1 & 0 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 & 0 \\ 0 & 0 & 1 & 0 & 0 \\ -\frac{a_3}{h_4} & -\frac{a_5}{h_4} & -\frac{a_6}{h_4} & 1 & 0 \\ 0 & 0 & 0 & 0 & 1 \end{bmatrix} $$

is an automorphism of $\mathfrak{t}$ such that $\psi_1^{-1} \cdot \mathcal{H}_4(\mathbf{1}) = \mathcal{H}_4(\mathbf{1})$ and $\mathbf{g}_1^2 = (\psi_1)^* \mathbf{g}_1^1$ has matrix

$$ \mathbf{g}_1^2 = \begin{bmatrix} h'_1 & a'_1 & a'_2 & 0 \\ a'_1 & h'_2 & a'_4 & 0 \\ a'_2 & a'_4 & h'_3 & 0 \\ 0 & 0 & 0 & h'_4 \end{bmatrix} $$

with respect to $(E_1, E_2, E_3, E_4)$ for some constants $a'_1, a'_2, a'_4, h'_1, \dots, h'_4 \in \mathbb{R}$. Note here that $\mathbf{g}_1^1 = (\psi_1^{-1})^* \mathbf{g}_1^2$, or equivalently $\mathbf{g}_1^2(A, B) = (\psi_1)^* \mathbf{g}_1^1(A, B) = \mathbf{g}_1^1(\psi_1 \cdot A, \psi_1 \cdot B)$ for $A, B \in \mathcal{H}_4(\mathbf{1})$. That is to say, $(\mathfrak{t}, \mathcal{H}_4, \mathbf{g}^1)$ is isometric to $(\mathfrak{t}, \mathcal{H}_4, \mathbf{g}^2)$ by Proposition 3.1.

Continuing on in this way, we have

$$ \psi_2 = \begin{bmatrix} 1 & 0 & 0 & 0 & 0 \\ \frac{a'_1 h'_3 - a'_2 a'_4}{a'_4^2 - h'_2 h'_3} & 1 & -\frac{a'_4}{h'_2} & 0 & 0 \\ \frac{a'_2 h'_2 - a'_1 a'_4}{a'_4^2 - h'_2 h'_3} & 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 1 & -\frac{a'_4}{h'_2} \\ 0 & 0 & 0 & 0 & 1 \end{bmatrix} \in \mathrm{Aut}(\mathfrak{t}), \quad \mathbf{g}_1^3 = \psi_2^* \mathbf{g}_1^2 = \begin{bmatrix} b_1 & 0 & 0 & 0 \\ 0 & b_2 & 0 & 0 \\ 0 & 0 & b_3 & 0 \\ 0 & 0 & 0 & b_4 \end{bmatrix} $$

for some $b_1, \dots, b_4 \in \mathbb{R}$. Note that $a'_4^2 - h'_2 h'_3 \neq 0$ and $h'_2 \neq 0$ since $\mathbf{g}_1^2$ is positive definite. Finally,

$$ \psi_4 = \mathrm{diag} \left( \sqrt{\frac{b_2}{b_4}}, \sqrt{\frac{b_1}{b_4}}, \sqrt{\frac{b_1 b_2}{b_3 b_4}}, \sqrt{\frac{b_1 b_2}{b_4^2}}, \sqrt{\frac{b_1 b_2^2}{b_3 b_4^2}} \right) $$

is an automorphism such that $\mathbf{g}_1^4 = \psi_3^* \mathbf{g}_1^3 = \frac{b_1 b_2}{b_4} I_4 = \mathbf{h}_1^{4,\alpha}$ with $\alpha = \frac{b_1 b_2}{b_4}$. It therefore follows by transitivity that $(\mathfrak{t}, \mathcal{D}, \mathbf{g})$ is isometric to $(\mathfrak{t}, \mathcal{H}_4, \mathbf{h}^{4,\alpha})$ for some $\alpha > 0$.

Now suppose $(\mathfrak{t}, \mathcal{H}_4, \mathbf{h}^{4,\alpha})$ and $(\mathfrak{t}, \mathcal{H}_4, \mathbf{h}^{4,\beta})$ are isometric for some $\alpha, \beta > 0$. By Proposition 3.1 there exists $\psi \in \mathrm{Aut}(\mathfrak{t})$ such that $\psi \cdot \mathcal{H}_4(\mathbf{1}) = \mathcal{H}_4(\mathbf{1})$ and $\mathbf{h}^{4,\alpha} = \psi^* \mathbf{h}^{4,\beta}$. Utilizing Lemma 2.1 and computing these conditions in coordinates, it is fairly straightforward to show that this implies that $\alpha = \beta$. Hence, each different $\alpha > 0$ yields a non-isometric structure. $\square$
Since isometries preserving the identity element are automorphisms of the group, it is not difficult to find the (linearized) isotropy subgroup of identity (i.e., the subgroup of the isometry group fixing the identity).

**Corollary 3.4.** *The isotropy subgroups of identity associated to the respective left-invariant sub-Riemannian structures on $\mathcal{T}$ are given by*

$$(i) \operatorname{Iso}_1(\mathcal{T}, \mathcal{H}_3, \mathbf{h}^3) \cong \mathbb{Z}_2 \times \mathcal{O}(2),$$

$$(ii) \operatorname{Iso}_1(\mathcal{T}, \mathcal{H}_4, \mathbf{h}^{4,\alpha}) \cong \mathbb{Z}_2 \times \mathbb{Z}_2 \times \mathbb{Z}_2,$$

$$(iii) \operatorname{Iso}_1(\mathcal{T}, \mathcal{H}_5, \mathbf{h}^{5,(\alpha,\beta)}) \cong \mathbb{Z}_2 \times \mathbb{Z}_2 \times \mathbb{Z}_2, \text{ for } \alpha > \beta > 0, \\ \operatorname{Iso}_1(\mathcal{T}, \mathcal{H}_5, \mathbf{h}^{5,(\alpha,\alpha)}) \cong \mathbb{Z}_2 \times \mathcal{O}(2) \text{ where } \alpha > 0.$$

**Remark 3.5.** The isotropy groups of $(\mathcal{T}, \mathcal{H}_5, \mathbf{h}^{5,(\alpha,\beta)})$, $\alpha \ge \beta > 0$ correspond to the result in [15, Proposition 7] for invariant Riemannian structures on $\mathcal{T}$.

## Acknowledgements

The research leading to these results has received funding from the National Research Foundation (NRF) of South Africa.

## References

[1] Adams M.R., Tie J.: On sub-Riemannian geodesics on the Engel groups: Hamilton's equations. *Math. Nachr.* 286 (14-15) (2013) 1381-1406.

[2] Agrachev A., Barilari D.: Sub-Riemannian structures on 3D Lie groups. *J. Dyn. Control Syst.* 18 (1) (2012) 21-44.

[3] Agrachev A., Barilari D., Boscain U.: A comprehensive introduction to sub-Riemannian geometry. Cambridge University Press (2020).

[4] Almeida D.M.: Sub-Riemannian homogeneous spaces of Engel type. *J. Dyn. Control Syst.* 20 (2) (2014) 149-166.

[5] Ardentov A.A., Sachkov Y.L.: Extremal trajectories in the nilpotent sub-Riemannian problem on the Engel group. *Mat. Sb.* 202 (11) (2011) 31-54.

[6] Ardentov A.A., Sachkov Y.L.: Conjugate points in nilpotent sub-Riemannian problem on the Engel group. *J. Math. Sci. (N.Y.)* 195 (3) (2013) 369-390.

[7] Bartlett C.E., Biggs R., Remsing C.C.: Control systems on nilpotent Lie groups of dimension $\le 4$: equivalence and classification. *Differential Geom. Appl* 54 (2017) 282-297.

[8] Biggs R.: Isometries of Riemannian and sub-Riemannian structures on three-dimensional Lie groups. *Commun. Math.* 25 (2) (2017) 99-135.

[9] Biggs R., Nagy P.T.: On sub-Riemannian and Riemannian structures on the Heisenberg groups. *J. Dyn. Control Syst.* 22 (3) (2016) 563-594.

[10] Boscain U., Rossi F.: Invariant Carnot-Caratheodory metrics on $S^3$, $SO(3)$, $SL(2)$, and lens spaces. *SIAM J. Control Optim.* 47 (4) (2008) 1851-1878.
[11] Butt Y.A., Sachkov Y.L., Bhatti A.I.: Extremal trajectories and Maxwell strata in sub-Riemannian problem on group of motions of pseudo-Euclidean plane. *J. Dyn. Control Syst.* 20 (3) (2014) 341–364.

[12] Calin O., Chang D.C.: *Sub-Riemannian geometry*. Cambridge University Press (2009).

[13] Ghanam R., Thompson G.: Minimal matrix representations of five-dimensional Lie algebras. *Extracta Math.* 30 (1) (2019) 95–133.

[14] Hilgert J., Neeb K.H.: *Structure and geometry of Lie groups*. Springer (2012).

[15] Homolya S., Kowalski O.: Simply connected two-step homogeneous nilmanifolds of dimension 5. *Note Mat.* 26 (1) (2006) 69–77.

[16] Kivioja V., Le Donne E.: Isometries of nilpotent metric groups. *J. Éc. polytech. Math.* 4 (2017) 473–482.

[17] Liu W., Sussman H.J.: Shortest paths for sub-Riemannian metrics on rank-two distributions. *Mem. Amer. Math. Soc.* 118 (564) (1995) x+104.

[18] Mazhitova A.D.: Sub-Riemannian geodesics on the three-dimensional solvable non-nilpotent Lie group $SOLV^-$. *J. Dyn. Control Syst.* 18 (3) (2012) 309–322.

[19] Moiseev I., Sachkov Y.L.: Maxwell strata in sub-Riemannian problem on the group of motions of a plane. *ESAIM Control Optim. Calc. Var.* 16 (2) (2010) 380–399.

[20] Montgomery R.: *A tour of subriemannian geometries, their geodesics and applications*. American Mathematical Society (2002).

[21] Sachkov Y.L.: Conjugate and cut time in the sub-Riemannian problem on the group of motions of a plane. *ESAIM Control Optim. Calc. Var.* 16 (4) (2010) 1018–1039.

[22] Šnobl L., Winternitz P.: *Classification and identification of Lie algebras*. American Mathematical Society (2014).

*Received:* December 10, 2020

Accepted for publication: July 03, 2021

Communicated by: Ilka Agricola