Geometry Informed Tokenization of Molecules for Language Model Generation

## Abstract

Abstract We consider molecule generation in 3D space using language models (LMs), which requires discrete tokenization of 3D molecular geometries. Although tokenization of molecular graphs exists, that for 3D geometries is largely unexplored. Here, we attempt to bridge this gap by proposing the Geo2Seq, which converts molecular geometries into S ⁢ E ⁢ ( 3 ) 𝑆 𝐸 3 SE(3) italic_S italic_E ( 3 ) -invariant 1D discrete sequences.
Geo2Seq consists of canonical labeling and invariant spherical representation steps, which together maintain geometric and atomic fidelity in a format conducive to LMs.
Our experiments show that, when coupled with Geo2Seq, various LMs excel in molecular geometry generation, especially in controlled generation tasks.

## 1 Introduction

The generation of novel molecules with desired properties is an important step in drug discovery. Specifically, the design of three-dimensional (3D) molecular geometries is particularly important because 3D information plays a critical role in determining many molecular properties.
Different generative models have been used for 3D molecule generation. Early studies such as G-SchNet [] use autoregressive generative models to generate 3D molecules by sequentially placing atoms in 3D space. It was observed that these models often yield results with low chemical validity. Recently, diffusion models [, ] achieve better performance in 3D molecule generation tasks. However, they typically need thousands of diffusion steps, resulting in long generation time.

Language models (LMs) [, , , ], with their streamlined data processing and powerful generation capabilities, have shown success across various domains, particularly in natural language processing (NLP). Recently, large language models (LLMs) [] show extraordinary capabilities in learning complex patterns []
and generating meaningful outputs [, , ].
Despite their potential, the application of LLMs to the direct generation of 3D molecules is largely under-explored. This is primarily due to the fact that geometric graph structures of molecular data are fundamentally different from texts.
However, 3D geometric information is crucial in molecular tasks,
since different conformations of the same molecule topology have different properties, such as per-atom forces.
This gap reveals a unique challenge of how to make use of the powerful pattern recognition and generative capabilities of LLMs to handle complicated molecular graph structures, especially geometries.

In this work, we bridge this gap by applying LMs to the task of 3D molecule generation. We employ a novel approach translating the intricate geometry of molecules into a format that can be effectively processed by LMs. This is achieved by our proposed tokenization method Geo2Seq, which converts 3D molecular structures into $SE(3)$-invariant one-dimensional (1D) discrete sequences. The transformation is based on canonical labeling, which allows dimension reduction with no information loss outside graph isomorphism groups, and invariant spherical representations, which guarantees $SE(3)$-invariance under the equivariant global frame. By doing so, we harness the advanced sequence-processing capabilities and efficiency of LMs while retaining essential geometric and atomic information.

Note that since Geo2Seq operates solely on input data, our method is agnostic to the subsequent LMs used.
When combined with powerful modern LLMs, Geo2Seq can achieve highly accurate modeling of 3D molecular structures. In addition, Geo2Seq can benefit conditional generation by including real-world chemical properties in sequences because modern LLMs are capable of capturing long-context correlations to comprehend global structure and information
in sequences.
Our experimental results demonstrate these advantages. We show that using different LMs with Geo2Seq can reliably produce valid and diverse 3D molecules and outperform the strong diffusion-based baselines by a large margin in conditional generation.
These results validate the feasibility of using LMs for 3D molecule generation and highlight their potential to aid in the discovery and design of new molecules, paving the way for applications such as drug development and material science.

## 2 Preliminaries and Related Work

### 2.1 3D Molecule Generation

In this work, we study the problem of generating 3D molecules from scratch. Note that this problem is different from the 3D molecular conformation generation problem studied in the literature [, , , , , , , , ], where 3D molecular conformations are generated from 2D molecular graphs. We represent a 3D molecule with $n$ atoms in the form of a 3D point cloud (*i.e.*, a set of points with different positions in 3D Euclidean space) as $G=({\bm{z}},{\bm{R}})$. Here, ${\bm{z}}=[z_{1},\cdots,z_{n}]\in\mathbb{Z}^{n}$ is the atom type vector where $z_{i}$ is the atomic number (nuclear charge number) of the $i$-th atom, and ${\bm{R}}=[{\bm{r}}_{1},\cdots,{\bm{r}}_{n}]\in\mathbb{R}^{3\times n}$ is the atom coordinate matrix, where ${\bm{r}}_{i}$ is the 3D coordinate of the $i$-th atom. Note that 3D atom coordinates ${\bm{R}}$ are commonly called 3D molecular conformations or geometries in chemistry. We aim to solve the following two generation tasks in this work:

- •
Random generation. Given a 3D molecule dataset $\mathcal{G}=\{G_{j}\}_{j=1}^{m}$, we aim to learn an unconditional generative model $p_{\theta}(\cdot)$ on $\mathcal{G}$ so that the model can generate valid and diverse 3D molecules.
- •
Controllable generation. Given a 3D molecule dataset $\mathcal{G}=\{(G_{j},s_{j})\}_{j=1}^{m}$ where $s_{j}$ is a certain property value of $G_{j}$, we aim to learn a conditional generative model $p_{\theta}(\cdot|s)$ on $\mathcal{G}$ so that for a given $s$, the model can generate 3D molecules whose quantum property values are $s$.

A major technical challenge of 3D molecule generation lies in maintaining invariant to $SE(3)$ transformations, including rotation and translation. In other words, ideal models should assign the same probability to $G=({\bm{z}},{\bm{R}})$ and $G^{\prime}=({\bm{z}},{\bm{R}}^{\prime})$ if ${\bm{R}}^{\prime}={\bm{Q}}{\bm{R}}+{\bm{b}}\bm{1}^{T}$, where $\bm{1}$ is an $n$-dimensional vector whose elements are all one, ${\bm{b}}\in\mathbb{R}^{3}$ is an arbitrary translation vector, and ${\bm{Q}}\in\mathbb{R}^{3\times 3}$ is a rotation matrix satisfying ${\bm{Q}}{\bm{Q}}^{T}={\bm{I}},|{\bm{Q}}|=1$. To achieve $SE(3)$-invariance in 3D molecule generation, existing studies have proposed various strategies. Early studies propose to generate 3D atom positions by $SE(3)$-invariant features, such as interatomic distances, angles and torsion angles. They construct 3D molecular structures through either atom-by-atom generation [, ] or generating full distance matrices [] in one shot. Recently, more and more studies have applied generative models to generate 3D atom coordinate directly. These studies include E-NFs [] and EDM [], which combine equivariant atom coordinate alignment process with equivariant EGNN [] model for 3D molecule generation. Following EDM, many other studies have proposed to improve diffusion-based 3D molecule generation frameworks by stochastic differential equation (SDE) based diffusion models [, ] or latent diffusion models []. Besides, some recent studies [] have explored generating 3D molecules through generating and connecting fragments first, then aligning atom coordinates with software like RDKit. We refer readers to Du et al. [], Zhang et al. [] for a comprehensive review.

While generating 3D molecules in the form of 3D point clouds have been well studied, few studies have tried applying powerful language models to this problem. In this work, different from mainstream methods, we convert 3D point clouds to $SE(3)$-invariant 1D discrete sequences, and show that generating sequences by LMs achieves promising performance in the 3D molecule generation task.

### 2.2 Chemical Language Model

LMs
have catalyzed significant advancements across a spectrum of fields. Recently, LLMs have revolutionized the landscape of NLP and beyond [, , ]. Drawing inspiration from NLP methodologies, chemical language models (CLMs) have emerged as a competent way for representing molecules [, , , ]. Due to the superiority LMs show in generation tasks, most CLMs are designed as generative models.
Variants of LMs have been adapted for molecular science, producing a variety of works such as MolGPT [], MolReGPT [], MolT5 [], MoleculeGPT [], InstructMol [], DrugGPT [], and many others.

CLMs learn the chemical vocabulary
and syntax used to represent molecules, as well as the conditional probabilities of character occurrence at given positions of sequences depending on preceding characters. This vocabulary covers all characters from the adopted molecule representation. All inputs including chemical structures and properties should be converted into sequence form and tokenized for compatibility with language models.
Commonly, SMILES [] is used for this sequential representation, although other formats like SELFIES [], atom type strings, and custom strings with positional or property values are also viable options.
To learn representations, CLMs are usually pre-trained on extensive molecular sequences through self-supervised learning. Subsequently, models are fine-tuned
on more focused datasets with desired properties, such as activity against a target protein.
Generative CLMs generally adopt an autoregressive training approach of next token prediction, i.e., iteratively predicting each subsequent token in a sequence based on the preceding tokens. Traditional autoregressive models use the Transformer architecture with causal self-attention [] due to its superior efficacy, while other sequence models like recurrent neural networks (RNNs) and state space models (SSMs) [, , ] also show considerable functionality.

Given a dataset of sequences, ${\bm{U}}=\{U_{1},U_{2},\cdots,U_{N}\}$, where $U_{i}$ is transformed from the representation, property conditions and/or descriptions of a molecule $G_{i}$ with $n_{i}$ nodes, let $U_{i}=\{u_{1},u_{2},\cdots,u_{n_{i}}\}$ and all tokens $u_{i}$ belong to vocabulary $V$. An autoregressive CLM has parameters $\theta$ encoding a distribution with conditional probabilities of each token given its predecessors,
$p(U_{i};\theta)=\prod_{j=1}^{n_{i}}p(u_{j}|u_{0}:u_{j-1};\theta).$
The optimization process involves maximizing the probabilities of the entire dataset $p({\bm{U}};\theta)=\prod_{i=1}^{N}p(U_{i};\theta)$. Each conditional distribution $p(u_{j}|u_{0}:u_{j-1};\theta)$ is a categorical distribution over the vocabulary size $|V|$; thus the loss for each term aligns with the standard cross-entropy loss.
To generate new sequences, the model samples each token sequentially from these conditional distributions. To introduce randomness and control into generation, the sampling process is typically modulated with Top-K $(k)$ and temperature $(\tau)$ hyperparameters, enabling a balance between adherence and diversity.

Most existing CLM works consider chemical structures as well as other modalities such as natural language captions [, , , , , , , , , , , ], while some focus on pure text of chemical literature [] or molecule strings [, , , , , , , , , ]. Notably, all these works solely consider 2D molecules for representation learning and downstream tasks, overlooking 3D geometric structures which is crucial in many molecular predictive and generative tasks.
For example, different conformations of the same 2D molecule have different potentials and per-atom forces. In order to use pivotal 3D information, another line of work incorporate geometric models such as GNNs in parallel with the CLM [, , , , , ], which requires additional design and training techniques to mitigate alignment issues.
Some works extend the architecture of CLM to include 3D-geometric-model-like modules in the attention block [, , , , , , , ], capturing 3D information as positional encodings with considerable computations and framework design.
In contrast, Flam-Shepherd and Aspuru-Guzik [] make an initial attempt showing language models trained directly on contents of XYZ format chemical files can generate molecules with three coordinates, implying pure LMs’ potential to directly explore 3D chemical space.
In this work, we propose an invariant 3D molecular sequencing algorithm, Geo2Seq, to empower CLMs with structural completeness and geometric invariance, showing LMs’ capabilities of understanding molecules precisely in 3D space. We extend beyond the conventional Transformer architecture of CLMs and additionally employ SSMs as LM backbones. Furthermore, Geo2Seq operates solely on the input data, which allows independence from model architecture and training techniques and provides reuse flexibility.

Figure: Figure 1: Overview of Geo2Seq. We use the canonical labeling order to arrange nodes in a row, fill in the place of each node with vector $[z_{i},d_{i},\theta_{i},\phi_{i}]$, and concatenate all elements into a sequence. Each node vector contains atom type and spherical coordinates. Notably, the spherical coordinates are $SE(3)$-invariant.
Refer to caption: x1.png

## 3 Tokenization of 3D Molecules

A fundamental difference between LMs and other models is that LMs
use discrete inputs, *i.e.*, tokens. In this section, we introduce our tokenization method to map input 3D molecules with atomic coordinates to discrete token sequences appropriate for LM learning.

A main challenge in tokenization design is to develop bijective mappings between 3D molecules and token sequences, *i.e.*, obtaining the same token sequence for the same input 3D molecule, while obtaining different sequences for different inputs.
In this section, we present our solutions to tackle this challenge. We first reorder the atoms in the input molecule to a canonical order (Section 3.1), such that any two isomorphic graphs result in the same canonical form, and any non-isomorphic graphs yield different canonical forms. We then convert 3D Cartesian coordinates to $SE(3)$-invariant spherical representations, including distances and angles (Section 3.2). Combining them together, we obtain our geometry informed tokenization method Geo2Seq (Section 3.3). We provide rigorous proof of all theorems supporting the bijective mapping relation in Appendix B.

### 3.1 Serialization via Canonical Ordering

As the first step in 3D molecule tokenization, we need to transform a graph to a 1D sequential representation. We resort to canonical labeling as a solution for dimension reduction without information loss.

Canonical labeling (CL), in the context of graph theory, is a process to assign a unique form to each graph in a way that two graphs receive the same canonical form only if they are isomorphic [].
The canonical form is a re-indexed version of a graph, which is unique for the whole isomorphism class of a graph.
The new indexes naturally establish the order of nodes in the graph. The order, which we refer to as canonical labels, is not necessarily unique if the graph has symmetries and thus has an automorphism group larger than 1. However, all canonical labels are strictly equivalent when used for serialization.
The canonical label essentially re-assigns an index $\ell_{i}$ to each node originally indexed with $i$ in graph $G$. Since canonical labeling can precisely distinguish non-isomorphic graphs, it fully contains the structure information of a graph $G$. Thus, by arranging nodes with attributes in the labeling order $\ell_{1},\ell_{2},\cdots$, we obtain a sequential representation of attributed graphs with all structural information preserved.

The Nauty algorithm [], tailored for CL and computing graph automorphism groups, presents a rigorous implementation of CL.
In this paper, we adopt the Nauty algorithm for CL calculation, while all analyses and derivations apply to other rigorous algorithms. The bijective mapping relation between CL-obtained sequential representation and graphs can be be proved based on graph isomorphism. We first formally define graph isomorphism as below.
Due to the geometric needs in our case, we move a step forward and define the isomorphism problem for attributed graphs.

###### Definition 3.1 .

[Graph Isomorphism]
Let $G_{1}=(V_{1},E_{1},A_{1})$ and $G_{2}=(V_{2},E_{2},A_{2})$ be two graphs, where $V_{i}$ denotes the set of vertices, $E_{i}$ denotes the set of edges, and $A_{i}$ denotes the node attributes of $G_{i}$ for $i=1,2$. Let $attr(v)$ denote the node attributes of vertex $v$.
The graphs $G_{1}$ and $G_{2}$ are said to be isomorphic, denoted as $G_{1}\cong G_{2}$, if there exists a bijection $b:V_{1}\rightarrow V_{2}$ such that for every vertex $v\in V_{1}$,
$attr(v)\in A_{1}=attr(b(v))\in A_{2}$, and
for every pair of vertices $u,v\in V_{1}$,

$$ $(u,v)\in E_{1}\Leftrightarrow(b(u),b(v))\in E_{2}.$ $$

CL processes can also be extended to node/edge-attributed graphs,
leading us to the guarantee below.

###### Lemma 3.2 .

[Canonical Labeling for Colored Graph Isomorphism]
Let $G_{1}=(V_{1},E_{1},A_{1})$ and $G_{2}=(V_{2},E_{2},A_{2})$ be two finite, undirected graphs where $V_{i}$ denotes the set of vertices, $E_{i}$ denotes the set of edges, and $A_{i}$ denotes the node attributes of the graph $G_{i}$ for $i=1,2$. Let ${\bm{L}}:\mathcal{G}\rightarrow\mathcal{L}$ be a function that maps a graph $G\in\mathcal{G}$, the set of all finite, undirected graphs, to its canonical label ${\bm{L}}(G)\in\mathcal{L}$, the set of all possible canonical labels, as produced by the Nauty algorithm. Then the following equivalence holds:

$$ ${\bm{L}}(G_{1})={\bm{L}}(G_{2})\Leftrightarrow G_{1}\cong G_{2}$ $$

where $G_{1}\cong G_{2}$ denotes that $G_{1}$ and $G_{2}$ are isomorphic.

Lemma 3.2 indicates that the CL process is both complete (sufficient to distinguish non-isomorphic graphs) and sound (not distinguishing actually isomorphic graphs).
Note that if ${\bm{L}}(G)$ corresponds to multiple automorphic labels, we can randomly select one since they are all equivalent and produce the same sequence later through Geo2Seq, as detailed in Appendix B. However, this is a very uncommon case for real-world 3D attributed graphs like molecules.

###### Definition 3.1 .

###### Lemma 3.2 .

### 3.2 Invariant Spherical Representations

Figure: Figure 2: Illustrations of the equivariant frame and invariant spherical representations. If the molecule is rotated and translated by a rotation matrix ${\bm{Q}}$ and a translation vector ${\bm{b}}$, the atom coordinates change accordingly. But our spherical representations remain invariant since the frame is equivariant to the $SE(3)$-transformation.
Refer to caption: x2.png

In this section, we describe how to incorporate 3D structure information into our sequences. One main challenge here is to ensure the $SE(3)$-invariance property described in Section 2.1.
Specifically, given a 3D molecule, if it is rotated or translated in the 3D space, its 3D representation should be unchanged. Another challenge is to ensure no information loss [, ]. Specifically, given the 3D representation, we can recover the given 3D structure. If two 3D structures cannot be matched via a $SE(3)$ transformation, the representations should be different. This property is important to the discriminative ability of models.

We address these challenges by spherical representations, *i.e.*, using spherical coordinates to represent 3D structures.
Compared to Cartesian coordinates, spherical coordinate values are bounded in a smaller region, namely, a range of $[0,\pi]$ or $[0,2\pi]$. This makes spherical coordinates advantageous in discretized representations and thus easier to be modeled by LMs. Given the same decimal place constraints, spherical coordinates require a smaller vocabulary size, and given the same vocabulary size, spherical coordinates present less information loss. This is also supported by empirical results and analysis when using different methods to represent 3D molecular structures, as detailed in Appendix C.

We propose to maintain $SE(3)$-invariance while ensuring no information loss.
Given a 3D molecule $G$ with atom types ${\bm{z}}$ and atom coordinates ${\bm{R}}$, we first build a global coordinate frame ${\bm{F}}=({\bm{x}},{\bm{y}},{\bm{z}})$ based on the input. Specifically, as shown in Figure 1, the frame is built based on the first three non-collinear atoms in the canonical ordering ${\bm{L}}(G)$. Let $\ell_{1},\ell_{2}$, and $\ell_{F}$ be the indices of these three atoms.
Then the global frame ${\bm{F}}=({\bm{x}},{\bm{y}},{\bm{z}})$ is calculated as

$$ $\displaystyle{\bm{x}}$ $\displaystyle=\text{normalize}({\bm{r}}_{\ell_{2}}-{\bm{r}}_{\ell_{1}}),$ (1) $\displaystyle{\bm{y}}$ $\displaystyle=\text{normalize}\left(\left({\bm{r}}_{\ell_{F}}-{\bm{r}}_{\ell_{ 1}}\right)\times{\bm{x}}\right),$ $\displaystyle{\bm{z}}$ $\displaystyle={\bm{x}}\times{\bm{y}}.$ $$

Here $\text{normalize}(\cdot)$ is the function to normalize a vector to unit length.
Note that the global frame is equivariant to the rotation and translation of the input molecule, as shown in Figure 2 and Appendix B.2. After obtaining the global frame, we use a function $f(\cdot)$ to convert the coordinates of each atom to spherical coordinates $d,\theta,\phi$ under this frame.
Specifically, for each node ${\ell_{i}}$ with coordinate ${\bm{r}}_{\ell_{i}}$, the corresponding spherical coordinate is

$$ $\displaystyle d_{\ell_{i}}$ $\displaystyle=||{\bm{r}}_{\ell_{i}}-{\bm{r}}_{\ell_{1}}||_{2},$ (2) $\displaystyle\theta_{\ell_{i}}$ $\displaystyle=\arccos\left(\left({\bm{r}}_{\ell_{i}}-{\bm{r}}_{\ell_{1}}\right )\cdot{\bm{z}}/d_{\ell_{i}}\right),$ $\displaystyle\phi_{\ell_{i}}$ $\displaystyle=\operatorname{atan2}\left(\left({\bm{r}}_{\ell_{i}}-{\bm{r}}_{ \ell_{1}}\right)\cdot{\bm{y}},\left({\bm{r}}_{\ell_{i}}-{\bm{r}}_{\ell_{1}} \right)\cdot{\bm{x}}\right).$ $$

The spherical coordinates show the relative position of each atom in the global frame ${\bm{F}}$. As shown in Figure 2, if the input coordinates are rotated by a matrix ${\bm{Q}}$ and translated by a vector ${\bm{b}}$, the transformed spherical coordinates remain the same, so the spherical coordinates are $SE(3)$-invariant.

Next, we demonstrate that there is no information loss in our method. We show that given our $SE(3)$-invariant spherical representations, we can recover the given 3D structures. For each node ${\ell_{i}}$, we convert the spherical coordinate $[d_{\ell_{i}},\theta_{\ell_{i}},\phi_{\ell_{i}}]$ to coordinate ${\bm{r}}^{\prime}_{\ell_{i}}$ in 3D space as

$$ $[d_{\ell_{i}}\sin(\theta_{\ell_{i}})\cos(\phi_{\ell_{i}}),d_{\ell_{i}}\sin( \theta_{\ell_{i}})\sin(\phi_{\ell_{i}}),d_{\ell_{i}}\cos{\theta_{\ell_{i}}}].$ (3) $$

Note that our reconstructed coordinate ${\bm{r}}^{\prime}_{\ell_{i}}$ may not be exactly the same as the original coordinate ${\bm{r}}_{\ell_{i}}$. However, there exists a $SE(3)$-transformation $g$, such that $g({\bm{r}}^{\prime}_{\ell_{i}})={\bm{r}}_{\ell_{i}}$ for all $i$. Note that the same transformation $g$ is applied to all nodes. Formally, by applying the function $f(\cdot)$ to the 3D coordinate matrix ${\bm{R}}$, we can demonstrate the following properties of spherical representations.

###### Lemma 3.3 .

Let $G=({\bm{z}},{\bm{R}})$ be a 3D graph with node type vector ${\bm{z}}$ and node coordinate matrix ${\bm{R}}$. Let ${\bm{F}}$ be the equivariant global frame of graph $G$ built based on the first three non-collinear nodes in ${\bm{L}}(G)$.
$f(\cdot)$ is our function that maps 3D coordinate matrix ${\bm{R}}$ of $G$ to spherical representations ${\bm{S}}$ under the equivariant global frame ${\bm{F}}$. Then for any 3D transformation $g\in SE(3)$, we have $f({\bm{R}})=f(g({\bm{R}}))$. Given spherical representations ${\bm{S}}=f({\bm{R}})$, there exist a transformation $g\in SE(3)$, such that $f^{-1}({\bm{S}})=g({\bm{R}})$.

Lemma 3.3 indicates that our spherical representation is $SE(3)$-invariant, and we can reconstruct (a transformation of) the original coordinates. Therefore, our method can convert 3D structures into $SE(3)$-invariant representations with no information loss. Detailed proofs are provided in Appendix B.

###### Lemma 3.3 .

### 3.3 Geo2Seq: Geometry Informed Tokenization

In this section, we describe the process and properties of our 3D tokenization method, Geo2Seq.
Equipped with canonical labeling that reduces graph structures to 1D sequences with no information loss regarding graph isomorphism, and $SE(3)$-invariant spherical representations that ensure no 3D information loss, we develop Geo2Seq, a reversible transformation from 3D molecules to 1D sequences.
Figure 1 shows an overview of Geo2Seq.
Specifically, given a graph $G$ with $n$ nodes, Geo2Seq concatenates the node vector $[z_{i},d_{i},\theta_{i},\phi_{i}]$ of every node in $G$ to a 1D sequence by its canonical order, $\ell_{1},\cdots,\ell_{n}$.
To formulate the properties of Geo2Seq, we extend the concept of graph isomorphism in Definition 3.1 to 3D graphs.

###### Definition 3.4 .

[3D Graph Isomorphism]
Let $G_{1}=({\bm{z}}_{1},{\bm{R}}_{1})$ and $G_{2}=({\bm{z}}_{2},{\bm{R}}_{2})$ be two 3D graphs, where ${\bm{z}}_{i}$ is the node type vector and ${\bm{R}}_{i}$ is the node coordinate matrix of the molecule $G_{i}$.
Let $V_{i}$ denote the set of vertices, $A_{i}$ denote node attributes, and no edge exists.
Two 3D graphs $G_{1}$ and $G_{2}$ are 3D isomorphic, denoted as $G_{1}\cong_{3D}G_{2}$, if there exists a bijection $b:V_{1}\rightarrow V_{2}$ such that $G_{1}\cong G_{2}$ given $A_{i}=[{\bm{z}}_{i},{\bm{R}}_{i}]$, and
there exists a 3D transformation $g\in SE(3)$ such that ${\bm{r}}^{G_{1}}_{i}=g({\bm{r}}^{G_{2}}_{b(i)})$.
If a small error $\bm{\epsilon}$ is allowed such that $|{\bm{r}}^{G_{1}}_{i}-g({\bm{r}}^{G_{2}}_{b(i)})|\leq\bm{\epsilon}$, we call the two 3D graphs $\bm{\epsilon}$-constrained 3D isomorphic.

Considering Lemma 3.2, we specify $G=(V,E,A)$ with $A=[{\bm{z}},{\bm{R}}]$ and define the CL function for 3D molecules as ${\bm{L}}_{m}$, which extends the equivalence of Lemma 3.2 to ${\bm{L}}_{m}$ with 3D isomorphism.
We formulate Geo2Seq and our major theoretical derivations below.

###### Theorem 3.5 .

[Bijective Mapping between 3D Graph and Sequence]
Following Definition 3.4, let $G_{1}=({\bm{z}}_{1},{\bm{R}}_{1})$ and $G_{2}=({\bm{z}}_{2},{\bm{R}}_{2})$ be two 3D graphs.
Let ${\bm{L}}_{m}(G)$ be the canonical label for 3D graph $G$ and $f:\mathcal{R}\rightarrow\mathcal{S}$ be the function that maps 3D coordinates to its spherical representations.
Given a graph $G$ with $n$ nodes and ${\bm{X}}=[{\bm{x}}_{1},\cdots,{\bm{x}}_{n}]^{T}\in\mathbb{R}^{n\times m}$, where $m\in\mathbb{Z}$,
we define ${\bm{L}}_{m}(G)\otimes{\bm{X}}=\text{concat}({\bm{x}}_{\ell_{1}},...,{\bm{x}}_
{\ell_{n}})$, where $\ell_{i}$ is the index of the node labeled $i$ by ${\bm{L}}_{m}(G)$, and $\text{concat}(\cdot)$ concatenates elements as a sequence.
We define

$$ $\text{Geo2Seq}(G)={\bm{L}}_{m}(G)\otimes({\bm{z}},f({\bm{R}}))={\bm{L}}_{m}(G) \otimes{\bm{X}},$ $$

where ${\bm{x}}_{i}=[z_{i},d_{i},\theta_{i},\phi_{i}]$.
Then $\text{Geo2Seq}:\mathcal{G}\rightarrow\mathcal{U}$ is a surjective function, and the following equivalence holds:

$$ $\text{Geo2Seq}(G_{1})=Geo2Seq(G_{2})\Leftrightarrow G_{1}\cong_{3D}G_{2},$ $$

where $G_{1}\cong_{3D}G_{2}$ denotes $G_{1}$ and $G_{2}$ are 3D isomorphic.

Theorem 3.5 establishes the following guarantees for Geo2Seq:
(1) Given a 3D molecule, we can uniquely construct a 1D sequence using Geo2Seq. (2) If two molecules are 3D isomorphic, their sequence outputs from Geo2Seq are identical.
(3) Given a sequence output of Geo2Seq, we can uniquely reconstruct a 3D molecule. (4) If two constructed sequences from Geo2Seq are identical, their corresponding molecules must be 3D isomorphic. This enable sequential tokenization of 3D molecules, preserving structural completeness and geometric invariance.

Due to the necessity of discreteness in serialization and tokenization for LMs, in reality, numerical values need to be discretized before concatenation.
In practice, we round up numerical values to certain decimal places. Thus Theorem 3.5 can be extended with constraints, as below.

###### Corollary 3.6 .

[Constrained Bijective Mapping between 3D Graph and Sequence]
Following the notations and definitions of Theorem 3.5, let spherical coordinate values be rounded up to $b$ decimal places.
Then $\text{Geo2Seq}:\mathcal{G}\rightarrow\mathcal{U}$ is a surjective function, and the following equivalence holds:

$$ $Geo2Seq(G_{1})=Geo2Seq(G_{2})\Leftrightarrow G_{1}\cong_{3D-|10^{-b}|/2}G_{2},$ $$

where $G_{1}\cong_{3D-|10^{-b}|/2}G_{2}$ denotes graphs $G_{1}$ and $G_{2}$ are $(|10^{-b}|/2)$-constrained 3D isomorphic.

Corollary 3.6 extends Theorem 3.5’s guarantees for the practical use of Geo2Seq. If we allow a round-up error below $|10^{-b}|/2$ for coordinates when distinguishing 3D isomorphism, all properties still hold.
This implies that the practical Geo2Seq implementation retains near-complete geometric information and invariance, with numerical precision of $\bm{\epsilon}\leq|10^{-b}|/2$.

With discreteness incorporated, we can collect a finite vocabulary covering all accessible molecule samples to enable tokenization for LMs.
Specifically, we use vocabularies of approximately 1K-16K tokens consisting of atom type tokens $C,N,O\cdots$, and spherical coordinate tokens such as $-1.98$, $1.57^{\circ}$ or $-0.032^{\circ}$. The numerical tokens range from the smallest to the largest distance and angle values with restricted precision of 2 or 3 decimal places. Experimental results show the benefits in using this level of tokenization, as detailed in Appendix C.

###### Definition 3.4 .

###### Theorem 3.5 .

###### Corollary 3.6 .

## 4 3D Molecule Generation

Training and Sampling.
Now that we have defined a canonical and robust sequence representation for 3D molecules, we turn to the method of modeling such sequences, $U$. Here, we attempt to train a model $M$ with parameters $\theta$ to capture the distribution of such sequences, $p_{\theta}(U)$, in our dataset. As this is a well-studied problem within language modeling, we opt to use two language models, GPT [] and Mamba [], which have shown effective sequence modeling capabilities on a range of tasks. Both models are trained using a standard next-token prediction cross-entropy loss $\ell$ for all elements in the sequence:

$$ $\min_{\theta}\underset{u\in U}{\mathbb{E}}\left[\sum_{i=1}^{|u|-1}\ell\left(M_ {\theta}(u_{1},\cdots,u_{i}),u_{i+1}\right)\right].$ $$

To sample from a trained model, we first select an initial atom token by sampling from the multinomial distribution of first-tokens in the training data (we note that in almost all cases this is ‘H’). We then perform a standard autoregressive sampling procedure by iteratively sampling from the conditional distribution $p_{\theta}(u_{i+1}|u_{1},\cdots,u_{i})$ until the stop token or max length is reached. We sample from this distribution using top-$k$ sampling [] and a softmax temperature $\tau$ [, ]. Unless otherwise noted, $\tau=0.7$ and $k=80$.

Controllable Generation.
For controllable generation, we follow Bagal et al. [] and use a conditioning token for the desired property. This token is created by projecting the desired properties through a trainable linear layer to create a vector with the model’s initial token embedding space. This property token is then used as the initial element in the molecular sequence. Training and sampling are performed as before with this new sequence formulation. Sampling begins with the desired property’s token as input.

## 5 Experimental Studies

In this section, we evaluate the method of generating 3D molecules in the form of our proposed Geo2Seq representations by LLMs. We show that in the random generation task (see Section 2.1), the performance of Geo2Seq with GPT [] or Mamba [] models is better than or comparable with state-of-the-art 3D point cloud based methods, including EDM [] and GEOLDM []. In addition, in the controllable generation task (see Section 2.1), we show that Geo2Seq with Mamba models outperform previous 3D point cloud based methods by a large margin.

### 5.1 Random Generation

**Table 1: Random generation performance on QM9 and GEOM-DRUGS datasets. Here, larger numbers indicate better performance. bold and underline highlight the best and second best performance, respectively. Note that for GEOM-DRUGS dataset, molecule stability and unique percentage are close to 0% and 100% for all methods so they are not presented. Following Hoogeboom et al. [] and Xu et al. [], we report the mean and standard deviation over three runs on QM9 dataset.**
| Method | QM9 | GEOM-DRUGS |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Atom Sta (%) | Mol Sta (%) | Valid (%) | Valid & Unique (%) | Atom Sta (%) | Valid (%) |  |
| Data | 99.0 | 95.2 | 97.7 | 97.7 | 86.5 | 99.9 |
| E-NFs | 85.0 | 4.9 | 40.2 | 39.4 | - | - |
| G-SchNet | 95.7 | 68.1 | 85.5 | 80.3 | - | - |
| GDM | 97.0 | 63.2 | - | - | 75.0 | 90.8 |
| GDM-AUG | 97.6 | 71.6 | 90.4 | 89.5 | 77.7 | 91.8 |
| EDM | 98.7$\pm$0.1 | 82.0$\pm$0.4 | 91.9$\pm$0.5 | 90.7$\pm$0.6 | 81.3 | 92.6 |
| EDM-Bridge | 98.8$\pm$0.1 | 84.6$\pm$0.3 | 92.0$\pm$0.1 | 90.7$\pm$0.1 | 82.4 | 92.8 |
| GEOLDM | 98.9$\pm$0.1 | 89.4$\pm$0.5 | 93.8$\pm$0.4 | 92.7$\pm$0.5 | 84.4 | 99.3 |
| Geo2Seq with GPT | 98.3$\pm$0.1 | 90.3$\pm$0.1 | 94.8$\pm$0.2 | 80.6$\pm$0.4 | 82.6 | 87.4 |
| Geo2Seq with Mamba | 98.9$\pm$0.2 | 93.2$\pm$0.2 | 97.1$\pm$0.2 | 81.7$\pm$0.4 | 82.5 | 96.1 |

Data. We adopt two datasets, QM9 [] and GEOM-DRUGS [], to evaluate performances in the random generation task. The QM9 dataset collects over 130k 3D molecules with 3D structures calculated by density functional theory (DFT). Each molecule in QM9 has less than 9 heavy atoms and its chemical elements all belong to H, C, N, O, F. Following [], we split the dataset into train, validation and test sets with 100k, 18k and 12k samples, separately. The GEOM-DRUGS dataset consists of over 450k large molecules with 37 million DFT-calculated 3D structures. Molecules in GEOM-DRUGS has up to 181 atoms and 44.2 atoms on average. We follow Hoogeboom et al. [] to select 30 3D structures with the lowest energies per molecule for model training.

Setup. On the QM9 dataset, we set the training batch size to 32, base learning rate to 0.0004, and train a 12-layer GPT model and a 26-layer Mamba model by AdamW [] optimizers. On the GEOM-DRUGS dataset, we set the training batch size to 32, base learning rate to 0.0004, and train a 14-layer GPT model and a 28-layer Mamba model by AdamW optimizers. See Appendix D for more information about hyperparameters and other settings. When model training is completed, we randomly generate 10,000 molecules, and evaluate the performance on these molecules. Specifically, we first transform 3D molecular structures to 2D molecular graphs using the bond inference algorithm implemented in the official code of EDM. Then, we evaluate the performance by atom stability, which is the percentage of atoms with correct bond valencies, and molecule stability, which is the percentage of molecules whose all atoms have correct bond valencies. In addition, we report the percentage of valid molecules that can be successfully converted to SMILES strings by RDKit, and the percentage of valid and unique molecules that can be converted to unique SMILES strings.

Baselines. We compare GPT and Mamba models with several strong baseline methods. Specifically, we compare with an autoregressive generation method G-SchNet [] and an equivariant flow model based method E-NFs []. We also compare with some recently proposed diffusion based methods, including EDM [], GDM (the non-equivariant variant of EDM) and GDM-AUG (GDM trained with random rotation as data augmentation). Besides, we compare with EDM-Bridge [] and GEOLDM [], which are two latest 3D molecule generation methods improving EDM by SDE based diffusion models and latent diffusion models, respectively. To ensure that the comparison is fair, our methods and baseline methods use the same data split and evaluation metrics.

**Table 2: Controllable generation performance in terms of property MAE on QM9 datasets. Here, smaller numbers indicate better performance, and bold and underline highlight the best and second best performance, respectively.**
| Property (Units) | $\alpha$ (Bohr^3) | $\Delta\epsilon$ (meV) | $\epsilon_{\text{HOMO}}$ (meV) | $\epsilon_{\text{LUMO}}$ (meV) | $\mu$ (D) | $C_{v}$ ($\frac{\text{cal}}{\text{mol}}$K) |
| --- | --- | --- | --- | --- | --- | --- |
| Data | 0.10 | 64 | 39 | 36 | 0.043 | 0.040 |
| Random | 9.01 | 1470 | 645 | 1457 | 1.616 | 6.857 |
| $N_{\text{atoms}}$ | 3.86 | 866 | 426 | 813 | 1.053 | 1.971 |
| EDM | 2.76 | 655 | 356 | 584 | 1.111 | 1.101 |
| GEOLDM | 2.37 | 587 | 340 | 522 | 1.108 | 1.025 |
| Geo2Seq with Mamba | 0.46 | 98 | 57 | 71 | 0.164 | 0.275 |

Results. We present the random generation results of different methods on QM9 and GEOM-DRUGS datasets in Table 1. Note that for GEOM-DRUGS dataset, all methods achieve nearly 0% molecule stability percentage and 100% uniqueness percentage. Thus, following previous studies, these two metrics are omitted. According to the results in Table 1, on QM9 dataset, generating 3D molecules in Geo2Seq representations with either GPT or Mamba models achieve better performance than all 3D point cloud based baseline methods in molecule stability and valid percentage, and achieves atom stability percentages close to the upper bound (99%). This demonstrates that our method can model 3D molecular structure distribution and capture the underlying chemical rules more accurately. It is worth noticing that our method does not achieve very high uniqueness percentage, showing that it is not easy for our method to generate a large number of diverse molecules. We believe this is due to that the conversion from real numbers to discrete tokens limits the search space of 3D molecular structures, especially on a small dataset like QM9, while it is easier to generate more diverse molecules for 3D point cloud based methods as they directly generate real numbers. This is reflected by the fact that our method achieves nearly 100% uniqueness percentage on the large GEOM-DRUGS dataset. On GEOM-DRUGS dataset, both GPT and Mamba models achieve reasonably high atom stability and valid percentage. The performance of our method is comparable with strong diffusion based baseline methods, showing that LLMs have the potential to model very complicated drug molecular structures well. We will explore further improving the performance on GEOM-DRUGS dataset with larger LLMs in the future.

See Appendix D.3 for additional experiments on more baselines [, ], and metrics including percentage of novel/complete molecules. See Appendix C for ablation studies about atom order, 3D representation and tokenization, Appendix D for generation complexity analysis and results with pretraining, and Appendix F for token embedding and molecule visualization.

### 5.2 Controllable Generation

Data. In the controllable generation task, we train our models on molecules and their property labels in the QM9 [] dataset. Specifically, we try taking a certain quantum property value as the conditional input to LLMs, and train LLMs to generate molecules with the conditioned quantum property values. Following Hoogeboom et al. [], we split the training dataset of QM9 to two subsets where each subset has 50k samples, and train our conditional generation models and an EGNN [] based quantum property prediction models on these two subsets, respectively. We conduct the controllable generation experiments on six quantum properties from QM9, including polarizability ($\alpha$), HOMO energy ($\epsilon_{\text{HOMO}}$), LUMO energy ($\epsilon_{\text{LUMO}}$), HOMO-LUMO gap ($\Delta\epsilon$), dipole moment ($\mu$) and heat capacity at 298.15K ($C_{v}$).

Setup. For the controllable generation experiment, we train 16-layer Mamba [] models with the same hyperparameters as the random generation experiments in Section 5.1. To evaluate the performance, we sample 10000 quantum property values, generate molecules conditioned on these property values by trained models, and compute the mean absolute difference (MAE) between the given property values and the property values of the generated molecules. Note that we use the trained EGNN based property prediction models to calculate the property values of the generated molecules.

Baselines. We compare our models with two equivariant diffusion models, EDM [] and GEOLDM []. In addition, we use several baselines that are based on dataset molecules. One baseline (Data) is directly taking the molecules from the QM9 dataset and use their property values as conditions. The MAE metric simply reflects the prediction error of the trained property prediction model, which can be considered as a lower bound. The second baseline (Random) is taking the molecules from the dataset but uses the randomly shuffled property values as conditions, and its MAE can be considered as an upper bound. The third baseline ($N_{\text{atoms}}$) uses the molecules from the dataset but uses property values predicted from the number of atoms as conditions. Achieving better performance than this baseline shows that models can use conditional information beyond the number of atoms.

Results. Controllable generation results of different methods are summarized in Table 2. As shown in the table, among all six properties, our method outperforms the strong diffusion based baseline methods EDM and GEOLDM by a large margin. Our method moves a significant step in pushing the performance of controllable generation task towards the lower bound, *i.e.*, Data baseline. As we use the same training set as EDM and GEOLDM to train the conditional generation model, the good performance of our method shows that LLMs have more powerful capacity in incorporating conditional information into the 3D molecular structure generation process. We believe that the powerful long-context correlation capturing structures from LLMs, *e.g.*, attention mechanism, play significant roles in achieving the good control of 3D molecule generation by the conditioned property values. The huge success of LLMs in controllable molecule generation will motivate broader applications of LLMs in goal-directed or constrained drug design. See Appendix F for visualization of molecules generated from some given polarizability values.

## 6 Discussion

Geo2Seq showcases the potential of pure LMs in revolutionizing molecular design and drug discovery when geometric information is properly transformed. The framework has certain limitations, particularly in the generalization abilities across the continuous domain of real numbers. Due to the discrete nature of vocabularies, LMs rely on large pre-training corpus, fine-grained tokenization or emergent abilities for better generalization, as a trade-off to high precision and versatility. Future works points towards several directions, such as expanding on conditional tasks and exploring advanced tokenization techniques.

## Acknowledgments

This work was supported partially by National Science Foundation grant IIS-2006861 and National Institutes of Health grant U01AG070112 (to S.J.), and by the Molecule Maker Lab Institute (to H.J.): an AI research institute program supported by NSF under award No. 2019897. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

## Appendix A Broader Impacts and Limitations

Our work demonstrates the significant potential of pure language models (LMs) in revolutionizing molecular design and drug discovery by effectively transforming geometric information. The challenge of molecule design is particularly daunting when scientific experiments are cost-prohibitive or impractical. In many real-world scenarios, data collection is confined to specific chemical domains, yet the ability to generate molecules for broader tasks where experimental validation is difficult remains crucial. Traditional diffusion-based models fall short in terms of efficiency, scalability, and the ability to learn from extensive databases or transfer knowledge across different tasks. In contrast, LMs exhibit inherent advantages in these areas. We envision the development of efficient, large-scale models trained on vast chemical databases that can function across multiple datasets and molecular tasks. By introducing LMs into the 3D molecule generation field, we unlock substantial potential for broad scientific impact.

Our research adheres strictly to ethical guidelines, with no involvement of human subjects or potential privacy and fairness issues. This work aims to advance the field of Machine Learning and AI for drug discovery, with no immediate societal consequences requiring specific attention. We foresee no potential for malicious or unintended usage beyond known chemical applications. However, we recognize that all technological advancements carry inherent risks, and we advocate for ongoing evaluation of the broader implications of our methodology in various contexts.

We admit certain limitations, including that rounding up numerical values to certain decimal places bring information loss and discretized numbers impair generalization abilities across the continuous domain of real numbers. However, this is a trade-off betweeen advantages brought by our model-agnostic framework. Due to the discrete nature of vocabularies, LMs depend on extensive pre-training corpora, fine-grained tokenization, or emergent abilities for better generalization, balancing high precision and versatility.
Geo2Seq operates solely on the input data, which allows independence from model architecture and training techniques and provides reuse flexibility. This also means that we can effortlessly apply Geo2Seq on the latest generative language models, making seamless use of their capabilities.
Future work points towards expanding on conditional tasks and exploring advanced tokenization techniques to enhance the model’s performance and applicability.

## Appendix B Proofs

### B.1 Proof of Lemma 3.2

###### Lemma (Colored Canonical Labeling for Graph Isomorphism) .

Let $G_{1}=(V_{1},E_{1},A_{1})$ and $G_{2}=(V_{2},E_{2},A_{2})$ be two finite, undirected graphs where $V_{i}$ denotes the set of vertices, $E_{i}$ denotes the set of edges, and $A_{i}$ denotes the node attributes of the graph $G_{i}$ for $i=1,2$. Let ${\bm{L}}:\mathcal{G}\rightarrow\mathcal{L}$ be a function that maps a graph $G\in\mathcal{G}$, the set of all finite, undirected graphs, to its canonical labeling ${\bm{L}}(G)\in\mathcal{L}$, the set of all possible canonical labelings, as produced by the Nauty algorithm. Then the following equivalence holds:

$$ ${\bm{L}}(G_{1})={\bm{L}}(G_{2})\iff G_{1}\cong G_{2}$ $$

where $G_{1}\cong G_{2}$ denotes that the graphs $G_{1}$ and $G_{2}$ are isomorphic.

The Nauty algorithm, tailored for CL and computing graph automorphism groups, presents rigorous mathematical underpinnings to guarantee the CL properties.
Here we leave out the proof of Nauty algorithm’s rigor for canonical labeling, which is detailed in the work of McKay and Piperno []. The key is the refinement process ensuring that the partitioning of the graph’s vertices is done in such a way that any two isomorphic graphs will end with the same partition structure.

###### Lemma (Colored Canonical Labeling for Graph Isomorphism) .

### B.2 Proof of Lemma 3.3

###### Lemma .

Let $G=({\bm{z}},{\bm{R}})$ be a 3D graph with node type vector ${\bm{z}}$ and node coordinate matrix ${\bm{R}}$. Let ${\bm{F}}$ be the equivariant global frame of graph $G$ built based on the first three non-collinear nodes in ${\bm{L}}(G)$.
$f(\cdot)$ is our function that maps 3D coordinate matrix ${\bm{R}}$ to spherical representations ${\bm{S}}$ under the equivariant global frame ${\bm{F}}$. Then for any $SE(3)$ transformation $g$, we have $f({\bm{R}})=f(g({\bm{R}}))$. Given spherical representations ${\bm{S}}=f({\bm{R}})$, there exist a $SE(3)$ transformation $g$, such that $f^{-1}({\bm{S}})=g({\bm{R}})$.

###### Proof.

Let $\ell_{1},\ell_{2}$, and $\ell_{F}$ be the indices of the first three non-collinear atoms in $G$.
Then the global frame ${\bm{F}}=({\bm{x}},{\bm{y}},{\bm{z}})$ is

$$ $\displaystyle{\bm{x}}$ $\displaystyle=\text{normalize}({\bm{r}}_{\ell_{2}}-{\bm{r}}_{\ell_{1}})$ $\displaystyle{\bm{y}}$ $\displaystyle=\text{normalize}\left(\left({\bm{r}}_{\ell_{F}}-{\bm{r}}_{\ell_{ 1}}\right)\times{\bm{x}}_{1}\right)$ $\displaystyle{\bm{z}}$ $\displaystyle={\bm{x}}\times{\bm{y}}$ $$

For a $SE(3)$ transformation $g$, let ${\bm{R}}^{\prime}=g({\bm{R}})={\bm{Q}}{\bm{R}}+{\bm{b}}$. Then the global frame ${\bm{F}}^{\prime}=({\bm{x}}^{\prime},{\bm{y}}^{\prime},{\bm{z}}^{\prime})$ is

$$ $\displaystyle{\bm{x}}^{\prime}$ $\displaystyle=\text{normalize}({\bm{r}}_{\ell_{2}}-{\bm{r}}_{\ell_{1}})=\text{ normalize}(g({\bm{r}}_{\ell_{2}})-g({\bm{r}}_{\ell_{1}}))={\bm{Q}}{\bm{x}}$ $\displaystyle{\bm{y}}^{\prime}$ $\displaystyle=\text{normalize}\left(\left({\bm{r}}_{\ell_{F}}-{\bm{r}}_{\ell_{ 1}}\right)\times{\bm{x}}_{2}\right)=\text{normalize}\left(\left(g({\bm{r}}_{ \ell_{F}})-g({\bm{r}}_{\ell_{1}})\right)\times{\bm{x}}_{2}\right)={\bm{Q}}{\bm {y}}$ $\displaystyle{\bm{z}}^{\prime}$ $\displaystyle={\bm{x}}^{\prime}\times{\bm{y}}^{\prime}=({\bm{Q}}{\bm{x}}) \times({\bm{Q}}{\bm{y}})={\bm{Q}}{\bm{z}}$ $$

Thus ${\bm{F}}^{\prime}={\bm{Q}}{\bm{F}}$.
Here $\text{normalize}(\cdot)$ is the function to normalize a vector to the corresponding unit vector. Then $\forall i$, the spherical representations $f({\bm{R}})_{\ell_{i}}$ is

$$ $\displaystyle d_{\ell_{i}}$ $\displaystyle=||{\bm{r}}_{\ell_{i}}-{\bm{r}}_{\ell_{1}}||_{2}$ $\displaystyle\theta_{\ell_{i}}$ $\displaystyle=\arccos\left(\left({\bm{r}}_{\ell_{i}}-{\bm{r}}_{\ell_{1}}\right )\cdot{\bm{z}}/d_{\ell_{i}}\right)$ $\displaystyle\phi_{\ell_{i}}$ $\displaystyle=\operatorname{atan2}\left(\left({\bm{r}}_{\ell_{i}}-{\bm{r}}_{ \ell_{1}}\right)\cdot{\bm{y}},\left({\bm{r}}_{\ell_{i}}-{\bm{r}}_{\ell_{1}} \right)\cdot{\bm{x}}\right)$ $$

Similarly, the spherical representations $f({\bm{R}}^{\prime})_{\ell_{i}}$ is

$$ $\displaystyle d^{\prime}_{\ell_{i}}$ $\displaystyle=||{\bm{r}}^{\prime}_{\ell_{i}}-{\bm{r}}^{\prime}_{\ell_{1}}||_{2 }=||g\left({\bm{r}}_{\ell_{i}}\right)-g\left({\bm{r}}_{\ell_{1}}\right)||_{2}= d_{\ell_{i}}$ $\displaystyle\theta^{\prime}_{\ell_{i}}$ $\displaystyle=\arccos\left(\left({\bm{r}}^{\prime}_{\ell_{i}}-{\bm{r}}^{\prime }_{\ell_{1}}\right)\cdot{\bm{z}}^{\prime}/d^{\prime}_{\ell_{i}}\right)=\arccos \left(\left(g\left({\bm{r}}_{\ell_{i}}\right)-g\left({\bm{r}}_{\ell_{1}}\right )\right)\cdot{\bm{z}}^{\prime}/d^{\prime}_{\ell_{i}}\right)=\theta_{\ell_{i}}$ $\displaystyle\phi^{\prime}_{\ell_{i}}$ $\displaystyle=\operatorname{atan2}\left(\left({\bm{r}}_{\ell^{\prime}_{i}}-{ \bm{r}}_{\ell^{\prime}_{1}}\right)\cdot{\bm{y}}^{\prime},\left({\bm{r}}_{\ell^ {\prime}_{i}}-{\bm{r}}_{\ell^{\prime}_{1}}\right)\cdot{\bm{x}}^{\prime}\right) =\operatorname{atan2}\left(\left(g\left({\bm{r}}_{\ell_{i}}\right)-g\left({\bm {r}}_{\ell_{1}}\right)\right)\cdot{\bm{y}}^{\prime},\left(g\left({\bm{r}}_{ \ell_{i}}\right)-g\left({\bm{r}}_{\ell_{1}}\right)\right)\cdot{\bm{x}}^{\prime }\right)=\phi_{\ell_{i}}$ $$

Therefore, we show that $f({\bm{R}})=f(g({\bm{R}}))$.
Next, we consider the function $f^{-1}(\cdot)$. For all $i$, the three terms in $f^{-1}({\bm{S}})_{\ell_{i}}$ are

$$ $\displaystyle d_{\ell_{i}}\sin(\theta_{\ell_{i}})\cos(\phi_{\ell_{i}})$ (4) $\displaystyle d_{\ell_{i}}\sin(\theta_{\ell_{i}})\sin(\phi_{\ell_{i}})$ $\displaystyle d_{\ell_{i}}\cos{\theta_{\ell_{i}}}$ $$

Then we have ${\bm{r}}_{\ell_{i}}=f^{-1}({\bm{S}})_{\ell_{i}}^{T}{\bm{F}}+{\bm{r}}_{\ell_{0}}$. Therefore, we show that there exist a $SE(3)$ transformation $g$, such that $g(f^{-1}({\bm{S}}))={\bm{R}}$.
∎

###### Lemma .

###### Proof.

### B.3 Proof of Theorem 3.5

First we establish a lemma and provide its proof.

###### Lemma B.1 .

Let $G_{1}=({\bm{z}}_{1},{\bm{R}}_{1})$ and $G_{2}=({\bm{z}}_{2},{\bm{R}}_{2})$ be two 3D graphs, where ${\bm{z}}_{i}$ is the node type vector and ${\bm{R}}_{i}$ is the node coordinate matrix of the molecule $G_{i}$ for $i=1,2$. Let ${\bm{L}}(G)$ be the canonical label of graph $G$. We have $G_{1}\cong G_{2}$. Let $\ell_{i}$ and $\ell^{\prime}_{i}$ denote the indexes of the node labeled $i$ correspondingly in ${\bm{L}}(G_{1})$ and ${\bm{L}}(G_{2})$, respectively.
Let ${\bm{F}}$ be the equivariant global frame of graph $G$ built based on the first three non-collinear atoms in ${\bm{L}}(G)$.
Let $f:\mathcal{G}\rightarrow\mathcal{S}$ be a surjective function that maps a 3D graph $G\in\mathcal{G}$ to its spherical representations $S=f(G)\in\mathcal{S}$ under the equivariant global frame ${\bm{F}}$.
Then the following equivalence holds:

$$ $\forall i\in V_{1},f(G_{1})_{\ell_{i}}=f(G_{2})_{\ell^{\prime}_{i}}\iff G_{1} \cong_{3D}G_{2}$ $$

where $G_{1}\cong_{3D}G_{2}$ denotes that the graphs $G_{1}$ and $G_{2}$ are 3D isomorphic.

###### Proof.

Let ${\bm{L}}(G)$ be the canonical labeling of graph $G$. Let $\ell_{i}$ and $\ell^{\prime}_{i}$ denote the index of the node labeled $i$ correspondingly in ${\bm{L}}(G_{1})$ and ${\bm{L}}(G_{2})$, respectively.
We have

$$ $G_{1}\cong_{3D}G_{2}\iff\begin{cases}&G_{1}\cong G_{2}\text{, and}\\ &\text{there exists a 3D transformation }g\in SE(3)\text{ such that }{\bm{r}}^ {G_{2}}_{\ell^{\prime}_{i}}=g({\bm{r}}^{G_{1}}_{\ell_{i}}).\end{cases}$ $$

Specifically, $g({\bm{r}}_{\ell_{i}})={\bm{Q}}{\bm{r}}_{\ell_{i}}+{\bm{b}}$. Here ${\bm{Q}}$ is a rotation matrix, and ${\bm{b}}$ is a translation vector.

Let $\ell_{1},\ell_{2}$, and $\ell_{F}$ be the indices of the first three non-collinear atoms in $G_{1}$.
Then the equivariant global frame ${\bm{F}}_{1}=({\bm{x}}_{1},{\bm{y}}_{1},{\bm{z}}_{1})$ is

$$ $\displaystyle{\bm{x}}_{1}$ $\displaystyle=\text{normalize}({\bm{r}}_{\ell_{2}}-{\bm{r}}_{\ell_{1}})$ $\displaystyle{\bm{y}}_{1}$ $\displaystyle=\text{normalize}\left(\left({\bm{r}}_{\ell_{F}}-{\bm{r}}_{\ell_{ 1}}\right)\times{\bm{x}}_{1}\right)$ $\displaystyle{\bm{z}}_{1}$ $\displaystyle={\bm{x}}_{1}\times{\bm{y}}_{1}$ $$

Here $\text{normalize}(\cdot)$ is the function to normalize a vector to the corresponding unit vector. Then $\forall i$, the spherical representations $f(G_{1})_{\ell_{i}}$ is

$$ $\displaystyle d_{\ell_{i}}$ $\displaystyle=||{\bm{r}}_{\ell_{i}}-{\bm{r}}_{\ell_{1}}||_{2}$ $\displaystyle\theta_{\ell_{i}}$ $\displaystyle=\arccos\left(\left({\bm{r}}_{\ell_{i}}-{\bm{r}}_{\ell_{1}}\right )\cdot{\bm{z}}_{1}/d_{\ell_{i}}\right)$ $\displaystyle\phi_{\ell_{i}}$ $\displaystyle=\operatorname{atan2}\left(\left({\bm{r}}_{\ell_{i}}-{\bm{r}}_{ \ell_{1}}\right)\cdot{\bm{y}}_{1},\left({\bm{r}}_{\ell_{i}}-{\bm{r}}_{\ell_{1} }\right)\cdot{\bm{x}}_{1}\right)$ $$

Similarly, for $G_{2}$, let $\ell^{\prime}_{1},\ell^{\prime}_{2}$, and $\ell^{\prime}_{F}$ be the indices of the first three non-collinear atoms. Then the equivariant global frame ${\bm{F}}_{2}=({\bm{x}}_{2},{\bm{y}}_{2},{\bm{z}}_{2})$ is

$$ $\displaystyle{\bm{x}}_{2}$ $\displaystyle=\text{normalize}({\bm{r}}_{\ell^{\prime}_{2}}-{\bm{r}}_{\ell^{ \prime}_{1}})=\text{normalize}(g({\bm{r}}_{\ell_{2}})-g({\bm{r}}_{\ell_{1}}))= {\bm{Q}}{\bm{x}}_{1}$ $\displaystyle{\bm{y}}_{2}$ $\displaystyle=\text{normalize}\left(\left({\bm{r}}_{\ell^{\prime}_{F}}-{\bm{r} }_{\ell^{\prime}_{1}}\right)\times{\bm{x}}_{2}\right)=\text{normalize}\left( \left(g({\bm{r}}_{\ell_{F}})-g({\bm{r}}_{\ell_{1}})\right)\times{\bm{x}}_{2} \right)={\bm{Q}}{\bm{y}}_{1}$ $\displaystyle{\bm{z}}_{2}$ $\displaystyle={\bm{x}}_{2}\times{\bm{y}}_{2}=({\bm{Q}}{\bm{x}}_{1})\times({\bm {Q}}{\bm{y}}_{1})={\bm{Q}}{\bm{z}}_{1}$ $$

Then $\forall i$, the spherical representations $f(G_{2})_{\ell^{\prime}_{i}}$ is

$$ $\displaystyle d_{\ell^{\prime}_{i}}$ $\displaystyle=||{\bm{r}}_{\ell^{\prime}_{i}}-{\bm{r}}_{\ell^{\prime}_{1}}||_{2 }=||g\left({\bm{r}}_{\ell_{i}}\right)-g\left({\bm{r}}_{\ell_{1}}\right)||_{2}= d_{\ell_{i}}$ $\displaystyle\theta_{\ell^{\prime}_{i}}$ $\displaystyle=\arccos\left(\left({\bm{r}}^{\prime}_{\ell_{i}}-{\bm{r}}^{\prime }_{\ell_{1}}\right)\cdot{\bm{z}}_{2}/d_{\ell^{\prime}_{i}}\right)=\arccos\left (\left(g\left({\bm{r}}_{\ell_{i}}\right)-g\left({\bm{r}}_{\ell_{1}}\right) \right)\cdot{\bm{z}}_{2}/d_{\ell^{\prime}_{i}}\right)=\theta_{\ell_{i}}$ $\displaystyle\phi_{\ell^{\prime}_{i}}$ $\displaystyle=\operatorname{atan2}\left(\left({\bm{r}}_{\ell^{\prime}_{i}}-{ \bm{r}}_{\ell^{\prime}_{1}}\right)\cdot{\bm{y}}_{2},\left({\bm{r}}_{\ell^{ \prime}_{i}}-{\bm{r}}_{\ell^{\prime}_{1}}\right)\cdot{\bm{x}}_{2}\right)= \operatorname{atan2}\left(\left(g\left({\bm{r}}_{\ell_{i}}\right)-g\left({\bm{ r}}_{\ell_{1}}\right)\right)\cdot{\bm{y}}_{2},\left(g\left({\bm{r}}_{\ell_{i}} \right)-g\left({\bm{r}}_{\ell_{1}}\right)\right)\cdot{\bm{x}}_{2}\right)=\phi_ {\ell_{i}}$ $$

Therefore, we show that $G_{1}\cong_{3D}G_{2}\iff\forall i,\in V_{1},f(G_{1})_{\ell_{i}}=f(G_{2})_{\ell
^{\prime}_{i}}$ holds.
∎

Then we prove Theorem 3.5.

###### Theorem (Bijective mapping between 3D graph isomorphism and sequence) .

Let $G_{1}=({\bm{z}}_{1},{\bm{R}}_{1})$ and $G_{2}=({\bm{z}}_{2},{\bm{R}}_{2})$ be two 3D graphs, where ${\bm{z}}_{j}$ is the node type vector and ${\bm{R}}_{j}$ is the node coordinate matrix of the molecule $G_{j}$ for $j=1,2$.
Let ${\bm{L}}_{m}(G)$ be the canonical label for 3D graph and $f:\mathcal{G}\rightarrow\mathcal{S}$ be the function that maps a 3D graph $G$ to its spherical representations.
Given graph $G$ with $n$ nodes and ${\bm{X}}=[{\bm{x}}_{1},...,{\bm{x}}_{n}]^{T}\in\mathbb{R}^{n\times m}$, where $m\in\mathbb{Z}$,
we define ${\bm{L}}_{m}(G)\otimes{\bm{X}}=\text{concat}({\bm{x}}_{\ell_{1}},...,{\bm{x}}_
{\ell_{n}})$, where $\ell_{i}$ is the node index of the node labeled $i$ in ${\bm{L}}_{m}(G)$, and $\text{concat}(\cdot)$ concatenates elements as a sequence.
Define

$$ $\text{Geo2Seq}(G_{i})={\bm{L}}_{m}(G)\otimes({\bm{z}},f(G))={\bm{L}}_{m}(G) \otimes{\bm{X}},$ $$

where ${\bm{x}}_{i}=[z_{i},d_{i},\theta_{i},\phi_{i}]$.
Then $\text{Geo2Seq}:\mathcal{G}\rightarrow\mathcal{U}$ is a surjective function, and the following equivalence holds:

$$ $\text{Geo2Seq}(G_{1})=Geo2Seq(G_{2})\iff G_{1}\cong_{3D}G_{2}$ $$

where $G_{1}\cong_{3D}G_{2}$ denotes that the graphs $G_{1}$ and $G_{2}$ are 3D isomorphic.

###### Proof.

First, we prove that $\text{Geo2Seq}:\mathcal{G}\rightarrow\mathcal{U}$ is a surjective function.
Given the definition

$$ $\text{Geo2Seq}(G_{i})={\bm{L}}_{m}(G)\otimes({\bm{z}},f(G))={\bm{L}}_{m}(G) \otimes{\bm{X}},$ $$

where ${\bm{x}}_{i}=[z_{i},d_{i},\theta_{i},\phi_{i}]$,
we need to prove that all operations are deterministic.
$\otimes$ and ${\bm{z}}$ are defined to be deterministic, and $f:\mathcal{G}\rightarrow\mathcal{S}$ is a function.
${\bm{L}}_{m}(G_{j})$ outputs the automorphism group of $G_{j}$’s canonical label. By definition, the automorphism group contain different labels of the strictly identical graph. Let $\ell_{i}$ and $\ell^{\prime}_{i}$ describe two different sets of labels of the same automorphism group with $n$ nodes; since the graphs are identical,

$$ $[z_{\ell_{i}},d_{\ell_{i}},\theta_{\ell_{i}},\phi_{\ell_{i}}]=[z_{\ell^{\prime }_{i}},d_{\ell^{\prime}_{i}},\theta_{\ell^{\prime}_{i}},\phi_{\ell^{\prime}_{i }}]\text{for}i=1,...,n.$ $$

Thus $\text{concat}({\bm{x}}_{\ell_{1}},...,{\bm{x}}_{\ell_{n}})=\text{concat}({\bm{
x}}_{\ell^{\prime}_{1}},...,{\bm{x}}_{\ell^{\prime}_{n}})$, i.e., different labels of one automorphism group produce identical sequences with Geo2Seq.
Therefore, $\text{Geo2Seq}:\mathcal{G}\rightarrow\mathcal{U}$ is a well-defined function; given a 3D molecule, we can uniquely construct a 1D sequence from Geo2Seq.

Next we prove Geo2Seq’s surjectivity. Given any output sequence $q\in\mathcal{U}$ of Geo2Seq, the sequence is in the format

$$ $q=concat([z_{1},d_{1},\theta_{1},\phi_{1}],...,[z_{n},d_{n},\theta_{n},\phi_{n }]).$ $$

For the nodes in $q$, we denote with $S=[[d_{1},\theta_{1},\phi_{1}],...,[d_{n},\theta_{n},\phi_{n}]]$. Given the surjectivity of the spherical representation function $f:\mathcal{G}\rightarrow\mathcal{S}$ and the defined $f^{-1}:\mathcal{S}\rightarrow\mathcal{G}$, there must be a unique $G({\bm{z}},{\bm{R}})\in\mathcal{G}$ where $S=f(G)$.
Therefore, $\forall$ output sequence $q\in\mathcal{U}$ there exists

$$ $G({\bm{z}},{\bm{R}})\in\mathcal{G}\quad\textit{s.t.}\quad q=\text{Geo2Seq}(G),$ $$

i.e., Geo2Seq is surjective; given a sequence output of Geo2Seq, we can uniquely reconstruct a 3D molecule.

Now we prove the equivalence $\text{Geo2Seq}(G_{i})=Geo2Seq(G_{i})\iff G_{1}\cong_{3D}G_{2}$, starting from right to left.
Considering Lemma 3.2 for molecule $G=({\bm{z}},{\bm{R}})$, we specify $G=(V,E,A)$ with $A=[{\bm{z}},{\bm{R}}]$ and define the CL function for 3D molecule graphs as ${\bm{L}}_{m}$, which extends the equivalence in Lemma 3.2 to ${\bm{L}}_{m}$ on molecules with 3D isomorphism.
If $G_{1}\cong_{3D}G_{2}$, i.e., graphs $G_{1}$ and $G_{2}$ are 3D isomorphic, then from Lemma 3.2 we know the canonical forms ${\bm{L}}_{m}(G_{1})={\bm{L}}_{m}(G_{2})$. Let graphs $G_{1}$ and $G_{2}$ have numbers of node $n$.
Let $\ell_{i}$ and $\ell^{\prime}_{i}$ be the denotations of a corresponding pair of canonical labelings from ${\bm{L}}_{m}(G_{1})$ and ${\bm{L}}_{m}(G_{2})$, respectively.
Since graphs $G_{1}$ and $G_{2}$ are 3D isomorphic, from Def.3.4 we know $\forall i\in\mathcal{V}(G_{1}),z_{\ell_{i}}=z_{l^{\prime}_{i}}$;
and from Lemma B.1 we know $\forall i\in\mathcal{V}(G_{1}),f(G_{1})_{\ell_{i}}=f(G_{2})_{\ell^{\prime}_{i}}$. Thus, we have

$$ $\displaystyle\text{Geo2Seq}(G_{1})$ $\displaystyle={\bm{L}}_{m}(G_{1})\otimes({\bm{z}}_{1},f(G_{1}))$ (5) $\displaystyle=\text{concat}_{z_{j}\in{\bm{z}}_{1},d_{j},\theta_{j},\phi_{j}\in f (G_{1}),i=1,...n}([z_{\ell_{i}},d_{\ell_{i}},\theta_{\ell_{i}},\phi_{\ell_{i}}])$ $\displaystyle=\text{concat}_{z_{j}\in{\bm{z}}_{2},d_{j},\theta_{j},\phi_{j}\in f (G_{2}),i=1,...n}([z_{\ell^{\prime}_{i}},d_{\ell^{\prime}_{i}},\theta_{\ell^{ \prime}_{i}},\phi_{\ell^{\prime}_{i}}])$ $\displaystyle={\bm{L}}_{m}(G_{2})\otimes({\bm{z}}_{2},f(G_{2}))=\text{Geo2Seq} (G_{2}).$ $$

Note that if ${\bm{L}}_{m}(G_{1})$ and ${\bm{L}}_{m}(G_{2})$ contain automorphism groups larger than 1, we can include all possible labelings, which will all produce the same sequence later through Geo2Seq, as we have shown in detail above. However, this is a very rare case for real-world 3D graphs like molecules.
Therefore, we have shown that if two molecules are 3D isomorphic considering atoms, bonds, and coordinates, their sequences resulting from Geo2Seq must be identical.

Finally, we prove the equivalence from left to right. We provide proof by contradiction.
Given that $\text{Geo2Seq}(G_{1})=Geo2Seq(G_{2})$, we assume that the graphs $G_{1}$ and $G_{2}$ are not 3D isomorphic.
We denote with $G_{1}=({\bm{z}}_{1},{\bm{R}}_{1})$ and $G_{2}=({\bm{z}}_{2},{\bm{R}}_{2})$.
If $G_{1}$ and $G_{2}$ are not even isomorphic for $A_{i}={\bm{z}}_{i}$, then from Def.3.1, there does not exist a node-to-node mapping from $G_{1}$ to $G_{2}$, where each node is identically attributed and connected. And from Lemma 3.2, we know the canonical forms ${\bm{L}}_{m}(G_{1})\neq{\bm{L}}_{m}(G_{2})$. Thus for

$$ $\text{Geo2Seq}(G_{1})=\text{concat}_{z_{j}\in{\bm{z}}_{1},d_{j},\theta_{j}, \phi_{j}\in f(G_{1}),i=1,...n}([z_{\ell_{i}},d_{\ell_{i}},\theta_{\ell_{i}}, \phi_{\ell_{i}}]),$ $$

and

$$ $\text{Geo2Seq}(G_{2})=\text{concat}_{z_{j}\in{\bm{z}}_{2},d_{j},\theta_{j}, \phi_{j}\in f(G_{2}),i=1,...n}([z_{\ell^{\prime}_{i}},d_{\ell^{\prime}_{i}}, \theta_{\ell^{\prime}_{i}},\phi_{\ell^{\prime}_{i}}]),$ $$

there must be at least one pair of $z_{\ell_{i}},z_{\ell^{\prime}_{i}}$ where $z_{\ell_{i}}\neq z_{\ell^{\prime}_{i}}$. Therefore, $\text{Geo2Seq}(G_{1})\neq\text{Geo2Seq}(G_{2})$, which is a contradiction to the initial condition that $\text{Geo2Seq}(G_{1})=Geo2Seq(G_{2})$ and ends the proof.

If $G_{1}$ and $G_{2}$ are isomorphic for $A_{i}={\bm{z}}_{i}$, we continue with the following analyses.
Let $\ell_{i}$ and $\ell^{\prime}_{i}$ be the denotations of a corresponding pair of canonical labelings from ${\bm{L}}_{m}(G_{1})$ and ${\bm{L}}_{m}(G_{2})$, respectively.
Let $f:\mathcal{G}\rightarrow\mathcal{S}$ be the surjective function mapping a 3D graph to its spherical representations.
Since $G_{1}$ and $G_{2}$ are not 3D isomorphic, from Lemma B.1, we know there exists at least one

$$ $i\in V_{1},s.t.f(G_{1})_{\ell_{i}}\neq f(G_{2})_{\ell^{\prime}_{i}};$ $$

otherwise, we would have

$$ $\forall i\in V_{1},f(G_{1})_{\ell_{i}}=f(G_{2})_{\ell^{\prime}_{i}}\Rightarrow G _{1}\cong_{3D}G_{2},$ $$

contradicting the above condition. Thus for

$$ $\text{Geo2Seq}(G_{1})=\text{concat}_{z_{j}\in{\bm{z}}_{1},d_{j},\theta_{j}, \phi_{j}\in f(G_{1}),i=1,...n}([z_{\ell_{i}},d_{\ell_{i}},\theta_{\ell_{i}}, \phi_{\ell_{i}}]),$ $$

and

$$ $\text{Geo2Seq}(G_{2})=\text{concat}_{z_{j}\in{\bm{z}}_{2},d_{j},\theta_{j}, \phi_{j}\in f(G_{2}),i=1,...n}([z_{\ell^{\prime}_{i}},d_{\ell^{\prime}_{i}}, \theta_{\ell^{\prime}_{i}},\phi_{\ell^{\prime}_{i}}]),$ $$

$G_{1}$ and $G_{2}$ are isomorphic, so

$$ $\forall i=1,...n,z_{\ell_{i}}=z_{\ell^{\prime}_{i}};$ $$

at least one pair of spherical coordinates does not correspond, so there must be at least one pair of $(d_{\ell_{i}},\theta_{\ell_{i}},\phi_{\ell_{i}})$ and $(d_{\ell^{\prime}_{i}},\theta_{\ell^{\prime}_{i}},\phi_{\ell^{\prime}_{i}})$ where

$$ $(d_{\ell_{i}},\theta_{\ell_{i}},\phi_{\ell_{i}})\neq(d_{\ell^{\prime}_{i}}, \theta_{\ell^{\prime}_{i}},\phi_{\ell^{\prime}_{i}}).$ $$

Thus, $\text{Geo2Seq}(G_{1})\neq\text{Geo2Seq}(G_{2})$, which contradicts the initial condition that $\text{Geo2Seq}(G_{1})=Geo2Seq(G_{2})$.
Therefore, we have shown that if two constructed sequences from Geo2Seq are identical, their corresponding molecules must be 3D isomorphic considering atoms, bonds, and coordinates.
This ends the proof.

∎

###### Lemma B.1 .

###### Proof.

###### Theorem (Bijective mapping between 3D graph isomorphism and sequence) .

###### Proof.

### B.4 Proof of Corollary 3.6

###### Corollary (Constrained bijective Mapping between 3D graph and sequence) .

Let $G_{1}=({\bm{z}}_{1},{\bm{R}}_{1})$ and $G_{2}=({\bm{z}}_{2},{\bm{R}}_{2})$ be two 3D graphs, where ${\bm{z}}_{j}$ is the node type vector and ${\bm{R}}_{j}$ is the node coordinate matrix of the molecule $G_{j}$ for $j=1,2$.
Let ${\bm{L}}_{m}(G)$ be the canonical labeling for 3D graph and $f:\mathcal{G}\rightarrow\mathcal{S}$ be the function that maps a 3D graph $G$ to its spherical representations.
Given graph $G$ with $n$ nodes and ${\bm{X}}=[{\bm{x}}_{1},...,{\bm{x}}_{n}]\in\mathbb{R}^{n\times m}$, where $m\in\mathbb{Z}$,
we define ${\bm{L}}_{m}(G)\otimes{\bm{X}}=\text{concat}({\bm{x}}_{\ell_{1}},...,{\bm{x}}_
{\ell_{n}})$, where $\ell_{i}$ is the node index of the node labeled $i$ in ${\bm{L}}_{m}(G)$, and $\text{concat}(\cdot)$ concatenates elements as a sequence.
Define

$$ $\text{Geo2Seq}(G_{i})={\bm{L}}_{m}(G)\otimes({\bm{z}},f(G))={\bm{L}}_{m}(G) \otimes{\bm{X}},$ $$

where ${\bm{x}}_{i}=[z_{i},d_{i},\theta_{i},\phi_{i}]$.
Let the truncation of spherical coordinate values be after $b$ decimal digits.
Then $\text{Geo2Seq}:\mathcal{G}\rightarrow\mathcal{U}$ is a surjective function, and the following equivalence holds:

$$ $Geo2Seq(G_{i})=Geo2Seq(G_{i})\iff G_{1}\cong_{3D-\frac{|10^{-b}|}{2}}G_{2}$ $$

where $G_{1}\cong_{3D-\frac{|10^{-b}|}{2}}G_{2}$ denotes that the graphs $G_{1}$ and $G_{2}$ are $\frac{|10^{-b}|}{2}$-constrained 3D isomorphic.

###### Proof.

First, we prove that $\text{Geo2Seq}:\mathcal{G}\rightarrow\mathcal{U}$ is a surjective function, which resembles the proof for Theorem 3.5.
Given the definition

$$ $\text{Geo2Seq}(G_{i})={\bm{L}}_{m}(G)\otimes({\bm{z}},f(G))={\bm{L}}_{m}(G) \otimes{\bm{X}},$ $$

where ${\bm{x}}_{i}=[z_{i},d_{i},\theta_{i},\phi_{i}]$,
we need to prove that all operations are deterministic.
$\otimes$ and ${\bm{z}}$ are defined to be deterministic, and $f:\mathcal{G}\rightarrow\mathcal{S}$ with truncation after certain decimal places is still a well-defined function.
${\bm{L}}_{m}(G_{j})$ outputs the automorphism group of $G_{j}$’s canonical label. By definition, the automorphism group contain different labels of the strictly identical graph. Let $\ell_{i}$ and $\ell^{\prime}_{i}$ describe two different sets of labels of the same automorphism group with $n$ nodes; since the graphs are identical,

$$ $[z_{\ell_{i}},d_{\ell_{i}},\theta_{\ell_{i}},\phi_{\ell_{i}}]=[z_{\ell^{\prime }_{i}},d_{\ell^{\prime}_{i}},\theta_{\ell^{\prime}_{i}},\phi_{\ell^{\prime}_{i }}]\text{for}i=1,...,n.$ $$

Thus $\text{concat}({\bm{x}}_{\ell_{1}},...,{\bm{x}}_{\ell_{n}})=\text{concat}({\bm{
x}}_{\ell^{\prime}_{1}},...,{\bm{x}}_{\ell^{\prime}_{n}})$, i.e., different labels of one automorphism group produce identical sequences with Geo2Seq.
Therefore, $\text{Geo2Seq}:\mathcal{G}\rightarrow\mathcal{U}$ is still a well-defined function; given a 3D molecule, we can uniquely construct a 1D sequence from Geo2Seq.

Next we prove Geo2Seq’s surjectivity.
Given any output sequence $q\in\mathcal{U}$ of Geo2Seq, the sequence is in the format

$$ $q=concat([z_{1},d_{1},\theta_{1},\phi_{1}],...,[z_{n},d_{n},\theta_{n},\phi_{n }]).$ $$

For the nodes in $q$, we define $S_{trun}=[[d_{1},\theta_{1},\phi_{1}],...,[d_{n},\theta_{n},\phi_{n}]]$. Given the surjectivity of the spherical representation function $f:\mathcal{G}\rightarrow\mathcal{S}$ and the defined $f^{-1}:\mathcal{S}\rightarrow\mathcal{G}$, there must be a unique $G({\bm{z}},{\bm{R}})\in\mathcal{G}$ where $S_{trun}=f(G)$.
Therefore, $\forall$ output sequence $q\in\mathcal{U}$ there exists

$$ $G({\bm{z}},{\bm{R}})\in\mathcal{G}\quad\textit{s.t.}\quad q=\text{Geo2Seq}(G),$ $$

i.e., Geo2Seq is surjective; given a sequence output of Geo2Seq, we can uniquely reconstruct a 3D molecule.

Now we prove the equivalence $\text{Geo2Seq}(G_{i})=Geo2Seq(G_{i})\iff G_{1}\cong_{3D}G_{2}$, starting from right to left.
When a number is truncated after $b$ decimal places, according to the rounding principle, the maximum error caused is $\bm{\epsilon}\leq\frac{|10^{-b}|}{2}$.
Considering Lemma 3.2 for molecule $G=({\bm{z}},{\bm{R}})$, we specify $G=(V,E,A)$ with $A=[{\bm{z}},{\bm{R}}]$ and define the CL function for 3D molecule graphs as ${\bm{L}}_{m}$, which extends the equivalence in Lemma 3.2 to ${\bm{L}}_{m}$ on molecules with 3D isomorphism.
If $G_{1}\cong_{3D-\frac{|10^{-b}|}{2}}G_{2}$, i.e., graphs $G_{1}$ and $G_{2}$ are $\frac{|10^{-b}|}{2}$-constrained 3D isomorphic, then from Lemma 3.2 we know $G_{1}$ and $G_{2}$ are still isomorphic for $A_{i}={\bm{z}}_{i}$, and the canonical forms ${\bm{L}}_{m}(G_{1})={\bm{L}}_{m}(G_{2})$. Let graphs $G_{1}$ and $G_{2}$ have numbers of node $n$.
Let $\ell_{i}$ and $\ell^{\prime}_{i}$ be the denotations of a corresponding pair of canonical labelings from ${\bm{L}}_{m}(G_{1})$ and ${\bm{L}}_{m}(G_{2})$, respectively.
Since graphs $G_{1}$ and $G_{2}$ are $\frac{|10^{-b}|}{2}$-constrained 3D isomorphic, from Def.3.4 we know $\forall i\in\mathcal{V}(G_{1}),z_{\ell_{i}}=z_{l^{\prime}_{i}}$;
and from Lemma B.1 we know $\forall i\in\mathcal{V}(G_{1}),f(G_{1})_{\ell_{i}}=f(G_{2})_{\ell^{\prime}_{i}}$ with $\frac{|10^{-b}|}{2}$ error range allowed for each numerical value. Thus, we still have

$$ $\displaystyle\text{Geo2Seq}(G_{1})$ $\displaystyle={\bm{L}}_{m}(G_{1})\otimes({\bm{z}}_{1},f(G_{1}))$ (6) $\displaystyle=\text{concat}_{z_{j}\in{\bm{z}}_{1},d_{j},\theta_{j},\phi_{j}\in f (G_{1}),i=1,...n}([z_{\ell_{i}},d_{\ell_{i}},\theta_{\ell_{i}},\phi_{\ell_{i}}])$ $\displaystyle=\text{concat}_{z_{j}\in{\bm{z}}_{2},d_{j},\theta_{j},\phi_{j}\in f (G_{2}),i=1,...n}([z_{\ell^{\prime}_{i}},d_{\ell^{\prime}_{i}},\theta_{\ell^{ \prime}_{i}},\phi_{\ell^{\prime}_{i}}])$ $\displaystyle={\bm{L}}_{m}(G_{2})\otimes({\bm{z}}_{2},f(G_{2}))=\text{Geo2Seq} (G_{2}).$ $$

Note that if ${\bm{L}}_{m}(G_{1})$ and ${\bm{L}}_{m}(G_{2})$ contain automorphism groups larger than 1, we can include all possible labelings, which will all produce the same sequence later through Geo2Seq, as we have shown in detail above. However, this is a very rare case for real-world 3D graphs like molecules.
Therefore, we have shown that if two molecules are 3D isomorphic considering atoms, bonds, and coordinates within the round-up error range $\frac{|10^{-b}|}{2}$, their sequences resulting from Geo2Seq must be identical.

Finally, we prove the equivalence from left to right. We provide proof by contradiction.
Given that $\text{Geo2Seq}(G_{1})=Geo2Seq(G_{2})$, we assume that the graphs $G_{1}$ and $G_{2}$ are not $\frac{|10^{-b}|}{2}$-constrained 3D isomorphic.
We denote with $G_{1}=({\bm{z}}_{1},{\bm{R}}_{1})$ and $G_{2}=({\bm{z}}_{2},{\bm{R}}_{2})$.
If $G_{1}$ and $G_{2}$ are not even isomorphic for $A_{i}={\bm{z}}_{i}$, then from Def.3.1, there does not exist a node-to-node mapping from $G_{1}$ to $G_{2}$, where each node is identically attributed and connected. And from Lemma 3.2, we know the canonical forms ${\bm{L}}_{m}(G_{1})\neq{\bm{L}}_{m}(G_{2})$. Thus for

$$ $\text{Geo2Seq}(G_{1})=\text{concat}_{z_{j}\in{\bm{z}}_{1},d_{j},\theta_{j}, \phi_{j}\in f(G_{1}),i=1,...n}([z_{\ell_{i}},d_{\ell_{i}},\theta_{\ell_{i}}, \phi_{\ell_{i}}]),$ $$

and

$$ $\text{Geo2Seq}(G_{2})=\text{concat}_{z_{j}\in{\bm{z}}_{2},d_{j},\theta_{j}, \phi_{j}\in f(G_{2}),i=1,...n}([z_{\ell^{\prime}_{i}},d_{\ell^{\prime}_{i}}, \theta_{\ell^{\prime}_{i}},\phi_{\ell^{\prime}_{i}}]),$ $$

there must be at least one pair of $z_{\ell_{i}},z_{\ell^{\prime}_{i}}$ where $z_{\ell_{i}}\neq z_{\ell^{\prime}_{i}}$. Therefore, $\text{Geo2Seq}(G_{1})\neq\text{Geo2Seq}(G_{2})$, which is a contradiction to the initial condition that $\text{Geo2Seq}(G_{1})=Geo2Seq(G_{2})$ and ends the proof.

If $G_{1}$ and $G_{2}$ are isomorphic for $A_{i}={\bm{z}}_{i}$, we continue with the following analyses.
Let $\ell_{i}$ and $\ell^{\prime}_{i}$ be the denotations of a corresponding pair of canonical labelings from ${\bm{L}}_{m}(G_{1})$ and ${\bm{L}}_{m}(G_{2})$, respectively.
Let $f:\mathcal{G}\rightarrow\mathcal{S}$ be the surjective function mapping a 3D graph to its spherical representations.
Since $G_{1}$ and $G_{2}$ are not $\frac{|10^{-b}|}{2}$-constrained 3D isomorphic, from Lemma B.1, we know there exists at least one

$$ $i\in V_{1},s.t.f(G_{1})_{\ell_{i}}\neq f(G_{2})_{\ell^{\prime}_{i}},$ $$

even with error range $\frac{|10^{-b}|}{2}$ allowed;
otherwise, we would have

$$ $\forall i\in V_{1},f(G_{1})_{\ell_{i}}=f(G_{2})_{\ell^{\prime}_{i}}\Rightarrow G _{1}\cong_{3D}G_{2},$ $$

contradicting the above condition. Thus for

$$ $\text{Geo2Seq}(G_{1})=\text{concat}_{z_{j}\in{\bm{z}}_{1},d_{j},\theta_{j}, \phi_{j}\in f(G_{1}),i=1,...n}([z_{\ell_{i}},d_{\ell_{i}},\theta_{\ell_{i}}, \phi_{\ell_{i}}]),$ $$

and

$$ $\text{Geo2Seq}(G_{2})=\text{concat}_{z_{j}\in{\bm{z}}_{2},d_{j},\theta_{j}, \phi_{j}\in f(G_{2}),i=1,...n}([z_{\ell^{\prime}_{i}},d_{\ell^{\prime}_{i}}, \theta_{\ell^{\prime}_{i}},\phi_{\ell^{\prime}_{i}}]),$ $$

$G_{1}$ and $G_{2}$ are isomorphic, so

$$ $\forall i=1,...n,z_{\ell_{i}}=z_{\ell^{\prime}_{i}};$ $$

at least one pair of spherical coordinates does not correspond, so there must be at least one pair of $(d_{\ell_{i}},\theta_{\ell_{i}},\phi_{\ell_{i}})$ and $(d_{\ell^{\prime}_{i}},\theta_{\ell^{\prime}_{i}},\phi_{\ell^{\prime}_{i}})$ where

$$ $\text{min}(|d_{\ell_{i}},\theta_{\ell_{i}},\phi_{\ell_{i}}|-|d_{\ell^{\prime}_ {i}},\theta_{\ell^{\prime}_{i}},\phi_{\ell^{\prime}_{i}}|)>\frac{|10^{-b}|}{2}.$ $$

Thus $\text{Geo2Seq}(G_{1})\neq\text{Geo2Seq}(G_{2})$, which contradicts the initial condition that $\text{Geo2Seq}(G_{1})=Geo2Seq(G_{2})$.
Therefore, we have shown that if two constructed sequences from Geo2Seq are identical, their corresponding molecules must be 3D isomorphic considering atoms, bonds, and coordinates within the round-up error range $\frac{|10^{-b}|}{2}$.
This ends the proof.

∎

###### Corollary (Constrained bijective Mapping between 3D graph and sequence) .

###### Proof.

## Appendix C Ablation Studies

**Table 3: Random generation performance with different atom generation orders.**
| Order | Atom Sta (%) | Mol Sta (%) | Valid (%) | Valid & Unique (%) |
| --- | --- | --- | --- | --- |
| Canonical-locality | 97.39 | 86.77 | 92.97 | 84.71 |
| Canonical-nonlocality | 96.45 | 81.36 | 90.89 | 83.37 |
| DFS [] | 95.95 | 81.54 | 90.45 | 82.48 |
| BFS [] | 96.85 | 80.92 | 90.49 | 76.13 |
| Dijkstra [] | 95.29 | 77.25 | 88.97 | 73.52 |
| Cuthill–McKee [] | 93.56 | 71.57 | 85.36 | 76.23 |
| Hilbert-curve [] | 90.11 | 64.99 | 80.40 | 67.83 |
| Random | 64.87 | 20.14 | 43.16 | 38.44 |

**Table 4: Random generation performance with different 3D representations.**
| 3D representation | Atom Sta (%) | Mol Sta (%) | Valid (%) | Valid & Unique (%) |
| --- | --- | --- | --- | --- |
| Original coordinates | 91.1 | 58.1 | 75.6 | 55.1 |
| Invariant coordinates | 96.0 | 78.5 | 89.7 | 74.1 |
| Spherical coordinates | 97.0 | 82.2 | 91.0 | 75.5 |

**Table 5: Random generation performance with different tokenization.**
| Tokenization | Atom Sta (%) | Mol Sta (%) | Valid (%) | Valid & Unique (%) |
| --- | --- | --- | --- | --- |
| Sub-tokenization | 96.4 | 80.3 | 89.9 | 74.4 |
| Comp-tokenization | 97.0 | 82.2 | 91.0 | 75.5 |

To study the effects of atom order, 3D representations and tokenization of Geo2Seq on the generation performance of LLMs, we conduct a series of ablation experiments. Among all ablation experiments, we train 8-layer GPT models on QM9 dataset for 250 epochs with the same hyperparameters as Section 5.1 and use the random generation metrics in Section 5.1 to compare the performance under different settings.

Ablation on atom order. First, we show that our proposed canonical order of atoms in Geo2Seq sequence representation is significant for LLMs to achieve good 3D molecular structure modeling.
Specifically, we conduct an extended study of ordering algorithms, comparing our Geo2Seq with alternative canonicalization strategies as well as established traversing baselines.
As we specified in Sec 3.1, theoretically, analyses and derivations apply to all rigorous CL algorithms. In the paper, we select Nauty Algorithm because its implementation has the best time efficiency among all existing CL algorithms. We implemented Nauty Algorithm for 3D molecules, where multiple strategies can be applied for the partitioning of graph vertices (a step in Nauty). We compare canonicalization strategies with/without locality considered.
The traversing baselines includes Breadth-First Search (BFS) [], Depth-First Search (DFS) [], Dijkstra’s algorithm [], Cuthill–McKee algorithm [], and Hilbert curve [].
We also compare with a Random sequence representation where atoms are randomly ordered.
All the other settings of sequence representations remain the same.
As Table 3 shows, canonicalization with locality considered can lead to better results, due to the importance of neighboring atom interactions in molecular evaluations. In addition, we can clearly observe that well-designed canonical ordering as in Geo2Seq significantly outperforms basic traverse strategies and the random order, which validates the significance of canonical order.

Ablation on 3D representation. Besides, we explore using different methods to represent 3D molecular structures. We compare the spherical coordinates in Geo2Seq with directly using the 3D Cartesian coordinates of atoms from QM9 xyz data files in sequences. Additionally, we also compare with using the $SE(3)$-invariant Cartesian coordinates that are projected to the equivariant frame proposed in Section 3.2. Results in Table 4 demonstrate that LLMs achieve the best performance on spherical coordinates. We believe this is due to that the numerical values of distances and angles of spherical coordinates lie in a smaller region than coordinates, which reduces outliers and makes it easier for LLMs to capture their correlation.

Advantage of invariant spherical representations. The above experiments show the superiority of invariant spherical coordinates over invariant Cartesian coordinates. While invariant Cartesian coordinates when our proposed equivariant frame is applied can also $SE(3)$-invariance, spherical coordinates are advantageous in discretized representations. Compared to Cartesian coordinates, spherical coordinate values are bounded in a smaller region, namely, a range of $[0,\pi]$ or $[0,2\pi]$. Given the same decimal place constraints, spherical coordinates require a smaller vocabulary size, and given the same vocabulary size, spherical coordinates present less information loss. This makes spherical coordinates advantageous in discretized representations and thus easier to be modeled by LMs. Lemma 3.3 and its proof aim to guarantee the validity that our proposed invariant spherical representations possess $SE(3)$-invariance. We consider it as a part of our theoretical contribution towards the derivation of Theorem 3.5.

Ablation on tokenization. Finally, we try a new way to tokenize real numbers in spherical coordinates. Instead of simply taking the complete real number as a token (Comp-tokenization), we try splitting it by the decimal point and treat every part as an individual token (Sub-tokenization). We compare these two tokenization methods in Table 5. Results show that our used Comp-tokenization leads to better performance. This shows that treating the complete real number as an individual token enables LLMs to capture 3D molecular structures more effectively.

Overall, through a series of ablation experiments, we show that canonical atom order, spherical coordinate representation and Comp-tokenization in Geo2Seq are all very useful in parsing 3D molecules to good sequence representations.

## Appendix D Experimental Details and Additional Results

### D.1 Hyperparameters and Experimental Details

In the random generation experiment (Section 5.1), we apply two LMs, GPT [] and Mamba [], to our proposed Geo2Seq representations. For GPT models, we adopt the architecture of GPT-1, set the hidden dimension to 768, the number of attention head to 8, and the number of layers to 12 and 14 for QM9 and GEOM-DRUGS datasets, respectively. For Mamba models, we set the hidden dimension to 768 and the number of layers to 26 and 28 for QM9 and GEOM-DRUGS datasets, respectively. On QM9 dataset, we set the batch size to 32, base learning rate to 0.0004, the number of training epochs to 300 and 210 for GPT and Mamba models, respectively. On GEOM-DRUGS dataset, we set the batch size to 32, base learning rate to 0.0004, the number of training epochs to 20 and 25 for GPT and Mamba models, respectively. During model training, we use AdamW [] optimizer and follow the commonly used linear warm up and cosine decay scheduler to adjust learning rates. Specifically, the learning rate first linearly increases from zero to the base learning rate 0.0004 when handling the first 10% of total training tokens, then gradually decreases to 0.00004 by the cosine decay scheduler. Besides, the tokenization of real numbers uses the precision of two and three decimal places for QM9 and GEOM-DRUGS datasets, respectively. In the controllable generation experiment (Section 5.2), we train 16-layer Mamba models for 200 epochs, and all the other hyperparameters and settings are the same as the random generation experiment.
Based on data statistics, we set the context length to 512 for QM9 dataset and 744 for GEOM-DRUGS dataset throughout the experiments.
All experiments on the QM9 dataset are conducted using a single NVIDIA A6000 GPU. Experiments on the GEOM-DRUGS dataset are deployed on 4 NVIDIA A100 GPUs.

### D.2 Licenses

We strictly follow all licenses when using the public assets in this work.
The QM9 dataset is under license CC-BY 4.0. The GEOM-DRUGS dataset is under license CC0 1.0. The code of EDM, GEOLDM, JODO, and MiDi is under MIT License.

### D.3 Experiments on additional baselines and metrics

We extend our experiments with two more baselines, JODO [] and MiDi [], which are diffusion models jointly generating 2D and 3D molecular information. We exclude them in experiments of the main paper, since the setting is not the same as ours. Our method follows works on 3D molecule generation without 2D information, such as bonds.

We extend the metrics of our evaluation for more comprehensive comparisons on random generation of QM9 dataset. We report the percentage of valid, unique and novel molecules, i.e., that are not present in the training set. We also report the percentage of complete molecules in which all atoms are connected. Following JODO [], we also include 2D metrics. Frechet ChemNet Distance (FCD) measures the distance between the test set and the generated set with the activation of the penultimate layer of ChemNet. Lower FCD values indicate more similarity between the two distributions. Similarity to the nearest neighbor (SNN) calculates an average Tanimoto similarity between the fingerprints of a generated molecule and its closest molecule in the test set. Fragment similarity (Frag) compares the distributions of BRICS fragments in the generated and test sets, and Scaffold similarity (Scaf) compares the frequencies of Bemis-Murcko scaffolds between them. Additionally, we include alignment metrics. For RDKit generated bonds, we compute the Maximum Mean Discrepancy (MMD) distances of the bond length (Bond), bond angle (Angle), and dihedral angle (Dihedral) distributions, and report their mean MMD distances.
To ensure fair comparison, we evaluate the metrics of all methods on the generated 3D structures, and use RDKit to convert 3D structures to 2D graphs if needed. We use the same model and settings as the main paper for Geo2Seq, and follow the released codes for the baselines’ respective hyperparameter and settings.
Table 6 reports the random generation results on QM9 dataset.
According to the results, though our model is not designed to directly learn 2D information, the performance of our method is better than or comparable with baseline methods on all metrics including the 2D metrics, which demonstrates the effectiveness of our design.

**Table 6: Additional random generation results on QM9 dataset.**
| Metric | EDM | GEOLDM | JODO | MiDi | Geo2Seq with Mamba |
| --- | --- | --- | --- | --- | --- |
| Atom Sta (%) | 98.7 | 98.9 | 98.9 | 98.2 | 98.9 |
| Mol Sta (%) | 82.0 | 89.4 | 89.0 | 83.5 | 93.2 |
| Valid (%) | 91.9 | 93.8 | 94.9 | 95.2 | 97.1 |
| Valid & Unique (%) | 90.7 | 91.8 | 92.8 | 92.8 | 81.7 |
| Valid & Unique & Novel (%) | 83.0 | 83.1 | 85.2 | 85.5 | 71.2 |
| Complete (%) | 90.9 | 93.3 | 94.4 | 94.4 | 97.3 |
| Bond Length MMD | 0.18 | 0.12 | 0.27 | 1.09 | 0.08 |
| Bond Angle MMD | 0.04 | 0.04 | 0.05 | 0.05 | 0.04 |
| Dihedral Angle MMD | 0.003 | 0.003 | 0.0022 | 0.0033 | 0.0011 |
| FCD | 1.16 | 0.94 | 1.55 | 1.28 | 2.04 |
| SNN | 0.47 | 0.49 | 0.47 | 0.47 | 0.49 |
| Frag | 0.94 | 0.94 | 0.94 | 0.94 | 0.83 |
| Scaf | 0.29 | 0.33 | 0.25 | 0.26 | 0.38 |

Reporting the percentage of novel molecules is important in showing that language models can generate new molecules instead of merely memorizing the training dataset.
Given our improvements on controllable generation is significant, we explore whether the generated molecules are different from the molecules in the training set.
Thus we also extend the metric on controllable generation experiments. We use the same model and setting as the main paper.
Table 7 presents the novelty results of controllable generation compared with EDM and JODO.
Results show that our method achieves reasonably high novelty scores, which demonstrates that our method is not simply memorizing training data.

**Table 7: Additional controllable generation results for the percentage of valid, unique, and novel molecules on QM9 dataset.**
| Method | $\alpha$ | $\Delta\epsilon$ | $\epsilon_{\text{HOMO}}$ | $\epsilon_{\text{LUMO}}$ | $\mu$ | $C_{v}$ |
| --- | --- | --- | --- | --- | --- | --- |
| EDM | 87.0% | 84.1% | 79.8% | 84.7% | 73.0% | 68.0% |
| JODO | 86.5% | 87.3% | 86.7% | 86.2% | 86.8% | 85.6% |
| Geo2Seq with Mamba | 82.8% | 82.8% | 83.6% | 83.0% | 83.3% | 83.6% |

In addition, following Hoogeboom et al. [], we compare negative log-likelihood (NLL) performance on the random generation of QM9 dataset for Geo2Seq and baseline models that reports this metric. For this experiment, we use the same model and setting as the main paper. From Table 8, we can see the performance of our method is better than or comparable with all baseline methods, evidencing the validity of our model.

**Table 8: Additional Negative Log Likelihood (NLL) comparisons of random generation on QM9 dataset.**
| Method | NLL |
| --- | --- |
| E-NF | -59.7 |
| GDM | -94.7 |
| EDM | -110.7 |
| GEOLDM | -335.0 |
| Geo2Seq with Mamba | -242.0 |

### D.4 Generation Efficiency Analysis

We compare the generation efficiency of our method and the diffusion-based methods using a single NVIDIA A100 GPU and a batch size of 32. The results in Table 6 show that our method is much faster than diffusion-based methods, indicating the great efficiency of our method. Though we have take more memory compared to diffusion-based methods, our time efficiency is much better than diffusion-based methods. Throughput, or samples per second, is one of the most important metrics to measure generation efficiency. In particular, Geo2Seq with Mamba is more than 100 times faster than diffusion-based methods, indicating the high throughput of our method, a significant advantage in practical applications where speed is crucial.

**Table 9: Generation efficiency comparison between diffusion-based methods and our LM-based method.**
| Method | QM9 | DRUG |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Parameters | Memory | Sample/second | Parameters | Memory | Sample/second |  |
| EDM | 5.3M | 1.5GB | 1.4 | 2.4M | 7.4GB | 0.1 |
| GeoLDM | 11.4M | 1.5GB | 1.4 | 5.5M | 8.4GB | 0.1 |
| Geo2Seq with GPT | 87.7M | 2.4GB | 8.3 | 105.4M | 3.1GB | 0.2 |
| Geo2Seq with Mamba | 91.8M | 2.2GB | 100.0 | 108.4M | 2.6GB | 16.7 |

### D.5 Results with Pretraining

To show the advantage of pretraining, we compare the random generation performance on QM9 for models with and without pretraining on Molecule3D [] dataset, which includes around 4M molecules. Specifically, we conduct experiments on an 8-layer GPT model and a 20-layer Mamba model. The models are pretrained for 20 epochs and then finetuned for 200 epochs. The results in Table 10 demonstrate the advantage of pretraining. Future studies could explore pretraining on larger datasets.

**Table 10: Random generation performance on QM9 for models with and without pretraining on Molecule3D dataset.**
| Method | Atom Sta (%) | Mol Sta (%) | Valid (%) | Valid & Unique (%) |
| --- | --- | --- | --- | --- |
| Geo2Seq with GPT | 97.0 | 82.2 | 91.0 | 75.5 |
| Geo2Seq with GPT + pretraining | 98.5 | 89.7 | 94.8 | 76.6 |
| Geo2Seq with Mamba | 97.4 | 86.8 | 93.0 | 78.8 |
| Geo2Seq with Mamba + pretraining | 98.3 | 89.4 | 94.9 | 83.5 |

## Appendix E Extended Studies

### E.1 Scaling Laws

Scaling law refers to the relations between functional properties of interest, performance metrics in our case, and properties of the architecture or optimization process.
In this section we explore the scaling laws of our models, specifically regarding parameter size, since they provide typical insights for LMs.
Scaling laws in 3D molecule generation appears similar to that in NLP. We provide experiments on both GPT and Mamba in Table 11 and 12, respectively. As can be observed, LMs’ performances on molecules grow significantly with parameter size increase, similar to the emergence abilities widely-recognized in NLP tasks. As known from NLP studies [], model capabilities grow consistently with model size, while emergence abilities are largely caused by nonlinear metrics. This matches our observations, since the chemical metrics are hardly linear.

**Table 11: Scaling laws on Geo2Seq with GPT model.**
| Parameter size - GPT | 2556532 | 31309824 | 61650944 | 88012800 | 116342688 |
| --- | --- | --- | --- | --- | --- |
| Atom sta(%) | 76.2 | 89.6 | 96.5 | 98.3 | 98.5 |
| Mol sta(%) | 5.1 | 42.4 | 81.3 | 89.1 | 90.6 |
| Valid(%) | 45.5 | 73.1 | 90.9 | 94.3 | 95.1 |
| Valid & Unique(%) | 43.4 | 66.7 | 83.6 | 74.9 | 78.6 |

**Table 12: Scaling laws on Geo2Seq with Mamba model.**
| Parameter size - Mamba | 2180352 | 31458048 | 61631232 | 93088512 | 121977600 |
| --- | --- | --- | --- | --- | --- |
| Atom sta(%) | 81.6 | 95.7 | 97.4 | 97.8 | 97.9 |
| Mol sta(%) | 13.6 | 79.2 | 86.8 | 88.3 | 89.0 |
| Valid(%) | 51.2 | 89.4 | 93 | 93.7 | 94.4 |
| Valid & Unique(%) | 49.6 | 78.7 | 78.8 | 82.6 | 83.5 |

Note that we evaluate all models after 250 epochs for fairness concerns, while this fixed hyperparameter setting is not optimal for performances at all parameter sizes. Other settings are the same as the ablation studies.

### E.2 Error Case Analysis

In the natural language domain, trained language models can produce error cases showing repetition or hallucinations. This is also a problem that often arises with LLMs. In this section, we provide the analysis of some error cases to introduce more insights into the field.

Similarly to NLP cases, our trained language models are showing repetition or hallucinations, especially when not trained to best convergence. This happens to both GPT and Mamba models. Below we show some error cases from a 16-layer Mamba model trained 150 epochs on the QM9 dataset.
The error case below shows a typical repetition problem. The model generates repeated tokens for several periods, resulting in an invalid sample.

- •
H 0.00 0.00° 0.00° C 1.09 1.57° 0.00° N 2.02 2.15° 0.00° C 3.39 1.99° -0.02° H 3.98 2.10° 0.23° C 4.34 2.11° -0.35° H 4.43 2.38° -0.46° H 5.41 2.09° -0.29° H 4.29 1.96° -0.59° C 4.05 1.63° 0.04° H 4.09 4.09 4.09 4.09 4.09 4.09 4.09 4.09 4.09 4.09 4.09 4.09 4.09 4.09 4.09 4.09 4.09 4.09 4.09 4.09 4.09 1.01° H 4.96 4.96 4.96 H 1.23° 1.23° 1.23° C 3.27 1.74° -0.03° H 4.02 1.83° -0.23° H 3.79 1.91° 0.17° H 3.79 1.53° -0.03°

For hallucination, our tokenization design actually prevents token-level hallucination by defining elements and whole-numerical-values as tokens, instead of using single characters. This prevents token-level hallucination, i.e., non-existent elements or numbers such as ‘Hr’ or ‘-0..15’. However, there can still be sequence-level hallucinations, such as the error case below. The model generates distance values in the place the should be angle values (and vice versa).

- •
H 0.00 0.00° 0.00° N 1.01 1.57° 0.00° H 1.70 2.14° 0.00° C 2.06 1.13° -0.48° O 3.13 1.34° -0.42° N 2.49 0.62° -0.87° C 2.94 0.10° -1.64° H 3.20 0.39° 3.91 H 2.81 0.33° -3.14° C 4.43 0.10° -1.77° H 5.15 0.22° 0.22° H 2.78 0.21° -0.95° C 1.86° 1.86° 4.84 H 0.19° 1.89° -2.06° H 8.28 1.94° -1.69° H 6.24 1.71° 5.70 C 3.97 0.67° -0.90° H 5.02 0.68° -0.77° H 4.15 0.91° -1.11° C 2.93 0.50° 6.97 H 3.54 0.42° 0.42° H 2.74 0.88° 0.88°

These error cases will be rarer if the model well converges.

## Appendix F Visualization Results

### F.1 Visualization of Generated Molecules

In this section, we provide visualizations of molecules generated from Geo2Seq with Mamba conditionally on the property of Polarizability $\alpha$ in Figure 3. The Polarizability of a molecule is the tendency to acquire an electric dipole moment when the molecule is subject to an external electric field. Large $\alpha$ values usually correspond to less isometrically molecular geometries. This is consistent with our generated examples.

In addition, we provide visualizations of molecules generated from Geo2Seq with Mamba trained on QM9 and DRUG in Figure 4 and Figure 5, respectively. These examples are randomly generated without any cherry pick.
From the figures, we can see that the model can generate realistic molecular geometries for both small and large size molecules. However, similar to previous methods [, ], there are disconnected components, especially for larger molecules. A possible future direction is to apply fragment-based methods to reduce the sequence length, thus benefiting the training of language models.

Figure: Figure 3: Visualization of generated molecules condition on the property of Polarizability $\alpha$.
Refer to caption: x3.png

Figure: Figure 4: Visualization of molecules generated from Geo2Seq with Mamba trained on QM9.
Refer to caption: x4.png

Figure: Figure 5: Visualization of molecules generated from Geo2Seq with Mamba trained on GEOM-DRUGS.
Refer to caption: x5.png

### F.2 Visualization of Learned Token Embeddings

In this section, we provide UMAP visualizations of different token embeddings learned by Mamba models trained on QM9 and GEOM-DRUGS datasets. Patterns of the embeddings indicate that the model has successfully learned structure information from the sequence data, showcasing LMs’ capabilities to understanding molecules precisely in 3D space. We provide analyses of the visualization results in the caption of each figure.

Figure: Figure 6: UMAP visualization of element token embeddings learned by a Mamba model trained on GEOM-DRUGS. Red groups indicate columns in the periodic table and blue groups indicate rows, which are both numbered. Points are colored by atomic weight. Overall, the model appears to capture the structure of the periodic table. The column generally increases from top to bottom, and the row generally increases from left to right.
Refer to caption: extracted/5800645/fig/mamba_drug_ele_anno.png

Figure: Figure 7: UMAP visualization of element token embeddings learned by a Mamba model trained on QM9. Points are colored by atomic weight. Overall, the model appears to distinguish well between different elements. All different elements are distributed distantly from each other in the embedding space.
Refer to caption: extracted/5800645/fig/mamba_qm9_ele-2.png

Figure: Figure 8: 2D and 3D UMAP visualization of angle token embeddings learned by a Mamba model trained on GEOM-DRUGS. It can be observed that similar tokens (e.g., ‘1.41°’ and ‘1.42°’) are placed next to each other and the overall structure is a loop. Further, $\pi$-out-of-phase angles are placed near each other, such as ‘3.14°’, ‘-3.14°’, and ‘0°’.
Refer to caption: extracted/5800645/fig/mamba_drug_angle.png

Figure: Figure 9: 2D and 3D UMAP visualization of distance token embeddings learned by a Mamba model trained on QM9. Representations of distances lower than 6 form relatively distinct patterns. This is likely because these values are much more frequently seen in the training data. Values over 20 cluster into a clump, suggesting that they are also recognized by the model.
Refer to caption: extracted/5800645/fig/mamba_qm9_dist.png

Figure: Figure 10: 2D and 3D UMAP visualization of distance token embeddings learned by a Mamba model trained on GEOM-DRUGS. It is notable that the best and most distinct representations seem to arise from between 5 and 20. This is likely because these values are much more frequently seen in the training data. Values over 20 form an indistinct clump. Interestingly, values $>20$ are near values $<3$, which is initially unintuitive; however, they are likely placed in a similar location in the embedding space since both small and large distances are rarely seen in the data.
Refer to caption: extracted/5800645/fig/mamba_drug_dist.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: How would the method perform if the number of tokens in the tokenizer would be restricted to 50, 100, 500, 1000 ?
Why does mamba seems to perform better than gpt in your evaluation of random generation (table 1)? Why is gpt not featured in table 2?
Would adding the position information on embedding level make more sense?
How is the Geo2Seq canonicalization different from canonical SMILES?
Why would the cartesian coordinate tokens with the same frame as Geo2Seq not be rotation invariant?
soundness: 2
strengths: The work provides a way to generate 3D conformers using autoregressive generation. 
The bijective mapping between the molecular graph and the sequences produced by Geo2Seq is demonstrated.
summary: This paper proposes Geo2Seq tokenization scheme that is meant to enable generation of 3D molecules using autoregressive LLMs.
It canonicalizes the graph and creates an input dependent frame of reference with spherical coordinates to encode positions as strings to be provided for the autoregressive model.
weaknesses: A few minor stylistic points:
- Line 111: what is 'molecular science' ? 
- Line 80, 401: perhaps "conditional generation" would be a more precise term than "Controllable generation".
- Line 107: "LLMs have revolutionized the landscape of NLP and beyond." - what is the "NLP and beyond" field?

The proposed tokenization scheme relies on the model to be able to 'understand' the 'number tokens'. The appendix shows UMAP plots demonstrating that the models learn some spherical-like structure from those tokens. Could adding the SE(3) invariant position embeddings be a more efficient way to achieve this?
The vocabulary size is reported to be '1.8K for QM9 dataset' and '16K for Geom-Drug' for problems with single digit number of possible atoms. Could this be a sign of overfitting? 
The autoregressive models used are also much larger than the GNN baselines (~90M parameters vs ~5M), could the performance gains be a result of this?

---

confidence: 3
contribution: 3
initial_rating: 10
presentation: 4
questions: I could find the answers to most of my questions in paper. Thank you for providing such a detailed report of the work.
soundness: 4
strengths: - Proposed tokenization method Geo2Seq is agnostic to the used LMs, which provides flexibility in method usage [SIgnificance].
- [Clarity] The paper outline and structure are well defined and easy to follow.
- [Quality] Provision of additional random generation results for QM9 dataset, as well as additional experiments on more baselines and metrics perfectly visualize the novel method performance compared with existing methodologies.
- [Originality] Usage of spherical representations for 3D structures without information loss. 

- This work is provided with a substantial number of definitions and lemmas that help navigate the problem.
summary: This paper tackles the problem of 3D molecular generation from scratch. The authors propose the tokenization method Geo2Seq, that converts 3D molecular graphs into SE(3)-invariant sequences for further uses in LMs. 
It is shown that by using this novel tokenization method, various LMs can generate 3D molecular structures with high validity, outperforming diffusion-based baselines.
weaknesses: I believe this article has everything to be called a good paper without any major concerns.

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: 1.  What do you think the use cases are for your work?
2. I was somewhat confused by line 258 - why would the coordinates being bound in a smaller region be helpful?  A simple clarification would be satisfactory
3. Could you be more specific about, of the proofs in the appendix, which are taken from existing literature, and which are novel to this paper?
4. When it says in line 1735 "π-out-of-phase angles are placed near each other, such as ‘3.14°’, ‘-3.14°’, and ‘0°’" - isn't this a sign that you're not capturing some aspect of invariance?
soundness: 4
strengths: This work enables bridging geometric approaches to molecule representation with sequence-based molecule representation in a highly original and apparently successful manner.  The authors correctly identify the reasons why it can be difficult to translate inherently graphical information into a sequence representation, and then present both rigorous theoretical and extensive experimental evidence to show that they have arrived at a better solution than those existing to this problem.  I particularly appreciate the ablations and would suggest putting them earlier in the paper, as they convincingly show that these differences actually matter in real LLM's (not a trivial claim, since often apparently problematic objects to serialize/tokenize can be tokenized effectively given large enough models, e.g., in computer vision).  

This paper is also particularly compelling as a work which includes extensive mathematical foundations, a very significant use case for the mathematical framework built up, and evidence that the mathematical framework is effective at solving that real-world problem.  I am not fully qualified to comment on the correctness of the proofs; however, to the best of my knowledge they seemed both valid and novel.  

I appreciated the attention brought not just to the number of valid molecules, but also to the number of valid and unique and novel molecules, as arguably the most common "hidden failure mode" of transformers is merely regurgitating their training data.  Overall, the experiment section was clear and convincing.
summary: This paper provides a novel approach to tokenization of molecules for the purposes of conditional and unconditional generation, focusing not on the model, but on ways of representing molecules as symbols for any model requiring a symbolic approach in a manner that is SE-3 invariant and efficient.
weaknesses: I felt the controllable generation experiment was under-explained.  While they do list the six quantum properties in line 491 and discuss per-quantum-property efficacy in line 437, I would have preferred an analysis for each of these cases of instances where it succeeded and other methods failed or where their method failed and other methods succeeded.  More importantly, since the paper itself was on the mathematical foundation of a 1-d tokenization of 3-d molecules, I would have liked some explanation for each property of how the property relates to the 3-d configuration and what information relevant to that configuration is preserved or removed under the new framework, especially for readers with general knowledge of LLM's but without knowledge of chemistry.

Creating novel molecules is an interesting task; however, its usefulness was not made obvious in the paper (when would an applied researcher benefit from being able to generate a random valid molecule, or a molecule with a particular LUMO energy)?  The benefits of models such as AlphaFold were well documented in terms of medicinal chemistry; I was not able to find a similar reference explaining the benefits of random molecule generation, While it may be self-evident to researchers in the field of computational chemistry, to general readers of ICLR it would help to provide such a use case.  More ambitiously, it would be fascinating to see whether this approach could be applied to other geometric tokenization problems, such as generating valid representations of social networks with 2-d or 3-d constraints, or generating valid configurations of cellular automata.

Most LLM papers benefit strongly from looking at performance statistics as a function of model size and other hyperparameters; which I did not see here (there is a section on hyperparameters in the appendix, but not a comprehensive study involving the type of LLM used).

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Other than the points mentioned in the weakness section, I have the following questions:

-- The proposed combination of canonical representation and spherical representation is nothing very specific to 3D molecules but can also be used by any 3D point cloud, did the authors try to see whether their proposed method works for other available 3D point cloud generation tasks (see pointflow or related papers for the tasks)?
soundness: 4
strengths: -- The problem description of converting 3D graphs to 1D tokens preserving SE(3)-invariance is very interesting and relevant to the community. 

-- Moreover their proposed method Geo2Seq, which combines the two ideas from Graph theory (canonicalization and spherical representation) is novel (even though parts of it might be used in previous applications, for example canonicalization is used in SMILES).

-- Experimental studies on random and controlled generations across two dataset showcases the efficacy of the proposed method

-- Detailed Ablation Studies (sec C). One of the immediate questions I had was understanding the importance of using different types of ordering (one of which could be canonical) and different types of representation (including the proposed spherical). These questions have been asked and answered in the ablation (shown in the appendix).
summary: This paper explores language model (LM) based  generation of molecules in 3D space. To that end the paper proposes tokenization of molecular 3D graphs that converts molecular geometries into SE(3)-invariant 1D discrete sequences to be used as input to a LM. The paper proposes to a) first use graph canonicalization to create a 1D sequential ordering of the 3D molecule. b) use spherical representation to represent the 3D coordinate of the atoms. This ensures that any two isomorphic graphs have the same representation and different graphs have different representations. The LM model is next trained using typical next token prediction on the created token.  The experimental results on QM9 and GEOM-DRUGS dataset shows that their proposed Geo2Seq performs better than baselines both for random and controlled generations.
weaknesses: -- As stated in the strength section, even though the combination is novel, the individual components have been used before.

-- One thing that is not clear to me from the very thorough experiment is from the following observation: From table-4 the gap between the top 3-4 methods is not very large. Under that circumstance, should efficiency and memory required to perform the ordering be used to pick a winner? Did the author do such an analysis?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: For this work, the LM is used as a generative model for learning patterns of the proposed representation from data. There can be more case studies and discussion on how additional capabilities (e.g. in-context learning) of LLMs can be leveraged with this representation. Do the authors see any emergent abilities? Are prompt-based approaches possible?
soundness: 3
strengths: - The method and derivations are sound and rigorous
- The ablations are comprehensive and justify the method's design choices
- The random generation results are promising
- The presentation is beautiful
summary: The paper proposes a way to tokenize 3D molecules into sequences for language model generation which preserve structural completeness and SE(3)-invariance. They include detailed derivations and proofs for the claims. The method demonstrates promising results for random generation and controllable generation. They also include comprehensive ablations showing the importance of the canonicalization and 3D representation.
weaknesses: - There should be more LM-based baselines for random generation
- There should be more baselines for controllable generation

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Would using SMILES canonicalization be a better approach? How does it differ from the method you've employed?
2. Since the LLM is already generating iteratively, I believe adopting a local coordinate system could be more effective, as it would help keep distances manageable.
3. I'm curious whether normalizing the xyz coordinates (for example, by subtracting the center of mass coordinates) could achieve a similar effect. According to the authors, normalization could also assist in reducing the range to some extent.
soundness: 2
strengths: 1. This is a very interesting task, and it’s great to see a method for integrating multimodal data, especially 3D molecular structures.
2. The motivation is clear and straightforward.
3. The experiments in this paper are comprehensive, covering various LMs and ablations.
4. The writing is clear and accessible.
summary: The paper proposes a novel tokenization method, Geo2Seq, for generating 3D molecular structures using language models (LMs). Geo2Seq transforms molecular geometries into SE(3)-invariant 1D discrete sequences, making them compatible with LMs while preserving essential structural details. The method uses canonical labeling to establish a unique node order and spherical coordinates for SE(3)-invariance, enabling robust encoding of molecular geometry. This technique allows LMs to handle 3D molecule generation more effectively, surpassing existing models in generating diverse, chemically valid 3D structures.
weaknesses: I believe there is still room for improvement in the paper:

1. The canonical labeling method discussed in Section 3.1 resembles the SMILES canonicalization commonly used in cheminformatics. Intuitively, this functionality could be implemented using SMILES canonicalization in RDKit—why not utilize that?
2. The molecular bonds are generated using RDKit, also there is no strict definition of bonds in quantum chemistry. What is the necessity of introducing graph isomorphism in this paper?
3. The authors suggest that limiting the range could enhance the language model’s performance. However, in spherical coordinates, only the azimuth and polar angles are restricted in range, while the distance remains a continuous variable. This could still limit applications for larger molecules, such as proteins.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Although it may compromise invariance, would training directly on centered Cartesian coordinates improve performance on this task?
2. How does Geo2Seq ensure that generated molecules adhere to chemical and physical constraints (e.g., bond lengths and steric hindrance)? Are there methods to introduce explicit constraints?
3. Would increasing coordinate precision (i.e., more decimal places) impact the quality or accuracy of the model's generated outputs?
4. How does the GPT model perform in controllable generation? The model structure does not appear to be significantly different.
soundness: 3
strengths: The benchmarks show strong performance, especially in controllable generation.
summary: This paper proposes Geo2Seq, which extracts invariant coordinates of molecules and supports language model-based 3D molecular structure modeling through digit-level encoding. The method demonstrates strong performance across datasets.
weaknesses: 1. This is a technically solid paper but with relatively limited contributions.
2. This paper does not require extensive theoretical exposition. Both isomorphism and invariance are well-studied concepts with no new conclusions presented here, and the heavy use of theoretical formulations affects readability.
3. From the existing experimental analysis, it remains unclear whether the model is merely replicating token patterns or genuinely learning underlying molecular principles.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: Please refer to Weaknesses section.
soundness: 3
strengths: 1. Understanding how to use a sequential representation for modeling a 3D molecule is critical to language-based molecule generation.
2. The paper theoretically demonstrates that the application of canonical and spherical representations preserves SE(3)-invariance without any loss of information.
3. The paper is well-written and easy to understand.
summary: This paper introduces Geo2Seq, a language model-based method for generating molecules. The method converts a 3D molecular graph into a 1D sequential representation through canonical ordering. It employs a spherical representation to encapsulate the 3D structural information, preserving SE(3)-invariance while ensuring no information loss. Additionally, the paper presents a specialized 3D tokenization approach tailored for generating 3D molecules.
weaknesses: 1. My primary concern is that since the use of canonical ordering and spherical representation is not novel in the context of 3D molecular tasks, it would be beneficial to thoroughly discuss and compare their applications as documented in existing literature. This analysis should be included in the preliminary and related work section of the paper. This comparison will help in highlighting the unique contributions and potential advancements introduced by the Geo2Seq method.
2. Given that different tokenization approaches may significantly affect the generation quality, it would be beneficial to have additional ablation studies compare the proposed tokenization method with some traditional word-based or subword tokenization methods.

Meta-review: **Summary**
This work proposes a geometry-informed tokenization method Geo2Seq to enable 3D molecule generation with language models (LMs). Specifically, Geo2Seq employs canonical labeling and invariant spherical representations to transform, reversibly, 3D structures into SE(3)-invariant sequences, which can be fed into LMs. Experimental validation is provided on GEOM-Drugs and QM9 datasets. 

**Strengths**
The reviewers appreciated several merits of this work including clarity of presentation, flexibility of the method in terms of being agnostic to the choice of LM, solid theoretical arguments, and the empirical performance.

**Weaknesses**
Some reviewers raised concerns about the novelty of the approach (while disagreeing with others), pointing out that canonical labeling and invariant spherical coordinates have already been extensively studied in the prior work, and the submission spent disproportionate space on related work and theoretical results that they felt were rather straightforward did not provide any new or interesting insights. 

During the author-reviewer discussion period and the follow-up with AC, some of the reviewers also raised concerns about the possibility that the improvement in results might be due to memorization in spherical coordinates and/or overparameterization compared to the much smaller diffusion-based models. They also attributed the performance improvements due to using decimals in tokens, and the apparent arbitrariness in the choice of decimal precision, as factors that might help tailor to the specific datasets, potentially limiting the generalizability of the findings. One of these reviewers mentioned that while their concerns were addressed to some extent, they were not fully resolved;  another was not convinced by the response at all.   

Two of the reviewers specifically expressed dissatisfaction with a perfect 10 score by one of the reviewers, arguing that their review did not provide substantial arguments to support such a high score and overly inflated the perceived impact of this paper.  

**Recommendation**
During the rebuttal period, the authors provided additional experimental evidence and clarifications that were largely appreciated by the reviewers. I commend the authors for investing time and effort in preparing thoughtful responses. 

Noting the discrepancy in reviewers’ perception of this work, the AC decided to carefully review the work.  Despite the merits of this work, I agree with the dissenting reviewers about their concerns regarding the (methodological) novelty of this work, and believe that further empirical investigations are required to substantiate this work. 

Notably, there is a large body of prior work, beginning with [1], on using language models for generating  (macro-)molecules such as proteins given the 3D-structure.  In fact, [1] proposes an efficient Transformer-based approach that uses a quaternion-based representation, guaranteeing SE(3)-invariance and reconstruction of the local neighborhoods (up to isomorphism). See also [2] for a more recent approach that follows [1] to incorporate symmetries.    

A comprehensive empirical comparison with  [1] and [2], adapting them to the current setting (by leveraging their reconstructions), would address the concerns about the fairness of evaluation.  I also recommend that the authors reposition (the novelty of) their contributions accordingly. 

In the current form, however, the work does not meet the bar for acceptance at ICLR. 

[1] Ingraham et al. Generative models for graph-based protein design. NeurIPS 2019.

[2] Dauparas et al. Robust deep learning-based protein sequence design using ProteinMPNN. Science 2022.

Decision: Reject