Accelerating 3D Molecule Generation via Jointly Geometric Optimal Transport

## Abstract

Abstract This paper proposes a new 3D molecule generation framework, called GOAT, for fast and effective 3D molecule generation based on the flow-matching optimal transport objective. Specifically, we formulate a geometric transport formula for measuring the cost of mapping multi-modal features (e.g., continuous atom coordinates and categorical atom types) between a base distribution and a target data distribution. Our formula is solved within a unified, equivalent, and smooth representation space. This is achieved by transforming the multi-modal features into a continuous latent space with equivalent networks. In addition, we find that identifying optimal distributional coupling is necessary for fast and effective transport between any two distributions. We further propose a flow refinement and purification mechanism for optimal coupling identification. By doing so, GOAT can turn arbitrary distribution couplings into new deterministic couplings, leading to a unified optimal transport path for fast 3D molecule generation. The purification filters the subpar molecules to ensure the ultimate generation performance. We theoretically prove the proposed method indeed reduced the transport cost. Finally, extensive experiments show that GOAT enjoys the efficiency of solving geometric optimal transport, leading to a double speedup compared to the sub-optimal method while achieving the best generation quality regarding validity, uniqueness, and novelty.

## 1 Introduction

The problem of 3D molecule generation is essential in various scientific fields, such as material science, biology, and chemistry ; ; . Typically, 3D molecules can be represented as atomic geometric graphs ; ; , where each atom/node is embedded in the Cartesian coordinates and encompasses rich features, such as atom types and charges. There has been fruitful research progress on geometric generative modeling for facilitating the process of 3D molecule generation. Specifically, geometric generative models are proposed to estimate the distribution of complex geometries and are used for generating feature-rich geometries. The success of these generative modeling mainly comes from the advancements in using the notion of probability measurement transport for generating samples. Generative modeling aims to generate samples via mapping a simple prior distribution, e.g., Gaussian, to a desired target probability distribution. This mapping process can be termed as a distribution transport/generative problem .

Recent representative models for sampling 3D molecules in silicon include diffusion-based models ; ; and flow matching-based models ; . Diffusion-based models have shown superior results on molecule generation tasks ; . They simulate a stochastic differential equation (SDE) to transport a base distribution (e.g., Gaussian) to the data distribution. However, a major drawback of diffusion-based models is their slow inference speed with the learned stochastic transport trajectory ; ; ; they typically need approximately $1,000$ sampling steps to produce effective samples regarding molecule validity, uniqueness, and novelty. This could make large-scale inference prohibitively expensive. Accordingly, flow matching — built upon continuous normalizing flows — has emerged as a new paradigm that could potentially provide effective density estimation and fast inference ; ; ; ; ; .

This paper aims to deal with the problem of fast and effective 3D molecule generation based on flow matching optimal transport principle. In particular, our objective is to obtain a distribution transport trajectory with optimal transport cost and generation quality regarding molecule validity, uniqueness, and novelty. A few recent works have been proposed to improve the sampling efficiency of the geometrical domains via flow matching principle.  was proposed for efficient, simulation-free training of equivariant continuous normalizing flows, which can produce samples from the equilibrium Boltzmann distribution of a molecule in Cartesian coordinates. However, it can only work for many-body molecular systems and does not consider atomic features.

Table: Figure 1: Illustration of Probability Paths Learned by Different Molecule Generative Models. 1. The diffusion path ; , which typically deviates from a straight line map, necessitates a large number of sampling steps. 2. The hybrid transport ensures straight transport for atomic coordinates, but it does not guarantee the same for atom features. Furthermore, this cost does not consider the optimal distribution couplings, leading to suboptimal transport between distributions. 3. GOAT simultaneously considers the optimal transport for atom coordinates and features, providing a unified and straight path for fast sampling.

In the context of molecule generation, properly characterizing the transport cost to optimize over is indispensable and challenging. There are two main challenges. Firstly, the multi-modal property of atomic feature space, typically consisting of continuous atom coordinates and categorical atom types, makes the transport cost measurement hard to optimize. Secondly, the optimal transport problem essentially is to search optimal distribution couplings for mapping. This process typically requires similarity computation of the two distributions. However, the various sizes of geometrical graphs to transport introduce difficulties in evaluating the distribution similarity. The closest to ours is EquiFM , which attempts to address the multi-modality issue by using different probability paths to transport each modality separately. The proposed equivariant optimal transport (OT) for transporting atom coordinates indeed forms a straight-line trajectory for training, while the variance-preserving principle could not ensure a straight-line trajectory for atom features. Therefore, the fused flow paths might deviate strongly from the OT paths and could not ensure optimal coupling between two probability measurements, leading to large computational costs and numerical errors.

In this work, we propose a new framework for fast 3D molecule generation based on a novel and principled optimal transport flow-matching objective, dubbed as unified Geometric OptimAl Transport (GOAT). In particular, we formulate a geometric transport cost measurement for optimally transporting continuous atom coordinates and categorical features, which is inherently a Bilevel optimization problem. To deal with the first challenge induced by transporting multiple modalities, GOAT leverages a latent variable model equipped with equivariant networks to map the multi-modal features into a unified, equivalent, and smooth representation space. This equivariant latent variable model has been proven to be flexible and expressive for modeling complex 3D molecules ; . A latent flow matching then operates over the latent space, which can provide distributional coupling estimation.

To tackle the second challenge — obtaining the optimal distribution couplings, we propose to refrain from directly working with distribution similarity computation. Instead, we propose to rectify the flow with the estimated ones with the latent flow matching. Because the estimated distributional couplings are identified based on the synthesized samples, they might deviate from the real-world samples in terms of quality. We provide a purification process for high-quality samples regarding validity, uniqueness, and novelty. With this process, we can turn arbitrary couplings into deterministic and causal ones, which can be used to rectify the flow, leading to the optimal transport path for fast and effective generation.

We theoretically prove the decrease in geometric transport cost by the proposed framework. Moreover, we empirically highlight the superiority of GOTA by conducting experiments on widely used benchmarks. The proposed method reduced the transport cost by nearly 89.65%, halving the sampling times compared to EquiFM. Regarding the generation quality, the proposed method obtained an improvement in the ultimate significance, which measures the proportion of valid, unique, and novel molecules, up to 17.1% compared with existing algorithms.

## 2 Problem Setup

Notations. A three-dimensional (3D) molecule with $N$ atoms can be represented as a geometric graph denoted as $\mathbf{g}=\langle\mathbf{x},\mathbf{h}\rangle$, where $\mathbf{x}=(\mathbf{x}_{1},\dots,\mathbf{x}_{N})\in\mathbb{R}^{N\times 3}$ represents the atom coordinates and $\mathbf{h}=(\mathbf{h}_{1},\dots,\mathbf{h}_{N})\in\mathbb{R}^{N\times d}$ is the atom features containing atomic types, charges, etc. $d$ is the dimensionality of the atom features. A zero center-of-mass (Zero CoM) space is defined as $\mathbb{X}=\{\mathbf{x}\in\mathbb{R}^{N\times 3}:\frac{1}{N}\sum_{i=1}^{N}
\mathbf{x}^{i}=\mathbf{0}\}$, indicating that the mean of the $N$ atoms’ coordinates should be 0. In what follows, we will introduce some necessary concepts, including flow matching and optimal transport to facilitate the definition of our problem.

General Flow Matching (^1^11We are aware of the drawbacks of reusing the notation $\mathbf{x}$, which represents a general data point here.). Given noise $\mathbf{x}_{0}\in R^{N}\sim p_{0}$ and data $\mathbf{x}_{1}\in R^{N}\sim p_{1}$, the general flow-based model considers a mapping $f:\mathbb{R}^{N}\to\mathbb{R}^{N}$ as a smooth time-varying vector field $u:[0,1]\times\mathbb{R}^{N}\to\mathbb{R}^{N}$, which defines an ordinary differential equation (ODE):

$$ $d\mathbf{x}=u_{t}(\mathbf{x})dt.$ (1) $$

Continuous normalizing flows (CNFs) were introduced with black-box ODE solvers to train approximate $u_{t}$. However, CNFs are hard to train as they need numerous evaluations of the vector field. Flow matching , a simulation-free approach for training CNFs, is proposed to regress the neural network $v_{\theta}(\mathbf{x},t)$ to some target vector field $u_{t}(\mathbf{x})$:

$$ $\mathcal{L}_{\textrm{FM}}(\theta):=\mathbb{E}_{t\sim\mathcal{U}(0,1),\mathbf{x }\sim p_{t}(\mathbf{x})}\|v_{\theta}(\mathbf{x},t)-u_{t}(\mathbf{x})\|^{2},$ (2) $$

where $p_{t}(\mathbf{x})$ is the corresponding probability path which conditioned on $\mathbf{x}_{1}\sim p_{1}$ and then defined as $p_{t}(\mathbf{x}|\mathbf{x}_{1})=\int p_{t}(\mathbf{x}|\mathbf{x}_{1})p_{1}(
\mathbf{x}_{1})d\mathbf{x}_{1}$. In implementation, common probability paths include variance exploding (VE) diffusion path , variance preserving (VP) diffusion path , and straight transport path ; .

Optimal Transport (OT). Transport plan between $p_{0}$ and $p_{1}$ is also called coupling and we denoted it as $\Gamma(p_{0},p_{1})$ . OT addresses the problem of finding the optimal coupling that minimizes the transport cost, typically in the form of $\mathbb{E}[c(\mathbf{x}_{1}-\mathbf{x}_{0})]$, where $c:\mathbb{R}^{N}\to\mathbb{R}$ is a cost function, such as $c(\cdot)=\|\cdot\|^{2}$. Formally, a coupling $\Gamma(p_{0},p_{1})$ is called optimal only if it achieves the minimum value of $\mathbb{E}[c(\mathbf{x}_{1}-\mathbf{x}_{0})|(\mathbf{x}_{1},\mathbf{x}_{0})\in
\Gamma(p_{0},p_{1})]$ among all couplings that share the same marginals. An ideal optimal transport trajectory is a map of optimal couplings.

Geometric Optimal Transport. In our task, we consider a pair of geometric probability distributions over $\mathbb{R}^{N\times(3+d)}$ with densities $p(\mathbf{g}_{0})$ and $p(\mathbf{g}_{1})$ (or denoted as $p_{0}$ and $p_{1}$).
Geometric generative modeling considers the task of fitting a mapping $f$ from $\mathbb{R}^{N\times(3+d)}$ to $\mathbb{R}^{N\times(3+d)}$ that transforms $\mathbf{g}_{0}$ to $\mathbf{g}_{1}$. More specifically, if $\mathbf{g}_{0}$ is distributed with density $p_{0}$ then $f(\mathbf{g}_{0})$ is distributed with density $p_{1}$. Typically, $p_{0}$ is an easily sampled density, such as a Gaussian.

In our specific problem, beyond geometric distribution transport, we concentrate on fast 3D molecule generation based on the flow model with optimal transport (OT), which has been proven effective in accelerating non-geometric flow models ; . Therefore, our problem is defined as geometric optimal transport flow matching.

## 3 Our Method: GOAT

### 3.1 Formulating Geometric Optimal Transport Problem

Our objective is to obtain an optimal transport trajectory for fast 3D molecule geometry generation based on optimal transport flow models. In this regard, we can reformulate our problem into searching for optimal coupling for geometric optimal transport. Specifically, it involves the transportation of molecules via optimal coupling, where each atom follows a straight and shortest path. In other words, each molecule is coupled with a noisy sample that incurs the minimum cost, and each atom within the target molecule is paired with its counterpart in the noise, leading to the minimum cost. To encapsulate the optimal scenario, we consider the problem of geometric optimal transport with two components: 1) optimal molecule transport (OMT) with equivariant OT for atom coordinates and invariant OT for atom features; 2) optimal distribution transport (ODT) with optimal molecule coupling.

Geometric Transport Cost.
Transporting a molecule includes transforming atom coordinates and features. We can depict the molecule transport cost as below:

$$ $\displaystyle c_{g}(\mathbf{g}_{0},\mathbf{g}_{1})=\|\mathbf{x}_{1}-\mathbf{x} _{0}\|_{2}+\|\mathbf{h}_{1}-\mathbf{h}_{0}\|_{2},$ (3) $$

where $\mathbf{g}_{0}\sim p_{0}$ and $\mathbf{g}_{1}\sim p_{1}$. In addition, given coupling $\Gamma(p_{0},p_{1})$, we measure the distribution similarity between two distributions denoted as $p_{0}$ and $p_{1}$ based on the probability transport cost as follows:

$$ $\displaystyle C_{g}=$ $\displaystyle\mathbb{E}[c_{g}(\mathbf{g}_{0},\mathbf{g}_{1})],(\mathbf{g}_{0}, \mathbf{g}_{1})\in\Gamma(p_{0},p_{1}).$ (4) $$

Geometric Optimal Transport Problem. Building upon the established transport cost, we can formulate the geometric optimal transport problem for fast and effective 3D molecule generation. In particular, a molecule remains invariant for any rotation, translation, and permutation, while the transport cost is not invariant or equivariant to the above operations. Therefore, there exists a minimum molecule transport cost
with 1) optimal rotation and translation transformations such that the molecules from the data and noise are nearest to each other at the atomic coordinate level; and 2) optimal permutation transformation such that the atomic features of the two are also nearest.We supplement the detailed analysis of equivariance and invariance in Appendix A and present geometric optimal transport as follows:

$$ $\displaystyle\min_{\Gamma}\mathbb{E}[\hat{c}_{g}(\mathbf{g}_{0},\mathbf{g}_{1}$ $\displaystyle)],$ (5) $\displaystyle\textbf{s. t. }~{}~{}~{}(\mathbf{g}_{0},\mathbf{g}_{1})$ $\displaystyle\in\Gamma(p_{0},p_{1}),$ $\displaystyle\hat{c}_{g}(\mathbf{g}_{0},\mathbf{g}_{1})$ $\displaystyle=\min_{\mathbf{R},\mathbf{t},\pi}\|\pi(\mathbf{R}\mathbf{x}^{1}_{ 1}+\mathbf{t},\mathbf{R}\mathbf{x}^{2}_{1}+\mathbf{t},\dots,\mathbf{R}\mathbf{ x}^{N}_{1}+\mathbf{t})-(\mathbf{x}^{1}_{0},\mathbf{x}^{2}_{0},\dots,\mathbf{x} ^{N}_{0})\|_{2}$ $\displaystyle+\min_{\pi}\|\pi(\mathbf{h}^{1}_{1},\mathbf{h}^{2}_{1},\dots, \mathbf{h}^{N}_{1})-(\mathbf{h}^{1}_{0},\mathbf{h}^{2}_{0},\dots,\mathbf{h}^{N }_{0})\|_{2},\forall\pi,\mathbf{R},~{}\textrm{and}~{}\mathbf{t},$ $$

where $\hat{c}_{g}$ denotes optimal molecule transport cost, $\pi$ represents a permutation of $N$ elements, and $\mathbf{R}$ and $\mathbf{t}$ denote a rotation matrix and a translation, respectively. The defined geometric optimal transport problem forms a bi-level optimization problem that involves two levels of optimization tasks. Specifically, minimizing molecule transport cost is nested inside the optimizing distribution transport cost.

The Challenges of Solving the Geometric Optimal Transport Problem. First, optimal molecule transport involves searching for a unified optimal permutation for atom coordinates and features with minimum transport cost. The paths for transporting continuous coordinates and categorical features are incompatible and require sophisticated, hybrid modeling of multi-modal variables  , leading to a sub-optimal solution.
Second, a molecular distribution comprises molecules with diverse numbers of atoms, introducing difficulties in quantifying the transport cost for searching optimal coupling.
As a result, the minimization of geometric transport $C_{g}$ within molecular distributions poses a more significant challenge compared to other domains such as computer vision  or many-body systems .
Moreover, the proposed geometric optimal transport problem, which involves a nested optimization structure, presents a significant computational challenge for optimization.

### 3.2 Solving Geometric Optimal Transport Problem

In this section, we will address the above challenges under the depicted problem for fast and effective 3D molecule generation, from two aspects, including optimal molecule transport and optimal distribution transport.

#### 3.2.1 Solving Optimal Molecule Transport

As illustrated in Eq. (5), our objective for optimal molecule transport is to find $\hat{\mathbf{R}}$, $\hat{\mathbf{t}}$, and $\hat{\pi}$ that minimize the transport cost denoted as $c_{g}$ for two given molecule geometries represented as $\mathbf{g}_{0}$ and $\mathbf{g}_{1}$:

$$ $\hat{\mathbf{R}},\hat{\mathbf{t}},\hat{\pi}=\arg\min_{\mathbf{R},\mathbf{t}, \pi}c_{g}(\mathbf{g}_{0},\mathbf{g}_{1}),\forall\pi,\mathbf{R},~{}\textrm{and} ~{}\mathbf{t}.$ (6) $$

As per Eq. (5), the optimal molecule transport problem entails the consideration of both atom coordinates and features for comprehensive representations of 3D molecules.
Though coordinates and features represent different modalities, they need to be considered in tandem. Previous research ; either solely focused on equivariant optimal transport for coordinates or transported atom features via a distinct yet non-optimal path. In contrast, we propose to unify the transport of atom coordinates and features. If we can unify different modalities within an equivariant and smooth representation space, optimally transport from a base distribution to the data distribution leading to fast and effective molecule generation is possible.

Specifically, we map the atom coordinates and features from the data space into a latent space with an equivariant autoencoder , which enables us to compute a unified optimal permutation ($\hat{\pi}$).
After the mapping, we ascertain the optimal rotation ($\hat{\mathbf{R}}$) for two atomic coordinate sets for transportation, utilizing the Kabsch algorithm  as did in ; . The computed rotation matrix ensures that the coordinates of the target molecules in the latent space are in closest proximity to the noise molecules. In addition, to achieve optimal translation ($\hat{\mathbf{t}}$), we establish the data distribution $p_{1}$ and base distribution $p_{0}$ in the zero CoM space ; . Equipped with the equivariant autoencoder, the latent representation also resides in the zero CoM space, thus ensuring optimal translation in the latent space.

Implementation. Initially, the distributions $p_{0}$ and $p_{1}$ are aligned to the Zero CoM by subtracting the center of gravity using $\hat{\mathbf{t}}$.
Subsequently, an equivariant autoencoder is designed to project $\mathbf{g}_{1}\sim p_{1}$ into the latent space. Here, the encoder $\mathcal{E}_{\phi}$ transforms $\mathbf{g}_{1}$ into latent domain $\mathbf{z}_{1}=\mathcal{E}_{\phi}(\mathbf{g}_{1})$, where $\mathbf{z}_{1}=\langle\mathbf{z}_{\mathbf{x},1}\in\mathbb{R}^{N\times 3},
\mathbf{z}_{\mathbf{h},1}\in\mathbb{R}^{N\times k}\rangle$ and $k$ represents the latent dimensionality for the atomic features. The decoder $\mathcal{D}_{\epsilon}$ then learns to decode $\mathbf{z}_{1}$ back to molecular domain formulated as $\hat{\mathbf{g}_{1}}=\mathcal{D}_{\epsilon}(\mathbf{z}_{1})$. The EAE can be trained by minimizing the reconstruction objective which is $d(\mathcal{D}(\mathcal{E}(\mathbf{g}_{1})),\mathbf{g}_{1})$.
With the encoded $\mathbf{z}_{1}$ and the sampled noise $\mathbf{z}_{0}=\langle\mathbf{z}_{\mathbf{x},0}\in\mathbb{R}^{N\times 3},
\mathbf{z}_{\mathbf{h},0}\in\mathbb{R}^{N\times k}\rangle$ from $p_{0}$, we then formulate the atom-level cost matrix as $M_{c_{g}}[i,j]=\|\mathbf{z}_{1}^{i}-\mathbf{z}_{0}^{j}\|^{2}=\|\mathbf{z}_{
\mathbf{x},1}^{i}-\mathbf{z}_{\mathbf{x},0}^{j}\|^{2}+\|\mathbf{z}_{\mathbf{h}
,1}^{i}-\mathbf{z}_{\mathbf{h},0}^{j}\|^{2}$, which is the 2-norm distance between $i$-th atom of $\mathbf{z}_{1}$ and $j$-th atom of $\mathbf{z}_{0}$ including the latent coordinates $\mathbf{z}_{\mathbf{x}}$ and the latent features $\mathbf{z}_{\mathbf{h}}$. With $M_{c_{g}}$, the optimal permutation $\hat{\pi}$ is induced with the Hungarian algorithm . The coordinates of the noise molecule $\mathbf{z}_{\mathbf{x},0}$ and the latent coordinates of the target molecule $\mathbf{z}_{\mathbf{x},1}$ are then aligned through rotation $\hat{\mathbf{R}}$ solved by the Kabsch algorithm . In summary, we perform the above-calculated translation, encoding, rotation, and permutation on the target molecule $\mathbf{g}_{1}$ to obtain $\hat{\mathbf{z}}_{1}$, which forms the optimal molecule transport cost with the sampled noise $\mathbf{z}_{0}$. The complete process is denoted as $\hat{\mathbf{z}}_{1}=\pi(\hat{\mathbf{R}}\mathcal{E}_{\phi}(\mathbf{g}_{1}+
\hat{\mathbf{t}}))$.

#### 3.2.2 Searching Optimal Coupling for Optimal Distribution Transport

By solving Eq. (6), we can obtain $\hat{\mathbf{R}}$, $\hat{\mathbf{t}}$, and $\hat{\pi}$ yielding an optimal molecule transport trajectory — a straight one — given two data points from the base distribution and target distribution, respectively (see the gray trajectory in Figure 2). Nevertheless, ensuring a straight trajectory does not necessarily yield optimal transport for generative modeling because a straight map for two data points does not indicate a straight map between two distributions. Figure 2 depicts two possible trajectories for generative modeling. The gray one shows a straight but not the shortest map, while the red one represents the shortest map indicating an optimal trajectory. As discussed in Sec. 3.1, an optimal trajectory can only be achieved with optimal coupling, leading to the shortest path for mapping the base distribution to the target distribution. To approximate optimal coupling and further boost the sampling speed, we introduce the second part of our framework dedicated to solving optimal distribution transport as follows.

Figure: Figure 2: An Illustration of the Difference Between Straight Coupling and Optimal Coupling. GOAT approximates optimal coupling for a fast generation.
Refer to caption: x4.png

The pathway to optimal distribution transport is to identify the optimal coupling $\hat{\Gamma}(p_{0},p_{1})$ that satisfies the condition formulated as:

$$ $\displaystyle\hat{\Gamma}=\arg\min_{\Gamma}\mathbb{E}[\hat{c}_{g}(\mathbf{z}_{ 0},\mathbf{z}_{1})]$ (7) $\displaystyle\textbf{s. t. }(\mathbf{z}_{0},\mathbf{z}_{1})\in\Gamma(p_{0},p_{ 1}),$ $$

where $\Gamma$ is an arbitrary coupling plan between $p_{0}$ and $p_{1}$ and $\hat{c}_{g}$ is optimal molecule transport cost defined in Eq. (5).

However, measuring the distribution transport cost for searching for optimal coupling is challenging due to the various sizes of molecules.
Inspired by , we bypass the process of quantifying the transport cost and estimate optimal coupling $\hat{\Gamma}(p_{0},p^{\prime}_{1})$ based on a trained flow model with the initial coupling denoted as $\Gamma(p_{0},p_{1})$.

The estimated optimal coupling can minimize the transport cost but may introduce generation error for the following reasons. The first type of error arises from estimating the flow path between $p_{0}$ to $p_{1}$ via a neural network $v_{\theta}$, implying that $p^{\prime}_{1}$, characterized by $v_{\theta}$, does not perfectly match the data distribution $p_{1}$. The second type of error stems from the discreteness of molecular data and the continuity of the distribution. In essence, an intermediate value between two similar and valid molecules, which are closely distributed, may not be biochemically valid. To compensate for such discrepancies, we implement a purification process on the generated coupling to ensure effective generation. We present a detailed implementation below.

Implementation.
First, based on Sec. 3.2.1, we can obtain a set of noise and target molecule pairs with optimal molecule transport cost. We leverage the corresponding transport path as the conditional probability path for training the flow with the loss formulated as:

$$ $\mathcal{L}_{F1}(\theta)=\mathbb{E}_{t,p_{0},p_{1}}\|v_{\theta}(\hat{\mathbf{z }}_{t},t)-(\hat{\mathbf{z}}_{1}-\mathbf{z}_{0})\|^{2},$ (8) $$

where $\hat{\mathbf{z}}_{t}=t\hat{\mathbf{z}}_{1}+(1-t)\mathbf{z}_{0},t\in[0,1]$. Compared with using conditional optimal transport path and variance-preserving path in a hybrid fashion , our method employs a unified linear interpolation of $\hat{\mathbf{z}}_{1}-\mathbf{z}_{0}$ as the flow probability path. Such straight trajectory adheres to the naive ODE formula denoted as $d\mathbf{z}_{t}=(\hat{\mathbf{z}}_{1}-\mathbf{z}_{0})dt$, thereby providing a more straight flow path for fast sampling. The optimum of $\mathcal{L}_{F1}$ is achieved when $v_{\hat{\theta}}(\mathbf{z}_{t},t)=\mathbb{E}[\mathbf{z}_{1}-\mathbf{z}_{0}|
\mathbf{z}_{t}]$.

The proposed framework then samples data pairs ($\mathbf{z}_{0},\mathbf{z}^{\prime}_{1}$) via trained flow model $\hat{\theta}_{1}$ as the estimated optimal coupling. Specifically, $\mathbf{z}^{\prime}_{1}$ can be sampled following $d\mathbf{z}_{t}=v_{\hat{\theta}_{1}}(\mathbf{z}_{t},t)dt$ starting from $\mathbf{z}_{0}\sim p_{0}$ and the process of sampling is denoted as $\textrm{ODE}_{v_{\hat{\theta}}}$. Pair of $\mathbf{z}_{0}$ and $\mathbf{z}^{\prime}_{1}$ is set as fixed and a batch of pairs will be generated as the estimated optimal coupling represented as $\hat{\Gamma}(p_{0},p^{\prime}_{1})=\{(\mathbf{z}_{0},\mathbf{z}^{\prime}_{1})\}$, where $\mathbf{z}^{\prime}_{1}=\textrm{ODE}_{v_{\hat{\theta}}}(\mathbf{z}_{0})$.

Finally, we purify and refine the flow. Specifically, $\mathbf{z}^{\prime}_{1}$ is decoded by $\mathcal{D}_{\epsilon}$ for $\mathbf{g}^{\prime}_{1}$ and evaluated in terms of stability and validity by RdKit . This provides a criterion for filtering out invalid molecules to purify the coupling. Subsequently, the flow is refined using the loss in Eq. (8) with purified coupling.

Provably Reduced Geometric Transport Cost. The flow refined by the estimated optimal coupling $\hat{\Gamma}$ can boost generation only when geometric transport cost is reduced. We theoretically show our approach indeed can reduce geometric transport costs as follows.

###### Theorem 3.1 .

The coupling $\hat{\Gamma}$ incurs no larger geometric transport cost than the random coupling $\Gamma(p_{0},p_{1})$ in that $\mathbf{E}[\hat{c}_{g}(\mathbf{z}_{0},\mathbf{z}^{\prime}_{1})]\leq\mathbf{E}[
\hat{c}_{g}(\mathbf{z}_{0},\mathbf{z}_{1})]$, where ($\mathbf{z}_{0},\mathbf{z}^{\prime}_{1})=\hat{\Gamma}(p_{0},p^{\prime}_{1})$, ($\mathbf{z}_{0},\mathbf{z}_{1})=\Gamma(p_{0},p_{1})$, and $\hat{c}_{g}$ is optimal molecule transport cost.

With this theorem, the proposed GOAT is guaranteed a Pareto descent on the geometric transport cost leading to faster generation. A comprehensive proof is given in Appendix B and the pseudocode for training and sampling is presented in Appendix C.

###### Theorem 3.1 .

## 4 Experimental Studies

Datasets. We evaluate over benchmark datasets for 3D molecule generation, including QM9  and the GEOM-DRUG . QM9 is a standard dataset that contains 130k 3D molecules with up to 29 atoms. GEOM-DRUG encompasses around 450K molecules, each with an average of 44 atoms and a maximum of 181 atoms. More dataset details are presented in Appendix E.

Baselines. We compare GOAT with several competitive baseline models. G-Schnet  and Equivariant Normalizing Flows (ENF)  are equivariant generative models utilizing the autoregressive models and continuous normalizing flow, respectively. Equivariant Graph Diffusion Model (EDM) and its variant GDM-Aug , EDM-Bridge , GeoLDM  are diffusion-based approaches. GeoBFN  leverages Bayesian flow networks for distributional parameter approximation. EquiFM  is the first flow-matching method for 3D molecule generation.

### 4.1 Evaluation Metrics

Evaluating Generation Quality. Without loss of generality, we use validity, uniqueness, and novelty to evaluate the quality of the generated molecules . Existing calculates validity, uniqueness, and novelty, which are nested; novelty measures novel molecules among unique and valid molecules. However, such a calculation cannot reflect the ultimate quality among all samples. We further propose a new metric toward the ultimate significance of the generative models .

Below, we provide the detailed definitions of these metrics. 1) Validity. An essential criterion for molecule generation is that the generated molecules must be chemically valid, which implies that the molecules should obey chemical bonds and valency constraints. We use RdKit  to check if a molecule obeys the chemical valency rules. Validity calculates the percentage of valid molecules among all the generated molecules; 2) Uniqueness. An important indicator of a molecule generative model is whether it can continuously generate different samples, which is quantified by the uniqueness. We evaluate uniqueness by measuring the fraction of unique molecules among all the generated valid ones; 3) Novelty. An ideal generative model for de novo molecule design should be able to generate novel molecular samples that do not exist in the training set. Therefore, we report novelty that quantifies the percentage of novel samples among all the valid and unique molecules; 4) Significance. To comprehensively evaluate the molecule generative models, we represent a new metric, significance, to quantify the percentage of valid, unique, and novel molecules among the generated samples.

Evaluating Generation Efficiency. In addition to the above quantity metrics, we report sampling steps to measure the generation speed. The time cost of each sampling step in most baselines, including EDM, EDM-Bridge, GeoBFN, GeoLFM, and EquiFM, is identical because they all applied EGNN  with the same layers and parameters. Fewer steps indicate higher generation efficiency. For EquiFM and the proposed GOAT, we applied the same adaptive stepsize on ODE solver Dopri5  for a fair comparison. Furthermore, we measure the generation efficiency by comparing geometric transport cost, which is calculated by Eq. (4).

### 4.2 Results and Analysis

Table: Table 1: Comparisons of generation quality (larger is better) regarding Atom Stability, Validity, Uniqueness, Novelty, and Significance. And comparisons of generation efficiency regarding sampling steps (less is better). The best results are highlighted in bold.

In this study, we generate 10K molecular samples for each method and compute the aforementioned metrics for comparisons. The evaluation results are presented in Table 1 and Figure 3.

Performance Comparisons with Diffusion-Based Methods. We observe that all diffusion-based generation methods indeed need $1000$ sampling steps to achieve comparable generation quality. Surprisingly, with the least sampling steps, GOAT achieves the best uniqueness, novelty, and significance over QM9. Specifically, it improves novelty by up to 35.4% and significance by up to 31.4%, respectively. Among these diffusion models, GeoLDM achieves the best atom stability and validity performance. However, it owns the worst novelty and significance, 58.1% and 53.9% on QM9, respectively. These results indicate that the latent diffusion models can model the complex geometric 3D molecules well but introduce a serious overfitting problem — generating more molecules that are the same as the training samples. Though GDM-Aug can achieve the second-best novelty among all methods, it needs $1000$ sampling steps for 3D molecule generation. As for GEOM-DRUG, we directly compare the validity as ultimate significance since all compared methods achieved almost 100% uniqueness . Table 1 shows that the proposed algorithm also achieves competitive performance while maintaining a leading edge in generation speed on such a large-scale dataset. Specifically, GOAT only spends 0.90 seconds for each molecule on average and reaches 96% validity, while GeoLDM takes 11.84 seconds to reach 99.3%. We believe this performance is acceptable and competitive.

Performance Comparisons with Flow-Matching-Based Methods. EquiFM and GOAT are all based on flow matching, using an ODE solver for generation. We can see that flow-matching-based methods can obtain faster generation speeds than diffusion models. In particular, GOAT only needs $90$ steps, while EquiFM requires $200$ steps for sampling. EquiFM solely considers optimal transport for atom coordinates. Therefore, the generation speed is still inferior to ours. Because the proposed GOAT solves optimal molecule transport and optimal distribution transport together, the number of sampling steps is further reduced by 2X compared to EquiFM with the same ODE solver. This verifies our hypothesis that a unified optimal transport path can further boost the generation efficiency.

Though EquiFM can perform well in terms of atom stability and molecule validity, it achieves the worst performance in novelty and significance on QM9 among all methods. More specifically, nearly half of the generated samples are the same as the training samples, which is unacceptable in the context of de novo molecule design. In contrast, GOAT can obtain 78.7% novelty with 37% improvement and 70.8% significance with 31.8% improvement compared to EquiFM. On GEOM-DRUG, the proposed method achieves approximate performance compared to EquiFM while taking only half the sampling steps. GeoBFN  can have comparable sampling efficiency to ours, which is neither diffusion-based nor flow-matching based methods. We find that its generation quality over GEOM-DRUG is around 3% lower than GOAT regarding the validity, and it owns around 8% decrease in novelty with a similar sampling speed.

Geometric Transport Cost Comparisons. As EquiFM and GOAT are both flow-matching-based transport methods, we compare their transport costs and present the visualized results in Figure 3. We present distribution transport cost ($p_{0}\to p_{1}$) in blue bars and molecule transport cost averaged over the number of atoms ($\mathbf{g}_{0}\to\mathbf{g}_{1}$) in red lines. Compared to EquiFM transports with a hybrid method, the proposed method reduced the geometric transport cost with 1) unified transport (Unified), 2) optimal molecule transport (1-OMT), and 3) optimal distribution transport cost (1-ODT), thereby achieving a significant reduction in geometric transport cost by nearly 89.65%, leading to faster generation. We further minimize molecule and distribution transport costs (2-OMT and 2-ODT) and observe that the transport cost is reduced marginally, indicating a nearly optimum of the proposed method. The above analysis reveals that the proposed method indeed reduced the geometric transport cost by unifying transport, minimizing molecule transport cost, and estimating optimal couplings. The most intuitive manifestation of the reduction in transport cost is the boost in generation speed, which has been demonstrated in the previous section.

Figure: Figure 3: The blue histogram plots the comparisons of distribution transport cost. The red line chart depicts the average transport cost per atom (best view in color).
Refer to caption: x5.png

Controllable Molecule Generation.
Without loss of generality, GOAT can be readily adapted to perform controllable molecule generation with a desired property $s$ by modeling the neural network as $v_{\theta}(\mathbf{z},t|s)$. We evaluated the performance of GOAT on generating molecules with properties including $\alpha$, $\Delta\varepsilon$, $\varepsilon_{\textrm{HOMO}}$, $\varepsilon_{\textrm{LUMO}}$, $\mu$, and $C_{v}$. The quality of the generated molecules concerning their desired property was assessed using the Mean Absolute Error (MAE) between the conditioned property and the predicted property. This measure helps to determine how closely the generated molecules align with the desired property.

Compared to existing methods, our proposed method demonstrates competitive performance in controllable generation tasks. Notably, it achieves the fastest generation for all properties and attains state-of-the-art performance in generating molecules with desired $C_{v}$, with 6.9% improvement. We provide detailed descriptions of these properties and present the complete experimental results in Appendix H.

Figure: Table 2: Ablation Studies. OMT represents optimal molecule transport and ODT stands for optimal distribution transport.

Ablation Studies. Initially, without any consideration of optimal transport, the model trained solely with flow matching in the latent space (w/o OMT) exhibits a significant boost in training speed. This can be attributed to the diminished transport cost resulting from the unified space, although the performance remains suboptimal. Subsequently, when solving OMT without ODT (w/o ODT), both performance and speed see improvements, but they still fall short of the final result, which takes into account both molecule and distribution in geometric optimal transport.

Limitations. Addressing the optimal transport costs, particularly those involving rotation and permutation aspects, can be computationally intensive ; . However, these operations can be efficiently parallelized on CPUs to enhance the training speed. Besides, refining the flow may require additional time-consuming training, but such an operation boosts the generation speed and improves the novelty without compromising the quality. In summary, the above-mentioned operations will accelerate the generation of molecules once and for all after the training, which is prioritized in this research. We leave improvements concerning training efficiency and other methods for boosting the generation speed, such as distillation , for our future work.

## 5 Conclusion

This paper introduces GOAT, a 3D molecular generation framework that tackles optimal transport for enhanced speed and efficiency in molecule design. Recognizing that in silico molecule generation is a problem of probability distribution transport and the key to accelerating this lies in minimizing transport cost. To this end, we formulated the geometric optimal transport problem tailored for molecular distribution. This proposed problem led us to consider the transport cost of molecular coordinates, features, and distribution in unison. This consideration has further inspired the design of unified transport, the algorithm for solving optimal molecule transport, and the framework for minimizing the distribution transport cost. Both theoretical and empirical validations confirm that GOAT reduces the geometric transport cost, resulting in faster and more efficient molecule generation. Our method achieves state-of-the-art speed and performance in generating valid, unique, and novel molecules, thereby enhancing the ultimate significance of in silico molecule generation.

## Appendix

## Appendix A Equivariance and Invariance in Geometric Optimal Transport

Equivariance. Molecules, typically existing within a three-dimensional physical space, are subject to geometric symmetries, including translations, rotations, and potential reflections. These are collectively referred to as the Euclidean group in 3 dimensions, denoted as $\mathrm{E(3)}$ [].

A function $F$ is said to be equivariant to the action of a group $G$ if $T_{g}\circ F(\mathbf{x})=F\circ S_{g}(\mathbf{x})$ for all $g\in G$, where $S_{g}$, $T_{g}$ are linear representations related to the group element $g$ [].
Invariance. A function $F$ is said to be invariant to the action of a group $G$ if $F\circ\pi(\mathbf{x})=F(\mathbf{x})$ for all $g\in G$ and every permutation $\pi\in S_{n}$.

Equivariance and Invariance in Molecules. For geometric graph generation, we consider the special Euclidean group $\mathrm{SE(3)}$, involving translations and rotations. Moreover, the transformations $S_{g}$ or $T_{g}$ can be represented by a translation $\mathbf{t}$ and an orthogonal matrix rotation $\mathbf{R}$.

For a molecule $\mathbf{g}=\langle\mathbf{x},\mathbf{h}\rangle$, the node features $\mathbf{h}$ are $\mathrm{SE(3)}$-invariant while the coordinates $\mathbf{x}$ are $\mathrm{SE(3)}$-equivariant, which can be expressed as $\mathbf{R}\mathbf{x}+\mathbf{t}=(\mathbf{R}\mathbf{x}_{1}+\mathbf{t},\dots,
\mathbf{R}\mathbf{x}_{N}+\mathbf{t})$.

Equivariance and Invariance in Geometric Optimal Transport.
For non-topological data, such as images, the transport cost between two given data points is fixed. However, this does not apply to topological graphs. For instance, when a topological graph (molecule) undergoes rotation or translation, the inherent properties of the molecule remain unchanged, but the cost of transporting coordinates may vary. Similarly, if the atom order in one of the molecules changes in silico, the molecule remains constant, but the transport cost of coordinates and features may alter. Therefore, the proposed optimal molecule transport problem aims to find an optimal rotation, translation, and permutation transformation for one molecule to minimize the distance, considering both coordinates and features, from another molecule.

## Appendix B Proof for Theorem 3.1

The theorem 3.1 is reproduced here for convenience:

Theorem 3.1
The coupling $\hat{\Gamma}$ incurs no larger geometric transport costs than the arbitrary coupling $\Gamma(p_{0},p_{1})$ in that $\mathbf{E}[\hat{c}_{g}(\mathbf{z}_{0},\mathbf{z}^{\prime}_{1})]\leq\mathbf{E}[
\hat{c}_{g}(\mathbf{z}_{0},\mathbf{z}_{1})]$ where ($\mathbf{z}_{0},\mathbf{z}^{\prime}_{1})\in\hat{\Gamma}(p_{0},p^{\prime}_{1})$, ($\mathbf{z}_{0},\mathbf{z}_{1})\in\Gamma(p_{0},p_{1})$, and $\hat{c}_{g}(\mathbf{z}_{0},\mathbf{z}_{1})=\min\|\pi(\mathbf{R}\mathbf{z}^{1}_
{\mathbf{x},1}+\mathbf{t},\mathbf{R}\mathbf{z}^{2}_{\mathbf{x},1}+\mathbf{t},
\dots,\mathbf{R}\mathbf{z}^{N}_{\mathbf{x},1}+\mathbf{t})-(\mathbf{z}^{1}_{
\mathbf{x},0},\mathbf{z}^{2}_{\mathbf{x},0},\dots,\mathbf{z}^{N}_{\mathbf{x},0
})\|_{2}+\min\|\pi(\mathbf{z}^{1}_{\mathbf{h},1},\mathbf{z}^{2}_{\mathbf{h},1}
,\dots,\mathbf{z}^{N}_{\mathbf{h},1})-(\mathbf{z}^{1}_{\mathbf{h},0},\mathbf{z
}^{2}_{\mathbf{h},0},\dots,\mathbf{z}^{N}_{\mathbf{h},0})\|_{2},\forall\pi,
\mathbf{R},~{}\textrm{and}~{}\mathbf{t}.$

$\mathbf{z}$ is geometry $\mathbf{g}$ in the latent space, which is composed of $\mathbf{z}_{\mathbf{x}}\in\mathbb{R}^{N\times 3}$ and $\mathbf{z}_{\mathbf{h}}\in\mathbb{R}^{N\times k}$, where $k$ is the latent dimension characterized by $\mathcal{E}_{\phi}$.

With node-granular optimal transport $\hat{\mathbf{R}},\hat{\mathbf{t}}$ and $\hat{\pi}$ we have:

$$ $\displaystyle\mathbb{E}[\hat{c}_{g}(\mathbf{z}_{0},\mathbf{z}^{\prime}_{1})]=$ $\displaystyle\mathbb{E}[\min\|\pi(\mathbf{R}\mathbf{z}^{\prime 1}_{\mathbf{x}, 1}+\mathbf{t},\mathbf{R}\mathbf{z}^{\prime 2}_{\mathbf{x},1}+\mathbf{t},\dots, \mathbf{R}\mathbf{z}^{\prime N}_{\mathbf{x},1}+\mathbf{t})-(\mathbf{z}^{1}_{ \mathbf{x},0},\mathbf{z}^{2}_{\mathbf{x},0},\dots,\mathbf{z}^{N}_{\mathbf{x},0 })\|_{2}$ $\displaystyle+\min\|\pi(\mathbf{z}^{\prime 1}_{\mathbf{h},1},\mathbf{z}^{ \prime 2}_{\mathbf{h},1},\dots,\mathbf{z}^{\prime N}_{\mathbf{h},1})-(\mathbf{ z}^{1}_{\mathbf{h},0},\mathbf{z}^{2}_{\mathbf{h},0},\dots,\mathbf{z}^{N}_{ \mathbf{h},0})\|_{2},\forall\pi,\mathbf{R},~{}\textrm{and}~{}\mathbf{t}]$ $\displaystyle=$ $\displaystyle\mathbb{E}[\|\hat{\pi}(\hat{\mathbf{R}}\mathbf{z}^{\prime 1}_{ \mathbf{x},1}+\hat{\mathbf{t}},\hat{\mathbf{R}}\mathbf{z}^{\prime 2}_{\mathbf{ x},1}+\hat{\mathbf{t}},\dots,\hat{\mathbf{R}}\mathbf{z}^{\prime N}_{\mathbf{x} ,1}+\hat{\mathbf{t}})-(\mathbf{z}^{1}_{\mathbf{x},0},\mathbf{z}^{2}_{\mathbf{x },0},\dots,\mathbf{z}^{N}_{\mathbf{x},0})\|_{2}$ $\displaystyle+\|\hat{\pi}(\mathbf{z}^{\prime 1}_{\mathbf{h},1},\mathbf{z}^{ \prime 2}_{\mathbf{h},1},\dots,\mathbf{z}^{\prime N}_{\mathbf{h},1})-(\mathbf{ z}^{1}_{\mathbf{h},0},\mathbf{z}^{2}_{\mathbf{h},0},\dots,\mathbf{z}^{N}_{ \mathbf{h},0})\|_{2}]$ $$

Let $\hat{\mathbf{z}}_{\mathbf{x}}=\hat{\pi}(\hat{\mathbf{R}}\mathbf{z}^{1}_{
\mathbf{x}}+\hat{\mathbf{t}},\hat{\mathbf{R}}\mathbf{z}^{2}_{\mathbf{x}}+\hat{
\mathbf{t}},\dots,\hat{\mathbf{R}}\mathbf{z}^{N}_{\mathbf{x}}+\hat{\mathbf{t}})$, $\hat{\mathbf{z}}_{\mathbf{h}}=\hat{\pi}(\mathbf{z}^{1}_{\mathbf{h}},\mathbf{z}
^{2}_{\mathbf{h}},\dots,\mathbf{z}^{N}_{\mathbf{h}})$, and $\hat{\mathbf{z}}=[\hat{\mathbf{z}_{x}},\hat{\mathbf{z}_{h}}]\in\mathbb{R}^{N
\times(3+k)}$, then we have:

$$ $\displaystyle\mathbb{E}[\hat{c}_{g}(\mathbf{z}_{0},\mathbf{z}^{\prime}_{1})]=$ $\displaystyle\mathbb{E}[\|(\hat{\mathbf{z}}^{\prime 1}_{\mathbf{x},1}+,\hat{ \mathbf{z}}^{\prime 2}_{\mathbf{x},1}+,\dots,\hat{\mathbf{z}}^{\prime N}_{ \mathbf{x},1})-(\mathbf{z}^{1}_{\mathbf{x},0},\mathbf{z}^{2}_{\mathbf{x},0}, \dots,\mathbf{z}^{N}_{\mathbf{x},0})\|_{2}$ $\displaystyle+\|(\hat{\mathbf{z}}^{\prime 1}_{\mathbf{h},1},\hat{\mathbf{z}}^{ \prime 2}_{\mathbf{h},1},\dots,\hat{\mathbf{z}}^{\prime N}_{\mathbf{h},1})-( \mathbf{z}^{1}_{\mathbf{h},0},\mathbf{z}^{2}_{\mathbf{h},0},\dots,\mathbf{z}^{ N}_{\mathbf{h},0})\|_{2}]$ $\displaystyle=$ $\displaystyle\mathbb{E}[\|(\hat{\mathbf{z}}^{\prime 1}_{1}+,\hat{\mathbf{z}}^{ \prime 2}_{1}+,\dots,\hat{\mathbf{z}}^{\prime N}_{1})-(\mathbf{z}^{1}_{0}, \mathbf{z}^{2}_{0},\dots,\mathbf{z}^{N}_{0})\|_{2}]$ $\displaystyle=$ $\displaystyle\mathbb{E}[\|\hat{\mathbf{z}}^{\prime}_{1}-\mathbf{z}_{0}\|_{2}].$ $$

Likewise, we have:

$$ $\mathbb{E}[\hat{c}_{g}(\mathbf{z}_{0},\mathbf{z}_{1})]=\mathbb{E}[\|\hat{ \mathbf{z}}_{1}-\mathbf{z}_{0}\|_{2}].$ (9) $$

At this point, what we aim to prove is simplified to:

$$ $\displaystyle\mathbb{E}[\|\hat{\mathbf{z}}^{\prime}_{1}-\mathbf{z}_{0}\|_{2}] \leq\mathbb{E}[\|\hat{\mathbf{z}}_{1}-\mathbf{z}_{0}\|_{2}]$ (10) $$

###### Proof.

Given that $\mathbf{z}^{\prime}_{1}=\textrm{ODE}_{\hat{\theta}}(\mathbf{z}_{0})$, $d\mathbf{z}_{t}=v_{\hat{\theta}}(\mathbf{z}_{t},t)dt$, we have:

$$ $\mathbb{E}[\hat{c}_{g}(\mathbf{z}_{0},\mathbf{z}^{\prime}_{1})]=\mathbb{E} \left[\left\|\int_{0}^{1}v_{\hat{\theta}}(\mathbf{z}_{t},t)dt\right\|_{2}\right]$ (11) $$

$\|\cdot\|_{2}:\mathbb{R}^{N\times(3+k)}\to\mathbb{R}_{+}$ is the Euclidean norm of $\cdot$ and it is convex, therefore, with ${\|\int_{\Omega}vdt\|\leq\int_{\Omega}\|v\|\,dt}$ induced by Jensen’s inequality we have:

$$ $\mathbb{E}[\hat{c}_{g}(\mathbf{z}_{0},\mathbf{z}^{\prime}_{1})]\leq\mathbb{E} \left[\int_{0}^{1}\left\|v_{\hat{\theta}}(\mathbf{z}_{t},t)\right\|_{2}dt \right].$ (12) $$

With defined $v_{\hat{\theta}}(\mathbf{z}_{t},t)=\mathbb{E}[\mathbf{z}_{1}-\mathbf{z}_{0}|
\mathbf{z}_{t}]$, we then have:

$$ $\mathbb{E}[\hat{c}_{g}(\mathbf{z}_{0},\mathbf{z}^{\prime}_{1})]=\mathbb{E} \left[\int_{0}^{1}\left\|\mathbb{E}[\mathbf{z}_{1}-\mathbf{z}_{0}|\mathbf{z}_{ t}]\right\|_{2}dt\right].$ (13) $$

Again, with the finite form of Jensen’s inequality, we have:

$$ $\displaystyle\mathbb{E}[\hat{c}_{g}(\mathbf{z}_{0},\mathbf{z}^{\prime}_{1})]$ $\displaystyle\leq\mathbb{E}\left[\int_{0}^{1}\mathbb{E}[\left\|\mathbf{z}_{1}- \mathbf{z}_{0}\right\|_{2}|\mathbf{z}_{t}]dt\right]$ // Jensen’s inequality (14) $\displaystyle=\int_{0}^{1}\mathbb{E}\big{[}\mathbb{E}[\left\|\mathbf{z}_{1}- \mathbf{z}_{0}\right\|_{2}|\mathbf{z}_{t}]\big{]}dt$ $\displaystyle=\int_{0}^{1}\mathbb{E}[\left\|\mathbf{z}_{1}-\mathbf{z}_{0} \right\|_{2}]dt$ $\displaystyle\textrm{//}~{}\mathbb{E}[\|\mathbf{z}_{1}-\mathbf{z}_{0}\|_{2}| \mathbf{z}_{t}]=\|\mathbf{z}_{1}-\mathbf{z}_{0}\|_{2}$ $\displaystyle=\mathbb{E}[\|\hat{\mathbf{z}}_{1}-\hat{\mathbf{z}}_{0}\|_{2}]$ $\displaystyle=\mathbb{E}[\hat{c}_{g}(\mathbf{z}_{0},\mathbf{z}_{1})]$ // By Eq. 9 $$

Combining equations 11 to 14, Eq. 10 is proved.
∎

It is important to note that solving the geometric optimal transport problem in the latent space does not necessarily ensure that the molecule itself or its distribution also satisfies the optimal transport in the original space. However, given that the proposed flow model is trained in the latent space, it is sufficient to ensure that latent molecules and distributions are transported with optimal cost, thereby accelerating the flow model in the generation of molecules.

###### Proof.

## Appendix C Algorithms

This section contains the main algorithms of the proposed GOAT.
First, we present the algorithm for solving optimal molecule transport and unified flow in Algorithm 1 and Algorithm 2, respectively.
Algorithm 3 presents the pseudo-code for training the GOAT.
Algorithm 4 presents the process of fast molecule generation with GOAT.

Figure: Algorithm 1 Optimal Molecule Transport

Figure: Algorithm 2 Equivariant Autoencoder

Figure: Algorithm 3 Geometric Optimal Transport

Figure: Algorithm 4 Sampling

## Appendix D Related Works

Molecule Generation Models.
Initial research in molecule generation primarily concentrated on the creation of molecules as 2D graphs [, , ]. However, the field has seen a shift in interest towards 3D molecule generation. Techniques such as G-SchNet [] and G-SphereNet [] employ autoregressive methods to incrementally construct molecules by progressively linking atoms or molecular fragments. These approaches necessitate either a detailed formulation of a complex action space or an ordering of actions.

Motivated by the success of Diffusion Models (DMs) in image generation, the focus has now turned to their application in 3D molecule generation from noise [, , , ]. To address the inconsistency of unified Gaussian diffusion across various modalities, a latent space was introduced by []. To resolve the atom-bond inconsistency issue,  [] proposed different noise schedulers for different modalities to accommodate noise sensitivity. However, diffusion-based models consistently face the challenge of slow sampling speed, resulting in a significant computational burden for generation. To enhance the speed, recent proposals have introduced flow matching-based [] and Bayesian flow network-based [] models. Despite these advancements, there remains substantial potential for improvement in these frameworks regarding speed, novelty, and ultimate significance.

Flow Models. Introduced in [], Continuous Normalizing Flows (CNFs) represent a continuous-time variant of Normalizing Flows []. Subsequently, flow matching [] and rectified flow [] were proposed to circumvent the need for ODE simulations during forward and backward propagation in CNF, and they introduced optimal transport for faster generation. Leveraging these advanced flow models,  [] pioneered the use of flow models for molecule generation, which was later followed by the proposal of  [], based on hybrid transport. Beyond the realm of 3D molecule generation, the concept of flow matching and optimal transport has also found applications in many-body systems [] and molecule simulations []. Despite these advancements, existing models primarily focus on atomic coordinates, leaving the challenge of geometric optimal transport unresolved.

## Appendix E Dataset

### E.1 QM9 Dataset

QM9 [] is a comprehensive dataset that provides geometric, energetic, electronic, and thermodynamic properties for a subset of the GDB-17 database [] comprises a total of 130,831 molecules (^2^22[https://springernature.figshare.com/ndownloader/files/3195389](https://springernature.figshare.com/ndownloader/files/3195389)). We utilize the train/validation/test partitions delineated in [], comprising 100K, 18K, and 13K samples for each respective partition.

### E.2 GEOM-DRUG Dataset

GEOM-DRUG (Geometric Ensemble Of Molecules) dataset [] encompasses around 450,000 molecules, each with an average of 44.2 atoms and a maximum of 181 atoms(^3^33[https://dataverse.harvard.edu/file.xhtml?fileId=4360331&version=2.0](https://dataverse.harvard.edu/file.xhtml?fileId=4360331&version=2.0)). We build the GEOM-DRUG dataset following [] with the provided code.

## Appendix F Implementation Details

In this study, all the neural networks utilized for the encoder, flow network, and decoder are implemented using EGNNs []. The dimension of latent invariant features, denoted as $k$, is set to 2 for QM9 and 1 for GEOM-DRUG, to map the molecule for a unified flow matching.

For the training of the flow neural network, we employ EGNNs with 9 layers and 256 hidden features on QM9, and 4 layers and 256 hidden features on GEOM-DRUG, with a batch size of 64 and 16, respectively.

In the case of equivariant autoencoders, the decoder is parameterized in the same manner as the encoder, but the encoder is implemented with a 1-layer EGNN. This shallow encoder effectively constrains the encoding capacity and aids in regularizing the latent space [].

All models utilize SiLU activations and are trained until convergence. Across all experiments, the Adam optimizer [] with a constant learning rate of $10^{-4}$ is chosen as our default training configuration. The training process for QM9 takes approximately 3000 epochs, while for GEOM-DRUG, it takes about 20 epochs.

With the flow model trained on QM9 or GEOM-DRUG, we then generate and purify the coupling to obtain a total of 100K molecular pairs, which form the estimated couplings.

Hardware Configuration

- 1.
GPU: NVIDIA GeForce RTX 3090
- 2.
CPU: Intel(R) Xeon(R) Platinum 8338C CPU
- 3.
Memory: 512 GB
- 4.
Time: Around 7 days for QM9 and 20 days for GEOM-DRUG.

## Appendix G More Experimental Results

We present the full results in Tables 3 and 4.
In our detailed experimental results on QM9, we reproduced EDM, GeoLDM, and EquiFM on the QM9 dataset to obtain the actual generation time consumption with the same compute configuration. As a result, the proposed method achieves the fastest sampling speed, which is consistent with the measurement of sampling steps. We also witness a huge generation speed improvement by the proposed GOAT for GEOM-DRUG.

In addition to supplementing the actual time used for generation, we also added the metrics of molecule stability, and it is obvious that all methods achieve nearly 0% molecule stability in GEOM-DRUG. This is because metrics, atom and molecule stability, create errors during bond type prediction based on pair-wise atom types and distances. Therefore, we concentrate on metrics measured by RdKit.

Lastly, we produced the full results of GeoBFN using sampling steps from 50 to 1,000. It is worth noting that the novelty and significance continue to decrease on QM9 datasets as sampling steps increase, which aligns with our conjecture in the experiments. Besides, we also observed that its performance on GEOM-DRUG also decreased in terms of validity. Combined with its efficiency and quality, we believe that our method GOAT has competitive performance compared with GeoBFN.

Table: Table 3: Comparisons of generation quality (larger is better) in terms of Atom Stability, Molecule Stability, Validity, Uniqueness, Novelty, and Significance. And comparisons of generation efficiency regarding generation time and sampling steps for one molecule (less is better). The best results are highlighted in bold.

**Table 4: Comparisons of generation quality (larger is better) in terms of Atom Stability, Molecule Stability, Validity, Uniqueness, Novelty, and Significance. And comparisons of generation efficiency regarding generation time and sampling steps per molecule (less is better). The best results are highlighted in bold.**
| GEOM-DRUG |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| # Metrics | Efficiency | Quality (%) |  |  |  |  |
| Time (s) | Steps | Atom Sta | Mol Sta | Valid | Uniqueness |  |
| Data | - | - | 86.5 | 0.0 | 99.9 | 100.0 |
| ENF | - | - | - | - | - | - |
| G-Schnet | - | - | - | - | - | - |
| GDM-aug | - | 1000 | 77.7 | - | 91.8 | - |
| EDM | 13.18 | 1000 | 81.3 | 0.0 | 92.6 | 99.9 |
| EDM-Bridge | - | 1000 | 82.4 | - | 92.8 | - |
| GeoBFN | - | 50 | 78.9 | - | 93.1 | - |
| - | 100 | 81.4 | - | 93.5 | - |  |
| - | 500 | 85.6 | - | 92.1 | - |  |
| - | 1000 | 86.2 | - | 91.7 | - |  |
| GeoLDM | 11.84 | 1000 | 84.4 | 0.0 | 99.3 | 99.9 |
| EquiFM | - | 200 | 84.1 | - | 98.9 | - |
| GOAT | 0.90 | 90 | 81.8 | 0.0 | 96.0 | 99.9 |

We present the visualization of generated molecules on QM9 and GEOM-DRUG in Figures 4 and 5.

Figure: Figure 4: Molecules Generated by GOAT trained on QM9.
Refer to caption: x6.png

Figure: Figure 5: Molecules Generated by GOAT trained on GEOM-DRUG.
Refer to caption: x7.png

## Appendix H Controllable Molecule Generation

### H.1 Properties

- 1.
$\alpha$ Polarizability: Tendency of a molecule to acquire an electric dipole moment when subjected to an external electric field.
- 2.
$\varepsilon_{\textrm{HOMO}}$: Highest occupied molecular orbital energy.
- 3.
$\varepsilon_{\textrm{LUMO}}$: Lowest unoccupied molecular orbital energy.
- 4.
$\Delta\varepsilon$ Gap: The energy difference between HOMO and LUMO.
- 5.
$\mu$: Dipole moment.
- 6.
$C_{v}$: Heat capacity at 298.15K

### H.2 Results and Analysis

We use the property classifier network $\varphi$ from []. We split the QM9 training partition into two halves with 50K samples each. The classifier $\varphi$ is trained in the first half, while the Conditional GOAT is trained in the second half. Then, $\varphi$ is applied to evaluate conditionally generated samples by the GOAT.

We report the numerical results in Table 5. Random means we simply do random shuffling of the property labels in the dataset and then evaluate $\varphi$ on it. $N_{\textrm{atoms}}$ predicts the molecular properties by only using the number of atoms in the molecule.

The proposed GOAT has the fastest speed and comparable performance in controllable generative tasks. It is worth noting that on $C_{v}$, the proposed method achieves the best results.

**Table 5: MAE for molecular property prediction. A lower number indicates a better controllable generation result.**
| Property | $\alpha$ | $\Delta\varepsilon$ | $\varepsilon_{\textrm{HOMO}}$ | $\varepsilon_{\textrm{LUMO}}$ | $\mu$ | $C_{v}$ | Steps |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Units | Bohr^3 | meV | meV | meV | D | $\frac{\textrm{cal}}{\textrm{mol}}$ K |  |
| QM9 | 0.10 | 64 | 39 | 36 | 0.043 | 0.040 | - |
| Random | 9.01 | 1470 | 645 | 1457 | 1.616 | 6.857 | - |
| $N_{\textrm{atoms}}$ | 3.86 | 866 | 426 | 813 | 1.053 | 1.971 | - |
| EDM | 2.76 | 655 | 356 | 583 | 1.111 | 1.101 | 1000 |
| GeoLDM | 2.37 | 587 | 340 | 522 | 1.108 | 1.025 | 1000 |
| GeoBFN | 2.34 | 577 | 328 | 516 | 0.998 | 0.949 | 500 |
| EquiFM | 2.41 | 591 | 337 | 530 | 1.106 | 1.033 | 220 |
| GOAT | 2.74 | 605 | 350 | 534 | 1.010 | 0.883 | 100 |

## Appendix I Impact Statements

This paper contributes to the advancement of generative Artificial Intelligence (AI) in scientific domains, including material science, chemistry, and biology. The insights gained will significantly enhance generative AI technologies, thereby streamlining the process of scientific knowledge discovery.

The application of machine learning to molecule generation expands the possibilities for molecule design beyond therapeutic purposes, potentially leading to the creation of illicit drugs or hazardous substances. This potential for misuse and unforeseen consequences underscores the need for stringent ethical guidelines, robust regulation, and responsible use of these technologies to safeguard individuals and society.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: Based on the listed weakness points above, I want to ask following questions:

1. Why using equivariance autoencoder to produce latent codes can yield in unified permutation? 
2. Can I assume your paper is based on the already on-the-shelf work which proposed EOT and EAE? If so, what is your major novel improvement scientifically based on that? If not, please clarify why.
3. Please clarify the terminology issues on point 2 and point 4 mentioned above in the weakness part.

I will adjust my rating to the paper based on the answers to the above questions during rebuttal and discussion phase.
soundness: 2
strengths: 1. **Clear problem statement**: optimal transport of multi-modal probability and distribution coupling in the specific case of molecule generation are well-known problems and to be addressed. The writing provides enough information for general readers to comprehend. 
The experiments seem abundant to make the effectiveness of the method stand. 
2. Table 4 is crucial for the core statement of this paper, which is that Optimal Molecule Transport (OMT) is important for faster and better molecule generation based on optimal transport (OT) of Flow Matching (FM). From the numbers there, it seems justified.
summary: Because of the known multi-modal (Cartesian coordinates and other features) distribution issue in molecule generation and distribution coupling issue, the paper proposed an optimal molecule transport (OMT) algorithm based on optimal transport (OT) in Flow Matching (FM) on latent codes encoded by equivariance autoencoder.
weaknesses: 1. I do not see why after equivariance autoencoder mapping to the latent space, we thus have a unified optimal permutation. In equation (5), we still have the same \pi for both coordinates and features right? The reasoning for this part is missing.
2. I am not sure the distribution coupling, $\Gamma$ is a correct terminology used in this paper. The distribution coupling should be referring to pairing points between two distributions, rather than pairing within one pair of data sample by permutation, rotation and translation. 
3. Maybe limited novelty: In EquiFM[1], we know the idea of equivariant optimal transport (EOT) is proposed, which has a large overlapping with the core contributions of this paper. Can this paper’s main contributions be concluded as  EOT + Equivariant Autoencoder? If so, the novelty of paper may be limited.
4. Is Geometric Probability Distribution a formal terminology in molecule generation community? I do not see it is widely adopted in other related work and it can be misunderstood with geometric distribution. 
5. The abbreviation EAE in line 251 pops out without specification (I am assuming it is referring to Equivaraint Autoencoder?)

[1] Song, Yuxuan, et al. "Equivariant flow matching with hybrid probability transport for 3d molecule generation." Advances in Neural Information

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: Please refer to weaknesses.

I honestly do not know much about the literature on molecule generation so my confidence would not be high. I am happy to check other reviewers' opinions.
soundness: 3
strengths: 1. The problem setup is well-motivated. The OT path and equivariance properties are indeed needed in diffusion models/flow matching methods for fast sampling process. 

2. The empirical performance is strong in terms of various metrics. The proposed methods is good in both generation quality and inference steps.
summary: This paper proposes a 3D molecule generation framework which formulates a geometric transport formula for
measuring the cost of mapping multi-modal features (e.g., continuous atom coordinates and categorical atom types) between a base distribution and a target data distribution. They further propose a flow refinement and purification mechanism for optimal coupling identification, which filters out the subpar molecules to ensure the ultimate generation quality.
weaknesses: 1. In Eq. (5), the authors seem to sample permutation, rotation, and translation matrix. Does this mean that the model is not strictly equivariant but approximately equivariant? In other words, the equivariance is learned from data augmentation but not learned by construction.

2. I am wondering maybe it is possible to distill the trained model, like rectified flow, to have even faster inference (1 to 5 steps). I am especially curious to see the generation quality of distillation + purification.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. It would be interesting if the paper could provide a comparison of the distance between generated molecules and the initial state, with/without the use of optimal transport.
2. The use of optimal transport may raise concerns about the diversity of the generated molecules. This could be addressed by generating N molecules and comparing the coverage of the reference conformation set within a specified distance threshold (you can refer to the design of this criterion in papers like GeoDiff[1]).
[1] Xu, Minkai, et al. "Geodiff: A geometric diffusion model for molecular conformation generation"
soundness: 3
strengths: 1. I think that using optimal transport to optimize the training process of diffusion models is a very clever and reasonable design, especially when applied to scenarios like molecular generation, where the generation efficiency of diffusion models can be a concern.
2. I'm glad that the paper points out the optimal transport optimization might bring extra computational cost, especially those involving rotation and permutation, can be computationally intensive. However , it's inevitable, but it's a great trial.
3. The framework achieves the satisfiable downstream generation quality regarding validity, uniqueness, and novelty, which are crucial metrics in molecule generation.
summary: This paper introduces a novel framework for accelerating 3D molecule generation, named GOAT, which leverages the principles of flow-matching optimal transport to efficiently generate molecules with improved quality and speed.
weaknesses: 1. While the paper includes some ablation studies, a more comprehensive set of experiments could provide further insights into the contribution of each component of the framework.
2. The training process itself may be time-consuming, which could be a drawback for some applications, though as mentioned in strengths, it's inevitable.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Could you provide a more comprehensive evaluation of the generated 3D molecular structures?
I acknowledge that determining the most effective 3D metric is an open and important question. There are several possible solutions you might consider:
- The approach uses the Wasserstein distance between the distributions of bond lengths and bond angles. Some papers, like MiDi, have implemented this method. However, be cautious, as a single poorly predicted molecule can significantly skew the metric due to the way MiDi implemented it.
- SemlaFlow, for example, proposes using the optimization energy drop with the MMFF94 force field as a measure. Given that your ground truth data is based on GFN-xTB calculations, it might be more appropriate to assess the energy drop using GFN-xTB, as it aligns with the potential energy landscape of your dataset.
- Models like JODO and others have utilized MMD for bond angles, bond lengths, and torsion distributions, focusing only on the most frequent bonds and angles. While none of these metrics are perfect, they are generally more informative and insightful than relying solely on 3D atom stability.

The important part here is to ensure that when comparing models, you use exactly the same implementation of these metrics to obtain a reliable comparison. By incorporating these additional evaluations, you can provide a more thorough and insightful assessment of the geometric accuracy of your generated molecular structures. If I am adding one of these, I would rather do it on GEOM-Drugs dataset because it is much more realistic, for QM9 it is quite fast to compute GFN-xTB geometry optimization, and a lot of these molecules do not have considerable variability of 3D structure. 

2. Could you perform comparisons with recent models like EQGAT-Diff, JODO, or SemlaFlow? I'm especially interested in a comparison with SemlaFlow because both papers place significant emphasis on improving the performance of molecule generation and utilize the flow matching objective. The main difference is that your approach performs flow matching in latent space, while SemlaFlow conducts diffusion explicitly on bonds, atoms, and coordinates.

It would be greatly appreciated if you could compare your method with SemlaFlow in terms of:
- Topological Metrics: Such as 2D molecule stability, validity, or introduced significance.
- 3D Metrics: Possibly using some of the metrics described in previous comments (e.g., bond angles, torsion distributions).
- Performance Metrics: Considering that both models generate molecules of similar size, metrics like the average time to generate a molecule would be insightful.

Additionally, could you discuss the advantages of your method over the version of flow matching used in SemlaFlow? Providing a brief discussion of the key algorithmic differences between your proposed method and SemlaFlow's flow matching approach—focusing on how these differences might impact performance or computational efficiency—would enhance the understanding of your contributions.
soundness: 3
strengths: 1. **Significant Theoretical Development in Joint Optimal Transport:**

The theoretical advancement of joint optimal transport presented in the paper has the potential to greatly impact future flow-matching (FM)-based model development. This innovation could propel the entire field of molecule and conformer generation toward faster and more efficient methods.

2. **Emphasis on Performance Improvement and Comprehensive Comparisons:**

The paper places a strong focus on performance enhancement and provides thorough performance comparisons with models like EquiFM, EDM, and others. Since dataset conformations are obtained through computational methods, there is an upper bound on computational cost beyond which applying AI is not justified. The proposed model addresses this by improving performance while staying within acceptable computational limits.
summary: The paper presents a novel approach to molecule generation by introducing a variation of optimal transport for multimodal features within a general flow-matching objective. An equivariant neural network is utilized to transform these multimodal features into a latent space, where multimodal data optimal transport is applied. The results show that this method outperforms existing models like EquiFM and EDM, and it also demonstrates superior computational speed compared to other models.
weaknesses: 1. **Comparison with Recent Edge-Modeling Methods and Potential Integration of Edge Features:**

Some recent models, such as JODO and EQGAT-Diff, have achieved better performance by explicitly modeling edges in molecular graphs. Including comparisons with these models would strengthen your paper. Moreover, adding edge features to your joint optimal transport framework seems feasible and could unlock the full potential of your method. Additionally, a more recent model called Semla Flow (a preprint published in June 2024) demonstrates superior accuracy and speed. Comparing your approach with these methods would greatly enhance the value of the proposed latent optimal transport (OT) technique.

2. **Concerns About Atom Stability Metric and 3D Evaluation:**

While the task is 3D molecule generation, there is no real evaluation of 3D coordinates for unconditional molecule generation in your paper. The term "atom stability" is mentioned without a clear definition. Given that you report an 86.5% figure for the GEOM Drugs dataset, I assume this refers to 3D atom stability. This metric is based on comparing bond lengths with tabulated values, allowing a tolerance of about 0.5 Å. The issue is that only 2.8% of GEOM Drugs molecules fully comply with these criteria, making the metric potentially misleading. I strongly encourage avoiding the propagation of this metric in new papers. The optimal distances between atoms are primarily defined by the potential energy landscape underlying the data—for GEOM Drugs, it's GFN2-xTB—and depending on atom configurations, deviations in bond lengths can exceed 10%.

3. **Difficulty in Claiming State-of-the-Art Performance:**

Related to the first point, it is difficult to claim that this method achieves state-of-the-art (SOTA) performance because it has not been compared with some other relevant methods mentioned above, including MiDi. Including such comparisons would strengthen the claim of achieving SOTA performance.

Meta-review: This paper introduces GOAT, a novel 3D molecule generation framework leveraging joint geometric optimal transport and equivariant autoencoders, which reviewers found innovative and effective in addressing multi-modal molecule properties while achieving impressive generation quality and speed. Strengths highlighted include clear motivation, significant theoretical contributions to flow matching methods, and competitive empirical performance, with notable efficiency gains over baseline models. However, reviewers raised concerns about limited comparisons to recent edge-modeling methods, potential ambiguities in terminology, and the adequacy of certain evaluation metrics, suggesting room for improvement in comprehensive benchmarking and clarity of presentation.

Decision: Accept (Poster)