FoldDiff: Folding in Point Cloud Diffusion
Authors: Yuzhou Zhao
Juan Matias Di Martino
Amirhossein Farzam
Guillermo Sapiro
Venue: Accepted by TMLR ()

Abstract: Diffusion denoising has emerged as a powerful approach for modeling data distributions, treating data as particles with their position and velocity modeled by a stochastic diffusion process. While this framework assumes data resides in a fixed vector spaces (e.g., images as pixel-ordered vectors), point clouds present unique challenges due to their unordered representation. Existing point cloud diffusion methods often rely on voxelization to address this issue, but this approach is computationally expensive, with cubically scaling complexity. In this work, we investigate the misalignment between point cloud irregularity and diffusion models, analyzing it through the lens of denoising implicit priors. First, we demonstrate how the unknown permutations inherent in point cloud structures disrupt denoising implicit priors. To address this, we then propose a novel folding-based approach that reorders point clouds into a permutation-invariant grid, enabling diffusion to be performed directly on the structured representation. This construction is exploited both globally and locally. Globally, \reviewcdmS{folded objects can represent point cloud objects} in a fixed vector space (like images), therefore it enables us to extend the work of denoising as implicit priors to point clouds. \reviewcdmS{Locally, the folded tokens are} efficient and novel token representations that can improve existing transformer-based point cloud diffusion models. Our experiments show that the proposed folding operation integrates effectively with both denoising implicit priors as well as advanced diffusion architectures, such as UNet and Diffusion Transformers (DiTs). Notably, DiT with \reviewcdmS{locally} folded tokens achieves competitive generative performance compared to state-of-the-art models while significantly reducing training and inference costs relative to voxelization-based methods.

Published in Transactions on Machine Learning Research (08/2025)
FoldDiff: Folding in Point Cloud Diffusion
Yuzhou Zhao
yz4450@princetonncde
Department of Electrical and Computer Engineering Princeton University
Duke University
J. Matías Di Martino
matias. di. www.mmmaaaakknkccoommm
Department of Computer Science Universidad Católica del Uruguay Duke University
Amirhossein Farzam
a.farzam@duke. edu
Department of Electrical and Computer Engineering
Duke University
Guillermo Sapiro
guillermos@ppricetton.dd
Department Electrical and Computer Engineering
Princeton Universily
Apple
Reviewed OpenReview: https: openrevieee net/ per/fo id= paRaabM1115
Abstract
Diffusion denoising has emerged a powerful approach for modeling data distributions, treating data particles with their position and velocity modeled by stochastic diffusion process. While this framework assumes data resides in fixed vector spaces (e.g., images as pixel-ordered vectors), point clouds present unique challenges due to their unordered repre- sentation. Existing point cloud diffusion methods often rely on voxelization to address this issue, but this approach is computationally expensive, with cubically scaling complexity. In this work, we investigate the misalignment between point cloud irregularity and diffusion models, analyzing it through the lens of denoising implicit priors. First, we demonstrate how the unknown permutations inherent in point cloud structures disrupt denoising implicit priors. To address this, we then propose novel folding-based approach that reorders point clouds into structured reprosentation. This construction is exploited both globally and locally. Glob- ally, folded objects can represent point cloud objects in a fixed vector space (like images), therefore enables us to extend the work of denoising as implicit priors to point clouds. Locally, the folded tokens are efficient and novel token representations that can improve existing transformer-basse proposed folding operation integrates ellectively with both denoising implicit priors well as advanced diffusion architectures, such as UNet and Diffusion Transformers (DiTs). No- tably, DiT with locally folded tokens achieves competitive generative performance compared to state-of-the-an models while significantly reducing training and inference costs relative to voxelization-base methods.
ioauutaaiio-invariann grid, enabling diffusion to be performed directly on the
point cloud diffusion models. Our experiments show that the
1 Introduction
Modern representations of images and surfaces are often high-dimensional, encompassing thousands of pixels 3D points. These representations typically reside low-dimensional manifolds, either explicitly defined

---

Published in Transactions on Machine Learning Research (08/2025)
inferred. Recent advancements in diffusion models (Ho et al., 2020; Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Song et al., 2021; Peebles & Xie, 2023) have demonstrated their efficacy in capturing implicit data distributions, and are conceptually linked to thermal systems. The forward diffusion process simulates thermal agitation, gradually transforming data into a standard Gaussian; while the reverse process mimics cooling, reconstructing data via Langevin-style probabilistic gradient ascent. However, this thermal- dynamics-inspire framework treats data, as particles and assumes that the data reside in a structured vector space (e.g., images as pixel-ordered vectors). The irregularity and lack of inherent order in point clouds pose significant challenges to the direct application of diffusion or Langevin dynamics.
The problems are illustrated in Fig. 1, where diffusion process or Langevin dynamics describes the motion of a particle in an arbitrary space. When we consider RGB images of height H and width W as particles, their positions are described by vectors in #x* x3 and updated by diffusion denoising or Langevin dynamics. Each entry in such vectors follows the same permutation across objects, and such a correspondence is preserved during denoising. This correspondence is missing in unordered point clouds. Thus, unlike image "particles," " the motions of "particles" that represent the point cloud objects cannol. be properly described due to the absence of this structured space. Permutation invariant choices for denoising loss further impose irretrievable permutation matrices on the noisy point clouds. This observation aligns with the findings reported in Zhou et al. (2021) and Mo et al. (2023), where similar challenges were encountered in designing diffusion models for unstructured point clouds. In our work, we build upon these observational results and present a formulated theoretical explanation of their failures based on "denoising implicit priors."
The theoretical framework of "denoising implicit priors" investigates diffusion from its fundamental building blocks: Denoising Autoencoders (DAEs). As first shown in Vincent (2011), a DAE that min- imizes the l2 norm between a recovered signal and a clean one is equivalent to an enorgy model that tries to match its score to that of a non-parametric Parzen density estimator of the data. The theoret- ical connection between DAEs and score-matching (Vincent, 2011) guarantees that each denoising step moves the data towards higher probability regions. Thus, authentic samples can be drawn from the im- plicit priors that DAEs learned from l2 noise re- gression. In this work, we employ the theoretical framework of denoising implicit priors to, for the first time, offer comprehensive theoretical analy- sis of diffusion denoising in unstructured spaces. In Sec. 3, we theoretically prove that such unknown permutations breaks the proportional relationship between an l2 denoising residual and the probabil- ity score, which are also empirically demonstrated in our experiments (Soc. 5.3).
X
X1
1lx
X/2
IIIx2
(a) Structured space with (b) Unstructured space with consistent axis.
unknown permutation.
(a) Particle motions can be properly de- scribed in shared and structured space (e.g., RGB images with the same height and width). (b) Particle motions cannot be properly described in unstructured spaces (e.g., unknown permutation between unordered point cloud objects) or during unstructured motion (e.g., unknown permutation imposed by permutation invariant denoising). Axes correspondence are noted by colors.
Figure 1:
Recent diffusion models for point clouds have opted for voxel grids (Zhou et al., 2021; Zeng et al., 2022; Mo et al., 2023) or triplanes (Shue et al., 2023) as structured representations for the diffusion processes. However, these methods suffers from cubically or quadratically scaling computational complexities as well as quantization errors. In this work, we proposed compact alternative based on the folding operation (Yang et al., 2018), which reconstructs a permutation-invarian grid-like representation of the input point cloud. The folded reconstruction can be viewed as a fixed permutation that approximates the original geometry, providing a structured representation for point cloud diffusion. Although folding is frequently employed to reconstruct local point patches in self-supervised learning models (Yu et al., 2022; Chen al., 2023), prior rescarch has neither examined nor leveraged the cermutation-invariaa properties of its reconstructions in depth. In our work, we provide a thorough discussion of these properties and demonstrate how they effectively resolve the permutation challenge in point cloud diffusion (see Sec. 4.2).
2

---

Published in Transactions on Machine Learning Research (08/2025)
Our novel framework, FoldDiff, is thus different from previous methods that performs diffusion voxelized space that incur cubic-scale computational costs and introduce quantization errors. It directly addresses the permutation challenge within the point clouds with a permutation-sqqiivriaat reconstruction, thereby being both minimalist and efficient, while fully inspired by our thcoretical contribution and exploiting the folding operation (Yang et al., 2018). FoldDijJ is a two-stages framework, with a folding-based slage for learning structured permutations, and a denoising- basod or DDPM-based stage for learning generative priors. It enjoys a linearly scaling computational complexity with the number of points in each object during diffusion denoising. The resulting grid-like representation, akin to Geometry Image (GI) (Gu et al., 2002), pairs effectively with image denoisers to learn implicit priors of the original manifold of 3D objects. By combining image denoisers trained on folded objects with the Langevin-style sampling algorithm from Kadkhodaie & Simoncelli (2021), 3D objects can be sampled from the denoisiig implicit priors. Additionally, we test our framework on the Denoising Diffusion Probabilistic Models (DDPMs) (Ho et al., 2020) with both UNet (Ronneberger et al., 2014) and DiT (Poebles Xic, 2023) backbones, and observed qualitatively and quantitatively competitive generative performances at significantly reduced cost. Notably, the use of folded tokens significantly reduces the token count in DiIs relative to its voxelization- based variant, leading to substantial improvements in training and sampling efficiency for point cloud diffusion.
Our main contributions can be summarized as follows:
As our core theoretical contribution, we present thorough analysis of the intractability of applying diffusion or Langevin dynamics to unordered data structures, grounded in the theory of "denoising implicit priors."
As our core technical contribution, we propose novel folding-based point cloud diffusion framework, dubbed FoldDiff', offering greater efficiency than popular exeelization-based met while enabling a seamless unification of 2D and 3D diffusion methods.
We empirically validate our novel framework on denoising implicit priors, UNet-based DDPMs and Daseb DDPMs, demonstrating competitive generative performance with lower training and inferencing cost.
2
Related work
Diffusion models and denoising implicit priors. Diffusion models (Ho et al., 2020; Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Song et al., 2021: Peebles & Xie, 2023) have recently emerged as powerful generative models. Standard Denoising Diffusion Probabilistic Models (DDPMs) (Ho et al., 2020) utilize a fixed Markovian forward process that incrementally corrupts the data with Gaussian noise, paired with a learned noise-conditioned denoiser that constructs the reverse Markovian process. DDPMs are theoretically equivalent to stacks of Denoising Autoencoders (DAEs) (Kadkhodaie et al., 2024), and are deeply connected to score matching and probability gradient ascent (Vincent et al., 2008). DAES estimate the data distribu- tion's score at varying noise levels, enabling denoising implicit priors on the data manifold. In particular, Kadkhodaie & Simoncelli (2021) proposed a stochastic gradient ascent procedure to sample from image DAÉS. Our work builds upon this idea, but addresses the unique challenges in extending it for the first time to the point cloud domain. We propose a novel folding- based solution to address point cloud irregularity (unordered data), which is compatible with any variants of diffusion-based modeling (e.g., standard DDPM (Ho al., 2020), score-based diffusion models (Song & Ermon, 2019; Song et al., 2021: 2023), or flow-based diffusion models (Lipman et al., 2022; Liu et al., 2022)); and also with multiple architectures (c.g., UNet (Ronneberger et al., 2014) or transformers (Peebles & Xie, 2023; Ma et al., 2024)).
A parallel line of research investigates relation-conditionee diffusion process (Gao et al., 2023; Xu et al., 2024; Yang et al., 2024), which conceptually aligns with our DiT-fold that models a combinatorial distribution of folded patches. Formally, a relation-conditioneed diffusion aims to learn a conditional score function Vy log qe (y, RR), where y = (Xx, X2, x2.... ) is a combinatorial structure (e.g., an image or a point cloud y as an assemble of local patches x:), and R the correlation matrix for all x,, Xj (Yang et al., 2024). To encourage
"Code is available at htap::://gthhubb..m/yyyaaaa////s
3

---

Published in Transactions on Machine Learning Research (08/2025)
capturing patch-wise correlations, Gao et al. (2023) integrates latent masking within DiT, while Xu et al. (2024) introduces asynchronous time scheduling for local patches. Notably, Xu et al. (2024) explored the generation of combinatorial 3D semantic parts with the more advanced flow-based diffusion (Ma et al., 2024). While these modifications may provide potential benefits for our aading-bascl DiT, we maintain an unmodified training scheme to ensure fair comparison our experiments.
Diffusion models for point clouds. Generalive models for point clouds aim to capture "distribution (objects in Rxx3) of distributions (points in IR*)." Diffusion process was first adopted in Luo & Hu (2021b) to model the distribution of points in IR conditioned on a PointNet (Qi et al., 2017a) encoded shape latent. Our work instead examines the diffusion models at the object level, where the "particles" are point cloud objects in Nx3
Due to the irregularity of point clouds, prior works perform diffusson either on a voxelized space following Point Voxel Diffusion (Zhou et al., 2021), or on encoded triplane representation following Triplane diffusion (Shue et al., 2023). These two strctured representations are easy to optimize and intuitively analogous to 2D pixcls, but they both suffer from quantization crrors and higher computational complexity. Comparing to generalive models modeled on raw point clouds in RNx33, voxelization- Zeng et al., 2022; Mo et al., 2023) perform diffusion in the voxel space in RCVVXVXV Here, C denotes the number of feature channels per voxel, and V represents the resolution of the voxel grid along cach spatial dimension. Similarly, triplane-based methods (Shue et al., 2023) perform diffusion in the space of encoded triplane latents in R3Cx xx where H and W specify the spatial resolution of each triplane feature map of dimension C. Our proposed folded structure fundamentally differs from previous structured representations, offering computational complexity at the order of the original point cloud (R while being fully compatible with established 2D diffusion architectures.
methods (Zhou el al., 2021;
A closely related area of point cloud diffusion is 3D molecule generation, which also faces the challenge of permutation equivariance. In this study, we address only permutation equivariance, deferring SE(3) equivariance- betttr managed by SE(3) diffusion models (Peng et al., 2024: Yim et al., 2023) to future works. Early studics primarily rely on Graph Neural Networks (Liu et al., 2018; Shi et al., 2020; Vignac et al., 2023) to ensure permulation equivariance in molecule graph generation. However, graph-based hods scale poorly for point cloud generation because of the lack of intrinsic connectivity in point clouds, which rules out the application of spectral- -basd graph diffusion methods (Elhagg et al., 2024); and point clouds, often consist of thousands of continuously valued points in R3, exhibiting far greater representational complexity than 3D molecules, which typically involve only tens to hundreds of atoms with limited atom and bond types. Notably, some graph-free techniques generate molecules in atomic densities within voxel grids (Masuda et al., 2020), which is less efficient comparing to our approach.
iomuutatiio-invarian manner by inferring
Recent works also explore the capability of diffusion models as powerful pretrained models for supervised learning tasks (Xiang et al., 2023; Zheng et al., 2023). In this work, we do not investigate this application in point cloud, but instead focus on the detrimental effects of an unstructured space on the diffusion process.
Point cloud denoisers. The unordered nature of point clouds poses a key challenge in applying image- based deep learning methods to 3D data. PointNet (Qi et al., 2017a) pioneered using permutation invari- ant operations, e.g., pointwise convolutions followed by channel-wise pooling, to address this. Subsoquent works improved the expressiveness (Qi et al., 2017b; Wang et al., 2018) or efficiency (Liu et al., 2019) of this approach. State-of-the-art point cloud denoisers (Rakotosaona et al., 2020; Luo & Hu, 2020: 2021a; de Silva Edirimuni et al., 2023) adopt similar permutation invariant encoder-decoder architecture, but focus on recovering local surface details rather than global shape of an object. We do not consider these local denoisers since we are interested in modeling the distribution of point cloud objects globally. As de- scribed in Sec. 5.3, we performed experiments on object-level denoisers paired with the Chamfer Distance loss. We provide theoretical and empirical analysis on why this object-level denoiser cannot be directly usod to sample shapes using "denoiser as prior" techniques (Sec. 3.1). We solve this problem with the approach here introduced (Sec. 4.1), and demonstrate the empirical applications of these ideas (Sec. 4.3).
4

---

Published in Transactions on Machine Learning Research (08/2025)
Folding-based autoencoders and geometry images. Folding-based decoders (Yang al., 2018:
Groueix et al., 2018; Pang et al., 2020) became a popular design choice due to their expressiveness power. In general, the decoding process simulates a deformation process from a genus-0 primitive surface (i.e., a 2D grid lattice) to a shape that reconstructs the imputting point cloud. This 2D-to-3D deformation is deeply connected with Geomelry Images (GIs) (Gu et al., 2002), which aims to simplify shape analysis with image- processing tools including CNNs (Zhang et al., 2022; Sinha et al., 2016; 2017; Maron et al., 2017). In our order of the inputting point cloud, creating a
work, folding adopted to obtain a permutational-invariae structured vector space for diffusion systems.
Recent works have also explored Geometry Images for 3D shape diffusion in the UV domain (Yan et al., 2024; Elizarov et al., 2025), where the surface of a 3D mesh is segmented into disjoint local patches and arranged into a geometry image in the UV domain. The segmented geometry images can be paired with texture maps and text prompts, enabling textured text-to-3D generation with well-ostablished text-to-image techniques. Though similar in concept, these methods mainly focus on the generation of UV images processed from 3D meshes, while our method focuses the generation of local or global geomelry images learned from 3D point clouds.
3 Denoising implicit priors on point clouds
3.1 Denoising on images and point clouds.
We represent the original input data (image or point cloud) as a vector RNxx3 The N dimensions, can be associated to an RGB image of N pixels a point cloud of N (3D) points. The goal is to efficiently represent the prior probability P(x) (i.e,, the manifold on the original space where data lies). In both image and point cloud denoising, it is common to assume that the noisy observation is a distribution conditioned on the clean observation, expressively y = x + Z, where y E Nx3 is the noisy observation, x E Nx3 is the clean observation, and z ~ (T,.)TTN) is the Gaussian noise modeled as both additive and Gaussian (Kadkhodaie Simoncelli, 2021). Thus, a denoiser is optimized to approximate the residual z to recover the clean image x given y. If is drawn from a prior distribution P(x), its created noisy version can be considered Gaussian convolved distribution P(y) that can be rewritten via marginalization,
ply) f P(ylx)P(x)ds S s x))
where g(z) is the multivariate Gaussian PDF with variance o² that models the noise residual
Even though the additive noise is modeled by a Gaussian distribution in both scenarios, point cloud denoising differs from image denoising in its optimization objectives. A key observation is that point clouds are unstructured data while images are structured. In image denoising, each noisy pixel y: has a one-to-on correspondence with its ground truth clean pixel x;. In this case, optimizing a denoiser foly) = - x is oquivalent minimizing the objective
argmin Ey-r(y) [xxx
(2)
where L is commonly the la norm, and x = y is the recovered signal. Unlike images, a clean point cloud a discrete set of samples from a surface S embedded in 3D space. Hence, each noisy point y; can have multiple origins from the clean surface S, which breaks the point-wist correspondence between noisy and clean point clouds. If we naively treat every point cloud with its given permutation as a vector and supervise denoising with the la norm, the point-wise correspondence is still unknown during test time. As reported in Rakolosaona et al. (2020), a denoiser optimized to minimize the objective given in Equation 2 is not recovering the original clean point but an average of all possible candidates that the noisy point originates from. This lcads to worse denoising performance as it does not guarantee the average to be a point on the surface. Another option is to minimize the distance from denoised points to the underlying surface through
argmin Eyy [(xx)))
(3)
5

---

Published in Transactions on Machine Learning Research (08/2025)
In practice, only have access to clean points discretized samples from the underlying surface. Prox- imity to the surface is thus the Euclidean distance between a noisy point yr and its nearest. neighbor (NN) in X,
ly: - Ny,,,,
(4)
A common loss function [(x, S) (Rakotosaona al., 2020; Roveri et al., 2018) is then
x NN x) (Ifo(y) (y NN x)
(5)
This strategy presents two main challenges: (1) nearest neighbors need to be computed online, which is computationally very demanding; and (2) the process of denoising introduces unknown permutation in the points representations. Next, we discuss the challenges of this permutation matrix.
3.2 Denoising implicit priors in structured and unstructured spaces.
Denoising CNNs can be viewed as least squares estiimators that recover the true signal by computing the conditional mcan of the posterior,
x(y) = / xP((xy)dr= P(ylx)P(x) dx,
(6)
Ply)
where xyy) is the best (in the le sense) approximation of the recovered signal. This solution can be expressed (Miyasawa, 1961)
x y o log Ply).
(7)
The predicted residual x(y) - y is then proportional to the score function Vy log P(y). This implies that the training objective a CNN denoiser equivalent to score matching. The noise residual provides a direction move up probability gradient toward clean image density. Thus, as proposed in Kadkhodaie & Simoncelli (2021), can gradually converge to the manifold P(x) using Langevin style gradient ascent.
Unfortunately, Equation 7 no longer holds when the data is unstructured, and therefore the whole idea of "denoiser as prior" collapses. To formalize this problem, let's consider a noisy observed point set y 1,...,NY ,,,,,, where we assume 2; ~ (O, o²) for all i, and x = (XI,..., XN) ) is the "correct"
{xi 22: 2; vectorization of clean points. Any vectorization of y can be rewritten as
y = II +z),
(8)
where II a permutation matrix. Since z, V(0,0²), we have IIz (O (I), i.c., z and Iz have the same distribution, and since is not directly observed, without loss of generality, we can write
y=
where x = IIx is a permutation of the original vectorization and o1) Note that II = NNN if and only if y; = x; + for all i. Suppose we naively vectorize y to obtain by a random permutation. Given a permutation II, we can rewrite the marginal distribution of the observation P(y[IT) and the least squares estimate x(y) as
IIy / P(y PI / /oly X))-x))((
(10)
x(y) = / xx((xly II)x = / P(y)),II) ((x dx.
(11)
IIy
The following proposition then follows. Proposition 1. The denoiser residual f(y) 2- y is proportional Vy log P(y) if and only if INXN
y-y
The proof is presented in the Appendix A.1. This proposition implics that Equation can only be applied structured data format that always follows its original permutation, which is not generally the case in point cloud denoising, validating the empirical observations in Sec. 5.3.

---

Published in Transactions on Machine Learning Research (08/2025)
4 FoldDiff for 3D shape generation
4.1 FoldingNet as reordering module
N'X2
Spatial coordinates
Point-wise
Spatial coordinates
Features
Latent vector
Encoder
pooling
Max
repeut
concat
Decoder
1xd
N
Nxd
N'x3
Chamfer Distance
(a) Original FoldingNet.
N'x2
Spatial coordinates
Point-wise Features
Spatial coordinates
flatten
Cross attention
Latent vector
Encoder
pooling
Max
repeat
concal
MLP
Decoder
1xd
X33
Nxd
x3
N'x(2
K.V
Sinkhorn Distance
(b) Modified FoldingNet.
Figure 2: Overviews of the original FoldingNet and our proposed modifications.
To solve the challenge of permutation and irregularity in point cloud denoising, we nced a canonical permu- tation II for all unordered point clouds x. FoldingNet (Yang al., 2018) aligns with our goal. The original FoldingNet architecture illustrated in Fig. 2(a), which encompasses permutation-invarian encoder (Qi et al., 2017a) E(x) that outputs 1D latent vector and folding-based decoder D(z, g) that simulates the z-conditioned "deforming forces" acting on the predefined 2D grid g and outputs reconstruction *. Since both and are permutationninvariie invariant to the original point cloud, & also inherits this property. Consoquently, & has a structurod ordering defined by g, cnabling us to treat the folded roconstruction as "particles" suitable for diffusion systems.
4.2 Modified FoldingNet
In our framework, introduce two modifications to the original FoldingNet. Fig. 2 shows overview of the original FoldingNet and our proposed alternative. First, while FoldingNet was originally optimized using Chamfer Distance (CD), we replace it with the Sinkhorn Distance (Cuturi, 2013; Feydy et al., 2019). Since our goal is to use the FoldingNet to find a fixed permutation that reconstructs the inputting point clouds, it can be formulated as an optimal transport problem that aims to transport the source unordered points to the target reordered reconstructions, and the Sinkhorn Distance is a differentiable solution. In Appendix A.2, provide a thorough theoretical analysis the optimal transport interpretation of FoldingNet. Second, incorporate attention block following Wen et al. (2020) to improve the reconstruction quality without breaking the permutation-invarian property. As illustrated in Fig. 2(b), the queries are generated by a multilayer perceptron (MLP) that processes the 2D coordinates concatenated with the 1D max-pooled feature. Subsequently, the attention module automatically selects the most informative encoded point-wise features corresponding to these queries, thereby establishing mapping-a permutation -orttt an input point to a specific position on the 2D grid. This process yields integrated features that guide the decoder in producing more precise reconstructions, while the decoder in the original design (Fig. 2(a)) relies solely

---

Published in Transactions on Machine Learning Research (08/2025)
Fold
Diff
Option
Sec. 4.3.1
Global
UNet Denoising Implicit Prior
Folding
Diffusion Sampling
reordered
Sec. 4.3.2
Unet Diffusion
Option
irregular
Local
Sec: 4,3.3
Folding
DIT Diffusion
Figure 3: An overview of the proposed FoldDiff framework. Folding operation provides a reordered recon- struction that is permutation invariant to the imput. The resulting structured global or local representation can be paired with various 2D diffusion techniques and enables efficient point cloud diffusion sampling with- out the need for voxelization.
on the global feature. Given that the attention mechanism is permutation-equivaattt with respect to the queries, and considering that queries are permutation-invariaat features, the permutation-invariaut of the reordered reconstruction is preserved. In Appendix A.2, we provide the training details of our local FoldingNet, and empirically justify the above modifications with experiments.
While the modificd NoldingNet demonstrated Fig. is sufficient for locally folded tokens, we use a hierarchical FoldingNet for globally folded objects. The archilecture is inspired by PointNet++ (Qi et al., 2017b) and the Laplacian Pyramid, and the details are covered in Appendix A.3. The resulting reconstruction is then reshaped into H W geometry image (GI), where each pixel encodes the (I, coordinates of the reconstructed geometry. The GI is then naturally compatible with 2D UNet denoiser and UNet-DDPMs trained with l2 noise residuals.
4.3 The FoldDiff framework
To address the irregularity in point clouds for diffusion models without incurring the high computational complexities and quantization errors of voxelization- based approaches, we propose FoldDiff (demonstrated in Fig. 3), which is a two-stages pipeline that first trains a folding-based autoencoder and then performs denoising diffusion on folded point clouds. We explore three variants of our framework: (1) 2D UNet denoising implicit priors paired with globally folded objects; (2) 2D UNet-based DDPMs paired with globally folded objects; and (3) DiT-based DDPMs paired with locally folded tokens. Here we define our naming schemes for better clarity and consistency:
Globally folded objects srrectssppintt pponnt clouds refers to the structured global И x W point cloud recon- structed by a hicrarchical FoldingNet with the architocture discussed Appendix A.3. These global representations are primarily paired with our UNet-based methods for 3D point cloud generation.
Locally folded tokens/patchnn refers to the structured local k x k point cloud reconstructed by a shared lightweight FoldingNet with the architecture discussed in Sec. 4.2. These token represen- lations are primarily paired with our DiT-based methods for 3D point cloud generation.
Sampling folded objects from UNet implicit priors. Modeling the manifold of reordered reconstruc- tions instead of unordered inputs circumvents the challenges posed by permulation matrices as shown in Proposition The globally folded reconstructions can be trivially reshaped into an image-like representa- tion (i.c., Geometry Image), making them compatible with 2D CNN-based denoiscrs. As demonstrated Equation 7, the predicted noise residual gives direction (i.c., score estimation) toward the clean manifold, thereby providing access to the density p(x). A general denoiser be considered score approximator
8

---

Published in Transactions on Machine Learning Research (08/2025)
adaptive to different noise levels. It's connection to denoising diffusion provides us proof-of-concept algo- rithm to illustrate the necessity of structured permutation. Following Kadkhodaie & Simoncelli (2021), we trained 2D CNN UNet denoiser (Ronneberger et al., 2014) as our implicit prior. Architectural details of the UNet are covered in Appendix A.3. The training objective is the Iz-norm between the denoised residuals and the ground truth additive Gaussian noise on each reordered point. To sample high-probability objects from a denoiser, we follow the Langevin-style stochastic gradient ascent algorithm in Kadkhodaie & Simoncelli (2021). A detailed explanation of the sampling algorithm is included in Appendix A.4.
Extending this approach, we perform Denoising Diffusion Probabilistic Models (DDPMs) sampling using noise-conditioned UNet denoisers trained on globally folded point clouds. Unlike general denoisers, noise-conditioned UNets are specifically trained to predict the score at fixed noise levels rather than adapting to a wide range of noise levels. This specialization allows for more accurate score estimation, enhancing the sampling quality and diversity.
Sampling folded objects from Unet-based DDPMs.
In DDPMs, the forward diffusion process corrupts data Xo by adding Gaussian noise over T' timesttps, forming latent variables X Xy. The joint distribution of the forward process is defined as ((x:r(X))
II ((xx/xxxxx/ (x)),, where (xx,|xxx1) = N(xx Vaix,-1, - -)) With the reparametrization trick, samples X ((x]|xx-)) can be rewritten as vāixo + vI aico, where eo ~ (N, The reverse process learns a score estimation network pe (xx-1|xx) (x)) such that po(Xo.T) = p(x)) I[== po |xx), where p(xr) ~ (0, I). The parameters 0 are optimized to minimize the evidence lower bound (ELBO) on a negative log-likel of xo under pe (x):r). The ELBO then reduces to minimizing the lg-norm between the ground truth noise and the predicted noise up to irrelevant constant multiplier, which coincides with the denoising score matching results introduced in Sec 3.
Sampling from DiT-based DDPMs with folded tokens. If UNet based diffusion predicts the dy- namics of "a particle" (i.e., an image or a point cloud), Diffusion Transformer (DiT) (Peebles & Xie, 2023) predicts the dynamics of "a particle group" (i.e., a group of tokens in an image or a point cloud). Here, an indestructible particle is no longer the entire object, but the tokens. In this case, the movement direction (i..., probability gradient) of each "particle" (i.e., each token) is dependent not only on noise levels, but also the interactions within "particle group" (i.c., group of tokens), and such interactions are captured by the transformer (Vaswani et al., 2017).
The original DIT for images consists of two main components. The first is a pretrained tokenizer, which compresses the high-rosolution image x into a lower-resolution latent image z. This latent image is then divided into patch embeddings in the raster order. The second component is the latent diffusion trans- former, which replaces the UNet in DDPMs. In the latent diffusion transformer, the 12 noise on each token is predicted conditioned on token interactions captured by self-attention. The original DiT added positional embeddings to the image patch embeddings to retain positional information.
We dub our DiT as DiT-fold, where the tokenizer is a lightweight FoldingNet as described in Sec. 4.2, and the folded reconstructions serve as our tokens. We applied sinusoidal positional embeddings on the barycenter of each folded token, allowing the transformer to capture the spatial relationship between tokens. Additionally, without the constraints of voxel grids, our approach allows for any integer mumber of token length L, with each token a k x & local gcometry image. We considered choices such that LX² x 2N for objects of N points. This design offers greater flexibility in sequence length compared to DłT architectures based on voxelization such as Mo et al. (2023), which is restricted to voxel dimensions V E {16, (1,,32,0)) 32, 64), and patch widths p E {2, 8)) resulting in token lengths L = (V/p)² E {22, 43, 83, ,2,2). In their largest configuration (V 64,p 2), the token count reaches 32,768 significantly exceeding the number of points (N 2048) in the ssperiments-leadiine FoldDiff provides a more compact and efficient alternative to point-voxel representations for point cloud diffusion, despite some computational overhead for training a lightweight folding lokenizer. Such overhead can be minimized by training a universal local patch tokenizer applicable across all object categories. A well- optimized, lightweight tokenizer can then be directly paired with DiTs of any size to model diverse target distributions efficiently. Aside from the above modifications, our implementations align with the original DiT (Peebles & Xie, 2023).
to highly inefficient training and inference. Our proposed
9

---

Published in Transactions on Machine Learning Research (08/2025)
Figure 4: Single-class point cloud gener- ation for airplanes, cars, and chairs. The proposed DiT-fold avoids the quantiza- tion error introduced from voxelization, thus generales smoother shapes.
PVD
LION
DiT-3D
DiT-fold
For fair comparisons, we employ the standard DDPM for both voxel- baseebbaed and our folding- based method- ologies. However, our proposed FoldDiff framework is also compatible with flow-based difffusion training and sampling techniques, evidenced by ablation studies.
5
Experiments
5.1 Experimental setup
Datasets. We compare the performance of different methods on single-category 3D shape generation using ShapeNet (Chang et al., 2015) chairs, cars, and airplanes as primary datasets. We use the same dataset splits of previous works (Luo & Hu, 2021b; Yang al., 2019; Zhou et al., 2021; Zeng et al., 2022; Mo et al., 2023). Throughout our experiments, each object contains 2048 uniformly sampled points. During evaluation, both the generated shapes and the reference shapes are inversoly transformed with the global mcan and variance of the training set.
Implementation details. All models trained with PyTorch. UNet implicit priors and DDPMs are trained globally folded objects. DiT-based DDPMs use the locally folded tokens with shared lightweight FoldingNet. (sce Sec. 4.2). Model details are covered in Appendix A.3.
Evaluation metrics. Following previous works (Luo & Hu, 2021b; Yang et al., 2019; Zhou et al., 2021; Zeng et al., 2022; Mo et al., 2023), we adopt coverage score (COV) and 1-Nearost- classifier Accuracy (I-NNA) to quantitatively evaluate the sampling quality and diversity of different methods. Specifically, 1-NNA evaluates whether the generated distribution is identical to the reference distribution (e.g. test-set distribution). Closer to 50% implies better similarity between distributions, thus capturing both the sampling quality and diversity. COV calculates the proportion of shapes from the reference dataset to be matched with at. least one generated shape. Higher COV implies better diversity. 1-NNAs and COVs are computed from the full pair-wise distance matrix of elements from both test set and the sampled set. Chamfer Distance (CD) and Earth Mover's Distance (EMD) were used our distance metrics to compute 1-NNA and COV, Yang et al. (2019) provided more detailed discussion over different evaluation metrics.
5.2 3D shape generation
We evaluate the generative performance and efficiency of DiT-DDPM-hased FoldDiff against previous point cloud generative frameworks, as demonstrated in Table We compared previous works with similar Gflops (Yang et al., 2019; Kim et al., 2020; 2021; Klokov et al., 2020: Luo & Hu, 2021b; Zhou et al., 2021; Mo et al., 2023). Among them, voxnt-voxel architectures (Zhou et al., 2021; Zeng et al., 2022; Mo et al., 2023) demonstrate the most competitive gencrative performance, with DiT-3D-XI, (Mo et al., 2023) recently achieving stale-of-the-art results. Since LION (Zeng el al., 2022) has Gflops of 247.38 and (6T32) in Mo et al. (2023) has Gflops of 1.12 10*, they are omitted in our experiments for fair comparison. The point-voxel representation provides a structured space for point cloud diffusion, yet suffers from a cubically scaling complexity and quantization errors. In contrast, our folded representation avoids the above restrictions. As illustrated in Fig. 4, framework reduces quantization noise, producing smoother point
10

---

Published in Transactions on Machine Learning Research (08/2025)
Table 1: Comparison results (%) on shape metrics of our DiT-fold and baseline models. (*) denotes re- implemented performances on 2048 uniformly sampled points. Here we considered the methods with Gftops
from 0 100. We report the average performance of generative runs.
Chaie
Airplane
Car
Method
Gfoops
I-NNA (1)
COV
I-NNA (1) EMD
COV
I-NNA (J)
COV (1)
CD
EMD
CD
EMD
CD
CD
EMD
CD
EMD
CD
EMD
[-GAN Achlioptas et al. (2017)
83.00
99.70
24.27
15.13
96.
96.79
30:12
14.32
94.46
99.01
19:03
6.539
I-GAN (CD) Achlioptas et al. (2017)
68.58
29.31
87.30
93.95
38.52
21.23
66.49
38.92
I-GAN (EMD; Achlioptas et ol. (2017)
71.90
64.65
38.07
44.86
89.49
76.91
36.27
36.52
71.16
66.19
37.78
45.17
PointPlow Yang et ul. (2019)
2.10
62.84
60.57
42 .0
50.00
75.68
70.74
47. 47.90
46.41
58.10
56.25
48.58
50.00
SoftFlow Kim
(2021))
59.21
60.05
11.39
76.05
46.91
61.77
SetVAE Kim et al. (2021)
1.61
58.84
60.57
46.83
44.26
76.54
67 65
43.770
48.400
59 94
59.94
49.15
46. 59
DeF-Not Klokov et al. (2020)
3.02
62.00
58.53
44.71
48.79
75.18
65.55
46.17
48.89
62:35
54.48
45.74
49.43
DPM Luo
(2021b)
60.05
74.77
76.42
68.89
PVD Zhou es al. (2021)
81.12
57.09
60.87
36.68
19.24
73.82
61.81
18.98
62.09
54.55
53.83
41.19
50. 56
PVD Zhon ot al. (2021)
81.12
56.85
55.91
46.68
51.41
73.57
67.97
47.22
48.20
57.79
53.07
47.51
53.30
DIT-ID-S(32/ /4) Mo al. (2023)
21.95
56.31
55.82
47:21
DIT-SD-B(32/1) Mo et al. (2023)
87.42
55.59
51.91
50.09
52.80
DIT-3D-S(32/ /4]* Mo et al. (2023)
21.95
59.19
55.82
44.96
51.16
78.57
67 .66
45.57
47.78
63.90
61.65
40.06
50.28
D (256/16)) (Ours)
12.80
55.48
53.56
48.03
72.19
66.06
47.28
51.11
58.74
52.26
48.53
53.79
DIT- (5122/))) (Ours)
23.92
55.06
53.25
26.83
53.78
67.24
61.78
50.99
51.97
56.63
52.85
47.73
52:40
clouds than PVD (Zhou et al., 2021), LION (Zeng et al., 2022). and DTT-3D (Mo et al., 2023). Moreover, we observed a generalized generative behavior as covered in Appendix A.5 by comparing generated shape to its nearest neighbors in the training set.
Due to constraints on training budget, we compare the point-voxel framework with our FoldDiff framework under the same model size: DiT-S. We denote different configurations of DiT-fold as DiT-fold(L/k3), where L stands for token length, and each token is A x & local geometry image. We denote different configurations of our baseline, DiT-3D (Mo et al., 2023), as DiT-3D(V/H)) where V is the voxel dimension, and p is the patch width. We re-implement point-voxel diffusion methods with similar Gflops including PVD and DiT- 3D-S for a more comprehensive comparison in ShapeNet (Chang et al., 2015) chairs, cars, and airplanes, while the results for other methods are reported as in their original paper. For fairness, we report both the available published performance metrics and re-implemented results (denoted with *).
In our default configuration, we use DiT-S with 256 tokens, each containing 16 folded points, which corre- sponds to 4 x 4 local geometry images. During training, each token is captured from 16 nearest neighbors of 256 farthest sampled centers. During sampling, tokens are directly inferred and contains 3D position of points. We train the DiT-S with folded tokens for 10,000 epochs with AdamW optimizers using a learning rale of 2 x 10-4, Following DiT (Peebles & Xie, 2023), we maintain an exponential ial moving average (EMA) of model weights over training with a decay of 0.9999 and the EMA weights were used during sampling for evaluation. This technique was also used in ro-implemented DiT-3D.
As shown in Table 1, with similar lower computational costs, our default configuration DiT-fold-S demonstrates superior performance over traditional methods (Yang al., 2019; Kim et al., 2020; 2021; Klokov et al., 2020; Luo & Hu, 2021b). More importantly, DiT-fold-S(256/11) surpasses the performances of point- voxel methods, PVD (Zhou et al., 2021) and DiT-3D-S (Mo et al., 2023), with equivalent or less Gfiops. In particular, DiT-fold-S(256/11) also achieves on-par performance on ShapeNet chairs with the reported DiT-3D-B(33) only 1/8 of the Gfiops and smaller model. We further boost the performance of DiT-fold by scaling to 512 tokens. Due to restrictions on our training budget, we didn't scale our models to DiT-B, DiT-L, and DiT-XL. Nevertheless, FoldDiff demonstrates its superiority over the point-voxel diffusion framework. Please refer to Appendix A. for more visualizations.
outperforms re-implemented DiT-3D-S(32/4) with only 1/2 of the Gflops. It
performance with
5.3 Implicit priors and DDPMs
Here we demonstrate the necessity of a structured space in both denoising implicit priors and DDPMs, also showcasing the flexibility of FoldDiff as general framework.
11

---

Published in Transactions on Machine Learning Research (08/2025)
Table 2: Generative performances of three FoldDiff variants using denoising implicit priors (DIPs) and denoising diffusion probabilistic models (DDPMs) with structured ShapeNet objects. UNet-based methods use globally folded reconstructions for training and sampling, while DiT-based methods use locally folded roconstructions.
Chair
Airplane
Car
Architecture
Method
1-NNA (L)
COV
(t)
1-NNA (L)
COV
(1)
1-NNA (4)
COV
(t)
CD
EMD
CD
EMD
CD
EMD
CD
EMD
CD
EMD
CD
EMD
2D UNet 2D UNet DIT-fold
DIP
85,73
87.24
28.85
28.40
96.13
96.09
20.82
22.63
90.62
92.14
34.09
36.93
DDPM
59.29
62.54
42.13
45.92
84.44
85.31
39.51
38.52
67.33
70.03
42.05
40.91
DDPM
55.48
53.56
48.03
51.69
72.19
66.06
47.28
51.11
58.74
52.26
48.53
53.79
Table 3: Ablation studies. MMD-CD is multiplied with 1 x 10-3, MMD-EMD is multiplied with x 10-2.
Fubling Condigs
I-NNA(())
COVT(N)
MXDX
Couliee
1-NNAJ(9]
COVT(5)
MMD1
takm source
attcation
Joss
CD
F3D
CD
EMD
CD
EMD
Meshod
EMA
EMD
EMD
EMD
Chais
Chair
CD
CD
56.35
56.38
54.39
35.83
47.71
47.83
51.00
50.45
2.59
2.55
1.45
1.47
D/T-SD-S(32/4)
X
08.35
69.18
39.73
00.33
2.90
1.60
Airplane,
D/T-3D-S(32/1
50.19
55.82
11.96
51.16
2.51
1.19
EMD
56.22
54.76
48.19
53.47
DDT-tald-S(256/10)
x
56.85
55.20
45.
50.60
2.62
1.48
Chair, Airplaze, Car
Stikhoon
55.48
53.56
48.03
31.69
2.51
1.44
D/T-66ld-S(258/11))
55.48
53.56
48.03
51.69
2.51
(a) Ablating Nets.
(b) Ablating EMA.
Reproduced failures in unstructured spaces. In Sec. 3, we demonstrated that the thcoretical foun- dation of denoising implicit, priors fails with unknown permutations. It supports the failures reported by Zhou et al. (2021) and Mo et al. (2023) when they perform diffusion unstructured point clouds. Here we reproduce these empirical results using object-level point cloud denoisers or diffusion models. In total, we experiment on four different options in the unstructured space: (1) PointNet denoising implicit prior, (2) PointNet++ denoising implicit prior, (3) PointNet+++ DDPM as reported by Zhou et al. (2021), and (4) DiT-raw DDPM as reported by Mo et al. (2023). Implementation details are covered in Appendix A.3. For the denoising implicit prior methods, the training loss is the surface proximily (Equation 5). For the DDPM methods, the training loss is the mean squared error (MSE) between the predicted clean point clouds and the ground truth clean point clouds. Langevin stochastic gradient ascent fails to generate any visible shape from the denoising prior trained unstructured point clouds also fail to generate any visible shape, agreeing with the empirical findings in Zhou et al. (2021) and Mo et al. (2023).
such unstructured space of point clouds. Similarly, DDPM methods trained
Implicit priors and DDPMs with FoldDiff. The FoldDiff framework offers a structured space for denoising implicit priors. We prepared dataset of folded objects folded token groups from ShapeNet Chang et al. (2015) chairs, airplanes, and cars. In total, we experiment on three different variants of FoldDiff (as described in Sec. 4.3) in the structured domain: (1) UNet denoising implicit prior with globally folded objects; (2) UNet DDPM with globally folded objects; and (3) DiT DDPM with locally folded tokens.
As reported in Table 2, all three variants of FoldDiff are capable of generating authentic shapes, and DDPMs provides better sampling quality and diversity than denoising implicit priors. Among the three variants, DiTs with locally folded tokens produce the most authentic and diverse shapes comparing to UNets with globally folded shapes. The better performance of the local folding paradigm can be explained by it avoiding rediscovering a globally autoencoded manifold learned by the hierarchical FoldingNet. This leaves more freodom for a combinatorial diffusion generation, as DiTs prodict the denoising direction for "particle groups" instead of fixed "particles" of the globally folded structure.
5.4 Ablation studies
Local FoldingNet. Here we ablate our DiT-fold with different local FoldingNets discussed in Sec. 4.2. We evaluate the default DrT-fold-S(256/1) /16) on the chair subset with different FoldingNet configurations. Minimum Matching Distance (MMD) were included as additional metrics to evaluate the quality of generated
12

---

Published in Transactions on Machine Learning Research (08/2025)
Configs
Sampling
I-NNA((%)
COV1(%)
MMD1
Backbone
Model
Speed (s/object)
CD
EMD
CD
EMD
CD
EMD
DiT-fold-S
DDPM
0.31
55.48
53.56
48.03
51.69
2.51
1.44
SiT-fold-S
PF-ODE
0.07
57.10
54.10
47.53
53.47
2.50
1.44
Table 4: The compatibility of FoldDiff with flow-based diffusion.
shapes for experiments. Lower MMD implies better fidelity. The experimental results, summarized in Table 3a, reveal several key insights. As observed, introducing attention mechanisms into the local folding tokenizers is necessary for accurately reconstructing local patches, leading to improved overall performance. Changing the optimization goal from the Chamfer Distance (CD) to the Sinkhorn Distance further boosts the performance, as the Sinkhorn Distance is better at continuously approxinating permutation from diverse set of object categories (e.g..
reconstructions. Moreover, training the local folding tokenizer chairs, airplanes, and cars) enhances both the accuracy and generalization capability of the learned tokens. This diversity proves to be a critical factor in achieving better performance in DiT-fold.
Exponential Moving Average. Exponential Moving Average (EMA) is a widely adopted weight aver- aging technique in deep learning, where an EMA of the raw weights is maintained during training and used for evaluation or inference. This approach typically leads to improved generalization compared to using the raw, last-step weights. On ShapeNet chairs, we examined the impact of this technique. As observed in Table 3b, EMA is crucial for enhancing generative quality and diversity for both the baseline (DiT-3D-S) and our model (DiT-fold-S). Notably, without using FMA weights, our model exhibits more robust performance compared to the point-voxel-based baseline.
Flow-based Diffusion. FoldDiff is compatible with various diffusion model variants. this study, we demonstrate its compatibility with Scalable Interpolant Transformers (SiT)(Ma et al., 2024), recently proposed flow-based diffusion transformer. Flow-matching an emerging variant of generative diffusson models that facilitates faster sampling via probability flow ordinary differential equation (PF-ODE). As presented in Table 4, compare the generative performance of both DiT-fold-S and SiT-fold-S models on ShapeNet chairs. The results indicate that while both models achieve comparable generative quality, the SiT model significantly outperforms in sampling speed. This finding underscores the efficacy of our proposed method when applied to advancod diffusion models such as SiT.
6 Conclusion
In this work, we provide a theoretical explanation the impact of irregularity in point cloud diffusion. To solve this problem, we propose FoldDiff, a novel framework for 3D point cloud diffusion with lincar complexity. By transforming point cloud diffusion from unstructured space to reordered structured space, FoldDiff eliminates the need for voxelization and integrates seamlessly with 2D denoising priors and DDPM-based methods. The framework leverages the folding operation to output a reordered reconstruction of the inputting geometry, enabling direct integration with off-the-shelf 2D denoising implicit priors and DDPM-based methods. This facilitates efficient modeling of the distribution of clean 3D objects and high- probability sampling from the learned distribution. Leveraging folded tokens with Diffusion Transformers, FoldDiff outperforms voxel-based approaches in both performance and efficiency, offering new direction for tokenizers and transformer architectures in point cloud modeling.
Future work includes exploring similar frameworks for meshes, as well as integrating texture information into the generative process. Additionally, our evaluations were conducted under a relatively small model setting due to computational constraints (i.e., DIT-S), and while sufficient to show the power of the proposed franework, mainly lays the groundwork for future rescarch to scale up our results or design optimized DrT variants tailored the folding operation, potentially unifying approaches for 3D generative modeling with 2D methods.
13

---

Published in Transactions on Machine Learning Research (08/2025)
Acknowledgments
Work partially supported by ONR, NSF, Simons Foundation, and gifts/ from Google, Amazon, and Apple.
References
Panos Achlioptas, Olga Diamanti, loannis Mitliagkas, and Leonidas Guibas. Learning Representations and Generative Models for 3D Point Clouds. In International Conference on Machine Learning, 2017.
Angel X. Chang, Thomas A. Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Rice 3D Model Repository. CoRR, abs/1512.03012, 2015.
Guangyan Chen, Meiling Wang, Yi Yang, Kai Yu, Li Yuan, and Yufeng Yuc. PointGPT: Auto-regrossively Generative Pre-training from Point Clouds. Advances in Neural Information Processing Systems, 36, 2023.
Marco Cuturi. Sinkhorn Distances: Lightspeed Computation of Optimal Transport. In C.J. Burges, L. Bot- tou, M. Welling, Z. Ghahramani, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
Dasith de Silva Edirimuni, Xucquan Lu, Zhiwen Shao, Gang Li, Antonio Robles-Kelly, and Ying He. Iter- alivePFN: True Iterative Point Cloud Filtering. 2023 IEEE/CVF Conference Compuler Vision and Pattern Recognition (CVPR), pp. 13530-13333 2023.
Ahmed Elhag, Yuyang Wang, Josh Susskind, and Miguel Angel Bautista Martin. Manifold diffusion fields. In ICLR, 2024.
Slava Elizarov, Ciara Rowles, and Simon Donné. Geometry Image Diffusion: Fast and Data-Efficient Text- to-3D with Image- image-Based Based Surface Representation. In International Conference on Learning Representations met/paff1d-GLnn7kk44777
(ICLR), 2025. URL
Haoqiang Fan, Hao Su, and Lconidas J. Guibas. A point set generation network for 3d object reconstruction from single image. 2017 IEEE Conference on Computer Vision and Pallern Recognition (CVPR), pp. 2463 2471, 2016.
Jean Feydy, Thibault Séjourné, François-Xavins Xavier Vialard, Shun-ichi Amari, Alain Trouve, and Gabriel Peyré, Interpolating between Optimal Transport and MMD using Sinkhorn Divergences. In The 22nd Interna- lional Conference on Arlificial Intelligence and Statislics, pp. 2681-2690, 2019.
Shanghua Gao, Pan Zhou, Mingg-Ming Cheng, and Shuicheng Yan. Masked Diffusion Transformer is a Strong Image Synthesizer. 2023 LEEE/CVF International Conference on Computer Vision (ICCV), pp. 23107-23116, 2023.
Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan C. Russell, and Mathieu Aubry. A Papier- Mache Approach to Learning 3D Surface Generation. 2018 IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 216 224, 2018
Xianfeng Gu, Steven J. Gortler, and Hugues Hoppe. Goometry Images. In Proceedings of the 29th annual conference on Compuler gruphics and interactive techniques, 2002.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In Advonces in Neural 2020.
Information Processing Systems, volume 33, pp.
Zahra Kadkhodaic and Ecro Simoncelli. Stochastic Solutions for Lincar Inverse Problems using the Prior Implicit in a Denoiser. In Advonces in Neural Information Processing Systems, volume 34, pp. 13242- 13254, 2021.
14

---

Published in Transactions on Machine Learning Research (08/2025)
Zahra Kadkhodaie, Florentin Guth, Eero P Simoncelli, and Stéphane Mallat. Generalization in Diffusion Models Arises from Geometry-adaptive Harmonic Representations. In The Twelfth International Confer- ence on Learning Representations, 2024.
Hyoongju Kim, Hyconseung Lee, Woo Hyun Kang, Joun Yeop Lee, and Nam Soo Kim. Flow: Proba- bilistic Framework for Normalizing Flow on Manifolds. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Newral Information Processing Systems, volume 33, pp. 16388- 16397. Curran Associates, Inc., 2020.
Jinwoo Kim, Jae Hyeon Yoo, Juho Lee, and Seunghoon Hong. SetV Learning Hierarchical Composition for Generative Modeling of Sct-Structurod Data. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15054-15063, 2021.
Roman Klokov, Edmond Boyer, and Jakob J. Verbeek. Discrete Point Flow Networks for Efficient Point Cloud Generation. In European Conference on Computer Vision, 2020.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben- Ben-Hamu, Maximilian Nickel, and Matt Le. Flow Matching for Generative Modeling. ArXiv, abs/2210.027477, 2022.
Qi Liu, Miltiadis Allamanis, Mare Brockschmidt, and Alexander L. Gaunt. Constrained Graph Variational Autoencoders for Molecule Design. In Newral Information Processing Systems, 2018.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. ArXiv, abs/2209.0300 03300, 2022.
Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han. Point-Voxel CNN for Efficient 3D Deep Learning. In Advances in Newral Information Processing Systems, volume 32, 2019.
Shitong Luo and Wei Hu. Differentiable Manifold Reconstruction for Point Cloud Denoising. Proceedings of the 28th ACM International Conference on Multimedia, 2020.
Shitong Luo and Wei Hu. Scsre-Based Point Cloud Denoising. 2021 IEEE/CVE International Conference on Computer Vision (ICCV), pp. 4563-4572, 2021a.
Shitong Luo and Wei Hu. Diffusion Probabilistic Models for 3D Point Cloud Generation. In Proceedings of the IEEE/CC IEEE/CVE Conference Computer Vision and Pattern Recognition (CVPR), June 2021b.
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M. Bolfi, Eric Vanden-Eijnden, and Saining Xie. SIT: Exploring Flow and biffusion-based Generative Models with Scalable Interpolant Transformers. In European Conference on Computer Vision, 2024.
Haggai Maron, Meirav Galun, Noam Aigerman, M. Trope, Nadav Dym, Ersin Yuner, Vladimir G. Kim, and Yaron Lipman. Convolutional Neural Networks on Surfaces via Seamless Toric Covers. ACM Transactions on Graphics (TOG), 36:1 - 10, 2017.
Tomohide Masuda, Matthew Ragoza, and David Ryan Koes. Generating 3D Molecular Structures Condi- tional Receptor Binding Site with Deep Generative Models. ArXiu, 022/0.2111444/ 2020.
Koichi Miyasawa. An Empirical Bayes Estimator of the Mean of a Normal Population. Bull. Inst. Internat. Statist,, 38:181-188, 1961.
Shentong Mo, Enze Xie, Ruihang Chu, Lanqing Hong, Matthias Niessner, and Zhenguo Li. DIT-3D: Ex- ploring Plain Diffusion Transformers for 3D Shape Generation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 67960-67971. Curran Associates, Inc., 2023.
Jiahao Pang, Duanshun Li, and Dong Tian. TearingNet: Point Cloud Autoencoder Learn Topology- Friendly Representations. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognation (CVPR), pp. 7449 74589,7788801 2020.
15

---

Published in Transactions on Machine Learning Research (08/2025)
William Peebles and Saining Xic. Scalable Diffusion Models with Transformers. In Proceedings of the October 2023.
IEEE/CVP International Conference on Compuler Vision (ICCV), pp.
Yong Peng, Linlin Tang, Qing Liao, Liu Yang. Shuhan Qi, and Jiajia Zhang. Diffusion Model for 3D Point Cloud Generation. International Conference on Genetic and Evolutionary Computing, Jan 2024. doi: 10.1007/9788 97778888-777006----111 48.
An Equivariant
Charles R. Qi Hao Su, Mo Kaichun, and Lconidas J. Guibas. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. In 2017 IEEE Conference on Computer Vision and Pattern Recogrétion (CVPR), pp. 77-85, 2017a. doi: 10.1109 /t/C//VVR........
Charles R. Qi, Li Yi, Hao Su, and Leonidas Guibas. Noi Deep Hierarchical Feature Learning on Point Sets in a Metric Space. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 5105-5114, 2017b.
Marie-Julie Rakotosaona, Vittorio La Barbera, Paul Guerrero, Niloy J Mitra, and Maks Ovsjanikov. POINT- CLEANNET: Learning to Denoise and Remove Outliers from Dense Point Clouds. In Computer Graphics Forum, volume 39, pp. 185 203, 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. Medical Image Compuling and Compuler-Assisled Intervention MICCAI, pp. 234 241, 2014. doi: 10.1007 10.1007/978-23 091999988-399-457744
28.
Riccardo Roveri, A. Cengiz Outireli, Toana Pandele, and Markus H. Gross. PointProNetts Consolidation of Point Clouds with Convolutional Neural Networks. Computer Graphics Forum, 37, 2018.
Yossi Rubner, Carlo Tomasi, and Leonidas J. Guibas. The earth mover's distance metric for image retrieval. International Journal of Computer Vision, 40:99 121, 2000.
Chence Shi, Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. GraphAF: a Flow- based Autoregressive Model for Molecular Graph Generation. ArXiv, 0os/2201.09335 2020.
J. Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3D Neural Field Generation Using Triplane Diffusion. In Proceedings of the IEEE/CVI Conference on Computer Vision and Pattern Recognétion (CVPR), pp.
June 2023.
Ayan Sinha, Jing Bai, and Karthik Ramani. Deep Learning 3D Shape Surfaces Using Geometry Images. European Conference on Computer Vision, 2016.
Ayan Sinha, Asim Unmesh, Qi-Xing Huang, and Karthik Ramani. SurINet: Generating 3D Shape Surfaces Using Deep Residual Networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 791 800, 2017.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Unsupervised Learning using Nonequilibriun Thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pp. 2256-2265, 2015.
Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribution. In Advances in Newral Information Processing Systems, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations. In 9th International Confer- ence on Learning Representations, ICLR 2021, 2021.
Yang Song, Prafulla Dhariwal, Mark Chen, and llya Sutskever. Consistency Models. In International Conference Machine Learning, 2023.
Ashish Vaswani, Noam M. Shazcer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, 2017.
16

---

Published in Transactions on Machine Learning Research (08/2025)
Clement Vignac, Igor Krawczuk, Antoine Sirandin, Bohan Wang, Volkan Cevher, and Pascal Frossard. Digress: Discrete denoising diffusion for graph generation. In The Eleventh International Conference on Learning Representations, 2023.
Pascal Vincent. A Connection Between Score Matching and Denoising Autoencoders. Newral Compulation, 23:1661-1677 1674, 2011.
Pascal Vincent, H. Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and Composing Robust Features with Denoising Autoencoders. In International Conference on Machine Learning. 2008.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. Dynamic Graph CNN for Learning on Point Clouds. ACM Transactions on Graphics (TOG), 38:1 12, 2018.
Xin Wen, Tianyang Li, Zhizhong Han, and Yu-Shen Liu. Point Cloud Completion by Skip-Attention Network With Hierarchical Folding. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognation (CVPR), pp. 1936 1945, 2020.
Weilai Xiang, Hongyu Yang, Di Huang, and Yunhong Wang. Denoising diffusion autoencoders are unified self-supervised learners. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp.
15756-15766, 2023.
Rui Xu, Jiepeng Wang, Hao Pan, Yang Liu, Xin Tong, Shiqing Xin, Changhe Tu, Taku Komura, and Wenping Wang. Combostoc: Combinatorial stochasticity for diffusion generative models, 2024. URL
httpa://arxavv.rr///ba//200/
Xingguang Yan, Han-Hung Lee, Ziyu Wan, and Angel X. Chang. An Object Worth 64x64 Pixels: Gener- ating 3D Object via Image Diffusion, 2024. URL https:://axxie http://axxxivoogrr////aa/e 03178.
Guandao Yang, Xun Huang, Zekun Hao, Ming- Yu Liu, Serge J. Belongie, and Bharath Hariharan. Point- Flow: 3D Point Cloud Generation With Continuous Normalizing Flows. 2019 IEEE/CVF International Conference Computer Vision (ICCV), pp. 4540 4549, 2019.
Ling Yang, Haotian Qian, Zhilong Zhang, Jingwei Liu, and Bin Cui. Guidded Adversarial Training of Diffusion Models. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7256-7266, 2024
Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. Foldingnet: Point Cloud Auto-encoder via Deep Grid Deformation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 206 215, 2018.
Jason Yim, Brian L. Trippe, Valentin De Bortoli, Emile Mathicu, Arnaud Doucet, Regina Barzilay, and Tommi Jaakkola. SE(3) Diffusson Model with Application to Protein Backbone Generation. In Proceedings of the 40th International Conference on Machine Learning, ICML 23. JMLR.org, 2023.
Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. BiittBBRRT: Pre-Training 3D Point Cloud Transformers with Masked Point Modeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognátion (CVPR), 2022.
Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, and Karsten Kreis. LION: Latent Point Diffusion Models for 3D Shape Generation. ArXio, abs/22 hts/2210.06975 2022.
Qijian Zhang, Junhui Hou, Yu Qian, Yiming Zeng, Juyong Zhang, and Ying Hc. Flattening-Net Deep Regular 2D Representation for 3D Point Cloud Analysis. IEEE Transactions on Pallern Amalysis and Machine Intelligence, 14:9726-97422 2022.
Xiao Zheng, Xiaoshui Huang, Guofeng Mei, Yuenan Hou, Zhaoyang Lyu, Bo Dai, Wanli Ouyang, and Yongshun Gong. Point Cloud Pre- TreTrrinn with Diffusion Models. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognátion (CVPR), pp. 20935-22945, 2023.
Linqi Zhou, Yilun Du, and Jiajun Wu. 3D Shape Generation and Completion through Point Voxel Diffusion. 2021 IEEE/CVT International Conference Computer Vision (ICCV), pp. 5806 5016-581, 2021.
17

---

Published in Transactions on Machine Learning Research (08/2025)
A Appendix
A.1 Proof of Proposition 1
In Sec. 3.2 we explained why gradient denoising in Equation cannot be naively extended to unstructured data. This is formalized through our Proposition 1 below, which implies that Equation holds only if we pick permutation the point set that exactly equal to a true underlying structure. For unstructured data, such permutation matrix non-retricvable, thus Equation no longer holds. In this appendix we include the proof of this proposition.
Proposition 1. The denoiser residual f(y) y is proportional to Vy log P(y) if and only if INXN.
Proof. Starting from Equation 10, we can write
VyP P) = -0² ( (x- - -
=03 / (IIx y) Ply,x|IT)ds
where we use x = Ilx and replace g(y - IIx) with P(y|x, II) to obtain the second row. Dividing by 0 P(yl[t gives
o² Vy log ITy IxP(xxly IIdx / yPxxly, IT)x,
II& -I
where in the first row we used Vy log IIy = Plylm gives the estimator xy))
Plylt) and in the second row we used Equation 11. This
x II + o'V, log P(y
(12)
Therefore, we evaluate how much f(y) & y differs from being proportional to Vy log P(y
fly) o2 log Ply (I x(y),,,
which is 0 if and only if - I.
A.2 Local Net
An optimal transport interpretation of folding reordering. For better understandings of our proposed folding reordering, we interpret it optimal transport problem that aims to transport the source unordered points to the target reordered reconstructions. Formally, the optimization goal can be written as
min II (0))
Σ1x: - -Ixx001,,000111 1.J
(13)
where II the global matching from the source input x to the target reconstruction &, and are the FoldingNet. parameters. One may notice that the objective exactly an Earth Moving Distance (EMD) (Rubner et al., 2000), which provides an explicit transport plan from the source to the target. However, EMD is not everywhere differentiable. When multiple permutations of the reconstruction x(0) give the same optimal transport cost, the gradient over 0 is not unique and thus not well-defined. Even though EMD may offer the best reconstruction results, the permutation implicitly learned with 0 may not be an ideal structurod representation for diffusion (see Table 5). One differentiable approxination to EMD is the Chamfer Distance (CD), which is the original loss function for FoldingNet (Yang et al., 2018). CD is still favored due to its better efficiency, as it does not explicitly compute an optimal transport plan, but approximates one via nearest neighbor matching. However, as pseudo-distance (Fan et al., 2016), CD produces less faithful point cloud reconstructions than EMD (Achlioptas et al., 2017).
18

---

Published in Transactions on Machine Learning Research (08/2025)
Table 5: Net reconstruction performance on small local patches of 16 points with different modifica- tions and the generative performance (L-NNA) on ShapeNet chairs.
Attention
Gflops
loss
CD(10
EMD(10
Laplacian(10
1-NNA-CD
1-NNA-EMD
x 1.6799
CD
3.0586
8.8583
4.0846
57.36
55.83
/
1.8716
CD
1.8379
5.7414
4.1840
55.93
54.15
/
1.8716
EMD
1.0152
1.0904
6.0602
56.22
54.76
/
1.8716
Sinkhorn
1.4429
3.3540
5.0698
55.48
53.56
Due to the above limitations of CD and EMD, we pivoted to the Sinkhorn distance (Cuturi, 2013; Feydy et al., 2019), which is entropy smoothed differentiable approximation of EMD,
1 Σ1x - - (I))
min ((x)
(14)
where HIII) - Mhe log Mi is the entropy of the transport plan II and À 2 0 is the regularization strength. With the entropy regularization introduced, Equation 14 is strictly convex and has a unique minimizer. In practice, we empirically obscrved that the Sinkhorn distance performs the best in balancing local patch reconstructions and the final generative performance (see Table 5).
Training details and experiments. To validate our introduced modifications to the local NoldingNet, we report the reconstruction errors in Chamfer Distances (CD) and Farth Moving Distances (FMD) on small local patches. We also compute the Laplacian of local folded patches to measure the smoothness of the local geometry image (GI) representation. Training local patches are extracted from chairs, cars, and airplanes in the ShapeNet dataset, where each object contains 2048 uniformly sampled points via. farthest point sampling. We also report the generative performance, in I-Nearest Neighbor Accuracy (1-NNA) computed with CD and EMD. In our default setting, each local patch resembles a 4 x 4 geometry image that aggregates 16 nearest neighbors of randomly chosen center point. The models were trained for 1,000 epochs with a batch size of 65,536 and a learning rate annealing from x 10 to 2 x 10-6 with a cosine schodule. The experimental results in Table demonstrates that our modificatt ions balance the reconstruction quality of folded local patches and boosts the final generative performances with only a 0.2 Gflops overhead.
ShapeNet chairs, of different modifications
A.3 Implementation details
Here cover the implementation details of each model used for experiments. We will discuss important components for each model. For more details, please refer to our public repo
Architectural details of unstructured PointNet denoiser. We use a PointNet-based denoiser for denoising implicit priors experiments in unstructured spaces as discussed in Sec. 5.3. The network design follows the eegmentatioon-vriiant of PointNet (Qi et al., 2017a). The encoder is composed of MLP layers with an input dimension of 3 (spatial coordinates) and output dimensions of [64, 64, 64, 128, 512]. The x 512-dimensional point-wise features then maxpooled over the N-points to achieve a 512-dimensional 1D vector. The ID vector then concatenated with the local feature at dimension 128, creating a global-local mixture of point-wise latent. code in N (512 + 128). The point-wise latent code then fed into a decoder of output dimensions [256, 128, 64, 3] predict the point-wise denoising direction. The loss is computed with the mean-squared-orro between the predicted denoising direction and the displacement from the noisy point to the nearest clean point.
Architectural details of hierarchical FoldingNet. This section continues Sec. 4.2 provide more architectural and training details of the proposed hierarchical folding autoencoder. We first adopt stack of
19

---

Published in Transactions on Machine Learning Research (08/2025)
Set Abstraction modules proposed in PointNet++ (Qi et al., 2017b) to encode information in different lev- els of resolution. After three Set Abstraction operations, acquire 3 sets of points in decreasing resolutions N2,N2 and their aggregated features N1 x C1 x C2,NN x C3. During decoding, the first folding block will reconstruct N3 points as our base geometry image (GI) Tu- The last-layer features before reconstruction are fed forward. Then we expand base GT Po to the same resolution as N2, and use the second folding block that outputs a difference image to add on Up(To) to achieve a higher resolution GI JT. We iterate this process in our decoder until we get GI with the same resolution as our inputting point cloud. In our implementation, N1 2048, N₂ 512, and N₂- 256. The corresponding resolution of GIs are To (16, 16), I = (32, 16), and 12- (64, 32). Set Abstraction blocks and Folding blocks follow the same design in SA-Net Wen et al. (2020). Note that in our architecture, we follow SA-Net to apply cross-attention layers Vaswani et al. (2017) analogously as skip-connections Ronneberger et al. (2014). This design preserves permutation invariance of resulting GIs, as the queries of the cross-attention layers are permutation-invarian features extracted by the encoder. Note that we train seperate hicrarchical foldingnet for cach category. This is less efficient than the DiT case where a lightweight folding lokenizer can be trained on multi-class objects as a generalized tokenizer and be used for different DiTs.
Architectural details of 2D UNet. This section continues Soc. 4 to provide more architectural and training details of the UNet (Ronneberger et al., 2014) model we applied to model denoiser implicit pri- ors. The UNet contains pairs of downsampingg-upamphn 256], each pair connected by skip connections. Each module contains convolutional layers with residual connections. The same architecture is used in all experiments for fair comparison. We first construct GI dataset converted from point clouds that contains 2048 points farthest sampled from ShapeNet (Chang el al., 2015) single-class objects, which further normalized into a bounding-box of [-1,1] across the dataset. In Kadkhodaie & Simoncelli (2021), the UNet is trained with Gaussian noise with standard deviations drawn from a ~ U[0, 0.4] (relative to image intensity range [0, 1]). Thus, given our GIs have an intensity range [],,1], we train the UNet with absolute noise intensity a ~ 0.8).
modules with feature dimensions [32, 64, 128,
The UNet used in the diffusion model have similar feature dimensions and architecture, except each layer is conditioned on diffusion time step ~ sigmoid(24[0, 1)).
Architectural details of DiT-fold and the folding tokenizer. This section continues Sec. 5 and covers implementation details of DiT with our FoldDiff framework. The lightweight folding tokenizer follows the modified architecture as shown in Sec. 4.2. We train a generalized tokenizer across the chair, airplane, car subsets of ShapeNet (Chang et al., 2015), and use this tokenizer for each of the following experiments related to DiT. The tokenizer contains shallow PointNet-basod (Qi et al., 2017a) encoder of output dimensions 164,128,128). The max-pooled 1 128-dimensional 1D vector is then concatenated with the coordinates of a k x k 2D grid, and fed into a MLP to output a point-wise latent code of dimension 128 that mixes the grid information with the patch-wise information. Then we compute the attention scores between the point-wise latent code and the 128-dimensional point-wise feature before max-pooling, further merging point-wise and patch-wise information. Two folding layers then takes the merged point-wise latent as input and output a reordered reconstruction of the inputting local patch.
A.4 Langevin gradient ascent details
This section continues Sec. 4 to provide more detailed explanation for the Langevine dynamics sampling algorithm in Kadkhodaie & Simoncelli (2021) which is detailed in Algorithm 1.
In the hyperparamcters, do is the initial noise level, or is the convergence threshold, ho controls the step size of each denoising correction, and 8 controls the proportion of injected noise in each iteration.
As shown in the algorithm, we update noisy image yt after each denoising step with
Yt-1 hed, 7424
20

---

Published in Transactions on Machine Learning Research (08/2025)
Here be controls the step size of the denoising correction d,, while "t controls the magnitude of noise injection after each denoising step. Thus the effective noise variance of of yt after each iteration is
of (1- + (1 (1- 3he)
(15)
In the first expression, the first term is the remaining noise variance after the denoiser correction, and the second term is the additional variance from the injected noise. To ensure convergence, of can be rewritten hut as the second expression in Equation 15 by enforcing 3h, 1 with h, = [+ given an initial parameter ho E [0, II, and [0, that controls the proportion of injected noise. When 3 = 1, we have oF 0, which indicates noise injection; when 8 = 0, noise with variance equivalent to the removed noise is injected. The noise injection amplitude Yi is relevant to both he and B with the expression:
[11 Bha)² (1 (1- he)a oi-11 [11 - Bh,)² - - (1-h l1f(y-1))l /N,
nt
(16)
where [ /N of-1 denotes the magnitude of predicted noise variance, and N is the number of pixels in the image.
To sample high-probability objects from denoiser, we first sample from random Gaussian noise yo (N0, 1) Then the residual of our 2D UNet denoiser is applied as the score V, log)(y). In each iteration, we take a small step loward the suggested direction, thus moving closer to the manifold of clean folded recoustructions, then inject an additional Gaussian perturbation to avoid getting stuck in local maxima (Kadkhodaic & Simoncelli, 2021).
Algorithm 1: Coarse-to-fine stochastic ascent method for sampling from the implicit prior of denoiser, using denoiser residual f(y) x y. (Kadkhodaie & Simoncelli, 2021)
Parameters: Go, OL, ho, 3
Initialization: 1, draw Yo (00
while 0:-1 S LL do
h 1tholt-1)
step size for denoising step // denoising direction // effective noise variance // shrinking noise variance // sampling Gaussian noise // update data with Langevin dynamics
de f(y
of ||d.11
o ((1 - -Bh) )2 (1 (1-)) of
Draw 2, NO, I):
Y1 f Y1-1i-1 + hid, YZZI
1 1+t41
end
A.5 Nearest neighbors of generated shapes
Fig. demonstrates the top nearest neighbors of a sampled chair using both Chamfer Distance (CD) and Earth Moving Distance (EMD). As one can notice, the sampled chair novel comparing to its nearest neighbors, showcasing generalized generative behavior of our DiT-fold.
A.6
Qualitative visualizations
A.6.1 More visualizations of synthesized shapes
Here we show more abundant synthesized shapes from DiT-fold trained on ShapeNet Chang et al. (2015) Airplane (Fig. 6), Car (Fig. 7), and Chair (Fig. 8).
21

---

Published in Transactions on Machine Learning Research (08/2025)
EMD
Figure 5: Top 5 nearest neigh- bors of the sampled chair from the training set. Top: near- est neighbors of Chamfer Dis- tance (CD). Bottom: near-
est neighbors of Earth Moving Distance (EMD).
Sampled Chair
Top 5 nearest shapes
I
i
Figure 6: Synthesized airplanes using DiT based on the proposed FoldDiff framework.
22

---

Published in Transactions on Machine Learning Research (08/2025)
Figure 7: Synthesized cars using DiT based on the proposed FoldDiff framework.
23

---

Published in Transactions on Machine Learning Research (08/2025)
Figure 8: Synthesized chairs using DiT based on the proposed FoldDiff framework.
24


Reviews:
[{"review_id": "PKE6nWeyOv", "reviewer": "Reviewer_CdmS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* The paper introduces a novel framework that enables the application of established 2D image generation methods to 3D point cloud data.\n* The overall idea is conceptually sound and well-motivated; leveraging 2D geometry image representations for diffusion modeling is a natural and appealing direction for addressing permutation invariance.\n* The introduction provides a clear and well-motivated discussion of the challenges inherent in 3D point cloud generation, particularly regarding permutation invariance.\n* The experimental section is comprehensive and effectively demonstrates the capabilities and performance of the proposed approach, FoldDiff.\n\n**Weaknesses:**\n\n1. While the authors frame their discussion as a novel theoretical contribution, the difficulty of applying diffusion or Langevin dynamics to unordered data such as point clouds is relatively intuitive and has been previously recognized in the community. As such, the theoretical exposition may overstate its originality.\n\n2. Tables 2 and 3 do not appear to offer new insights, as the challenges of applying diffusion models to point cloud data are already well recognized in the community. Additionally, the rationale for switching from PointNet in Table 2 to PointNet++ in Table 3 is not explained, which may confuse the interpretation of results. Given their limited contribution to the core claims of the paper, these results might be better placed in the appendix.\n\n3. Several key terms and architectural components are insufficiently defined, which hampers clarity and reproducibility:\n   \n   3.1 The distinction between “local” and “global” is introduced early but never formally defined. It remains unclear whether local patches are subsets of a global representation or produced independently via separate FoldingNet modules. The subsequent reference to “folded tokens” adds further ambiguity; a more consistent and clearly defined naming scheme would greatly improve clarity.\n\n   3.2 The term “folded objects” is used without explanation. It appears to refer to the points mapped to their geometry images, but this should be explicitly clarified.\n\n   3.3 Figure 2 depicts a modified FoldingNet architecture, yet Section 4.2 states that this version is not used in practice, with a hierarchical FoldingNet employed instead. This discrepancy should be addressed directly to avoid confusion about the actual method.\n\nConsider adding a more detailed illustration of the proposed workflow pipeline, annotated with the terminology used throughout the paper (e.g., \"folded tokens,\" \"local patches\"). This could significantly improve clarity and help readers better understand the structure and flow of the method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a novel approach to address the permutation invariance problem in 3D point cloud generation. Their method modify and utilize FoldingNet to effectively order the points, which allows for mesh flattening or surface parameterization. This mapping enables the point cloud to be represented as a 2D geometry image, allowing established 2D diffusion models to be applied for generation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major changes are mentioned in weaknesses. Minor additions: \n\n1. In Section 1, the method referred to as \"PointBERT\" should be written as Point-BERT, in accordance with the original authors' naming convention.\n2. Mention whether the pipeline is trained end-to-end or in separate stages.\n3. The paper appears to overstate the discretization error associated with point-voxel methods. In many such architectures, voxel features are combined with the original point features rather than replacing them. As a result, the role of voxels often resembles a grouping or abstraction mechanism, mimicking  the set abstraction in PointNet++, and the original point-level resolution is preserved.\n4. Consider acknowledging that the idea of representing 3D objects as local geometry image patches has been explored in prior work. For example, recent method have also leveraged local patch-based representations for 3D objects: \n\n@misc{yan2024objectworth64x64pixels,\n      title={An Object is Worth 64x64 Pixels: Generating 3D Object via Image Diffusion}, \n      author={Xingguang Yan and Han-Hung Lee and Ziyu Wan and Angel X. Chang},\n      year={2024},\n      eprint={2408.03178},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV},\n      url={https://arxiv.org/abs/2408.03178}, \n}", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "V5BHUJK8KR", "reviewer": "Reviewer_EgCF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The presentation of the paper is clear and its contribution easily understandable\n- Theory on denoising implicit priors shed light on the importance of correct permutation in the diffusion process\n- FoldDiff gives strong performance compared to baselines with much less compute\n\nWeaknesses:\n- During exposition, some more precise details on folding net is needed for those not familiar with this literature. For example, it is implied that the canonical point cloud after folding net exist on a grid so that the point clouds can be taken in as input to UNet (which typically takes in images). Does this mean the folded point clouds are now in $\\mathbb{R}^{H\\times W\\times 3}$ while the original point clouds are in $\\mathbb{R}^{N\\times 3}$ for some number $N$? Does each pixel in the new \"image\" contain xyz coordinates of point clouds in a canonical coordinate system? Some more explanation is appreciated.\n- The author mentions modified designs of FoldingNet by introducing different components and cross attention, and it is trained with Sinkhorn distance instead. Do these changes result in noticeable gains? Some ablation studies are needed to see why these components are advantageous.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The author presents a new framework for generating 3D point clouds that relies on a two-stage process. In the first stage, the author trains a folding network that permutes point clouds into a canonical ordering, and in the second stage, a standard UNet or DiT is trained on the ordered point cloud. In addition to the two-stage design, the author provides theoretical insight in why naively training diffusion on unordered point clouds does not work well with denoising implicit priors.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As noted in weaknesses in the previous section, some more expositions on the FoldingNet stage and ablation studies on FoldingNet designs can make the paper stronger.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oHqf7gXed3", "reviewer": "Reviewer_Maf8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n- The effort to incorporate permutation-equivariance into the point cloud diffusion model is highly appreciated. \n- The experimental validation is thorough, well-designed, and provides convincing results, demonstrating the robustness of the proposed method.\n\n\n\nWeaknesses:\n1. The theoretical proof presented may be perceived as somewhat trivial, as the challenges of unstructured space in diffusion models are already well-understood in the community.\n\n2. The technical contributions could be seen as limited, primarily consisting of: 1) Applying Sinkhorn Distance to train the FoldingNet\n2) add attention into the Folding-based autoencoder 3) Adapting UNet and DiT to work with the FoldingNet laten.\nWhile the approach is effective, it builds heavily on existing components rather than introducing entirely novel theoretical frameworks.\n\nTypos:\n1) Page 2, \"preview methods\" -> \"previous methods\"\n2) abstract, \"a stochastic diffusion processes\" -> \"a stochastic diffusion process\"\n3) intro, \"Recently advancements in diffusion models\" -> \"Recent advancements\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces FoldDiff, a new approach to 3D point cloud diffusion that tackles the inherent challenges of working with unordered point cloud data. The authors make several key contributions:\n- They provide a theoretical foundation for why traditional diffusion models struggle with point clouds, focusing on the permutation problem that disrupts denoising processes.\n- They propose a folding-based method that reorders point clouds into a structured grid format, making it possible to apply diffusion models more effectively.\n- They show how their method works well with existing architectures like UNet and Diffusion Transformers, even improving training and sampling efficiency.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Some of the related works have not been adequately discussed or compared [1][2].\n\n[1] Peng, Yong, et al. \"SE (3)-Diffusion: An Equivariant Diffusion Model for 3D Point Cloud Generation.\" International Conference on Genetic and Evolutionary Computing. Singapore: Springer Nature Singapore, 2023.\n\n[2] Zheng, Xiao, et al. \"Point cloud pre-training with diffusion models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bOJqCRxMP2", "reviewer": "Reviewer_Yw3X", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. Property 1 proposed in the paper is of interest because it sheds theoretical light on the challenges faced by vanilla DDPM in unstructured space.\n2. The FoldDiff framework makes special treatment for the characteristics of point cloud data, allowing the latest DiT architecture to be effectively applied to point cloud tasks with competitive performance.\n3. The authors conduct a comprehensive set of experiments and analyses that effectively demonstrate the validity of their method. The introduction of optimal transport is particularly interesting. Additionally, the architecture of DiT-fold provides a solid foundation for integrating the latest diffusion frameworks into point cloud tasks.\n\n## Weaknesses and Question\n\n1. This paper[1] also explores 3D structured shapes within the diffusion framework. It improves upon the original SiT (similar to DiT) from a timestep perspective. To enhance the clarity and contribution of this work, the authors should provide a detailed discussion that clearly delineates the differences between these methods. Moreover, can FoldDiff use SiT instead of DiT? Since Flow Matching[2] is a more advanced  framework than  DDPM.\n\n[1] ComboStoc: Combinatorial Stochasticity for Diffusion Generative Models\n\n[2] SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers\n\n2. Additionally, paper[1] introduces Structure-guided Adversarial Training of Diffusion Models. I believe this work is equally inspiring for 3D point cloud tasks and should be thoroughly discussed and analyzed by the authors to further enrich the context and implications of their findings.\n\n[1] Structure-Guided Adversarial Training of Diffusion Models\n\n3. The explanation of the advantages of Sinkhorn Distance over Chamfer Distance provided by the authors is overly simplistic. Given that Sinkhorn Distance has been introduced in the paper, the authors should elaborate on its benefits in detail and provide a comprehensive analysis to justify its inclusion and demonstrate its necessity.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces the FoldDiff framework for handling point cloud data, which inherently exists in an unstructured space. A significant challenge in this context is the unknown permutations of data. The authors provide a theoretical analysis of this challenge and propose a new framework, FoldDiff, to address it effectively. They empirically validate the proposed framework across various diffusion frameworks, demonstrating the efficacy of their method through the results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "see weaknesses", "broader_impact_concerns": "no ethical concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)