Pseudo-Non-Linear Data Augmentation via Energy Minimization

## Abstract

Abstract We propose a novel and interpretable data augmentation method based on energy-based modeling and principles from information geometry . Unlike black-box generative models, which rely on deep neural networks, our approach replaces these non-interpretable transformations with explicit, theoretically grounded ones, ensuring interpretability and strong guarantees such as energy minimization. Central to our method is the introduction of the backward projection algorithm, which reverses dimension reduction to generate new data. Empirical results demonstrate that our method achieves competitive performance with black-box generative models while offering greater transparency and interpretability.

## 1 Introduction

*Data augmentation* has advanced significantly in recent years, primarily due to the increasing use of generative models to meet the growing demand for large datasets (Feng et al., ; Wong et al., ). Despite their success, these generative models often rely on modern deep neural networks, which are typically treated as black boxes, raising concerns about their interpretability (Guidotti et al., ). For instance, the popular *autoencoder* model encodes original data into a compact latent representation and then decodes it back, with both processes usually handled by black-box neural networks (Kingma & Welling, ). Consequently, even when these models perform well, the lack of understanding of the underlying transformations makes it difficult to control the generated outputs, forcing researchers to depend heavily on empirical heuristics.

A natural approach to developing a more interpretable data augmentation method is to replace black-box transformations with more explicit ones (Rudin, ). In this work, we take inspiration from the autoencoder model, which consists of *encoder* and *decoder*. Encoder, when viewed as a form of *dimension reduction* (Wang et al., ), contributes to the model’s success by acting as a form of regularization and potentially avoiding sparsity through encoding data into a low-dimensional latent representation space. Indeed, various data augmentation methods adopt this philosophy, where compact representations are first learned via neural networks, incorporating dimension reduction as a key component of the pipeline (Maharana et al., ). However, while dimension reduction is a well-established field in data science, two main obstacles prevent its direct application to data augmentation. First, classical methods like Principal Component Analysis (Wold et al., ) and Singular Value Decomposition (Stewart, ) inherently rely on *linear* projection in the ambient space (e.g., Euclidean), making the straightforward application of these methods unsuitable for certain modality such as images. Second, the decoder — which aims to reverse the dimension reduction to generate new data — is highly non-trivial to design even for these classical linear methods. This is one of the main reasons why modern generative models rely on black-box transformations.

We address both issues by proposing a new framework and a data-centric algorithm. The framework introduces non-linearity through the well-known *energy-based model* (Xie et al., ), and is built upon recent developments in the *log-linear model on partially ordered sets (posets)* (Sugiyama et al., ; ) and *information geometry* (Amari, ; Amari & Nagaoka, ; Ay et al., ): the log-linear model on posets embeds structured data (e.g., tensors) as discrete probability distributions via an explicit mapping $\varphi$ into a statistical manifold $\mathcal{S}$, and subsequently, provides intricate geometric structure of the data that enables efficient dimension reduction method via projection in $\mathcal{S}$. Building on this, the proposed algorithm, termed *backward projection*, aims to reverse this *forward* projection process to generate new data via projection again. The core idea of backward projection is simple and general: given a new point in the low-dimensional latent representation space, we identify its $k$-nearest latent representations of the original data (obtained via forward projection) and use them to create a target subspace to project *backward* onto. A key insight of the proposal is its ability to exploit the interplay between linearity and non-linearity of projection: the linearity arises from the *divergence minimizing* property when projecting onto *flat* low-dimensional sub-manifolds defined by *linear* constraints on the coordinate systems provided by the log-linear model on posets; however, these projections are inherently non-linear as the space $\mathcal{S}$ is curved. This interplay leads to what we refer to as *pseudo-non-linear* data augmentation.

By combining backward projection with the log-linear model on posets, our approach benefits from explicit, energy-based transformations: these non-linear projections are interpretable, fully white-box, and energy-minimizing, while the framework offers the potential to capture intricate information beyond the ambient space structure. Our contributions are summarized as follows:

- •
We introduce a novel framework for modeling structured data (e.g., tensors) within a statistical manifold via energy-based modeling. Unlike previous works on information geometry, which focused on a single probability distribution, we consider multiple distributions simultaneously, offering a “meta” learning perspective that may be of independent interest.
- •
We propose the *backward projection* algorithm, a data-centric method that reverses dimension reduction, which we then utilize to develop a novel data augmentation method within our framework.
- •
We demonstrate the effectiveness of the proposed data augmentation method. Results show that our approach achieves competitive performance compared to black-box generative models such as autoencoder through simple, transparent, and interpretable algorithms, underscoring its interpretability.

## 2 Related Work

### 2.1 Data augmentation

#### In the Era of Deep Generative Models.

Data augmentation has proven to be highly effective in enhancing deep learning training by increasing dataset size, improving model robustness (Rebuffi et al., ), and introducing implicit regularization (Hernández-García & König, ). These techniques have been applied across various modalities, including text (Shorten et al., ; Feng et al., ; Li et al., ) and images (Shorten & Khoshgoftaar, ; Mumuni & Mumuni, ; Wang et al., ). Much of the recent progress in data augmentation has been driven by advancements in black-box generative models, such as autoencoders (Kingma & Welling, ; Chadebec et al., ) and generative adversarial networks (GANs) (Antoniou, ).

#### Interpretability.

Although there are data augmentation methods that do not rely on generative models (Maharana et al., ), these often depend on the knowledge of the underlying data generation mechanisms, which are typically unknown for complex datasets. As a result, creating interpretable augmented data involves interpreting black-box generative models, an area that remains an active research focus. To date, there is no fully satisfactory solution to this challenge. For example, the design of interpretable GANs is still evolving (Li et al., ; She et al., ) and remains largely limited to specific domains, such as image generation.

### 2.2 Dimension reduction and relation to data augmentation

#### Linear Methods.

Classical *linear* dimension reduction techniques, such as Principal Component Analysis (PCA) (Wold et al., ) and Singular Value Decomposition (SVD) (Stewart, ), work by identifying the optimal linear subspace that minimizes reconstruction error, typically through the orthogonal projection of data onto this subspace. These methods are not only straightforward and explicit, but they also provide valuable geometric insights. For instance, PCA highlights the principal directions that capture the most variance in the data, uncovering important structural patterns.

However, one of the challenges in applying linear dimension reduction methods to data augmentation is the *inverse* problem, where reconstructing the original data from the space of reduced dimension is highly non-trivial. While some studies have explored indirect approaches to using linear dimension reduction for data augmentation (Abayomi-Alli et al., ; Sirakov et al., ), they are often application-specific and hard to generalize, limiting their broader applicability.

#### Non-Linear Methods.

The *non-linear* generalizations, often called *manifold learning* (Meilă & Zhang, ), offer an alternative approach to dimension reduction. Popular methods like t-SNE (Hinton & Roweis, ; Van der Maaten & Hinton, ), Isomap (Tenenbaum et al., ), and UMAP (McInnes et al., ) are based on the manifold hypothesis, which suggests that high-dimensional data lie on a lower-dimensional manifold within the ambient space. The goal is to uncover this manifold and develop a smooth embedding that captures the data’s intrinsic low-dimensional structure. While classical manifold learning methods do not rely on black-box neural networks, they are computationally complex, prone to overfitting, and require careful hyperparameter tuning, making interpretation challenging (Han et al., ).

In theory, manifold learning avoids the *inverse* problem by aiming to recover the underlying low-dimension manifold of the data with near-zero information loss, making it conceptually appealing for data augmentation. However, this is rarely achieved in practice, hence solving the *inverse* problem is still necessary to generate realistic augmented data. Additionally, classical manifold learning methods that do not rely on black-box neural networks are often limited to providing fixed embeddings for training data and cannot perform out-of-sample extensions (Duque et al., ), further limiting their ability to augment data. Recent approaches to address this limitation involve more complex algorithms (Coifman & Lafon, ; Williams & Seeger, ; Vladymyrov & Carreira-Perpinán, ) or the introduction of black-box generative models (Duque et al., ), which reintroduces the concern about interpretability.

## 3 Preliminary

### 3.1 Dually-flatness in information geometry

Information geometry studies the structure of *statistical manifolds* $\mathcal{S}$ within the space of probability distributions. In this paper, we are primarily concerned with the space of exponential families $\{p_{\theta}(x)\mid\theta\in\mathbb{R}^{D}\}$, where each $p_{\theta}$ denotes a probability density function parameterized by $\theta$. We focus on the key concept in this field, *dually-flatness*, in this preliminary, while directing readers to Amari () for more comprehensive details.(^1^11We will assume some familiarity on the basic terminologies for manifold (Lee, , Chapter 1, 4).)

The starting point is the observation that the *log-partition function* $\psi(\theta)$ (also known as the *cumulant generating function* in statistics and *free energy* in physics) of an exponential family with density $p_{\theta}$ is convex in the *natural parameter* $\theta\in\mathbb{R}^{D}$. This convexity induces a natural coordinate system, $\theta$, on $\mathcal{S}$, defining both the Riemannian metric $g=\nabla^{2}\psi(\theta)$ and the Bregman divergence (Bregman, ) $D_{\psi}(p_{\theta},p_{\theta^{\prime}})$. With these structures, the manifold $(\mathcal{S},g)$ is flat, meaning that any curve $\theta(t)=at+b$ (where $a,b\in\mathbb{R}^{D}$ are constants) is a geodesic and lies entirely within $\mathcal{S}$. This flatness is known as *$e$-flatness*, and the geodesics are referred to as *$e$-geodesics* or *primal-geodesics*.

The dual structure arises from the *Legendre transform* (Legendre, ), which generates the dual function $\psi^{\ast}(\eta)$, where $\eta\in\mathbb{R}^{D}$ is the *expectation parameter*. This dual function is also convex, giving rise to the expectation coordinate system $\eta$, the dual Riemannian metric $g^{\ast}$, and also the dual Bregman divergence $D_{\psi^{\ast}}$ which is the well-known Kullback-Leibler divergence $D_{\mathrm{KL}}$. The corresponding flatness is termed *$m$-flatness*, with *$m$-geodesics* or *dual-geodesics* as its geodesics.

*Dually-flatness* then emerges from the interplay between these two structures. Specifically, for any point $p$ in $\mathcal{S}$, there is a unique point $p^{\ast}$ on an $e$-flat sub-manifold $\mathcal{B}\subseteq\mathcal{S}$ that minimizes the dual Bregman divergence $D_{\psi^{\ast}}(p,q)=D_{\mathrm{KL}}(p,q)$ (Amari, , Theorem 1.5). This process, known as the *$m$-projection*, can be efficiently solved via convex optimization. The dual holds when switching $e$ and $m$ in this context. Projection is a central tool in information geometry with profound implications for understanding the geometry of $\mathcal{S}$, which we will utilize later.

### 3.2 Statistical manifold on posets

A set $\Omega$ is a *partially ordered set (poset)* if it is equipped with a *partial order* “$\leq$”, a relation satisfying the following for all $x,y,z\in\Omega$:
1.) $x\leq x$(reflexivity);
2.) $x\leq y$and $y\leq x$ implies $x=y$ (antisymmetry); and
3.) $x\leq y$and $y\leq z$ implies $x\leq z$ (transitivity).
We focus on finite posets $\Omega$ with a bottom element $\bot$ such that $\bot\leq x$ for all $x\in\Omega$.

Given such a poset $\Omega$, consider a discrete random variable $X$ with finite support $\Omega$ with its probability mass function $p\colon\Omega\to\mathbb{R}_{\geq 0}$ being defined by $p(x)=\Pr(X=x)$ for $x\in\Omega$. A key observation is that for a discrete probability distribution $p$ over a poset $\Omega$, the *log-linear model on posets* recursively defines $\theta\colon\Omega\to\mathbb{R}$ as $\log p(x)\eqqcolon\sum_{y\leq x}\theta(y)$ for all $x\in\Omega$. Intuitively, one can think of $\theta(x)$ for each $x\in\Omega$ as specifying the correct energy for $x$ that correctly represents $p(x)$, where the dependence between $\theta$’s on different elements depends on the poset structure. This model belongs to the exponential family, with $\theta$ corresponding to the natural parameters, except for $\theta(\bot)$ which coincides with the partition function. Thus, all discrete probability distributions over $\Omega$ form a $(|\Omega|-1)$-dimensional dually-flat statistical manifold $\mathcal{S}\coloneqq\{p\colon\Omega\to\mathbb{R}_{\geq 0}\mid\sum_{x\in\Omega}
p(x)=1\}$, with dual coordinate systems $(\theta,\eta)$ defined by the poset structure.

## 4 Data augmentation with log-linear model on posets

We first present our proposed framework in Section 4.1 and the backward projection algorithm in Section 4.2, then, we combine and apply them to data augmentation in Section 4.3. Finally, we discuss an important feature of the proposed method regarding interpretability in Section 4.4.

### 4.1 Log-Linear model on posets framework

Given a dataset $\{z_{i}\}_{i=1}^{n}$, our proposed framework embeds the data into a statistical manifold $\mathcal{S}$ using an energy-based approach, leveraging the log-linear model on posets. This provides a geometric structure induced by the energy-based modeling, which is general and not restricted to any specific application, making it of broader interest. The process works in three steps:
1.) models each $z_{i}$ as a *real-valued poset*;
2.) embeds the data naturally into the statistical manifold $\mathcal{S}$;
3.) computes two coordinate representations of the embedded data using the log-linear model on posets.
See Figure 1 for an illustration. We now explain each step in detail below.

Figure: Figure 1: Given structured data, we design a corresponding poset $\Omega$ and embed the resulting real-valued poset as a discrete probability distribution $p_{\theta}(x)$ via a natural embedding $\varphi$ into the statistical manifold $\mathcal{S}$. Then the log-linear model on posets provides the dually-flat coordinates $(\theta,\eta)$ for $p_{\theta}$.
Refer to caption: log-linear-model.pdf

#### Real-Valued Poset.

In the usual machine learning pipeline, inputs are constrained to be vectors or matrices, failing to deal with more complex data. In contrast, posets are flexible enough to capture data with structures, including vectors and matrices. For instance, focusing on the underlying data structure for now (i.e., omitting the feature associated with individual entry), a $D$-dimensional vector $z\in\mathbb{R}^{D}$ can be modeled by the poset $\Omega\coloneqq[D]$ with the partial order being the natural order between positive integers. Similarly, other common data structures such as matrices or tensors can be treated in the same way, while capturing more complex structures potentially. In general, any data structure that naturally admits a partial order can be effectively modeled by a poset.

Now, considering the features associated with each entry in the data structure, we define the *real-valued poset*, which is a mapping from the poset $\Omega$ to, say, the set of real numbers $\mathbb{R}$ such that each entry (element) of the data structure (poset) $x\in\Omega$ is associated with a feature in $\mathbb{R}$. We denote the set of real-valued posets as $\Omega_{\mathbb{R}}$. In the $D$-dimensional vector example, $\Omega=[D]$, with each element $x\in\Omega$ corresponds to one of the $D$ dimensions. Associating a real number to each dimension (i.e., a $D$-dimensional vector) naturally corresponds to an element in $\Omega_{\mathbb{R}}$.

#### Natural Embedding.

To embed the data $\{z_{i}\in\Omega_{\mathbb{R}}\}_{i=1}^{n}$, which are now modeled as real-valued posets, to the statistical manifold $\mathcal{S}$ which concerns with discrete probability distributions, we want an embedding $\varphi\colon\Omega_{\mathbb{R}}\to\mathcal{S}$ such that $\sum_{x\in\Omega}(\varphi(z_{i}))_{x}=1$ for all $z_{i}$ with $\dim(\mathcal{S})=D-1$.(^2^22In fact, we can also consider the manifold of positive measures, which avoids the dimension being $D-1$ and the potential scaling issues. We omit this trivial extension in the presentation to prevent complications.) From the perspective of energy-based modeling, $\varphi$ is oftentimes naturally induced, e.g., for tabular frequency data. Moreover, $\varphi$ often admits a natural inverse $\varphi^{-1}$, or an empirical one based on the data. We will take both $\varphi$ and $\varphi^{-1}$ as granted from now on.

#### Dually-Flat Coordinates.

After defining $\varphi$, from the theory of information geometry and the log-linear model on posets, for each point $z_{i}^{\prime}\coloneqq\varphi(z_{i})\in\mathcal{S}$, we can associate the dually-flat coordinate systems $\theta(z_{i}^{\prime})\in\mathbb{R}^{D-1}$ and $\eta(z_{i}^{\prime})\in\mathbb{R}^{D-1}$. Such coordinate systems are with respect to the underlying poset structure of $\Omega$ and are driven by the principle of energy-based modeling.

### 4.2 Forward and backward projection

We now demonstrate how to incorporate projection theory to achieve data augmentation. As our algorithm is inspired from the architecture of autoencoders, we focus on two of the central building blocks: the *encoder* $\mathsf{Enc}(\cdot)$ and the *decoder* $\mathsf{Dec}(\cdot)$. First, for the encoding step, we formally explain how projection theory can be applied to perform dimension reduction within our framework. Next, for the decoding step, we introduce our proposed algorithm, termed *backward projection*, which serves as the *inverse* of dimension reduction. While our explanation is tailored to our proposed framework, i.e., the log-linear model on posets, the proposed backward projection algorithm itself is general and may be of independent interest as well.

#### Dimension Reduction: Forward Projection.

Given the log-linear model on posets framework in Section 4.1, the embedding from $\Omega_{\mathbb{R}}$ to the statistical manifold $\mathcal{S}$ does not achieve dimension reduction as $\dim(\mathcal{S})\approx\dim(\Omega_{\mathbb{R}})$. To achieve dimension reduction, we leverage the projection theory: by projecting $z_{i}^{\prime}=\varphi(z_{i})$ onto a low-dimensional flat *base sub-manifold* $\mathcal{B}\subseteq\mathcal{S}$ with $\dim(\mathcal{B})\ll\dim(\mathcal{S})$, we obtain the desired *encoding* $\mathsf{Enc}\coloneqq\operatorname{Proj}_{\mathcal{B}}\circ\varphi\colon\Omega
_{\mathbb{R}}\to\mathcal{B}$ that maps the data to a low-dimensional latent representation manifold. The encoding $\mathsf{Enc}(\cdot)$ is smooth and well-defined as the projection is unique when $\mathcal{B}$ is flat and minimizing either the primal or the dual Bregman divergence, depending on $\mathcal{B}$. These theoretical guarantees provide rigor and support reasoning through geometric intuition, which in turn offers interpretability.

#### Inverse Dimension Reduction: Backward Projection.

As we have hinted at, one of the technical burdens is that the encoding $\mathsf{Enc}(\cdot)$ is not invertible, hence no natural decoding $\mathsf{Dec}(\cdot)$ is available, even when $\mathsf{Enc}(\cdot)$ only involves traditional linear dimension reduction algorithm. While finding the exact inverse is mathematically impossible as the pre-image of the projection is not unique in any sense (even in Euclidean space), here, we propose a simple, geometrically intuitive, and data-centric solution that aims to find the *inverse* of the projection that is similar to the original data.

The high-level intuition is simple: if the result of the projection is close, then so is the original data, i.e., its *inverse*. Hence, given a point in the low-dimensional latent representation space, we try to “project it back” to approximate the original dataset by exploiting the fact that we have access to the *inverse* of the dataset’s projection, i.e., the dataset itself. Specifically, we can artificially create a *local* sub-manifold around a subset of the dataset, determined by the nearest neighbors of that given point in the latent representation space, and *backward* project onto it.

Formally, assuming that we have access to the embedded dataset $\{z^{\prime}_{i}=\varphi(z_{i})\}_{i=1}^{n}$ and their projected result $\{w_{i}=\operatorname{Proj}_{\mathcal{B}}(z^{\prime}_{i})\}_{i=1}^{n}$ for some base sub-manifold $\mathcal{B}$. To find the *inverse* of some given point $w^{\ast}\in\mathcal{B}$ assuming it comes from the projection on $\mathcal{B}$, we first find $w^{\ast}$’s $k$-nearest neighbors among $w_{i}$’s, obtaining a size $k$ index set $N$. Then we create a *local data sub-manifold* $\mathcal{D}$ based on the pre-images $z^{\prime}_{i}$’s of these $w_{i}$’s, and project $w^{\ast}$ on $\mathcal{D}$ to obtain the *inverse* $z^{\prime\ast}=\operatorname{Proj}_{\mathcal{B}}^{-1}(w^{\ast})\coloneqq
\operatorname{Proj}_{\mathcal{D}}(w^{\ast})$. Algorithm 4.1 summarizes this procedure, which we termed *backward projection*. With access to $\operatorname{Proj}_{\mathcal{B}}^{-1}(\cdot)$, decoding is simply $\mathsf{Dec}\coloneqq\varphi^{-1}\circ\operatorname{Proj}^{-1}_{\mathcal{B}}
\colon\mathcal{B}\to\Omega_{\mathbb{R}}$, serving as the *inverse* of $\mathsf{Enc}(\cdot)$ as desired.

Figure: Algorithm 4.1 Backward Projection

###### Remark 4.1 .

A flat sub-manifold can be defined by forcing linear constraints on the ($\theta$ or $\eta$) coordinates. For instance, given the nearest neighbors $z^{\prime}_{i^{\star}}$, one can define $\mathcal{D}\coloneqq\{\theta\in\mathbb{R}^{\dim(\mathcal{S})}\mid(\theta)_{x}=
\big{(}\theta(z^{\prime}_{i^{\star}})\big{)}_{x}\}$ for some $x\in\Omega$, namely, we fix some indexes to be the corresponding $\theta$-coordinate values of $z^{\prime}_{i^{\ast}}$. The quality of Algorithm 4.1 can be controlled by choosing appropriate linear constraints.

Algorithm 4.1 is a geometrically intuitive, data-centric algorithm with desirable theoretical guarantees such as divergence minimizing when projecting on the constructed local data sub-manifold $\mathcal{D}$. Its white-box nature ensures a level of interpretability, making it the cornerstone of our method, in contrast to black-box generative models.

###### Remark 4.1 .

### 4.3 Data augmentation with log-linear model on posets

With all the building blocks in place, we can now formally describe the complete data augmentation algorithm, which consists of three phases:
1.) encoding,
2.) generating, and
3.) decoding.

#### Encoding.

As described in Section 4.2, the encoding $\mathsf{Enc}\coloneqq\operatorname{Proj}_{\mathcal{B}}\circ\varphi$ is simply a combination of the natural embedding followed by a projection. Notation-wise, we write $w_{i}\coloneqq\mathsf{Enc}(z_{i})$.

#### Generating.

To generate new data $z^{\ast}$, we first generate a new point $w^{\ast}$ in the latent space, which in our case, is a pre-specified flat base sub-manifold $\mathcal{B}$. This can be done in various ways, such as using pure heuristics, controlled perturbations, or even black-box generative models. In our case, we focus on a simple, white-box generation method: *kernel density estimation* (Davis et al., ; Parzen, ). Specifically, we first fit a kernel density estimation model $M$ on either the $\theta$ or $\eta$ coordinate systems, then sample $m$ new points $w^{\ast}$ in the latent space $\mathcal{B}$ from $M$.

#### Decoding.

As described in Section 4.2, the decoding $\mathsf{Dec}\coloneqq\varphi^{-1}\circ\operatorname{Proj}^{-1}_{\mathcal{B}}$ is simply a combination of backward projection (Algorithm 4.1) with the inverse of the natural embedding. Notation-wise, we write $z^{\ast}\coloneqq\mathsf{Dec}(w^{\ast})=\varphi^{-1}(z^{\prime\ast})$ where $z^{\prime\ast}\coloneqq\operatorname{Proj}_{\mathcal{B}}^{-1}(w^{\ast})
\coloneqq\operatorname{Proj}_{\mathcal{D}}(w^{\ast})$.

We summarize the above procedure in Algorithm 4.2.

Figure: Algorithm 4.2 Data Augmentation with Log-Linear Model on Posets

In what follows, we use *positive tensors* as the running example for a better illustration.

###### Example 4.2 (Positive tensor) .

A $d^{\text{th}}$-order *tensor* $T\in\mathbb{R}^{I_{1}\times\cdots\times I_{d}}\eqqcolon\mathbb{R}^{D}$ is a multidimensional array with real entries for every index vector $v=(i_{1},\ldots,i_{d})\in[I_{1}]\times\cdots\times[I_{d}]\eqqcolon\Omega$ where for each $k$, $[I_{k}]\coloneqq\{1,2,\ldots,I_{k}\}$ for a positive integer $I_{k}$. Tensors with entries all being positive are called positive tensors, denoted as $P\in\mathbb{R}_{\geq 0}^{I_{1}\times\cdots\times I_{d}}$. For tensors, a natural partial order “$\leq$” one can impose on $\Omega$ between two index vectors $v=(i_{1},\ldots,i_{d}),w=(j_{1},\ldots,j_{d})$ is that $v\leq w$ if and only if $i_{k}\leq j_{k}$ for all $k=1,\ldots,d$. Finally, for positive tensors, a simple embedding $\varphi\colon\mathbb{R}_{\geq 0}^{I_{1}\times\cdots\times I_{d}}\to\mathcal{S}$ where $P^{\prime}\coloneqq\varphi(P)\colon\Omega\to\mathbb{R}_{\geq 0}$ such that $P^{\prime}_{v}\coloneqq P_{v}/\sum_{w\in\Omega}P_{w}$ for all $v\in\Omega$ can be defined.

Figure: Figure 2: Data augmentation for positive tensors via Algorithm 4.2 with $k=1$ and $\mathcal{D}=\mathcal{D}_{P_{2}^{\prime}}$.
Refer to caption: data-augmentation.pdf

We now illustrate Algorithm 4.2 with positive tensors. Following the notations in Example 4.2, let’s write $z_{i}$’s as $P_{i}$’s, $z_{i}^{\prime}$’s as $P_{i}^{\prime}$’s, and $w_{i}$’s as $Q_{i}$’s. Firstly, Example 4.2 provides one way to model positive tensors by real-valued posets and define a natural embedding $\varphi$ (i.e., normalization), giving $P_{i}^{\prime}$. To obtain the final encoding, we choose some base manifold $\mathcal{B}$ to project $P_{i}^{\prime}$ onto, giving $Q_{i}\coloneqq\operatorname{Proj}_{\mathcal{B}}(P_{i}^{\prime})$. For generation, we simply fit a kernel density estimation model $M$ to $\{w_{i}=Q_{i}\}_{i=1}^{n}$ and sample a new $Q^{\ast}\sim M$. Finally, for the decoding step, consider the case of $k=1$, $\mathcal{D}$ is created by some linear constraints w.r.t. one particular $P_{i^{\star}}^{\prime}$, where $i^{\star}$ corresponds to the nearest neighbor $Q_{i^{\star}}$ of $Q^{\ast}$ among $Q_{i}$’s. We then backward project $Q^{\ast}$ on $\mathcal{D}$ with Algorithm 4.1 to obtain $P^{\prime\ast}\coloneqq\operatorname{Proj}_{\mathcal{D}}(Q^{\ast})$, and we output $P^{\ast}\coloneqq\varphi^{-1}(P^{\prime\ast})$ as our generated result.(^3^33Empirically, we let $\varphi^{-1}$ to be *reversing the average of original scaling among the nearest neighbors*.) See Figure 2 for an illustration, where we let $k=m=1$.

Algorithm 4.2 integrates both forward (encoding) and backward (decoding) projections, which, as discussed in Section 4.2, are interpretable due to its white-box nature and come with strong theoretical guarantees. When the generating step is performed in a clear and white-box manner, Algorithm 4.2 retains its interpretability while continuing to benefit from these theoretical guarantees.

###### Example 4.2 (Positive tensor) .

### 4.4 Construction of sub-manifolds

For any sub-manifold $\mathcal{S}^{\prime}\subseteq\mathcal{S}$, as $\dim(\mathcal{S}^{\prime})$ increases, more information of the data is preserved after forward projection onto $\mathcal{S}^{\prime}$. In the case of constructing the base sub-manifold $\mathcal{B}$, the quality of the backward projection $\operatorname{Proj}_{\mathcal{B}}^{-1}(\cdot)$ (Algorithm 4.1) should increase along with $\dim(\mathcal{B})$ for the same reason. However, in the extreme case where $\dim(\mathcal{B})\approx\dim(\mathcal{S})$, Algorithm 4.2 becomes less effective due to the sparsity of the data, resulting in an intrinsic trade-off for choosing $\dim(\mathcal{B})$ (see Section A.3 for an empirical justification). In this section, we argue that by leveraging existing tools and understandings of the log-linear model on posets, such an intrinsic trade-off for constructing sub-manifolds (either $\mathcal{B}$ or $\mathcal{D}$) provides an additional layer of interpretability and control compared to black-box generative models like autoencoders.

To keep our presentation concise and concrete, we focus on positive tensors, although the argument and the high-level idea extend to more general cases. Firstly, the projection theory is well-explored for positive tensors within the log-linear model, where several established constructions for flat base sub-manifolds $\mathcal{B}\subseteq\mathcal{S}$ (Sugiyama et al., ; Ghalamkari et al., ) have proven powerful in capturing the non-trivial structure of positive tensors after the projection. One of which is called the *many-body tensor approximation* (Ghalamkari et al., ), which captures a *hierarchy* of mode interactions with different $\dim(\mathcal{B})$. Specifically, the $\ell$-body approximation considers projection on the following sub-manifold

$$ $\mathcal{M}_{\ell}\coloneqq\{\theta\in\mathbb{R}^{\dim(\mathcal{S})}\mid( \theta)_{x}=0\text{ for all {non} $\ell$-body parameters }x\in\Omega\},$ (1) $$

where the $\ell$-body parameter corresponds to $\ell$ non-one indices, acting as a generalization of one-body and two-body parameters (Ghalamkari & Sugiyama, ). Intuitively speaking, an $\ell$-body parameter captures the interaction among $\ell$ different modes, hence, when $\mathcal{B}=\mathcal{M}_{\ell}$, all interactions between modes of orders higher than $\ell$ are neglected. This offers a practical design choice for employing Algorithm 4.2. In particular, it allows us to leverage prior knowledge of the data to design an appropriate base sub-manifold $\mathcal{B}$ and also the local data sub-manifold $\mathcal{D}$ that captures different degrees of information with appropriate dimension. This approach provides a more principled way of defining the latent space, compared to black-box models like autoencoders, where the latent space dimensions are oftentimes tuned without a clear understanding of what those dimensions represent.

## 5 Experiment

In this section, we conduct a series of experiments to validate the efficacy of our proposed data augmentation method. We focus primarily on image tasks for a clear illustration, where we compare our method with autoencoder models. Additional experiments can be found in Appendix A.

### 5.1 Setup

Here, we briefly summarize the experimental setup, while directing readers to Section A.1 for more details. Consider the image classification task(^4^44As there are only finitely many labels (classes) in classification tasks, one can perform Algorithm 4.2 for each class separately without worrying about assigning labels.) on the MNIST dataset (LeCun, ), with the training set size being $1000$ ($200$ samples for each digit). Since MNIST images are in $\mathbb{R}_{\geq 0}^{28\times 28}$, we apply the log-linear model with posets for positive tensors as in Example 4.2. For the sub-manifold constructions, we utilize the many-body approximation (Equation 1) with its variances when constructing the base sub-manifold $\mathcal{B}$ and the local data sub-manifold $\mathcal{D}$, where we use $\dim(\mathcal{B})=17$ and $\dim(\mathcal{D})=767$. For a fair comparison with the autoencoder model, we consider a simple $2+2$ layers architecture with latent space dimension $17=\dim(\mathcal{B})$. Finally, when generating data, we first fit a kernel density estimation model $M$ on the latent representation of the training dataset, sample a new latent representation from which, and then decode it. We note that important hyperparameters ($k=8$ and the bandwidth $0.01$ of $M$) are chosen via a simple grid search in Section A.2.

### 5.2 Visual inspection

To illustrate how Algorithm 4.2 works in practice, Figure 3 shows the intermediate results after projection onto $\mathcal{B}$, while Figure 4 shows the results of Algorithm 4.2 after applying backward projection (Algorithm 4.1). We emphasize that the results from Figure 4 do not come from backward projecting the results of Figure 3; instead, they come from the latent representations sampled from $M$.

Figure: Figure 3: (*Top*) Forward projected data on $\mathcal{B}$. (*Bottom*) Heat map of corresponding $\theta$ values.
Refer to caption: x1.png

Figure: Figure 4: (*Top*) Augmented data via Algorithm 4.2. (*Bottom*) The closest training data.
Refer to caption: x3.png

For comparison, Figure 5 shows the data augmentation results generated by the autoencoder. Despite careful bandwidth tuning when fitting the kernel density model, the autoencoder results appear to overfit the training set. Finally, we note an interesting difference between the two approaches: our proposed method produces a *blurred* effect, while the autoencoder exhibits *hard-clipping*.

Figure: Figure 5: (*Top*) Augmented data via autoencoder. (*Bottom*) The closest training data.
Refer to caption: x5.png

### 5.3 Classification performance

We evaluate our proposed method on the downstream task, i.e., classification performance, in addition to visual inspection. Specifically, we train a linear classifier on three types of training datasets:
1.) original dataset,
2.) augmented dataset, and
3.) original dataset combined with the augmented dataset,
where the augmented dataset consists of $m=200$ augmented samples, which is $20\%$ of the original training set. For clarity, we refer to the original dataset as Original, the dataset augmented with Algorithm 4.2 as Ours, and the dataset augmented using the autoencoder model as AE.

The results are shown in Table 1, where each test set consists of $500$ samples, evaluated over $20$ bootstrapping runs. Firstly, observe that Original$+$Ours and Original$+$AE outperforms Original as expected, with the former outperforming the latter slightly. Moreover, we see that Ours outperforms AE by a large margin, which is surprising given the representation power of the autoencoder compared to our fully white-box, interpretable method. Overall, our method achieves competitive performance against black-box generative models in the downstream task while offering interpretability. We direct readers to Section A.5 for additional evaluations on other datasets.

**Table 1: Test accuracy of the linear classifier trained on different training sets.**
| Training Set | Original | Ours | AE | Original$+$Ours | Original$+$AE |
| --- | --- | --- | --- | --- | --- |
| Accuracy | $81.79\pm 4.57\%$ | $75.37\pm 2.89\%$ | $68.12\pm 3.96\%$ | $83.40\pm 3.22\%$ | $82.72\pm 3.50\%$ |

### 5.4 Interpretability with choices of sub-manifolds

As discussed in Section 4.4, constructing the base sub-manifold carefully allows for an additional layer of interpretability and control. In Section 5.1, the default base manifold $\mathcal{B}$, though implicit, is $\mathcal{M}_{1}$ for the tensor structure $\mathbb{R}_{\geq 0}^{7\times 2\times 2\times 7\times 2\times 2}$. We now consider $\mathcal{B}=\mathcal{M}_{\ell}$ for $\ell=1$ to $3$ for comparison, while direct readers to Section A.4 for a more in-depth experiment. Following the same setup as in Figures 3 and 4 for $\ell=1$, the results for $\ell=2,3$ are shown in Figures 6 and 7.

Figure: (a) Result with base sub-manifold $\mathcal{B}=\mathcal{M}_{2}$ ($\dim(\mathcal{B})=107$).
Refer to caption: x7.png

Figure: (a) Result with base sub-manifold $\mathcal{B}=\mathcal{M}_{2}$ ($\dim(\mathcal{B})=107$).
Refer to caption: x11.png

We observe several interesting phenomena. First, for $\ell=1$ with a small base sub-manifold dimension, the forward projection results (Figure 3) appear visually unclear, in contrast to the augmentation results (Figure 4). Note that throughout the experiment, the local data sub-manifold $\mathcal{D}$ has a dimension of $767$, indicating a high degree of freedom for backward projection. This suggests that $\mathcal{M}_{1}$ can effectively capture key features across signals, leading to non-trivial neighbor information and thus constructing a sufficiently good local data sub-manifold. Second, as expected, the higher the dimension of $\mathcal{M}_{\ell}$ (i.e., value of $\ell$), the more signal structures (in terms of mode interactions) are preserved as shown in Figures 3 and 6, resulting in better performance (Figures 4 and 7).

Based on the theory of many-body approximation, one can construct the base sub-manifold with a clear understanding of the trade-off between dimensionality and the performance of Algorithm 4.2. Unlike black-box generative models, which often rely on heuristics or blindly tuning the latent space dimension, our proposed method offers an additional layer of interpretability.

## 6 Discussion

#### Structural Limitation.

While the log-linear model is flexible to represent structural data, it still faces limitations. The key issue lies in the model’s reliance on a partial order of the index set, which makes it impossible to ensure invariance under the permutation of indexes. For instance, modeling graphical data usually requires non-invariance and non-equivariance of vertices (i.e., indexes), in this case, the log-linear model might not be the best model due to its structural limitations.

#### Meta-Perspective.

Classical information geometry typically involves learning a single distribution by manipulating a single point in the statistical manifold $\mathcal{S}$, as seen in tasks like learning the Boltzmann machine or finding the maximum likelihood estimation (Amari, ). In our case, however, we treat data as probability distributions within $\mathcal{S}$, offering a new perspective for applying the information geometry framework. With multiple distributions in $\mathcal{S}$, a natural extension would be to employ data-centric machine learning algorithms to learn the “data” distribution, i.e., the distribution of these *distributions*, thereby providing a *meta*-perspective.

## 7 Conclusion

In this paper, we proposed a novel data augmentation method that leverages several information geometric algorithms, incorporating interpretability while maintaining competitive performance. Our framework, built on the log-linear model on posets, equips data with information geometric structures, facilitating geometric reasoning and algorithm design. The proposed backward projection algorithm reverses the dimension reduction process in a geometrically intuitive and data-centric manner, which may be of independent interest.

We empirically demonstrated that our method achieves competitive performance compared to traditional autoencoder-based approaches on downstream tasks, even though the latter may possess greater representational power but lack interpretability, which is a crucial requirement in many practical applications. Overall, our work paves the way for further exploration of information geometric algorithms in various domains, not limited to data augmentation.

## Appendix A Omitted details from Section 5

### A.1 Details of image classification setup

#### Log-Linear Model on MNIST.

When we apply the log-linear model on posets for positive tensors, we first reshape every image into $\mathbb{R}_{\geq 0}^{7\times 2\times 2\times 7\times 2\times 2}$ and consider the natural poset $\Omega$ corresponding to this $6^{\text{th}}$-order tensor structure. From the discussion in Section 4.4, when considering the image as a tensor of shape $(7,2,2,7,2,2)$ instead of $(28,28)$, a finer hierarchy of projection is possible via many-body approximation (Ghalamkari et al., ). In Section 5, the default choice of the base sub-manifold is $\mathcal{B}=\mathcal{M}_{1}$ for the $6^{\text{th}}$-order tensor structure $\mathbb{R}_{\geq 0}^{7\times 2\times 2\times 7\times 2\times 2}$ as defined in Equation 1. On the other hand, the local sub-manifold $\mathcal{D}$ is constructed by fixing every one-body parameter: given a set $N$ of $k$ nearest neighbors, we let $\mathcal{D}$ as $\mathcal{M}_{1}^{\ast}(N)$ where in general, we define

$$ $\mathcal{M}_{\ell}^{\ast}(N)\coloneqq\left\{\theta\in\mathbb{R}^{\dim(\mathcal {S})}\mid(\theta)_{x}=\frac{1}{k}\sum_{i^{\ast}\in N}\big{(}\theta(z_{i^{\ast} }^{\prime})\big{)}_{x}\text{ for all $\ell$-body parameters }x\in\Omega\right\},$ (2) $$

where $\dim(\mathcal{S})=28\times 28=784$ for MNIST. This is like the dual notion of $\mathcal{M}_{\ell}$: in Equation 1, we allow all $\ell$-body parameters to vary; here, we allow all non $\ell$-body parameters to vary.

#### Kernel Density Estimation Model.

The default bandwidth for the kernel density estimation model is set to be $0.05$ to avoid overfitting. This is a fair comparison since the latent space dimensions for our proposed method and the autoencoder model we consider are the same.

#### Linear Classifier.

The classification task is conducted with a simple linear classifier trained with Stochastic Gradient Descent (SGD) (Ruder, ) till convergence with a learning rate of $0.01$.

### A.2 Sensitivity and robustness

We examine our proposed method’s robustness and sensitivity of the *bandwidth* used when fitting the kernel density model, and also the *number $k$ of the nearest neighbors* used in Algorithm 4.1.

#### Bandwidth of Kernel Density Estimation Model.

Consider varying the bandwidth we use when fitting the kernel density model, ranging among $\{0.01,0.05,0.1,0.2,0.5\}$. The results are shown in Figure 8, where we omit showing the closest training data as it is not important for the purpose here. We observe that Algorithm 4.2 is robust under different bandwidths when working with the kernel density estimation model in the generating step.

Figure: (a) Result with bandwidth $0.01$.
Refer to caption: x15.png

#### Number of Nearest Neighbors.

Next, we consider ranging $k$ among $\{1,4,8,16\}$. The results are shown in Figure 9. Observe that when $k$ is small, e.g., $1$, the result of Algorithm 4.2 tends to overfit since the local sub-manifold $\mathcal{D}$ in Algorithm 4.1 is defined using only the nearest neighbor. When $k$ goes up, a non-trivial augmentation emerges, robust across different $k$’s.

Figure: (a) Result with $k=1$.
Refer to caption: x20.png

### A.3 Necessity of dimension reduction

We demonstrate that dimension reduction, a key building block of our proposed method based on the intuition we have from autoencoder-like models, is necessary for Algorithm 4.2 to work.

#### Direct Fitting.

As discussed, naive perturbation-based data augmentation methods fall short of high-dimensional data due to the sparsity of the data. Figure 10 shows the results of directly fitting a kernel density estimation model on MNIST with $1000$ samples.

Figure: Figure 10: (*Top*) Augmented data via directly fitting a kernel density estimation model with a bandwidth $30$. (*Bottom*) The closest training data.
Refer to caption: x28.png

Observe that even with a large bandwidth ($30$) to introduce variability, we only see meaningless noisy perturbation on one of the exact training samples, indicating overfitting.

#### Local Data Sub-Manifold.

A potential problem related to the necessity of dimension reduction is that, if $\mathcal{D}$ captures too much local information about the data, backward projecting a random latent representation $w^{\ast}\in\mathcal{B}$ might already suffice to augment the data in a non-trivial way, without the need for knowing the latent representations of the training dataset. To this end, consider sampling uniformly random latent representations within the empirical range we observed from the latent representations of the training data and perform Algorithm 4.1. The results are shown in Figure 11.

Figure: (a) Result with $k=1$.
Refer to caption: x30.png

For $k=1$, Figure 11(a) shows that similar to Figure 9(a), it is possible to overfit one of the training data (i.e., the nearest neighbor of the randomly sampled latent representation). This is not surprising since the base sub-manifold is only of dimension $17$ and the local data sub-manifold is of dimension $767$, as the random latent representation is sufficiently close to one of the representations of the training data in $\mathcal{B}$, their backward projection result should not deviate too much. Furthermore, we observe the *fading effect*, which intuitively corresponds to *misspecification of the energy*, indicating that the sampled latent representation is fundamentally different from the one of the dataset.

As $k$ increases, the reason for getting informative and meaningful latent representations from the original dataset becomes clear. Specifically, we start to see *degeneration*: from unclear overlappings to collapsing (i.e., only a few pixels are showing). Intuitively speaking, it is because the random latent representation’s nearest neighbors appear to be significantly different, hence failing to provide a consistent local data sub-manifold. For instance, in the extreme case when $k=16$, the local data sub-manifold is completely not informative, resulting in collapsing. Overall, without dimension reduction, we will lose the reference of *realisitc latent representations* provided by the original dataset, which leads to bad performance once we are beyond the trivial overfitting regime.

### A.4 Choices of tensor structure and construction of sub-manifolds

In Section 5.4, we consider varying $\ell$ for $\mathcal{B}=\mathcal{M}_{\ell}$ with the tensor structure being $\mathbb{R}_{\geq 0}^{7\times 2\times 2\times 7\times 2\times 2}$. In this section, we further vary the tensor structure as well: in particular, we consider the tensor structure of the MNIST image being $\mathbb{R}_{\geq 0}^{28\times 28}$, $\mathbb{R}_{\geq 0}^{7\times 4\times 7\times 4}$, and $\mathbb{R}_{\geq 0}^{7\times 2\times 2\times 7\times 2\times 2}$. For notation convenience, we write their corresponding poset structures as $\Omega_{28\times 28}$, $\Omega_{7\times 4\times 7\times 4}$, and $\Omega_{7\times 2\times 2\times 7\times 2\times 2}$, and further write the many-body approximation sub-manifold (Equation 1) as $\mathcal{M}_{\ell}(\Omega)$ and its dual (Equation 2) as $\mathcal{M}^{\ast}_{\ell}(N,\Omega)$ for a particular poset $\Omega$ to emphasize the dependency. In particular, throughout this section, we consider the default local data sub-manifold, i.e., $\mathcal{D}=\mathcal{M}^{\ast}_{\ell}(N,\Omega_{7\times 2\times 2\times 7
\times 2\times 2})$, for consistency. Finally, we consider ranging $\ell$ from $1$ to at most $4$ where we neglect the degenerate case: for instance, in the case of $\Omega_{28\times 28}$, $\mathcal{M}_{2}(\Omega_{28\times 28})=\mathcal{S}$ as there are only two modes for a matrix, therefore degenerates to direct fitting which is not of interest (see Section A.3).

The results for the finest structure $\mathbb{R}_{\geq 0}^{7\times 2\times 2\times 7\times 2\times 2}$ are shown in Figures 12 and 13. As $\ell$ grows, the forward projection results in Figure 12 preserve the structure of the data better, subsequently improving the quality of the augmented data, as shown in Figure 13.

Figure: (a) Result with base sub-manifold $\mathcal{B}=\mathcal{M}_{1}(\Omega_{7\times 2\times 2\times 7\times 2\times 2})$ ($\dim(\mathcal{B})=17$).
Refer to caption: x38.png

Figure: (a) Result with base sub-manifold $\mathcal{B}=\mathcal{M}_{1}(\Omega_{7\times 2\times 2\times 7\times 2\times 2})$ ($\dim(\mathcal{B})=17$).
Refer to caption: x46.png

Similar trends can be found in the case of $\mathbb{R}_{\geq 0}^{7\times 4\times 7\times 4}$, as shown in Figures 14 and 15.

Figure: (a) Result with base sub-manifold $\mathcal{B}=\mathcal{M}_{1}(\Omega_{7\times 4\times 7\times 4})$ ($\dim(\mathcal{B})=19$).
Refer to caption: x50.png

Figure: (a) Result with base sub-manifold $\mathcal{B}=\mathcal{M}_{1}(\Omega_{7\times 4\times 7\times 4})$ ($\dim(\mathcal{B})=19$).
Refer to caption: x53.png

If we look at the results when using the original matrix structure $\mathbb{R}_{\geq 0}^{28\times 28}$ (Figures 16 and 17), some interesting comparison can be made. Firstly, if we compare the augmentation results for $\mathcal{B}=\mathcal{M}_{1}(\Omega_{28\times 28})$ (Figure 17) with the finer structures counterparts, e.g., Figure 13(a) for $\mathcal{M}_{1}(\Omega_{7\times 2\times 2\times 7\times 2\times 2})$, one can observe that the results are worse. However, the former requires more dimension ($\dim(\mathcal{M}_{1}(\Omega_{28\times 28}))=55>17=\dim(\mathcal{M}_{1}(\Omega_
{7\times 2\times 2\times 7\times 2\times 2}))$) for the base sub-manifold. Similarly, the augmentation results with $\mathcal{B}=\mathcal{M}_{1}(\Omega_{7\times 4\times 7\times 4})$ (Figure 15(a)) also achieve better performance with a lower base sub-manifold dimension.

Figure: Figure 16: Result with base sub-manifold $\mathcal{B}=\mathcal{M}_{1}(\Omega_{28\times 28})$ ($\dim(\mathcal{B})=55$). (*Top*) Forward projected data with tensor structure $\mathbb{R}_{\geq 0}^{28\times 28}$. (*Bottom*) Heat map of corresponding $\theta$ values.
Refer to caption: x55.png

Figure: Figure 17: Augmented data via Algorithm 4.2 with tensor structure $\mathbb{R}_{\geq 0}^{28\times 28}$ and base sub-manifold $\mathcal{B}=\mathcal{M}_{1}(\Omega_{28\times 28})$ ($\dim(\mathcal{B})=55$).
Refer to caption: x57.png

###### Remark A.1 .

Many-body approximation is a type of “oblivious” construction for base sub-manifolds that is expected to work well for general positive tensor data. However, alternative choices for $\mathcal{B}$, not necessarily $\mathcal{M}_{\ell}(\Omega)$ for some $\ell$, could be investigated when specific knowledge about the underlying data is available. Nonetheless, our approach demonstrates non-trivial performance, both in downstream tasks and through visual inspection.

###### Remark A.1 .

### A.5 Additional dataset

In addition to Section 5.3, we demonstrate our approach’s efficacy on various non-image datasets in this section. In particular, we consider the following UCI datasets:

- (a)
*Connectionist Bench (Sonar, Mines vs. Rocks)* (Sejnowski & Gorman, ) ($208$ samples, $60$ features, $2$ classes),
- (b)
*Taiwanese Bankruptcy Prediction* (Journal, ) ($6819$ samples, $95$ features, $2$ classes),
- (c)
*Musk (Version 2)* (Chapman & Jain, ) ($6497$ samples, $12$ features, $11$ classes).

We summarize the parameters we use for Algorithm 4.2 on each dataset in Table 2. In particular, for each dataset, the training set consists of $80\%$ of the total dataset, and the remaining $20\%$ is used for testing accuracy. Furthermore, the augmented dataset generated also consists of $20\%$ of the training data for each class, similar to the setting in Section 5.3. On the other hand, the autoencoder model trained in each experiment is with hidden-dimension $\dim(\mathcal{B})$ for a fair comparison.

**Table 2: Setting of Algorithm 4.2 for different UCI datasets.**
| Dataset | Poset $\Omega$ | Base $\mathcal{B}$ | Local Data $\mathcal{D}$ | Bandwidth | $k$ |
| --- | --- | --- | --- | --- | --- |
| a | $\Omega_{2\times 2\times 3\times 5}$ | $\mathcal{M}_{2}(\Omega_{2\times 2\times 3\times 5})$ | $\mathcal{M}^{\ast}_{1}(\Omega_{2\times 2\times 3\times 5})$ | $0.05$ | $4$ |
| b | $\Omega_{5\times 19}$ | $\mathcal{M}_{1}(\Omega_{5\times 19})$ | $\mathcal{M}^{\ast}_{1}(\Omega_{5\times 19})$ | $0.05$ | $8$ |
| c | $\Omega_{2\times 2\times 3}$ | $\mathcal{M}_{2}(\Omega_{2\times 2\times 3})$ | $\mathcal{M}^{\ast}_{1}(\Omega_{2\times 2\times 3})$ | $0.1$ | $10$ |

The results are shown in Table 3. The models perform similarly in most cases, with the fact that augmentation indeed helps with the downstream tasks’ performance. We conclude that Algorithm 4.2 is competitive compared to autoencoders regarding the quality of the downstream tasks.

**Table 3: Test accuracy of the linear classifier trained on different datasets.**
| Dataset | Training Set |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Original | Ours | AE | Original$+$Ours | Original$+$AE |  |
| a | $60.10\pm 11.46\%$ | $55.50\pm 14.92\%$ | $66.30\pm 10.76\%$ | $75.40\pm 12.40\%$ | $64.80\pm 12,85\%$ |
| b | $96.90\pm 4.55\%$ | $81.10\pm 11.12\%$ | $58.3\pm 14.04\%$ | $97.30\pm 4.34\%$ | $97.40\pm 4.82\%$ |
| c | $42.40\pm 14.96\%$ | $21.80\pm 11.95\%$ | $20.80\pm 9.79\%$ | $43.00\pm 11.19\%$ | $44.10\pm 14.43\%$ |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: - Could you comment on the practical applicability (e.g. performance gain, computational cost, scalability) of the method?
- Line 430, comparing the proposed method and AE: the paper says that the proposed method gives a blurred effect; could you clarify why this is desirable? 
  - Note also that blurring can also be obtained by using a VAE. Moreover, the paper seems to think of the blurred effect as a benefit that differentiate the proposed method from AE, whereas blurring is often considered an undesired behavior of VAE.
soundness: 3
strengths: - The proposed method has a motivation from information geometry.
- The paper provides the necessary background.
summary: The paper uses poset (partially-ordered set) to capture structures in the data.
Specifically, given a dataset, the data is modeled by a real-valued poset, where the partial ordering is chosen based on the data.

Then, data augmentation is performed in 5 steps:
1. map the poset to a statistical manifold $\mathcal{S}$ using an embedding function $\varphi$ (which can be nonlinear);
2. project to a lower-dimensional manifold $\mathcal{B}$ for dimension reduction;
3. sample from $\mathcal{B}$ (the distribution is estimated by e.g. a kernel density estimator);
4. backward project from $\mathcal{B}$ to $\mathcal{S}$; specifically, given a point $w^* \in \mathcal{B}$, let $w_1, \cdots, w_k \subset \mathcal{B}$ be the $k$-nearest neighbors of $w^*$ and let $z_1', \cdots, z_k' \subset \mathcal{S}$ be the corresponding pre-images, then the backward projection will map $w^*$ to a projection on the local manifold defined by $z_1', \cdots, z_k'$.
5. map to the space of the original data using the inverse of the embedding function, i.e. $\varphi^{-1}$.

The above method is _pseudo-non-linear_, in the sense that it is an interplay between linear constraints on the manifold (from log-linear energy models), and nonlinear mapping from the original data to the manifold.

The paper then empirically verifies the effectiveness of this data augmentation method. Using data augmented by the proposed method improves the MNIST accuracy from 81.8% to 83.4%. In comparison, data augmented using an autoencoder (AE) improves the accuracy to 82.7%.
weaknesses: - The proposed method provides limited empirical gain. The proposed method outperforms AE by 0.7% accuracy on MNIST and 6.6% gain on Connectionist Bench, but performs worse than AE on the other two datasets (Taiwanese Bankruptcy Prediction and Musk).
- The experiments are limited to small-scale datasets (with <10k samples and <1k dimension), and it's unclear whether the proposed method is scalable (for computational reasons) to more complex datasets.
- There's limited quantitative understanding on how much the data augmentation helps. For example, how does the performance change as a function of the number of augmented samples?
- The proposed framework requires partial ordering, and hence cannot capture invariant or equivariant structures, which the paper also mentions.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: **Suggestions** – Please refer to the Weaknesses section:
- Reorganize the method section to better guide the reader, starting with a global overview of the proposed approach, illustrated with a figure.
- Conduct a thorough experimental validation on **real, higher-resolution datasets, beyond simple linear classification, and integrating standard data augmentations.**
- Expand the evaluation to include structured data types beyond 2D images
soundness: 2
strengths: - The paper tackles a relevant research question through an elaborate and original angle. The approach is well-motivated and the paper is well written
- The proposed augmentation method is an interpretable generative approach that stands out from black-box generative models.
- The approach is applicable to any kind of structured data, which gives it a lot of potential.
summary: The authors propose a novel, interpretable data augmentation approach for structured data, that perturbs data within a dimensionality-reduced space.  Specifically,
- This space is constructed by i) embedding structured data into a *statistical manifold* (*i.e.*, discrete probability distributions) via an explicit mapping $\phi$ using an energy-based model (built upon the log-linear model on posets), and ii) projecting the embedded data into a low-dimensional sub-manifold. 
- The perturbation is performed by fitting a kernel density estimator to the data distribution in this sub-manifold and sampling from it. 
- The perturbed data is then *projected back* into the original space through a *backward projection* algorithm. This algorithm creates a local mapping around each perturbed point using the embeddings of the nearest known training points.

The proposed data augmentation is evaluated on MNIST by training a linear classifier, and compared with using an autoencoder model ($2+2$ layers) as data augmentation.
weaknesses: **Method section**
 
I am quite unfamiliar with the theory underlying Sections 3 and 4.1, and it is very difficult to parse these sections without a good knowledge of the domain. 
I believe that the paper would be more "accessible" if Section 4 started with a more global outline, where the different steps of the augmentation approach are mentioned: 1. Embedding on a statistical manifold, 2. Projection into a sub-manifold, 3. Perturbing the data, 4. Defining a backward projection. 
It would be good to guide the reader on the contents of the different subsections, such that the fact that section 4.1 focuses on step 1. One should not have to read the entire Section 4 to understand its global structure. 

Illustrative figures such as Figure 2 should be introduced early in Section 4 to directly provide a broad overview of the whole augmentation process, with a caption helping the figure to be self-contained. 


**Experimental validation**

My main concern regards the empirical validation of the proposed augmentation method.
The visual and quantitative evaluations are solely based on MNIST. This is very far from demonstrating the potential of the proposed approach.
The performance obtained using a simple linear classifier trained on $28 \times 28$ digit images will not give a clue on whether the augmentation helps in real scenarios with **natural images of higher resolution** and **beyond linear classification**. The paper should also evaluate whether the proposed augmentation **complements or outperforms standard data augmentation methods**. 

Even a small-scale evaluation would be valuable, like training a ResNet-18 on a small dataset such as CUB, with appropriate data augmentation (e.g., color jittering, random flipping, random resizing and cropping), and seeing whether adding the proposed augmentation improves results. The proposed method could also be compared to simple, image-level perturbations such as Gaussian noise or blurring. Along with quantitative evaluations, visualizations on data beyond MNIST would also be appreciated.

One of the strengths of the proposed approach being its applicability to any kind of structured data, I would have liked to see evaluations on data structures beyond 2D images.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: No.
soundness: 3
strengths: * As far as I know, the proposed approach is very original and novel.
* It does seem to provide some measure of interpretability (at least on MNIST, it wasn't demonstrated on the UCI datasets).
* Figures 1 and 2 are great and help to get a better intuition.
summary: The paper presents a new generative model based on information geometry principles. The main appeal of the method is that it is interpretable, unlike NN-based models. According to the proposed approach, data is first mapped to posets, then it is embedded into the statistical manifold having coordinate systems defined by natural and expectation parameters of exponential distributions. On this new representation encoding and decoding can be done. The authors suggest a projection scheme for encoding points, a method based on KDE for sampling novel points on the embedded submanifold, and a decoding scheme to project points backward to the manifold. The authors demonstrate their approach on MNIST and three UCI datasets.
weaknesses: Although novel and interesting, in my opinion, the paper is not ready for publication at ICLR and requires substantial work. Specifically,
* Clarity of exposition:
  * Information geometry is relatively a small niche in the field of ML. It is not a bad thing by any means, but it requires adding a substantial background on the field and the main concepts, even concepts that arrive from standard differential and Riemannian geometry. I do not think that simply referring to Amari (2016) as the authors did is nearly enough.
  * I believe that the paper needs to be restructured in a different way. Example 4.2 is great. I think that the authors should use it as a running example throughout the paper (from the start of section 4) and demonstrate each element of the method using it.
  * In my opinion the paper will be much clearer if the authors use a concrete example (i.e., vectors with numbers) to demonstrate every step of the algorithm (mapping to posets, mapping to the statistical manifold, etc.). Preferably it should be done with the running example.

* I think it would help to clarify the design choices in the paper and discuss their advantages and limitations. For instance, when is modeling with postes or using the many-body tensor approximation challenging, and why? In addition, perhaps the authors should clarify if there are alternatives to these choices and why not choose them.

* Experiments:
  * Evaluating the model on MNIST and UCI datasets is not enough. Even adding CIFAR-10 or data from other modalities (e.g. audio) will greatly improve the empirical evaluation part.
  * Perhaps I do not understand something, but I do not see an improvement in the generated augmented data when increasing the dimension of the submanifold. In addition, for all sizes, the image quality is not good. Do you think it can be made better?
  *  The authors state that AEs are overfitting the training set (Lines 430-431), yet to me it seems that this approach also overfits as the generated images look very similar to the closest images on the training data in all figures. Perhaps the improvements in test accuracy (Table 1) arise from the blurring effect (which acts as a form of standard data augmentation)? Further empirical evaluation will help clarify this ambiguity.

* Minor:
  * In line 29, the paper by Kingma & Welling, is from 2014.
  * In line 395, did you mean 100 images per digit?

---

confidence: 4
contribution: 1
initial_rating: 1
presentation: 2
questions: What benefit does the Riemannian framework add theoretically or practically? Couldn't the approach and your experiments be described using only simpler linear algebra?
soundness: 3
strengths: - The paper is mostly well-written and easy to follow
summary: The paper proposes a data augmentation setup by using a new encoder-decoder approach and then using a simple KDE model to sample from the latent state and use the reconstructed point as additional training data.
weaknesses: Main issues:
- The abstract promises "our approach replaces these non-interpretable transformations with explicit, theoretically grounded ones" but does not deliver any theoretical grounding, and as such this claim is very misleading. In the end, your reconstruction has the same ambiguity problem that it solves using the closest element in some arbitrary norm instead of another arbitrary norm. 

- Expanding on the previous point, the paper uses Riemannian geometry concepts but it isn't really clear why they are needed and what is the reasoning. It feels like these ideas were added for extra complexity without any added value. For example in Example 4.2 you start with a positive tensor as input and using your multi-set embedding and poset formulation you end up with an embedding that is simply the normalized flattened tensor. 

- Key parts of the algorithm weren't explained. For one you say you project to a flat base sub-manifold, but don't explain how this should be done or how one should identify such a submanifold as these are not trivial conditions. 

- Your reconstruction is based on a "local data submanifold" that again isn't explained properly. In the example, you take the nearest neighbors and (if I understand the notations correctly) constrain it to have some of the indices fixed to one of the nearest neighbors. This first of all is a linear subspace so again it raises the question of why do I need the whole Riemannian geometry mechanism. Second, this uses only one of the nearest neighbors so how do you pick the one? Why don't you use the others?

- Experiments are very nonconvincing, they use the very simple MNIST dataset (and even so with only 500 samples), they don't even compare to the basic PCA, and their gain is within the error margins so that the claim that this approach had any positive impact cannot be said with any statistical significance.

Meta-review: This paper explores data augmentation methods based on energy-based modelling.

Reviewers raised many concerns with the work, including that there was a lack of theoretical results contrary to the claim the work was theoretically grounded, that the methods section was not clearly laid out and summarized, there were no figures that help visualize the method, and that the experimental results were small-scale, did not show significant improvements, and did not consider ablations to provide quantitative understanding of the method

The authors did not respond to any of the reviews within the discussion period. Due to the author’s lack of engagement in the peer review process, I am recommending rejection.

Decision: Reject