SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kazu Ghalamkari
kazu.ghalamkari@riken.jp
Mahito Sugiyama
National Institute of Informatics
SOKENDAI
mahito@nii.ac.jp
Yoshinobu Kawahara
kawahara@ist.osaka-u.ac.jp
We present an alternative approach to decompose non-negative tensors, called many-body approximation. Traditional decomposition methods assume low-rankness in the representation, resulting in difficulties in global optimization and target rank selection. We avoid these problems by energy-based modeling of tensors, where a tensor and its mode correspond to a probability distribution and a random variable, respectively. Our model can be globally optimized in terms of the KL divergence minimization by taking the interaction between variables (that is, modes), into account that can be tuned more intuitively than ranks. Furthermore, we visualize interactions between modes as tensor networks and reveal a nontrivial relationship between many-body approximation and low-rank approximation. We demonstrate the effectiveness of our approach in tensor completion and approximation.
Tensors are generalizations of vectors and matrices. Data in fields such as neuroscience [8], bioinformatics [22], signal processing [7], and computer vision [30] are often stored in the form of tensors, and features are extracted from them. Tensor decomposition and its non-negative version [33] are popular methods to extract features by approximating tensors by the sum of products of smaller tensors. These methods usually seek to minimize the reconstruction error, the difference between an original tensor and the tensor reconstructed from obtained smaller tensors.
In tensor decomposition approaches, a low-rank structure is typically assumed, where a given tensor is essentially represented by a linear combination of a small number of bases. Such decomposition requires two kinds of information: structure and the number of bases used in the decomposition. The structure specifies the type of decomposition such as CP decomposition [15] and Tucker decomposition [37]. Tensor networks, which were initially introduced in physics, have become popular in the machine learning community recently because they help intuitive and flexible model design, including tensor train decomposition [29], tensor ring decomposition [41], and tensor tree decomposition [25]. Traditional tensor structures in physics, such as MERA [32] and PEPS [4], are also used in machine learning. The number of bases is often referred to as the rank. Larger ranks increase the capability of the model while increasing the computational cost, resulting in the tradeoff problem of rank tuning that requires a careful treatment by the user. Since these low-rank structure-based decompositions via reconstruction error minimization are non-convex, which causes initial value dependence [17, Chapter 3], the problem of finding an appropriate setting of the low-rank structure is highly nontrivial
C.
Figure 1: Interaction representations corresponding to (a) second- and third-order energy (b) two- and (c) three-body approximation.
in practice as it is hard to locate the cause if the decomposition does not perform well. As a result, in order to find proper structure and rank, the user must often perform decomposition multiple times with various settings, which is time- and memory-consuming. Although there is an attempt to avoid rank tuning by approximating it by trace norms [2], it also requires another hyperparameter: the weights of the unfolding tensor. Hence, such an approach does not fundamentally solve the problem.
Instead of the low-rank structure that has been the focus of attention in the past, in the present paper we propose a novel formulation of non-negative tensor decomposition, called many-body approximation, that focuses on the relationship among modes of tensors. The structure of decomposition can be naturally determined based on the existence of the interactions between modes. The proposed method requires only the decomposition structure and does not require the rank value, which traditional decomposition methods also require as a hyperparameter and often suffer to determine.
To describe interactions between modes, we follow the standard strategy in statistical mechanics that uses an energy function  $\mathcal{H}(\cdot)$  to treat interactions and considers the corresponding distribution  $\exp(-\mathcal{H}(\cdot))$ . This model is known to be an energy-based model in machine learning [19] and is exploited in tensor decomposition as Legendre decomposition [36]. Technically, it parameterizes a tensor as a discrete probability distribution and reduces the number of parameters by enforcing some of them to be zero in optimization. We explore this energy-based approach further and discover the family of parameter sets that represent interactions between modes in the energy function  $\mathcal{H}(\cdot)$ . How to choose non-zero parameters in Legendre decomposition has been an open problem; we start by addressing this problem and propose many-body approximation as a special case of Legendre decomposition. Moreover, although Legendre decomposition is not factorization of tensors in general, our proposal always offers factorization, which can reveal patterns in tensors. Since the advantage of Legendre decomposition is inherited to our proposal, many-body approximation can be achieved by convex optimization that globally minimizes the Kullback-Leibler (KL) divergence [18].
Furthermore, we introduce a way of representing mode interactions, that visualizes the presence or absence of interactions between modes as a diagram. We discuss the relation to the tensor network and point out that an operation called coarse-grained transformation [20] – in which multiple tensors are viewed as a new tensor – reveals unexpected relationship between the proposed method and existing methods such as tensor ring and tensor tree decomposition.
We summarize our contribution as follows:
- By focusing on the interaction between modes of tensors, we introduce rank-free tensor decomposition, called many-body approximation. This decomposition is achieved by convex optimization.
- We present a way of describing tensor many-body approximation called interaction representation, which is a diagram that shows interactions within a tensor and can be transformed into tensor networks.
- Many-body approximation can perform tensor completion via the em-algorithm, which empirically shows better performance than low-rank based existing methods.
2 Many-body Approximation for Tensors
Our proposal, many-body approximation, is based on the formulation of Legendre decomposition for tensors, which we first review in Section 2.1. Then we introduce interactions between modes and its visual representation in Section 2.2, many-body approximation in Section 2.3, and transformation of interaction representation into tensor networks in Section 2.4. In the following discussion, we consider  $D$ -order non-negative tensors with the size  $(I_1,\ldots ,I_D)$ , and we denote by  $[K] = \{1,2,\dots ,K\}$  for a positive integer  $K$ . We assume the sum of all elements in a given tensor to be 1 for simplicity,
while this assumption can be eliminated using the general property of the Kullback-Leibler (KL) divergence,  $\lambda D_{KL}(\mathcal{P},\mathcal{Q}) = D_{KL}(\lambda \mathcal{P},\lambda \mathcal{Q})$ , for any  $\lambda \in \mathbb{R}_{>0}$ .
2.1 Reminder to Legendre Decomposition and its optimization
Legendre decomposition is a method for decomposing a non-negative tensor by regarding the tensor as a discrete distribution and representing it with a limited number of parameters. We describe a non-negative tensor  $\mathcal{P}$  using parameters  $\pmb{\theta} = (\theta_{2,\dots,1},\dots,\theta_{I_1,\dots,I_D})$  and its energy function  $\mathcal{H}$  as
$$
\mathcal {P} _ {i _ {1}, \dots , i _ {D}} = \exp \left(- \mathcal {H} _ {i _ {1}, \dots , i _ {D}}\right), \quad \mathcal {H} _ {i _ {1}, \dots , i _ {D}} = - \sum_ {i _ {1} ^ {\prime} = 1} ^ {i _ {1}} \dots \sum_ {i _ {D} ^ {\prime} = 1} ^ {i _ {D}} \theta_ {i _ {1} ^ {\prime}, \dots , i _ {D} ^ {\prime}}, \tag {1}
$$
where  $\theta_{1,\dots,1}$  has a role of normalization. It is clear that the index set of a tensor corresponds to sample space of a distribution and the value of each entry of the tensor is regarded as the probability of realizing the corresponding index [35].
As we see in Equation (1), we can uniquely identify tensors from the parameters  $\theta$ . Inversely, we can compute  $\theta$  from a given tensor as
$$
\theta_ {i _ {1}, \dots , i _ {D}} = \sum_ {i _ {1} ^ {\prime} = 1} ^ {I _ {1}} \dots \sum_ {i _ {D} ^ {\prime} = 1} ^ {I _ {D}} \mu_ {i _ {1}, \dots , i _ {D}} ^ {i _ {1} ^ {\prime}, \dots , i _ {D} ^ {\prime}} \log \mathcal {P} _ {i _ {1} ^ {\prime}, \dots , i _ {D} ^ {\prime}} \tag {2}
$$
using the Möbius function  $\mu : S \times S \to \{-1, 0, +1\}$ , where  $S$  is the set of indices, defined inductively as follows:
$$
\mu_ {i _ {1}, \ldots , i _ {D}} ^ {i _ {1} ^ {\prime}, \ldots , i _ {D} ^ {\prime}} = \left\{ \begin{array}{l l} 1 & \text {i f} i _ {d} = i _ {d} ^ {\prime}, \forall d \in [ D ], \\ - \prod_ {d = 1} ^ {D} \sum_ {j _ {d} = i _ {d}} ^ {i _ {d} ^ {\prime} - 1} \mu_ {i _ {1}, \ldots , i _ {D}} ^ {j _ {1}, \ldots j _ {D}} & \text {e l s e i f} i _ {d} \leq i _ {d} ^ {\prime}, \forall d \in [ D ], \\ 0 & \text {o t h e r w i s e}. \end{array} \right.
$$
Since distribution described by Equation (1) belongs to the exponential family,  $\theta$  corresponds to natural parameters of the exponential family, and we can also identify each tensor by expectation parameters  $\pmb{\eta} = (\eta_{2,\dots,1},\dots,\eta_{I_1,\dots,I_D})$  using Möbius inversion as
$$
\eta_ {i _ {1}, \dots , i _ {D}} = \sum_ {i _ {1} ^ {\prime} = i _ {1}} ^ {I _ {1}} \dots \sum_ {i _ {D} ^ {\prime} = i _ {D}} ^ {I _ {D}} \mathcal {P} _ {i _ {1} ^ {\prime}, \dots , i _ {D} ^ {\prime}}, \quad \mathcal {P} _ {i _ {1}, \dots , i _ {D}} = \sum_ {i _ {1} ^ {\prime} = 1} ^ {I _ {1}} \dots \sum_ {i _ {D} ^ {\prime} = 1} ^ {I _ {D}} \mu_ {i _ {1}, \dots , i _ {D}} ^ {i _ {1} ^ {\prime}, \dots , i _ {D} ^ {\prime}} \eta_ {i _ {1} ^ {\prime}, \dots , i _ {D} ^ {\prime}}, \tag {3}
$$
where  $\eta_{1,\dots,1} = 1$  because of normalization. See Section A in Supplement for examples of the above calculation. Since distribution is determined by specifying either  $\theta$ -parameters or  $\eta$ -parameters, they form two coordinate systems, called the  $\theta$ -coordinate system and the  $\eta$ -coordinate system, respectively.
2.1.1 Optimization
Legendre decomposition approximates a tensor by setting some  $\theta$  values to be zero, which corresponds to dropping some parameters for regularization. It achieves convex optimization using the dual flatness of  $\theta$ - and  $\eta$ -coordinate systems. Let  $B$  be the set of indices of  $\theta$  parameters that are not imposed to be 0. Then Legendre decomposition coincides with a projection of a given nonnegative tensor  $\mathcal{P}$  onto the subspace  $\mathcal{B} = \{\mathcal{Q} \mid \theta_{i_1,\ldots,i_D} = 0 \text{ if } (i_1,\ldots,i_D) \notin B \text{ for } \theta$ -parameters of  $\mathcal{Q}\}$ .
Let us consider projection of a given tensor  $\mathcal{P}$  onto  $\mathcal{B}$ . The space of probability distributions, which is not a Euclidean space, is studied in information geometry. By taking geometry of probability distributions into account, we can introduce the concept of flatness for a set of tensors. The subspace  $\mathcal{B}$  is  $e$ -flat if the logarithmic combination, or called  $e$ -geodesic,  $\mathcal{R} \in \{(1 - t)\log \mathcal{Q}_1 + t\log \mathcal{Q}_2 - \phi(t) \mid 0 < t < 1\}$  of any two points  $\mathcal{Q}_1, \mathcal{Q}_2 \in \mathcal{B}$  is included in the subspace  $\mathcal{B}$ , where  $\phi(t)$  is a normalizer. There is always a unique point  $\overline{\mathcal{P}}$  on the  $e$ -flat subspace that minimizes the KL divergence from any point  $\mathcal{P}$ .
$$
\overline {{\mathcal {P}}} = \underset {\mathcal {Q}; \mathcal {Q} \in \boldsymbol {\mathcal {B}}} {\arg \min } D _ {K L} (\mathcal {P}, \mathcal {Q}). \tag {4}
$$
This projection is called the  $m$ -projection. The  $m$ -projection onto an  $e$ -flat subspace is convex optimization. We define two vectors  $\pmb{\theta}^{B} = (\theta_{b})_{b\in B}$  and  $\pmb{\eta}^{B} = (\eta_{b})_{b\in B}$  and write the dimensionality of these vectors as  $|B|$  since it is equal to the cardinality of  $B$ . The derivative of the KL divergence and the Hessian matrix  $G\in \mathbb{R}^{|B|\times |B|}$  are given as
$$
\frac {\partial}{\partial \boldsymbol {\theta} ^ {B}} D _ {K L} (\mathcal {P}, \mathcal {Q}) = \boldsymbol {\eta} ^ {B} - \hat {\boldsymbol {\eta}} ^ {B}, \quad \mathbf {G} _ {u, v} = \eta_ {\max  (i _ {1}, j _ {1}), \dots , \max  (i _ {D}, j _ {D})} - \eta_ {i _ {1}, \dots , i _ {D}} \eta_ {j _ {1}, \dots , j _ {D}}, \tag {5}
$$
where  $\pmb{\eta}^B$  and  $\hat{\pmb{\eta}}^B$  are the expectation parameters of  $\mathcal{Q}$  and  $\mathcal{P}$ , respectively, and  $u = (i_1,\dots ,i_D)$ ,  $v = (j_{1},\ldots ,j_{D})\in B$ . This matrix  $\mathbf{G}$  is also known as the negative Fisher information matrix. Using gradient descent with second-order derivative, we can update  $\theta^B$  in each iteration  $t$  as
$$
\boldsymbol {\theta} _ {t + 1} ^ {B} = \boldsymbol {\theta} _ {t} ^ {B} - \mathbf {G} ^ {- 1} \left(\boldsymbol {\eta} _ {t} ^ {B} - \hat {\boldsymbol {\eta}} ^ {B}\right). \tag {6}
$$
The distribution  $\mathcal{Q}_{t + 1}$  is calculated from the updated natural parameters  $\theta_{t + 1}$ . This step finds a point  $\mathcal{Q}_{t + 1} \in \mathcal{B}$  that is closer to the destination  $\overline{\mathcal{P}}$  along with the  $e$ -geodesic from  $\mathcal{Q}_t$  to  $\overline{\mathcal{P}}$ . We can also calculate the expected value parameters  $\eta_{t + 1}$  from the distribution. By repeating this process until convergence, we can always find the globally optimal solution satisfying Equation (4). See Section A in Supplement for more detail on the optimization. The problem of how to design and interpret  $B$  has not been explored before, and we firstly give the reasoning of  $B$  as the presence or absence of particular interactions between modes, which naturally leads to the design guideline of  $B$ . The computational complexity of Legendre decomposition is  $\mathcal{O}(\gamma |B|^3)$ , where  $\gamma$  is the number of iterations.
2.2 Interaction and its representation of tensors
In this subsection, we introduce interactions between modes and its visual representation to prepare for many-body approximation. The following discussion enables us to intuitively describe relationships between modes and formulate our novel rank-free tensor decomposition.
First, we introduce  $n$ -body parameters, a generalized concept of one-body and two-body parameters in [11]. In any  $n$ -body parameter, there are  $n$  non-one indices; for example,  $\theta_{1,2,1,1}$  is a one-body parameter,  $\theta_{4,3,1,1}$  is a two-body parameter, and  $\theta_{1,2,4,3}$  is a three-body parameter. We also use the following notation for  $n$ -body parameters:
$$
\theta_ {i _ {k}} ^ {(k)} = \theta_ {1, \dots , 1, i _ {k}, 1, \dots , 1}, \quad \theta_ {i _ {k}, i _ {m}} ^ {(k, m)} = \theta_ {1, \dots , 1, i _ {k}, 1, \dots , 1, i _ {m}, 1, \dots , 1}, \quad \theta_ {i _ {k}, i _ {m}, i _ {p}} ^ {(k, m, p)} = \theta_ {1, \dots , i _ {k}, \dots , i _ {m}, \dots , i _ {p}, \dots , 1},
$$
for  $n = 1,2$  and 3, respectively. We write the energy function  $\mathcal{H}$  with  $n$ -body parameters as
$$
\mathcal {H} _ {i _ {1}, \dots , i _ {D}} = H _ {0} + \sum_ {m = 1} ^ {D} H _ {i _ {m}} ^ {(m)} + \sum_ {m = 1} ^ {D} \sum_ {k = 1} ^ {m - 1} H _ {i _ {k}, i _ {m}} ^ {(k, m)} + \sum_ {m = 1} ^ {D} \sum_ {k = 1} ^ {m - 1} \sum_ {p = 1} ^ {k - 1} H _ {i _ {p}, i _ {k}, i _ {m}} ^ {(p, k, m)} + \dots + H _ {i _ {1}, \dots , i _ {D}} ^ {(1, \dots , D)}, \tag {7}
$$
where the  $n$ -th order energy is introduced as
$$
H _ {i _ {l _ {1}}, \dots , i _ {l _ {n}}} ^ {(l _ {1}, \dots , l _ {n})} = - \sum_ {i _ {l _ {1}} ^ {\prime} = 2} ^ {i _ {l _ {1}}} \dots \sum_ {i _ {l _ {n}} ^ {\prime} = 2} ^ {i _ {l _ {n}}} \theta_ {i _ {l _ {1}} ^ {\prime}, \dots , i _ {l _ {n}} ^ {\prime}} ^ {(l _ {1}, \dots , l _ {n})}. \tag {8}
$$
For simplicity, we suppose that  $1 \leq l_{1} < l_{2} < \dots < l_{n} \leq D$  holds. We set  $H_{0} = -\theta_{1,\ldots,1}$ . We say that an  $n$ -body interaction exists between modes  $l_{1}, \ldots, l_{n}$  if there are indices  $i_{l_{1}}, \ldots, i_{l_{n}}$  satisfying  $H_{i_{l_{1}}, \ldots, i_{l_{n}}}^{(l_{1}, \ldots, l_{n})} \neq 0$ . The first term,  $H_{0}$  in Equation (7), is called the normalized factor or the partition function. The terms  $H^{(k)}$  are called bias in machine learning and magnetic field or self-energy in statistical physics. The terms  $H^{(k,m)}$  are called the weight of the Boltzmann machine in machine learning and two-body interaction or electron-electron interaction in physics.
To visualize the existence of interactions within a tensor, we introduce a diagram called interaction representation, which is inspired by factor graphs in graphical modeling [3, Chapter 8]. The graphical representation of the product of tensors is widely known as tensor networks. However, displaying the relations between modes of a tensor as a factor graph is our novel approach. We represent the  $n$ -body interaction as a black square,  $\blacksquare$ , connected with  $n$  modes. We describe examples of the
Figure 2: (a)(left) Interaction representation of an example of cyclic two-body approximation and (a)(right) its transformed tensor network for  $D = 4$ . Each tensor is enclosed by a square and each mode is enclosed by a circle. A black circle  $\bullet$  is a hyperdiagonal tensor. Edges through  $\blacksquare$  between modes mean interaction existence. (b) Tensor network of tensor ring decomposition. (c) The relationship among solution spaces of tensor-ring decomposition  $\mathcal{B}_{\mathrm{ring}}$ , two-body approximation  $\mathcal{B}_2$ , and cyclic two-body approximation  $\mathcal{B}_{\mathrm{cyc}}$ .
two-body interaction between modes  $(k, m)$  and the three-body interaction among modes  $(k, m, p)$  in Figure 1(a). Combining these interactions, the diagram for the energy function including all two-body interactions is shown in Figure 1(b), and all two-body and three-body interactions are included in Figure 1(c) for  $D = 4$ . This visualization allows us to intuitively understand the relationship between modes of tensors. For simplicity, we abbreviate one-body interactions in the diagrams, while we always assume them. Once interaction representation is given, we can determine the corresponding decomposition of tensors.
In Boltzmann machines, we usually consider binary (two-level) variables and their second-order interactions. In our proposal, we consider multi-level  $D$  variables, each of which can take a natural number from 1 to  $I_{d}$  for  $d \in [D]$ . Moreover, higher-order interactions among them are allowed. Therefore, our proposal is a form of a multi-level extension of Boltzmann machines with higher-order interaction, where each node of Boltzmann machines corresponds to the tensor mode. In the following section, we reduce some of  $n$ -body interactions, - that is,  $H_{i_{l_1},\dots,i_{l_n}}^{(l_1,\dots,l_n)} = 0$  - by fixing each parameter  $\theta_{i_{l_1},\dots,i_{l_n}}^{(l_1,\dots,l_n)} = 0$  for all indices  $(i_{l_1},\ldots ,i_{l_n}) \in \{2,\ldots ,I_{l_1}\} \times \dots \times \{2,\ldots ,I_{l_n}\}$ .
2.3 Many-body approximation
We approximate a given tensor by assuming the existence of dominant interactions between the modes of the tensor and ignoring other interactions. Since this operation can be understood as enforcing some natural parameters of the distribution to be zero, it can be achieved by convex optimization through the theory of Legendre decomposition. We discuss how we can choose dominant interactions in Section 3.1.
As an example, we consider two types of approximations of a nonnegative tensor  $\mathcal{P}$  by tensors represented in Figure 1(b, c). If all energies greater than second-order or those greater than third-order in Equation (7) are ignored - that is,  $H_{i_{l_1},\dots,i_{l_n}}^{(l_1,\dots,l_n)} = 0$  for  $n > 2$  or  $n > 3$  - the tensor  $\mathcal{P}_{i_1,i_2,i_3,i_4}$  is approximated as follows:
$$
\mathcal {P} _ {i _ {1}, i _ {2}, i _ {3}, i _ {4}} \simeq \mathcal {P} _ {i _ {1}, i _ {2}, i _ {3}, i _ {4}} ^ {\leq 2} = \mathbf {X} _ {i _ {1}, i _ {2}} ^ {(1, 2)} \mathbf {X} _ {i _ {1}, i _ {3}} ^ {(1, 3)} \mathbf {X} _ {i _ {1}, i _ {4}} ^ {(1, 4)} \mathbf {X} _ {i _ {2}, i _ {3}} ^ {(2, 3)} \mathbf {X} _ {i _ {2}, i _ {4}} ^ {(2, 4)} \mathbf {X} _ {i _ {3}, i _ {4}} ^ {(3, 4)},
$$
$$
\mathcal {P} _ {i _ {1}, i _ {2}, i _ {3}, i _ {4}} \simeq \mathcal {P} _ {i _ {1}, i _ {2}, i _ {3}, i _ {4}} ^ {\le 3} = \chi_ {i _ {1}, i _ {2}, i _ {3}} ^ {(1, 2, 3)} \chi_ {i _ {1}, i _ {2}, i _ {4}} ^ {(1, 2, 4)} \chi_ {i _ {1}, i _ {3}, i _ {4}} ^ {(1, 3, 4)} \chi_ {i _ {2}, i _ {3}, i _ {4}} ^ {(2, 3, 4)},
$$
where each of matrices and tensors on the right-hand side is represented as
$$
\mathbf {X} _ {i _ {k}, i _ {m}} ^ {(k, m)} = \frac {1}{\sqrt [ 6 ]{Z}} \exp \left(\frac {1}{3} H _ {i _ {k}} ^ {(k)} + H _ {i _ {k}, i _ {m}} ^ {(k, m)} + \frac {1}{3} H _ {i _ {m}} ^ {(m)}\right),
$$
$$
\chi_ {i _ {k}, i _ {m}, i _ {p}} ^ {(k, m, p)} = \frac {1}{\sqrt [ 4 ]{Z}} \exp \bigg (\frac {H _ {i _ {k}} ^ {(k)}}{3} + \frac {H _ {i _ {m}} ^ {(m)}}{3} + \frac {H _ {i _ {p}} ^ {(p)}}{3} + \frac {1}{2} H _ {i _ {k}, i _ {m}} ^ {(k, m)} + \frac {1}{2} H _ {i _ {m}, i _ {p}} ^ {(m, p)} + \frac {1}{2} H _ {i _ {k}, i _ {p}} ^ {(k, p)} + H _ {i _ {k}, i _ {m}, i _ {p}} ^ {(k, m, p)} \bigg).
$$
The partition function, or the normalization factor, is given as  $Z = \exp(-\theta_{1,1,1,1})$ , which does not depend on indices  $(i_1, i_2, i_3, i_4)$ . Each  $\mathbf{X}^{(k,m)}$  (resp.  $\chi^{(k,m,p)}$ ) is a factorized representation for the relationship between  $k$ -th and  $m$ -th (resp.  $k$ -th,  $m$ -th and  $p$ -th) modes. Although our model can be
transformed into a linear model by taking the logarithm, our convex formulation enables us to find the optimal solution more stable than traditional linear low-rank-based nonconvex approaches. Since we do not impose any low-rankness, factorized representations, such as  $\mathbf{X}^{(k,m)}$  and  $\chi^{(k,m,p)}$ , can be full-rank matrices or tensors.
For a given tensor  $\mathcal{P}$ , we define its  $m$ -body approximation  $\mathcal{P}^{\leq m}$  as the optimal solution of Equation (4) - that is,  $\mathcal{P}^{\leq m} = \arg \min_{\mathcal{Q} \in \mathcal{B}_m} D_{KL}(\mathcal{P}, \mathcal{Q})$  - where the solution space  $\mathbf{B}_m$  is given as
$\mathcal{B}_m = \{\mathcal{Q} \mid \theta_{i_1, \dots, i_D} = 0$  if  $\theta_{i_1, \dots, i_D}$  is  $n (> m)$ -body parameters for  $\theta$ -parameters of  $\mathcal{Q}\}$ .
Note that  $\mathcal{P}^{\leq D} = \mathcal{P}$  always holds. For any natural number  $m < D$ , it holds that  $\mathcal{B}_m \subseteq \mathcal{B}_{m+1}$ . Interestingly, the two-body approximation for a non-negative tensor with  $I_1 = \dots = I_D = 2$  is equivalent to approximating the empirical distribution with the fully connected Boltzmann machine.
In the above discussion, we consider many-body approximation with all the  $n$ -body parameters, while we can relax this constraint and allow the use of only a part of  $n$ -body parameters in approximation. Let us consider the situation where only one-body interaction and two-body interaction between modes  $(d,d + 1)$  exist for all  $d\in [D]$  ( $D + 1$  implies 1 for simplicity). Figure 2(a) shows the interaction representation of the approximated tensor. As we can confirm by substituting 0 for  $H_{i_k,i_l}^{(k,l)}$  if  $l\neq k + 1$ , we can describe the approximated tensor as
$$
\mathcal {P} _ {i _ {1}, \dots , i _ {D}} \simeq \mathcal {P} _ {i _ {1}, \dots , i _ {D}} ^ {\text {c y c}} = \mathbf {X} _ {i _ {1}, i _ {2}} ^ {(1)} \mathbf {X} _ {i _ {2}, i _ {3}} ^ {(2)} \dots \mathbf {X} _ {i _ {D}, i _ {1}} ^ {(D)}, \tag {9}
$$