SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbf {X} _ {i _ {k}, i _ {k + 1}} ^ {(k)} = \frac {1}{\sqrt [ D ]{Z}} \exp \left(\frac {1}{2} H _ {i _ {k}} ^ {(k)} + H _ {i _ {k}, i _ {k + 1}} ^ {(k, k + 1)} + \frac {1}{2} H _ {i _ {k + 1}} ^ {(k + 1)}\right)
$$
with the normalization factor  $Z = \exp(-\theta_{1,\dots,1})$ . When the tensor  $\mathcal{P}$  is approximated by  $\mathcal{P}^{\mathrm{cyc}}$ , the set  $B$  contains only all one-body parameters and two-body parameters  $\theta_{i_d,i_{d+1}}^{(d,d+1)}$  for  $d \in [D]$ . We call this approximation cyclic two-body approximation since the order of indices in Equation (9) is cyclic. It holds that  $\mathbf{B}_{\mathrm{cyc}} \subseteq \mathbf{B}_2$  for the solution space of cyclic two-body approximation  $\mathbf{B}_{\mathrm{cyc}}$  (Figure 2(c)).
2.4 Connection to tensor networks
Our tensor interaction representation is an undirected graph that focuses on the relationship between modes. By contrast, tensor networks, which are well known as diagrams that focus on smaller tensors after decomposition, represent a tensor as an undirected graph, whose nodes correspond to matrices or tensors and edges to summation over a mode in tensor products [6].
We provide examples in our representation that can be converted to tensor networks, which implies a tight connection between our representation and tensor networks. For the conversion, we use a hyper-diagonal tensor  $\mathcal{I}$  defined as  $\mathcal{I}_{ijk} = \delta_{ij}\delta_{jk}\delta_{ki}$ , where  $\delta_{ij} = 1$  if  $i = j$  and 0 otherwise. The tensor  $\mathcal{I}$  is often represented by  $\bullet$  in tensor networks. In the community of tensor networks, the tensor  $\mathcal{I}$  appears in the CNOT gate and a special case of the Z spider [28]. The tensor network in Figure 2(a) represents Equation (9) for  $D = 4$ .
A remarkable finding is that the converted tensor network representation of cyclic two-body approximation and the tensor network of tensor ring decomposition, whose tensor network is shown in Figure 2(b), have a similar structure, despite their different modeling. If we consider the region enclosed by the dotted line in Figure 2(a) as a new tensor, the tensor network of the cyclic two-body approximation coincides with the tensor network of the tensor ring decomposition shown in Figure 2(b). This operation, in which multiple tensors are regarded as a new tensor in a tensor network, is called coarse-graining transformation [9].
Formally, cyclic two-body approximation coincides with tensor ring decomposition with a specific constraint as described below. Non-negative tensor ring decomposition approximates a given tensor  $\mathcal{P} \in \mathbb{R}_{\geq 0}^{I_1 \times \dots \times I_D}$  with  $D$  core tensors  $\chi^{(1)}, \ldots, \chi^{(D)} \in \mathbb{R}_{\geq 0}^{R_{d-1} \times I_d \times R_d}$  as
$$
\mathcal {P} _ {i _ {1}, \dots , i _ {D}} \simeq \sum_ {r _ {1} = 1} ^ {R _ {1}} \dots \sum_ {r _ {D} = 1} ^ {R _ {D}} \chi_ {r _ {D}, i _ {1}, r _ {1}} ^ {(1)} \dots \chi_ {r _ {D - 1}, i _ {D}, r _ {D}} ^ {(D)}, \tag {10}
$$
where  $(R_{1},\ldots ,R_{D})$  is called the tensor ring rank. The cyclic two-body approximation also approximates the tensor  $\mathcal{P}$  in the form of Equation (10), imposing an additional constraint that each core tensor  $\chi^{(d)}$  is decomposed as
$$
\chi_ {r _ {d - 1}, i _ {d}, r _ {d}} ^ {(d)} = \sum_ {m _ {d} = 1} ^ {I _ {d}} \mathbf {X} _ {r _ {d - 1}, m _ {d}} ^ {(d)} \mathcal {I} _ {m _ {d}, i _ {d}, r _ {d}} \tag {11}
$$
for each  $d \in [D]$ . We assume  $r_0 = r_D$  for simplicity. We obtain Equation (9) by substituting Equation (11) into Equation (10).
This constraint enables us to perform convex optimization. This means that we find a subclass  $\mathcal{B}_{\mathrm{cyc}}$  that can be solved by convex optimization in tensor ring decomposition, which has suffered from the difficulty of non-convex optimization. In addition, this is simultaneously a subclass of two-body approximation, as shown in Figure 2(c).
From Kronecker's delta  $\delta$ ,  $r_d = i_d$  holds in Equation (11); thus,  $\chi^{(d)}$  is a tensor with the size  $(I_{d - 1}, I_d, I_d)$ . Tensor ring rank after the cyclic two-body approximation is  $(I_1, \ldots, I_D)$  since the size of core tensors coincides with tensor ring rank. This result firstly reveals the relationship between Legendre decomposition and low-rank approximation via tensor networks.
Here we compare the number of parameters of cyclic two-body approximation and that of tensor ring decomposition. The number of elements of an input tensor is  $I_{1}I_{2}\dots I_{D}$ . After cyclic two-body approximation, the number  $|B|$  of parameters is given as
$$
| B | = 1 + \sum_ {d = 1} ^ {D} \left(I _ {d} - 1\right) + \sum_ {d = 1} ^ {D} \left(I _ {d} - 1\right) \left(I _ {d + 1} - 1\right), \tag {12}
$$
where we assume  $I_{D + 1} = I_1$ . The first term is for the normalizer, the second for the number of one-body parameters, and the final term for the number of two-body parameters. In contrast, in the tensor ring decomposition with the target rank  $(R_1,\dots ,R_D)$ , the number of parameters is given as  $\sum_{d = 1}^{D}R_{d}I_{d}R_{d + 1}$ . The ratio of the number of parameters of these two methods is proportional to  $I / R^2$  if we assume  $R_{d} = R$  and  $I_{d} = I$  for all  $d\in [D]$  for simplicity. Therefore, when the target rank is small and the size of the input tensor is large, the proposed method has more parameters than the tensor ring decomposition. The correspondence of many-body approximation to existing low-rank approximation is not limited to tensor ring decomposition. We also provide another example of the correspondence for tensor tree decomposition in Section C.2 in Supplement.
2.5 Many-body approximation as generalization of mean-field approximation
Any tensor  $\mathcal{P}$  can be represented by vectors  $x^{(1)},\ldots ,x^{(D)}\in \mathbb{R}^{I_d}$  as  $\mathcal{P}_{i_1,\dots,i_D} = x_{i_1}^{(1)}x_{i_2}^{(2)}\dots x_{i_D}^{(D)}$  if and only if all  $n(\geq 2)$ -body  $\theta$ -parameters are 0 [10, 12]. The right-hand side is equal to the Kronecker product of  $D$  vectors  $x^{(1)},\ldots ,x^{(D)}$ ; therefore, this approximation is equivalent to the rank-1 approximation since the rank of the tensor that can be represented by the Kronecker product is always 1, which is also known to correspond to mean-field approximation. In this study, we propose many-body approximation by relaxing the condition for the mean-field approximation that ignores  $n(\geq 2)$ -body interactions. Therefore many-body approximation is generalization of rank-1 approximation and mean-field approximation.
The discussion above argues that the one-body approximation – that is, the rank-1 approximation that minimizes the KL divergence from a given tensor – is a convex problem. Although finding the rank-1 tensor that minimizes the Frobenius norm from a given tensor is known to be an NP-hard problem [14], it has been reported that finding the rank-1 tensor that minimizes the KL divergence from a given tensor is a convex problem [16]. Therefore, our claim is not inconsistent with those existing studies. It should be also noted that, except for the rank-1 case, there is no guarantee that traditional low-rank decomposition that optimizes the KL divergence is a convex problem [5, 13].
2.6 Low-body tensor completion
Since our proposal is an alternative to the existing low-rank approximation, we can replace it with our many-body approximation in practical applications. As a representative application, we demonstrate
Algorithm 1: Low-body tensor completion
LBTC(T,B,Ω) // Ω is observed indices  
t←1  
Initialize Pt // e.g. a random tensor  
Pt $\Omega$ ←Tt  
repeat  
Pt+1← MANYBODYAPPROXIMATION (Pt,B) // m-step, See Algorithm 2  
Pt+1 $\Omega$ ←Tt  
rest←|Pt+1-Pt/F/|Pt|F// e-step  
t←t+1  
until rest-rest-1<ε and t>2 // We set ε = 10-5in our implementation;  
return P
Table 1: Recovery fit score for tensor completion.
Scenario 1	Scenario 2	Scenario 3
LBTC	0.948±0.00152	0.917±0.000577	0.874±0.00153
HaLRTC	0.864	0.842	0.825
SiLRTC	0.863	0.841	0.820
SiLRTCTT	0.948	0.915	0.868
PTRCRW	0.945±0.0000	0.901±0.0000	0.844±0.0000
missing value estimation by the  $em$ -algorithm [26], where we replace the  $m$ -step of low-rank approximation with many-body approximation. We call this approach low-body tensor completion (LBTC). The  $em$ -algorithm with tensor low-rank approximation does not guarantee the uniqueness of the  $m$ -step because the model space is not flat, while LBTC ensures that the model space is flat, and thus the  $m$ -step of LBTC is unique. See more discussion of information geometry for LBTC in Section C.4. We provide its pseudo-code in Algorithm 1. Instead of hyper-parameters related to ranks that traditional tensor completion methods require, LBTC requires only information about interactions to be used, but it can be intuitively tuned, as also seen in Section 3.1. We examine the performance of LBTC in Section 3.2.
We conducted three experiments to see the usefulness and effectiveness of many-body approximation. Datasets and implementation details are available in Supplement.
3.1 An example of tuning interaction
We extracted 10 color images from Columbia Object Image Library (COIL-100) [27] and constructed a tensor  $\mathcal{P} \in \mathbb{R}_{\geq 0}^{40 \times 40 \times 3 \times 10}$ , where each mode represents the width, height, color, or image index, respectively. Then we decomposed  $\mathcal{P}$  with varying chosen interactions and showed the reconstructed images. Figure 3(a) shows  $\mathcal{P}^{\leq 4}$ , which coincides with the input tensor  $\mathcal{P}$  itself since its order is four. Reconstruction quality gradually dropped when we reduced interactions from Figure 3(a) to Figure 3(d). We can explicitly control the trend of reconstruction by manually choosing interaction, which is one of characteristics of the proposal. In Figure 3(c), each reconstructed image has a monotone taste because the color is not directly interacted with the pixel position (width and height). In Figure 3(d), although the whole tensor is monotone because the color is independent of any other modes, the reconstruction keeps the objects' shapes since the interaction among width, height, and image index are retained. See Figure 7 in Supplement for the role of the interaction. Interactions between modes are interpretable, in contrast to the rank of tensors, which is often difficult to interpret. Therefore, tuning of interactions is more intuitive than that of ranks.
We also visualize the interaction between color and image as a heatmap of a matrix  $\mathbf{X}^{(c,i)}\in \mathbb{R}^{3\times 10}$  in Figure 3(e), which is obtained in the case of Figure 3(c). We normalize each column of  $\mathbf{X}^{(c,i)}$  with the
Figure 3: (a,b) Four-body and three-body approximation for the COIL dataset. (c,d) Appropriate selection of interactions controls the richness of color. Visualization of interaction between (e) color and image index, and (f) width, height, and image index.
L2 norm for better visualization. As expected, indices corresponding to red objects (that is, 1, 4, and 7) have large values in the third red-color row. We also visualize the interaction among width, height, and image index, which corresponds to a tensor  $\chi^{(w,h,i)}\in \mathbb{R}^{40\times 40\times 10}$  without any information of colors, as a series of heatmaps in Figure 3(f). They provide the gray-scale reconstruction of images. The above discussion implies that many-body approximation captures patterns or features of inputs.
3.2 Tensor completion by many-body approximation
To examine the performance of LBTC, we conducted experiments on a real-world dataset: the traffic speed dataset in District 7, Los Angeles County, collected from PeMS [31]. This data was obtained from fixed-point observations of vehicle speeds every five minutes for 28 days. We made  $28 \times 24 \times 12 \times 4$  tensor  $\mathcal{T}$ , whose modes correspond to date, hour, minute, and lane. As practical situations, we assumed that a disorder of a lane's speedometer causes random and continuous deficiencies. We prepared three patterns of missing cases: a case with a disorder in the speedometer in Lane 1 (Scenario 1), a case with disorders in the speedometers in Lanes 1, 2, and 3 (Scenario 2), and a case with disorders in all speedometers (Scenario 3). The missing rates for scenarios 1, 2, and 3 were 9 percent, 27 percent, and 34 percent, respectively. We evaluate the recovery fit score  $1 - \left\| \mathcal{T}_{\tau} - \overline{\mathcal{T}}_{\tau} \right\|_{F} / \left\| \mathcal{T}_{\tau} \right\|_{F}$  for the reconstruction  $\overline{\mathcal{T}}$  and missing indices  $\tau$ .
In the proposal, we need to choose interactions to use. Intuitively, the interaction between date and minute and that between minute and lane seem irrelevant. Therefore, we used all the two-body interactions except for the above two interactions. In addition, the interaction among date, hour, and minute is also included because it is intuitively relevant.
As baseline methods, we use SiLRTC, HaLRTC [21], SiLRTC-TT [2], and PTRCRW [38]. SiLRTC-TT and PTRCRW are based on tensor train and ring decompositions, respectively, and PTRCRW is known as the state of the art. The reconstructions are available in Figure 8 in Supplement with ground truth. We tuned hyperparameters of baseline methods and compared their best results with LBTC. The resulting scores (see Table 1) show that our method is superior to other methods. Moreover, we can intuitively tune interactions in the proposal, while other baselines need typical hyperparameter tuning as seen in Figure 14 in Supplement. In our method, interactions among non-associated modes could worsen the performance as seen in Table 2 in Supplement. Since the proposal and PTRCRW are non-convex optimizations, we ran each of them 10 times with random initialization and reported mean values and the standard error. Many-body approximation itself is always convex, while our tensor completion algorithm LBTC is non-convex due to the use of the em-algorithm.
3.3 Comparison with ring decomposition
As seen in Section 2.4, many-body approximation has a close connection to low-rank approximation. For example, in tensor ring decomposition, if we impose the constraint that decomposed smaller tensors can be represented as products with hyper-diagonal tensors  $\mathcal{I}$ , this decomposition is equivalent to a cyclic two-body approximation. Therefore, to examine our conjecture that cyclic two-body approximation is as capable of approximating as tensor ring decomposition, we empirically examined the effectiveness of cyclic two-body approximation compared with tensor ring decomposition. As baselines, we used five existing methods of non-negative tensor ring decomposition: NTR-APG, NTR-HALS, NTR-MU, NTR-MM, and NTR-lraMM [39, 40]. We also describe the efficiency in Supplement.
We evaluated the approximation performance by the relative error  $\| \mathcal{T} - \overline{\mathcal{T}} \|_F / \| \mathcal{T} \|_F$  for an input tensor  $\mathcal{T}$  and a reconstructed tensor  $\overline{\mathcal{T}}$ . Since all the existing methods are based on nonconvex optimization, we plotted the best score (minimum relative error) among five restarts with random initialization. In contrast, the score of our method is obtained by a single run as it is convex optimization and such restarts are fundamentally unnecessary.
Synthetic data We performed experiments on two synthetic datasets with low tensor ring rank. We create sixth- and seventh-order tensors with ring ranks (15, ..., 15) and (10, ..., 10), respectively. In Figure 4(a), relative errors are plotted with gradually increasing the target rank of the tensor ring decomposition, which is compared to the score of our method, plotted as the cross point of horizontal and vertical red dotted lines. Please note that our method does not have the concept of the rank, so the score of our method is invariant to changes of the target rank unlike other methods. If the cross point of red dotted lines is lower than other lines, the proposed method is better than other methods. In both experiments, the proposed method is superior to comparison partners in terms of the reconstruction error.
Figure 4: Experimental results for (a) synthetic data and (b) real datasets. The vertical red dotted line is  $|B|$  (see Equation (12)).
Real data Next, we evaluated our method on real data. TT_ChartRes and TT_Origami are seventh-order tensors that are produced from TokyoTech Hyperspectral Image Dataset [23, 24]. Each tensor has been reshaped to reduce the computational complexity. As seen in Fig-
ure 4(b), the proposed method keeps the competitive relative errors. In baselines, a slight change of the target rank can induce a significant increase of the reconstruction error due to the nonconvexity. These results mean that we eliminate the instability of non-negative tensor ring decomposition by our convex formulation.
We propose many-body approximation for tensors, which decomposes tensors by focusing on the relationship between modes represented by an energy-based model. This method approximates and regularizes tensors by ignoring the energy corresponding to certain interactions, which can be viewed as a generalized formulation of mean-field approximation that considers only one-body interactions. Our novel formulation enables us to achieve convex optimization of the model, while the existing approaches based on the low-rank structure are non-convex and empirically unstable with respect to the rank selection. Furthermore, we introduce a way of visualizing activated interactions between modes, called interaction representation. We have demonstrated transformation between our representation and tensor networks, which reveals the nontrivial connection between many-body approximation and the classical tensor low-rank tensor decomposition. We have empirically showed the intuitive model's designability and the better usefulness in tensor completion and approximation tasks compared to baseline methods.
**Limitation** Proposal only works on non-negative tensors. We have examined the effectiveness of LBTC on only traffic datasets.
This work was supported by RIKEN, Special Postdoctoral Researcher Program (KG), JST, CREST Grant Number JPMJCR22D3, Japan, and JSPS KAKENHI Grant Number JP21H03503 (MS), and JST, CREST Grant Number JPMJCR1913, Japan (YK).
[1] Amari, S. (2016). Information Geometry and Its Applications. Springer.
[2] Bengua, J. A., Phien, H. N., Tuan, H. D., and Do, M. N. (2017). Efficient tensor completion for color image and video recovery: Low-rank tensor train. IEEE Transactions on Image Processing, 26(5):2466-2479.
[3] Bishop, C. M. and Nasrabadi, N. M. (2006). Pattern recognition and machine learning, volume 4. Springer.
[4] Cheng, S., Wang, L., and Zhang, P. (2021). Supervised learning with projected entangled pair states. Physical Review B, 103(12):125117.
[5] Chi, E. C. and Kolda, T. G. (2012). On tensors, sparsity, and nonnegative factorizations. SIAM Journal on Matrix Analysis and Applications, 33(4):1272-1299.
[6] Cichocki, A., Lee, N., Oseledets, I., Phan, A.-H., Zhao, Q., Mandic, D. P., et al. (2016). Tensor networks for dimensionality reduction and large-scale optimization: Part 1 low-rank tensor decompositions. Foundations and Trends® in Machine Learning, 9(4-5):249-429.
[7] Cichocki, A., Mandic, D., De Lathauwer, L., Zhou, G., Zhao, Q., Caiafa, C., and Phan, H. A. (2015). Tensor decompositions for signal processing applications: From two-way to multiway component analysis. IEEE signal processing magazine, 32(2):145-163.
[8] Erol, A. and Hunyadi, B. (2022). Tensors for neuroimaging: A review on applications of tensors to unravel the mysteries of the brain. *Tensors for Data Processing*, pages 427–482.
[9] Evenbly, G. and Vidal, G. (2015). Tensor network renormalization. Physical review letters, 115(18):180405.
[10] Ghalamkari, K. and Sugiyama, M. (2021). Fast tucker rank reduction for non-negative tensors using mean-field approximation. In Advances in Neural Information Processing Systems, volume 34, pages 443-454, Virtual Event.
[11] Ghalamkari, K. and Sugiyama, M. (2022). Fast rank-1 NMF for missing data with KL divergence. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics, pages 2927–2940, Virtual Event.
[12] Ghalamkari, K. and Sugiyama, M. (2023). Non-negative low-rank approximations for multidimensional arrays on statistical manifold. Information Geometry.
[13] Hansen, S., Plantenga, T., and Kolda, T. G. (2015). Newton-based optimization for kullback-Leibler nonnegative tensor factorizations. Optimization Methods and Software, 30(5):1002-1029.
[14] Hillar, C. J. and Lim, L.-H. (2013). Most tensor problems are np-hard. Journal of the ACM (JACM), 60(6):1-39.
[15] Hitchcock, F. L. (1927). The expression of a tensor or a polyadic as a sum of products. Journal of Mathematics and Physics, 6(1-4):164-189.
[16] Huang, K. and Sidiropoulos, N. D. (2017). Kullback-leibler principal component for tensors is not np-hard. In 2017 51st Asilomar Conference on Signals, Systems, and Computers, pages 693-697. IEEE.
[17] Kolda, T. G. and Bader, B. W. (2009). Tensor decompositions and applications. SIAM review, 51(3):455-500.
[38] Yu, J., Zhou, G., Li, C., Zhao, Q., and Xie, S. (2021a). Low tensor-ring rank completion by parallel matrix factorization. IEEE Transactions on Neural Networks and Learning Systems, 32(7):3020-3033.
[39] Yu, Y., Xie, K., Yu, J., Jiang, Q., and Xie, S. (2021b). Fast nonnegative tensor ring decomposition based on the modulus method and low-rank approximation. Science China Technological Sciences, 64(9):1843-1853.
[40] Yu, Y., Zhou, G., Zheng, N., Qiu, Y., Xie, S., and Zhao, Q. (2022). Graph-regularized nonnegative tensor-ring decomposition for multiway representation learning. IEEE Transactions on Cybernetics.
[41] Zhao, Q., Zhou, G., Xie, S., Zhang, L., and Cichocki, A. (2016). Tensor ring decomposition. arXiv preprint arXiv:1606.05535.
A Projection theory in information geometry 14
B Formal definitions 16
C Theoretical remarks 16
C.1 Convexity and uniqueness of many-body approximation 16
C.2 Another example: Tensor tree decomposition 17
C.3 Reducing computational complexity by reshaping tensor 17
C.4 Information geometric view of LBTC 17
D Related work 18
E Additional numerical results 18
E.1 Additional results for COIL dataset 18
E.2 Additional results for LBTC 18
E.3 Additional results for comparison with ring decomposition 20
E.4 Comparison with traditional low-rank approximation 20
F Dataset details 22
F.1 For experiment with RGB images 22
F.2 For experiments with LRTC 22
F.3 For comparison with non-negative tensor ring decomposition 22
G Implementation detail 23
G.1 Proposed method 23
G.2Baselines 23
References for Supplementary Materials 25
A Projection theory in information geometry
We explain concepts of information geometry used in this study, including natural parameters, expectation parameters, model flatness, and convexity of optimization. In the following discussion, we consider only discrete probability distributions.
$(\theta, \eta)$ -coordinate and geodesics In this study, we treat a normalized  $D$ -order non-negative tensor  $\mathcal{P} \in \mathbb{R}_{\geq 0}^{I_1 \times \dots \times I_D}$  as a discrete probability distribution with  $D$  random variables. Let  $\mathcal{U}$  be the set of discrete probability distributions with  $D$  random variables. The entire space  $\mathcal{U}$  is a non-Euclidean space with the Fisher information matrix  $\mathbf{G}$  as the metric. This metric measures the distance between two points. In Euclidean space, the shortest path between two points is a straight line. In a non-Euclidean space, such a shortest path is called a geodesic. In the space  $\mathcal{U}$ , two kinds of geodesics can be introduced:  $e$ -geodesics and  $m$ -geodesics. For two points  $\mathcal{P}_1, \mathcal{P}_2 \in \mathcal{U}$ ,  $e$ - and  $m$ -geodesics are defined as