SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To see the effectiveness of proposal for data mining, we also plotted two-body interaction between day and lane extracted by the two-body approximation of the above traffic data. This is a matrix
$\mathbf{X}^{(d,l)} \in \mathbb{R}^{28 \times 4}$ . As seen in Figure 13, the value in the first column is always the largest and the value in the fourth column is always the smallest. From this result, we can interpret that the data corresponding to  $\mathcal{P}[:, :, :, 1]$  were taken in the overtaking lane and the data for  $\mathcal{P}[:, :, :, 4]$  were taken in the overtaken lane.
F Dataset details
We describe the details of each dataset below.
F.1 For experiment with RGB images
We downloaded the COIL-100 dataset from the official website. We used 10 images with object numbers 4, 5, 7, 10, 17, 23, 28, 38, 42, and 78. This selection aimed to include both monochromatic and multi-colored objects so that we can see how the proposed method changes the reconstruction if we deactivate the interactions related to color. We reshaped the size of each image to  $40 \times 40$ .
F.2 For experiments with LRTC
We downloaded traffic speed records in District 7, Los Angeles County, from PeMS. The data are in the public domain. The whole period of the data lasts for 28 days from February 4 to March 3, 2013. By measuring the speed of vehicles in four lanes at a certain point on the highway 12 times at five-minute intervals, a  $12 \times 4$  matrix is generated per hour. Twenty-four of these matrices are generated per day, so the dataset is a  $24 \times 12 \times 4$  tensor. Further, since we used these tensors for 28 days, the size of the resulting tensor is  $28 \times 24 \times 12 \times 4$ . We artificially generated defects in this tensor. As practical situations, we assumed that a disorder of a lane's speedometer causes random and continuous deficiencies. We prepared three patterns of missing cases: a case with a disorder in the speedometer in Lane 1 (Scenario 1), a case with disorders in the speedometers in Lanes 1, 2, and 3 (Scenario 2), and a case with disorders in all speedometers (Scenario 3). The missing rates for Scenarios 1, 2, and 3 were 9 percent, 27 percent, and 34 percent, respectively. The source code for imparting deficits are available in the Supplementary Materials.
F.3 For comparison with non-negative tensor ring decomposition
Synthetic datasets We created  $D$  core tensors with the size  $(R, I, R)$  by sampling from uniform distribution. A tensor with the size  $I^D$  and the tensor ring rank  $(R, \ldots, R)$  was then obtained by the product of these  $D$  tensors. Results for  $R = 15$ ,  $D = 5$ ,  $I = 30$  are shown in the left column in Figure 4, and those for  $R = 10$ ,  $D = 6$ ,  $I = 20$  in the right column in Figure 4. For all experiments on synthetic datasets, we change the target ring-rank as  $(r, \ldots, r)$  for  $r = 2, 3, \ldots, 9$  for baseline methods.
Real datasets TT_ChartRes is originally a (736, 736, 31) tensor, produced from TokyoTech 31-band Hyperspectral Image Dataset. We downloaded ChartRes.mat from the official website<sup>3</sup> and reshaped the tensor as (23, 8, 4, 23, 8, 4, 31). For baseline methods, we chose the target ring-rank as (2, 2, 2, 2, 2, 2, 2) (2, 2, 2, 2, 2, 2, 5), (2, 2, 2, 2, 2, 2, 8), (3, 2, 2, 3, 2, 2, 5), (2, 2, 2, 2, 2, 2, 9), (3, 2, 2, 3, 2, 2, 6), (4, 2, 2, 2, 2, 2, 6), (3, 2, 2, 4, 2, 2, 8), (3, 2, 2, 3, 2, 2, 9), (3, 2, 2, 3, 2, 2, 10), (3, 2, 2, 3, 2, 2, 12), (3, 2, 2, 3, 2, 2, 15), or (3, 2, 2, 3, 2, 2, 16). TT_Origami is originally (512, 512, 59) tensor, which is produced from TokyoTech 59-band Hyperspectral Image Dataset. We downloaded Origami.mat from the official website.<sup>4</sup> In TT_Origami, 0.0016 percent of elements were negative, hence we preprocessed all elements of TT_Origami by subtracting -0.000764, the smallest value in TT_Origami, to make all elements non-negative. We reshaped the tensor as (8, 8, 8, 8, 8, 8, 59). For baseline methods, we chose the target ring-rank as (2, 2, 2, 2, 2, 2, r) for  $r = 2$ ,  $3$ , ...,  $15$ . These reshaping reduces the computational complexity as described in Section C.3 to complete the experiment in a reasonable time.
<https://www1.cs.columbia.edu/CAVE/software/softlib/coil-100.php
http://pems.dot.ca.gov/
<sup>3</sup>http://www.ok.sc.e.titech.ac.jp/res/MSI/MSIdata31.html
<sup>4</sup>http://www.ok.sc.e.titech.ac.jp/res/MSI/MSIdata59.html
Figure 13: The obtained interaction between day and lane in two-body approximation.
G Implementation detail
We describe the implementation details of methods in the following. All methods are implemented in Julia 1.8.
Environment Experiments were conducted on Ubuntu 20.04.1 with a single core of  $2.1\mathrm{GHz}$  Intel Xeon CPU Gold 5218 and 128GB of memory.
G.1 Proposed method
Many-body approximation We use a natural gradient method for many-body approximation. The natural gradient method uses the inverse of the Fisher information matrix to perform second-order optimization in a non-Euclidean space. For non-normalized tensors, we conducted the following procedure. First, we computed the total sum of elements of an input tensor. Then, we normalized the tensor. After that, we conducted Legendre decomposition for the normalized tensor. Finally, we obtained the product of the result of the previous step and the total sum we computed initially. That procedure was justified by the general property of the KL divergence  $\lambda D_{KL}(\mathcal{P},\mathcal{Q}) = D_{KL}(\lambda \mathcal{P},\lambda \mathcal{Q})$  for any  $\lambda \in \mathbb{R}_{>0}$ . The termination criterion is the same as the original implementation of Legendre decomposition by [36]; that is, it terminates if  $||\pmb{\eta}_t^B -\hat{\pmb{\eta}}^B || < 10^{-5}$ , where  $\pmb{\eta}_t^B$  is the expectation parameters on  $t$ -th step and  $\hat{\pmb{\eta}}^B$  is the expectation parameters of an input tensor, which are defined in Section 2.1.1. The overall procedure is described in Algorithm 2. Note that this algorithm is based on Legendre decomposition by [36].
LBTC The overall procedure is described in Algorithm 1 with our tolerance threshold. We randomly initialized each missing element in  $\mathcal{P}^{(t = 1)}$  by Gaussian distribution with mean 50 and variance 10.
G.2Baselines
Baseline methods for tensor completion We implemented baseline methods for tensor completion by translating pseudo codes in original papers into Julia code. As baseline methods, we used HaLRTC, SiLRTC [21], SiLRTC-TT [2], and PTRCRW [38]. SiLRTC and HaLRTC are well-known classical methods based on HOSVD. SiLRTC-TT is based on tensor train decomposition. PTRCRW is based on tensor-ring decomposition and is recognized as the state of the art. HaLRTC requires a real value  $\rho$  and a weight vector  $\alpha$ . We used the default setting on  $\alpha$  described in the original paper and tuned the value  $\rho$  as  $\rho = 10^{-5}, 10^{-4}, \ldots, 10^{2}$ . SiLRTC requires a hyperparameter  $\gamma_{d}$  for  $d = 1, 2, 3, 4$ . This is the threshold of truncated SVD for the mode- $d$  matricization of the input tensor. Following the original paper, we assumed  $\tau = \tau_{1} = \dots = \tau_{4}$  and we tune  $\tau$  in the range of  $\{10^{-2}, 10^{-1}, \ldots, 10^{4}\}$ . SiLRTC-TT requires a weight vector  $\alpha$  and a real number  $f$ . We set the value of  $\alpha$  as the default setting described in the original paper. We tune the value of  $f$  in the range of  $\{0.001, 0.01, 0.025, 0.1, 0.25, 1\}$ . PTRC-RW requires two weight vectors,  $\alpha$  and  $\beta$ , step length  $d$ , and tensor ring rank as hyper parameters. Following the original paper, we used the default setting for  $\alpha$ ,  $\beta$ , and  $d$ . Then we tuned the tensor ring rank  $(R, R, R, R)$  within the range  $R \in \{1, 2, \ldots, 6\}$ , assuming that it has the same values in each element, which is also assumed in the original paper.
Algorithm 2: Many-body approximation
MANYBODYAPPROXIMATION(T,B)   
 $s\gets$  Total sum of  $\mathcal{T}$  .   
Obtain normalized input tensor  $\mathcal{P}\leftarrow \mathcal{T}. / s$  //“./”denotes element-wise division   
Compute  $\hat{\eta}$  of  $\mathcal{P}$  using Equation (3).   
Initialize  $\theta_{t = 1}^{B}$  //e.g.  $\theta_{b} = 0$  for all  $b\in B$ $t\gets 1$    
repeat   
Compute  $\mathcal{Q}_t$  using the current parameter  $\theta_t^B$  with Equation (1).   
Compute  $\pmb{\eta}_{t}^{B}$  from  $\mathcal{Q}_t$  using Equation (3).   
Compute the inverse of the Fisher information matrix G using Equation (5).   
 $\pmb{\theta}_{t + 1}^{B}\gets \pmb{\theta}_{t}^{B} - \mathbf{G}^{-1}(\pmb{\eta}_{t}^{B} - \hat{\pmb{\eta}}^{B})$ $t\gets t + 1$    
until  $||\pmb {\eta}_t^B -\hat{\pmb{\eta}}^B || <   \epsilon$  //We set  $\epsilon = 10^{-5}$  in our implementation;   
 $\overline{\mathcal{T}}\gets \mathcal{Q}_t.\ast s$  //“.*” denotes element-wise multiplication   
return T
Since PTRC-RW is not a convex method, we repeated it 10 times for each  $R$  with randomly sampled initial values by Gaussian distribution with mean 50 and variance 10. We provide Figure 14, which shows the sensitivity of the tensor completion performance with respect to changes of the above hyper-parameters. All of the above baseline methods include iterative procedures. These iterations were stopped when the relative error between the previous and the current iteration is less than  $10^{-5}$  or when the iteration count exceeds 1500 iterations.
**Baseline methods for tensor ring decomposition** We implemented baseline methods for tensor ring decomposition by translating MATLAB code provided by the authors into Julia code for fair comparison. As we can see from their original papers, NTR-APG, NTR-HALS, NTR-MU, NTR-MM, and NTR-lraMM have an inner and outer loop to find a local solution. We repeated the inner loop 100 times. We stopped the outer loop when the difference between the relative error of the previous and the current iteration is less than  $10^{-4}$ . NTR-MM and NTR-lraMM require a diagonal parameter matrix  $\Xi$ . We define  $\Xi = \omega I$  where  $I$  is an identical matrix and  $\omega = 0.1$ . The NTR-lraMM method performs low-rank approximation to the matrix obtained by mode expansion of an input tensor. The target rank was set to be 20. This setting is the default setting in the provided code. The initial positions of baseline methods were sampled from uniform distribution on  $(0,1)$ .
**Baseline methods for low-rank approximation** We implemented NNTF by translating the pseudocode in the original paper [34] into Julia code. The algorithm requires NMF in each iteration. For that, we also implement the most standard method, multiplicative update rules, following the paper [46]. We use default values of hyper parameters of sklearn [49] for NMF. For NTD, we translate TensorLy implementation [45] to Julia code, whose cost function is the Frobenius norm between input and output.
Figure 14: Hyper-parameter dependencies for the performance of baselines in the experiment in Section 3.2.
References for Supplementary Materials
[42] Amari, S. 1998. "Natural gradient works efficiently in learning." Neural computation 10(2):251-276.
[43] Cheng, Song, Lei Wang, Tao Xiang and Pan Zhang. 2019. "Tree tensor networks for generative modeling." Physical Review B 99(15):155131.
[44] Kim, Yong-Deok and Seungjin Choi. 2007. Nonnegative tucker decomposition. In 2007 IEEE Conference on Computer Vision and Pattern Recognition. IEEE pp. 1-8.
[45] Kossaifi, Jean, Yannis Panagakis, Anima Anandkumar and Maja Pantic. 2019. "TensorLy: Tensor Learning in Python." Journal of Machine Learning Research 20(26):1-6. URL: http://jmlr.org/papers/v20/18-277.html
[46] Lee, Daniel and H Sebastian Seung. 2000. Algorithms for non-negative matrix factorization. In Advances in Neural Information Processing Systems 13. Denver, United States: pp. 556-562.
[47] Malik, Osman Asif and Stephen Becker. 2021. A sampling-based method for tensor ring decomposition. In International Conference on Machine Learning. PMLR pp. 7400-7411.
[48] Murg, Valentin, Frank Verstraete, Reinhold Schneider, Peter R Nagy and O Legeza. 2015. "Tree tensor network state with variable tensor order: An efficient multireference method for strongly correlated systems." Journal of Chemical Theory and Computation 11(3):1027-1036.
[49] Pedregosa, F., G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournaepau, M. Brucher, M. Perrot and E. Duchesnay. 2011. "Scikit-learn: Machine Learning in Python." Journal of Machine Learning Research 12:2825–2830.
[50] Shi, Y-Y, L-M Duan and Guifre Vidal. 2006. "Classical simulation of quantum many-body systems with a tree tensor network." Physical review a 74(2):022320.
[51] Walczak, B. and D.L. Massart. 2001. "Dealing with missing data: Part I." Chemometrics and Intelligent Laboratory Systems 58(1):15-27. URL: https://www.sciencedirect.com/science/article/pii/S0169743901001319
[52] Zhang, Sheng, Weihong Wang, James Ford and Fillia Makedon. 2006. Learning from incomplete ratings using non-negative matrix factorization. In Proceedings of the 2006 SIAM International Conference on Data Mining. SIAM Bethesda, United States: pp. 549-553.