TT-NF: Tensor Train Neural Fields
Authors: Anton Obukhov
Mikhail Usvyatsov
Christos Sakaridis
Konrad Schindler
Luc Van Gool
Venue: Submitted to ICLR 2023 (2023)
Keywords: neural fields
deep learning
tensor train
tensor decompositions
sampling
radiance fields
voxels

Abstract: Learning neural fields has been an active topic in deep learning research, focusing, among other issues, on finding more compact and easy-to-fit representations. In this paper, we introduce a novel low-rank representation termed Tensor Train Neural Fields (TT-NF) for learning neural fields on dense regular grids and efficient methods for sampling from them. Our representation is a TT parameterization of the neural field, trained with backpropagation to minimize a non-convex objective. We analyze the effect of low-rank compression on the downstream task quality metrics in two settings. First, we demonstrate the efficiency of our method in a sandbox task of tensor denoising, which admits comparison with SVD-based schemes designed to minimize reconstruction error. Furthermore, we apply the proposed approach to Neural Radiance Fields, where the low-rank structure of the field corresponding to the best quality can be discovered only through learning. 

Under review as a conference paper at ICLR 2023
TT-NF: TENSOR TRAIN NEURAL FIELDS
Anonymous authors
Paper under double-blind review
ABSTRACT
Learning neural fields has been an active topic in deep learning research, focusing, among other issues, on finding more compact and easy-to-fit representations. In this paper, we introduce novel low-rank representation termed Tensor Train Neural Fields (TT-NF) for learning neural fields on dense regular grids and efficient methods for sampling from them. Our representation is TT parameterization of the neural field, trained with backpropagation to minimize a non-convex objective. We analyze the effect of low-rank compression on the downstream task quality metrics in two settings. First, we demonstrate the efficiency of our method in a sandbox task of tensor denoising, which admits comparison with SVD-base DyD-eaeed schemes designed to minimize reconstruction error. Furthermore, we apply the proposed approach to Neural Radiance Fields, where the low-rank structure of the field corresponding to the best quality can be discovered only through learning.
INTRODUCTION
Following the growing interest in deep neural networks, learning neural fields has become a promising research direction in areas concerned with structured representations, However, precision is usually odds with the computational complexity of these representations, which makes training them and sampling from them a challenge. In this paper, we investigate interpretable low-rank neural fields defined on dense regular grids and efficient methods for learning them. Since, in extreme cases, the dimensionality of such fields can exceed the memory size of typical computer by several orders of magnitude, we look at the problem learning such fields from the angle stochastic methods.
Tensor decompositions have become a ubiquitous tool for dealing with structured sparsity of in- tractable volumes of data. Within the large family tensor decompositions, we focus on the Tensor Train (TT) (Oseledets, 2011). also known the Matrix Product State in physics. TT is notable for its high-capacity representation, efficient algebraic operations in the low-rank space, and support of SVD-based algorithms for data approximation. As such, consider TT-SVD (Oseledets, 201 1) and TT-cross (Oseledets & Tyrtyshnikov, 2010) methods for obtaining a low-rank representation of the full tensor. While TT-SVD requires access to the full tensor at once (which might already be problematic in specific scenarios), TT-cross requires access to data through a black-box function, computing (or looking up) elements by their coordinates on demand. Both methods operate under the assumption of noise-free free data and are not guaranteed to output sufficiently good approximations in the presence of noise.
While noise in observations is challenging for SVD-based schemes and requires devising tailored approaches to different noise types and magnitude (Zhou et al., 2022), exploiting the low-rank structure of the field driven by data is even more challenging (Novikov et al., 2014; Boyko et al., 2020) and typically resorts to the paradigm of data updates through algebraic operations on TT.
In this work, we take step back and leverage the modern deep learning paradigm to parameterize neural fields as TT, coined TT-NE Through deep learning tooling with support for automatic differen- tiation and our novel sampling methods, obtain mini-batches of samples from the parameterized neural field and perform optimization of a non-convex objective defined by downstream task. The optimization comprises the computation of parameter gradients with backpropagation and parameter updates with any suitable technique, such as SGD.
range of problem sizes and provide reference We analyze TT-NF and several sampling techniques charts for choosing a sampling method based on memory and computational constraints. Next, we define synthetic task of low-rank tensor denoising and demonstrate the superiority of the proposed

---

Under review as a conference paper at ICLR 2023
optimization scheme over several SVD-based schemes. Finally, consider the formulation of Neural Radiance Fields (NeRF) introduced in Mildenhall et al. (2020), and propose a simple modification to TT-NE, termed QTT-NF, for dealing with hierarchical spaces.
Our contributions in this paper:
1. TT-NF compressed low-rank neural field representation defined on a dense grid;
2. QTT-NF modification of TT--FF for learning neural fields defined hierarchical spaces, such as 3D voxel grids seen in neural rendering;
3. Efficient algorithms for sampling from (Q)TT-NF and learning it from samples, designed for deep learning tooling.
The rest of the paper is organized as follows: Sec. 2 discusses the related work; Sec. 3 introduces notations from the relevant domains; Sec. 4 presents the proposed contributions; Sec. 5 demonstrates the practical use of the proposed methods; Sec. 6 concludes the paper. Many relevant details pertaining to our method, experiments, and extra discussion can be found in Appendix sections A, B, and C.
RELATED WORK
Tensor Decompositions Higher-order tensor decompositions have been found helpful for several data-based problems, as detailed by Kolda & Bader (2009). Oseledets (201 1) introduced the Tensor Train (TT) decomposition, which offers compressed low-rank tensor approximation that is stable and fast. The TT decompasition has also been used to approximate tensors with linear complexity in their dimensionality via the TT-cross approximation (Oseledets & Tyrtyshnikov, 2010). With the rise of deep learning, gensor-based methods have been integrated into neural networks, e.g.. Usvyatsov et al. (2021) explored the use of TT-cross approximation for gradient selection learning representations. We review tensor-based methods for network compression in the next paragraph and refer the reader to Panagakis et al. (2021) for a detailed overview similar works. On the software side, along with general deep learning frameworks (Paszke et al., 2019; Abadi et al., 2015), several tensor-centric frameworks have emerged (Kossaifi et al., 2019b; Usvyatsov et al., 2022: Novikov et al., 2020).
Neural Network Compression with Tensors Low-rank bases were utilized by Jaderberg al. (2014) to approximate convolutional filters and drastically speed up inference via separating filter depth from spatial dimensions. Lebedev et al. (2014) applied low-rank decomposition on all dimensions of the standard convolutional kernel tensors. Subsequent works employed more general tensor decompositions, notably the TT decomposition, to massively compress fully connected layers (Novikov et al., 2015) or both fully connected and convolutional layers (Garipov et al., 2016), with minor accuracy losses. Kossaifi et al. (2019a) applied the higher-order tensor factorization to the entire network instead of separately to individual layers. In similar vein, Li et al. (2019); Obukhov et al. (2020); Kanakis et al. (2020) propose to learn a basis and coefficients of each layer, thus enabling disentangled compression and multitask learning. While most of the aforementioned methods examine general convolutional networks, we focus specifically on compressing neural fields.
Tensor Decompositions in 3D Representations Early approaches in 3D volume representation used global low-rank tensor bases for reconstruction at multiple scales and resolutions (Suter et al., 2013). A review of compact representations and sampling techniques for compressed direct volume rendering (DVR) is made by Balsa Rodríguez et al. (2014), Ballester-Ripoll et al. (2015) analyzed multiple tensor approximation models regarding volume visualization. More recently, compression of 3D volumes over regular grids was addressed with a general higher-order singular value decomposition (SVD) by Millester-Ripoll et al. (2019). The TT decomposition has also been used by Boyko et al. (2020) to compress 3D scenes that are represented by volumetric distance functions. We review neural-field-hased methods separately in the next paragraph.
Neural Fields Neural fields implicit scene representations for geometry and radiance have recently attracted intense research activity, especially in the context of 3D. The application of neural fields to image compression is studied by Strümpler et al. (2021), who employ meta-leamed representations that increase efficiency in training. The usual volumetric type of representation is replaced by surface-based one by Zhang al. (2021a), who learned bidirectional reflectance
2

---

Under review as a conference paper at ICLR 2023
CO)
((2)
C(P)
C()
C(2)
C(D) RD-1 RD
R
Ra
RD-1
Rs
M1
M2
Mp
X, Y, Zn X2
Xp YD Zp
Figure I: Tensor diagram of the (Block) Tensor Train decomposition (Dolgov al., 2014). The low-rank tensor of shape Mi X x M x Rp represented as a product of D TT-cores CO,, each being tensor of shape R M; x R4- The TT-rank (1, Ri.... RD) defines the degree of approximation. The case of RD 1 corre- sponds to TT decomposition Oseledets (2011).
Figure 2: Tensor diagram of Quantized Ten- sor Train decomposition (Khoromskij, 2009) of 3D voxel grid of shape X x Y Z and Rd values in each voxel. All three dimensions ad- mit factorization into D levels of hierarchy (e.g.. X = X1 XD). We group factors by levels into tuples (X;, Y,, Z;), and introduce the low- rank bonds between the levels of the hierarchy.
distribution functions that enable novel view synthesis for unconstrained real-world scenes, and Zhang al. (2021b), who designed signed distance field to model the scene geometry. For a comprehensive overview of neural fields in visual computing, we refer the reader to Xie et al. (2022).
Neural Radiance Fields The seminal paper of Mildenhall et al. (2020) introduced the concept of NeRF as an end-to-end implicit differentiable function that maps viewpoints to scene renderings and showed that it could be effectively optimized. The area advances quickly; we further discuss works focusing on representation sparsity and efficiency. Yu et al. (2022) employed a sparse 3D-grid representation based on spherical harmonics and regularization for novel view synthesis, alleviating the need for neural components. Müller et al. (2022) improved the training efficiency by using hash encoding for the input feature vectors, which allows using a much smaller network. Reiser et al. (2021) improved the rendering efficiency by assigning different parts of the scene different small networks. Sun et al. (2022) achieved faster convergence to optimal solutions for volumetric representations by using a post-activation interpolation and guiding the optimization via prior knowledge of the problem. Chen et al. (2022) implemented low-rank constraints in 3D on the voxel grid parameterization and achieved impressive results in the NeRF setting.
NOTATION
Tensor Diagram Notation Diagrams are an efficient visualization tool for interactions between tensors. A tensor is drawn as node with the number of legs matching its number of dimensions. For example, a matrix W E Rmxx is drawn as -o-, and vector E R" looks as -o. Their product Wx x looks as a connection along the dimension being eliminated as a result of the operation: -0-0. A diagram of nodes with their connections reflects what is called "tensor network".
Tensor Contraction computes product of the entire tensor network. The result of this operation is a single tensor with dimensions corresponding to free legs inside the tensor network.
Tensor Train Decomposition The TT format represents D-dimensional array (tensor) A R M with modes M, R x called TT-cores (see Fig. 1). The tuple (Rg, R,. is called a TT-rank of the decomposition; it defines the degree of approximation of A. By convention (Oseledets, 201 1), Ro = RD = 1. Runx = max Rg. ..., RD)) is also called the rank of the decomposition. An element of A indices (ii,..,,i)) can be computed as follows:
product of D three- core tensors (() e
BI... R..
A A Σ
c.a) Cr.ia.0."" ... (D-L)
((2)
(D) BD-2AD-,, BD-1.in.
(1)
Dolgov et al. (2014) introduced Block TT, which attaches "block" dimension in place of the last rank Rp. The difference between the two is subtle, as both formats can be converted to each other. However, Block TT is more suitable for describing mului- neural fields. We thus assign special meaning to RD it will signify the "payload" dimension of our neural field. We will omit the word "block" in the remaining text and always assume Block TT. Refer Fig. for the diagram.
3

---

Under review as a conference paper at ICLR 2023
QTT
Mode Quantization refers to the introduction of artificial dimensions into the represented
tensor. For example, a 3D tensor of function values on the lattice 16 x 16 x 16 can be represented as tensor of shape (2) 22 x 23 24) x (2, x 22 23 x 24) x (2, x 22 x 23 x 24). Here color-coded factors denote one of the axes X, Y, and Z they explain, subscripts denote the artificiall--inrrdduced levels of hierarchy, delimits represented dimensions, denotes merged (flattened) dimensions, and parentheses are for visual convenience. This 12-dimensional cube with side 2 can then be represented using TT as in Fig. However, when the number of introduced levels of hierarchy is the same between modes of the original tensor (four in the given example), it is often beneficial (Oseledets, 2009) to introduce low-rank structure between levels of hierarchy, rather than individual factors. The resulting low-rank representation describes tensor of shape 8 x 8 x 8 x 8 with the following factorization pattern called QTT: (2) 12.,2,, 21 21) x (22 22 22) x (23 23 23) x (24 24 24). Notably, such permutation of dimension factors corresponds to 3D space traversal using Morton code (Morton, 1966), also known as Z-order. Connections with octrees used in rendering can also be drawn.
We define our neural field on 3D voxel grid shown in Fig. 2. Dimensions of the voxel grid are chosen equal to 2D, resulting in hierarchy of D levels. Together with the Block structure discussed above and QTT, arrive at the proposed representation, which has D cores with all modes equal to eight, and the payload dimension corresponding to the number of values to store in each voxel.
Tensor Decomposition The term may refer either to the decomposition scheme (e.g., in the figures above) or the process of obtaining values of the decomposition (e.g., TT-cores C) from elements of the full tensor A. While SVD-based schemes employ the latter meaning, we focus on the former by parameterizing decomposition given its configuration, which includes modes and TT-rank.
Sampling refers to obtaining elements of the full tensor from its decomposition at a given list of indices. This operation could be done in several ways, the simplest being Tensor Contraction followed by subsampling at the required indices. When the contraction is undesired or intractable, an alternative way is evaluating full tensor elements through the decomposition equation, such as Eq. 1. While mathematically straightforward, the subtleties of the chosen sampling algorithm result in a large variance in efficiency when used within the optimization loop due to the need for automatic differentiation, as we discuss in Sec. 4.2.
4 METHOD
We introduce TT-NF as parameterization of the TT decomposition discussed in See. and use it within deep learning framework with au- tomatic differentiation support. This paradigm change differs from previous methods for ob- taining tensor decompositions, relying on matrix decompositions (SVD, QR, etc.) and algebraic operations in the TT format. Each such scheme (collectively called "'VDD-aased") comes with its own set of limitations, summarized in Tab. 1. TT-SVD (Oseledets, 2011) assumes access to all elements of the full tensor A in memory, which may be intractable in specific large-scale scenar- ios, and does not support noise in observations. TT-Cross (Oseledets & Tyrtyshnikov, 2010) ac- cepts a black-box function for computing ele- ments of the full tensor on-demand, which suits
Table I: Comparison of methods for obtaining a TT decomposition from observations: TT-SVD (Os- eledets, 2011), TT-Cross (Oseledets & Tyrtysh- nikov, 2010), TT-OI (Zhou et al., 2022), TT-NE
Observation access pattern
Noise in
Method
observations
TT-SVD
Full tensor
Not supported
On-demand, pattern defined by dimen- sions and TT-rank
TT-Cross
Not supported
TT-OI
Full tensor
Sub-gaussian
On-demand,
Any supported by the choice of the loss function
TT-NF (our)
flexible batch size and access pattern
large-scale problems, but lacks the fiexibility in choosing the number of samples through which update is performed (batch size); this number is defined purely by the configuration of TT decompo- sition (dimensions and TT-rank). Likewise, it does not support noise in observations. TT-OI (Zhou et al., 2022) improves upon TT-SVD and supports zero-mean independent gaussian noise in observations but inherits scalability issues.
Our method stands out due to its flexibility of optimization parameters choice (c.g., batch size) and resilience various types of noise in observations, controlled through the choice of the loss function.
4

---

Under review as a conference paper at ICLR 2023
4.1 INITIALIZATION OF TT-NF
Given the field's dimensions, we first choose its TT-rank. For that, we choose the value Rmax and Set TT-rank to the maximum possible values according to Oseledets (2011), not exceeding Rmax-
Following the best practices in the deep learning literature, we initialize parameters of TT-NF from seratch using the normal distribution with scale â computed such that the full tensor elements computed using Eq. have a pre-defined scale o, as shown in Eq. 2:
log R ))
δ = exp
2 2 σ
(2)
2D
1=1
Alternatively, in the presence of access full tensor elements, parameters can be initialized using the output of any of the SVD-based schemes, leading to faster convergence.
4.2 SAMPLING FROM TT-NF
Obtaining samples from TT-NF is mathematically straightforward using Eq. 1. In deep learning frameworks, one way to obtain batch of B samples at indices (111
amounts to using index_select operation on each TT-core C, along mode M, to obtain batches of core slices (matrices) of shapes B x R x R2, and then applying bmm (batched matrix multiplication) operation to them. This sampling method (aliased v1) leads to space complexity of (BDR and quickly becomes unusable slices of parameterization are replicated in memory for each sample. Moreover, the scaling issue gets worse as we require keeping all intermediate computations after each bmm operation and allocating memory for gradients to enable automatic differentiation.
The prevention of model parameter replication enabled the scaling of modern neural networks with millions of parameters, which trained using minibatches thousands of samples. The cornerstone of efficient scaling is a set of specialized layers (e.g., Linear), which accept a batch of inputs and compute mappings using only one instance of parameters, During the backward pass, gradients w.r.t. parameters are accumulated from samples with predictable, constant space scaling.
With that in mind, we bootstrap our efficient sampling method (aliased v2) by leveraging Linear and produce a batch of intermediates u of shape B x Rí (the dimension corresponding to Ro = is ignored). For each subsequent TT-core C,, 2, D, we split the inputs according to which M, slices of C, will be used to perform metrox-matrix multiplication of each sample. Because Linear layers require samples in the minibatch be packed densely, we perform a permutation of align samples in the minibatch before multiplying them with the respective weight matrices. We additionally maintain restore the order of samples after processing e with the last TT-core. The output of each step has the shape B x R; compatible with the input to the next step until reaching the last step, where it is of the shape x Rp. Alg. outlines the details
layer functionality. Given a batch of indices above, start by taking (ift...
the inverse permutation
algorithm.
The resulting space complexity of v2 sampling is reduced to O(BDRmax). with the absorbed scaling factor just that of the vI method due to permutations. The memory footprint of is roughly Rmax times smaller than v1, which enables practical use for TT-NF optimization the rank increases.
Fig. 3 provides a reference chart for choosing an optimal sampling scheme based on an uncompressed field size of 2330 batch size, Rmaxx space, and time constraints. Refer additional charts for different field and batch sizes the Appendix, Figs. 7, 8. As can be seen, v2 consistently outperforms v1 in memory requirements and sampling through tensor contraction followed by indexing in both memory and FLOPs, given the batch sizes specified in the plot. Additionally, we introduce a reduced parameterization and an associated v3 sampling method in Sec. A.1.
5

---

Under review as a conference paper at ICLR 2023
Algorithm Memory-Efficient Sampling from TT-NF for Deep Learning Frameworks, Au- tomatic differentiation paths are highlighted in blue. Refer to Sec. 4.2 for more details.
Algorithm 2 Batched-Indexed Matrix-Vector
Permuted Product (BIMVP) for Deep Learning Frameworks, referenced in Alg. 1. Automatic differentiation paths are highlighted in blue.
Require:
Require:
D number of tensor dimensions,
C TT-core of shape R₁ x M x Rr,
B number of samples,
i indices [1, M..
(1, R1 ...y Rp) TT-rank,
" batch of vectors of shape B x Rg.
(M .... MD) TT-modes,
Ensure:
(C(1)
C(D)) TT-cores representing A,
T (v, C. Uig C.i..) permuted output,
m (....!?)) indices.
(8)
т permutation.
1: + argsort(i) compute permutation
Ensure:
b ... by unique(i) b count M unique
m 10
(8) (3)
samples
3: t T()) >> group vectors by matrices
without computing the whole A.
4: ",,.., UM f- split(r,b))
split groups
1: (1...., B)
I forward permutation
5: parfor + to M do
2: o + -1,..., B)
D inverse permutation
> Linear layer function without bias
3: (I)
7: Un + linear C.,..) bb R.
D B Rı
end parfor
4: for k +- to D do
5: is + x (ix)
> align mode indices > x Rg invert 4th permutation update forward perm. b update inverse perm.
+- (V1, vMr)
concatenate vectors
10: return U,
R,
6: U, T + BIMVP(C(()
ik. v)
7: ok K k
(π) 9: σ + (ok)
8:
10: end for
11: V (v))
I recover samples order D B x Rp
12: return
Computations (FLOPs)
Training memory (flots))
Inference memory (floats)
Sampling v1
1007
108
Sampling
108
18
Batch size: 1024
Sampling v3
10M
Contraction
100M
10M
1M
....
GPU RAM
275
10k
10k
10k
1000
parameters (floats)
100T
10T
38p
08 :9219 M23Pg)
2.50
108
1008
100
100M
1.58
1000
10M
10M
18
109
100k
0.58
104
1008
10k
74X
rank
rank
rank
rank
Figure 3: Space-time complexity of sampling from TT-NF of size 230 with various methods, batch sizes, and ranks. We compare three sampling schemes discussed Sec. 4, as well as the traditional tensor contraction scheme. As seen in the plots, v2 (Sec. 4.2) scheme requires orders of magnitude fewer floating point operations (FLOPs) and memory than the contraction scheme and outperforms naive v1 sampling in memory requirements. Additionally, v3 (Sec. A.1) offers extra speedup top of v2 without loss of representation capacity, bringing its parameterization closer to the theoretical number of degrees of freedom of the tensor train manifold. Lower is better. Best viewed in color.

---

Under review as a conference paper at ICLR 2023
Normal noise
Laplace noise
Rank:
0.6
0.4
0.5
TT-SVD
0.2
TT-O1
RMSE
TT-cross
Contraction Sampling v2
Rank: 128
Sampling v3
0.1
0.3
10 0.1
0.3
sigma
b
Figure 4: The resilience of various tensor regression methods size 2 to additive noise. We consider zero-mean Normal and Laplace noise with varying scales added to random ground truth with unit variance and known TT decomposition. We report root mean squared errors (RMSE) of tensors regressed by each method. Our sampling methods (v2 and v3) outperform other methods in noisy settings, including those specifically designed to work with noisy observations (TT-OI by Zhou et al. (2022)) and those computing fraction of elements each optimization step (TT-cross by Oseledets & Tyrtyshnikov (2010)). See discussion in Sec. 5.1. Lower is better. Best viewed in color.
5
EXPERIMENTS
5.1 TENSOR DENOISING
To compare TT-NF with SVD-based schemes, we use a synthetic task of tensor denoising: given noisy observation y E R M, x.. Mp of a tensor X' with a known TT structure as in Eq. 1, the task is to compute X = arg min || Al] For the experiments, we first choose tensor modes and TT-rank A sa Eq.
with Rp = 1, generate X' as in the TT-NF initialization scheme from Sec. 4.1 with perform tensor contraction. To simulate noisy observations V. we sample noise Z from independent zero-mean Normal or Laplace distributions with chosen scale and compute y X + Z.
1.0, and
TT-SVD (Oseledets, 2011) is a deterministic' algorithm; it does not take any hyperparameters and produces TT decomposition in single pass data. TT-OI (Zhou et al., 2022) performs several passes. TT-cross has several stopping settings, as runs an iterative maximum volume algorithm (Goreinov & Tyrtyshnikov, 2001) on each iteration of the main algorithm. We take the defaults provided by the tntorch package (Usvyatsov et al., 2022).
On the side of the non-convex optimization family, minimize loss function between samples from the observation and TT-NF. We choose a loss function best suited for the type of observation noise: Ll for Laplace and 1.2 for Normal. Contraction denotes the usage of all possible elements in the loss, whereas Sampling considers only mini- batche of samples.
Fig. 4 demonstrates the performance of all methods the problem of size 220 under varying noise types, scale, and TT-NF rank. Optimization lasts 1000 steps, with batch size of 4096 elements, Adam optimizer (Kingma & Ba, 2015) with default settings, learning rate warming up during the first 5% steps to 3e-2 and decaying exponentially to 3c-4. We ran all experiments 10 times and reported plots with 1-std error bars. runs are completed on single GPU.
TT-SVD provides reasonable baseline, which we use as initialization for TT-NF. TT-OI works better than TT-SVD on subset of Normal noise levels and ranks. TT-cross works only in noise-free setting, Contraction, which can be considered as full-batch gradient descent, does not improve much upon TT-SVD. We conjecture that gets stuck in saddle points, a recurring argument seen
'To the extent we call SVD deterministic.
7

---

Under review as a conference paper at ICLR 2023
Table 2: Comparison of QTT-NF with NeRF by Mildenhall et al. (2020) and TensoRF by Chen et al. (2022). In the latter, we disable grid masking and bounding box fitting for even comparison with the other methods. We consider two techniques for obtaining view-dependent color: spherical harmonies (SH) with 28 channels and MLP (NN), For QTT-NF, we use a grid size of 256 QTT-NF reaches performance competitive with the prior art. See Sec. 5.2 for more details.
Chair
Drums
Fieus
Hotdog
Lego
Materials
Mic
Ship
Method (shading)
Metric
Avg.
0806
NeRF
33.00
25.01
30.13
36.18
32.54
29.62
32.91
28.65
31.01
TensoRF (-mask)
32.19
25.01
30.81
35,28
33.54
28.81
31.72
28.90
30.78
PSNR 1
QTT-NF (SH)
32.09
24.96
30.89
35.49
32.48
28.22
31.50
27.55
30.40
QTT-NF (NN)
32.87
25.30
31.85
35.97
33.00
28.67
33.07
27.97
31.09
NeRF
0.967
0.925
0,964
0.974
0.961
0.949
0.980
0.856
0.947
TensoRF (-mask)
0.960
0.922
0.968
0.971
0.969
0.932
0.975
0.868
0.946
SSIM
QTT-NF (SI)
0.955
0.918
0.967
0.971
0.957
0.929
0.971
0.840
0,939
QTT-NF (NN)
0.965
0.923
0.971
0.972
0.962
0,934
0.979
0.850
0.945
NeRF
0.046
0.091
0.044
0.121
0.050
0.063
0.028
0.206
0.081
TensoRF (-mask)
0.053
0.094
0.037
0.055
0.040
0.075
0.029
0.170
0.069
LPIPS 4
QTT-NF (SH)
0.062
0.095
0.039
0.060
0.066
0.093
0.039
0.203
0,082
QTT-NF (NN)
0.050
0.094
0.037
0.055
0.052
0.089
0.032
0.201
0.077
in discussions of the advantages of SGD over GD in the deep learning literature. As can be seen, Sampling methods outperform all others in settings with the presence of noise.
5.2 NEURAL RADIANCE FIELDS
Following the fast-pacing domain, we test QTT-NF variant of representation (Fig. 2) in the neural radiance fields (NeRF) setting. The task of learning voxelized radiance fields from data assumes access to views of single scene with known camera poses in some frame of reference. The objective is to regress features of the voxel grid such that when passed through differentiable shading and ray marching algorithms, the projected images would correspond to the ground-truu data, and views from the held-out set would exhibit high PSNR. Such data-driven problem formulation does not permit the usage of existing SVD-based solutions and thus can only be solved through learning.
We start with the setting of Mildenhall et al. (2020) (recapped in Sec. A.2 of the Appendix) and replace the MLP converting coordinates and viewing direction into color and density with QTT-NF. Similar to Yu et al. (2022), we choose voxel grid resolution of 2563 and 28 channels to store 9 spherical harmonics (SH) coefficients per RGB channel and one voxel density value.
Next, we compare QTT-NF with a recent work TensoRF (Chen et al., 2022), which likewise uses tensor decompositions, albeit triplanar ones. This work achieves remarkable reconstruction quality by using several state-of-the-arr techniques on top of the proposed decomposition. We intentionally avoid such recipes that lead to better performance, voxel pruning, occupancy masks, extra losses (total variation), ray filtering, and progressive training schemes. One technique we borrow from that pool is the usage of a tiny MLP in place of SH for transforming voxel features into view-dependent color (NN), which retains the natural interpretability of the learned voxel grid features.
The standard testbed for neural rendering is the Blender dataset (Mildenhall et al., 2020).. It consists of 8 synthetic 3D scenes, each with hundred posed images of resolution 800 x 800. The scenes vary by complexity and include water and glossy surfaces, thus making it a challenging benchmark.
We train neural rendering experiments for S0K steps with an LR schedule similar the one from Sec. 5.1, but with base LR 3e-3, Rmax = 256, 4096 rays per batch, and 512 uniform samples per ray. The code of all experiments is implemented in PyTorch (Paszke et al., 2019). The observed variance of image quality metrics over five runs is not large: std(PSNR)~<A) 4e-2, std(SSIM) ~ 4c-4, std(LPIPS) 5e-4. Results from Tab. 2 attest that QTT-NF is capable of reaching performance competitive with the prior art.
Distributed under different Creative Commons licenses. Per scene: "chair", "ficus", "hotdog", "materials", "mic": CC-O; "drums": CC-BY; "lego": CC-BY-NC; "ship": CC-BY-SA.
8

---

Under review as a conference paper at ICLR 2023
32.5
PSNR
12
32
52.2
32
32
32
64
125
192
255
320
304
418
512
10c
20K
40K
B0C
100K
TT-rank
steps
Figure 5: Sweeps of TT-rank, number of training steps, and spherical harmonics components per channel when training QTT-NF on the "Lego" scene. The remaining parameters are fixed to the baseline values (represented with large dot plots); see Sec. 5.2. Higher better.
5.2.1 ABLATION STUDY
The ablation study of TT-rank, number of training steps, and SH basis in Fig. 5 shows that the results can be further improved, as most hyperparameters are not saturated. Additional studies of sampling schemes and pretraining with TT-SVD of full voxel grid can found in Sec. B of the Appendix.
Lego (axis-alignedd
-0.5
The reconstruction quality of QTT-NF depends pri- marily on the TT-rank and scene complexity. This contrary to TensoRF (Chen et al., 2022), which employs triplanar decompositions (CP, VM), and thus inevitably introduces preference for axis- 0.05
PSNR drop (dB)
-1.5
Hotdog (ordinary)
aligned content.
00.5
To demonstrate that, introduce rotation of the scene around the Z-axis into the ground-truth
-0.1
-0.15
-02
poses of scenes: "Lego" and "Hotdog". The former scene contains many axis-aligned primi- tives, which favorably utilize triplanar decompo- sition rank. We ensured that no rotation angle resulted in content out of voxel grid bounds. As can be seen from Fig. 6 (top), rotations of the same axis-aligned scene by 15, 30, and 45 degrees re- sult in significant performance drops with triplanar decomposition and all ranks, whereas the perfor- mance of QTT-NF drops more gracefully. An ordinary scene from Fig. (bottom) demonstrates the vanishing of this effect due to the absence of axis alignment in any of the rotations.
-.25
QTT-NF
-03
TensorF
-0.35
rotation angle
Figure 6: Sensitivity of QTT-NF and Ten- soRF (Chen et al., 2022) to context-to-axes align- ment. Scenes: top "Lego", " bottom "Hotdog". TensoRF exhibits a performance drop with all ranks when rotating axis axis-aligned scenes. Pattern: dot - 25%, dash - 50%, dashdot 75%, solid 100% of the rank. Higher better.
The average performance of QTT-NF (with 2.16M parameters) is slightly higher than TensoRF (with 3. 3.M7M parameters). This attests to better parameter-efficiency of QTT-NF for scene representation.
CONCLUSION
We presented TT-NF, a novel low-rank representation for neural fields that can be learned directly via backpropagation through samples and optimization. Our representation avoids instantiating the full uncompressed tensor but instead learns it in a compressed form by optimizing non-convex objective defined by the target task. We applied TT-NF to synthetic tensor denoising task, where we outperformed standard SVD-based approaches and the real-world novel view synthesis problem. For the latter, we proposed QTT-NF, modification of TT-NF that handles hierarchical spaces, such as 3D voxel grids. Last but not least, we proposed efficient sampling algorithms for our neural fields and showed that these algorithms increase efficiency both w.r.t. speed and memory, making (Q)TT-NF friendly for applications such as rendering. We further direct interested readers to Sec. of the Appendix for extra discussion.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Femaanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg. Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.or.
Olah,
Rafael Ballester-Ripoll, Susanne K. Suter, and Renato Pajarola, Analysis of tensor approximation for compression-domain volume visualization. Computers & Graphics, 47:34 2015. ISSN 0097-8493.
Rafael Ballester- Ripoll, Peter Lindstrom, and Renato Pajarola. Tthresh: Tensor compression for multidimensional visual data. IEEE transactions visualization and computer graphics, 26(9): 2891-2903, 2019.
M. Balsa Rodríguez, E. Gobbetti, J.A. Iglesias Guitián, M. Makhinya, F. Marton, R. Pajarola, and S.K. Suter. State-of-the-art in compressed gpu-based direct volume rendering. Computer Graphics Forum, 33(6):77-100,, 2014.
Susan Blackford, Antoine Petitet, Roldan Pozo, Karin Remington, R Clint Whaley, James Demmel, Jack Dongarra, Tain Duff, Sven Hammarling, Greg Henry, al. An updated set of basic linear algebra subprograms (blas). ACM Transactions on Mathematical Software, 18(2):135-151, 2002.
Alexey I. Boyko, Mikhail Matrosov, I. Oseledets, Dzmitry Tsetserukou, and Gonzalo Ferrer Tt-tsdf: Memory-efficient tsdf with low-rank tensor train decomposition. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 10116-10121,
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields, 2022.
V. Dolgov, B. N. Khoromskij. and V. Oseledets. Fast solution of parabolic problems in the tensor train/quantized tensor train format with initial application to the fokker-planck equation. SIAM Journal on Scientific Computing. 34(6)
2012.
S. V. Dolgov, B. N. Khoromskij, I. V. Oseledets, and D. V. Savostyanov. Computation of extreme cigenvalues in higher dimensions using block tensor train format. Computer Phys. Comm., 185(4): 1207-1216, 2014.
Timur Garipov, Dmitry Podoprikhin, Alexander Novikow, and Dmitry P. Vetrov. Ultimate tensoriza- tion: compressing convolutional and FC layers alike. CoRR, abs/161 0/2//6611032244 .03214, 2016.
Sergei A Goreinov and Eugene E Tyrtyshnikov. The maximal-volume concept in approximation by low-rank matrices. Contemporary Mathematics, 280:47-52, 2001.
Sebastian Holtz, Thorsten Rohwedder, and Reinhold Schneider. On manifolds of tensors of fixed tt-rank. Numerische Mathematik, 12014)
Andrey Ignatov, Radu Timofte, Andrei Kulik, Seungsoo Yang, Ke Wang, Felix Baum, Max Wu, Lirong Xu, and Lue Van Gool. Ai benchmark: All about deep learning on smartphones 2019. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 3617-3635. 2019.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference International Conference on Machine Learning Volume 37, ICML' 15, pp. 448-456. JMLR.org, 2015.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. In Proceedings of the British Machine Vision Conference. BMVA Press, 2014.
10

---

Under review as a conference paper at ICLR 2023
Menelaos Kanakis, David Bruggemann, Suman Saha, Stamatios Georgoulis, Anton Obukhov, and Luc Van Gool. Reparameterizing convolutions for incremental multi-tass learning without task interference. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX, volume 12365 of Lecture Notes in Computer Science, pp. 689-707. Springer, 2020.
Vladimir Kazeev, Ivan Oseledets, Maxim Rakhuba, and Christoph Schwab. Qu--fnite-element approx- imation for multiscale problems i: model problems in one dimension. Advances in Computational Mathematics, 43, 04 2017.
Andrew Kerr, Duane Merrill, Julien Demouth, and John Tran. Cutlass: Fast linear algebra in cuda C++, 12 2017.
B. Khoromskij. O(d log(nt)-quantics approximation of n-d tensors in high-dimensional numerical modeling. Constructive Approximation CONSTR APPROX, 34, 01 2009.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015.
Tamara G. Kolda and Brett W. Bader. Tensor decompositions and applications. SIAM Review, 51(3): 455-500, 09 2009.
Jean Kossaifi, Adrian Bulat, Georgios Tzimiropoulos, and Maja Pantic. T-net: Parametrizing fully convolutional nets with single high-order tensor. In CVPR, pp. 7822-7831, 2019a.
Jean Kossaifi, Yannis Panagakis. Anima Anandkumar, and Maja Pantic. Tensorly: Tensor learning in python. Journal of Machine Learning Research, 20(26):1-6, 2019b.
Vadim Lebedev, Yaroslav Ganin, Maxim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding up convolutional neural networks using fine-tuned CP-decompositioo, 2014.
Y. LeCun, B. Boser, S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1(4):541-551, 1989.
Donghyun Lee, Dingheng Wang, Yukuan Yang, Lei Deng, Guangshe Zhao, and Guoqi Li. Qttnet: Quantized tensor train neural networks for 3d object and video recognition. Neural Networks, 141: 420-432, 2021. ISSN 0893-6080.
Yawei Li, Shuhang Gu, Lue Van Gool, and Radu Timofte. Learning filter basis for convolutional neural network compression. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 10 2019.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Guy M Morton. A computer oriented geodetic data base and new technique in file sequencing. Ottawa, Canada, 1966.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. arXiv:2201.0544 January 2022.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML'10 10, pp. 807-814, Madison, WI, USA, 2010. Omnipress. ISBN 978 16055899977
Alexander Novikoy, Anton Rodomanov, Anton Osokin, and Dmitry Vetrov. Putting mrfs on a tensor train. In Eric P. Xing and Tony Jebara (eds.), Proceedings of the 31 International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp. 811-819, Bejing, China, 06 2014. PMLR.
11

---

Under review as a conference paper at ICLR 2023
Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
Alexander Novikov, Pavel Izmailov, Valentin Khrulkov, Michael Figurnov, and Ivan Oseledets. Tensor train decomposition on tensorflow (t3f). Journal of Machine Learning Research, 21(30):1-7, 2020.
Anton Obukhov, Maxim Rakhuba, Stamatios Georgoulis, Menelaos Kanakis, Dengxin Dai, and Luc Van Gool. T-basis: compact representation for neural networks. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 7392-7404. PMIR, 07 2020.
Anton Obukhov, Maxim Rakhuba, Alexander Liniger, Zhiwu Huang, Stamatios Georgoulis, Dengxin Dai, and Luc Van Gool. Spectral tensor train parameterization of deep learning layers. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp. 3547-3555. PMLR, 04 2021.
I. V. Oseledets. Approximation of matrices with logarithmic number of parameters. Doklady Mathematics, 80(2):653-654, October 2009. ISSN 1531-8362.
Ivan V Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5):
2295-2317, 2011.
Ivan V. Oseledets and Eugene E. Tyrtyshnikov. Tt-cross approximation for multidimensional arrays. Linear Algebra and its Applications, 432(():70-88, 2010. ISSN 0024-3795.
Ivan V. Oseledets and Eugene E. Tyrtyshnikov. Algebraic wavelet transform via quantics tensor train decomposition. SIAM Journal on Scientific Computing, 3313):1315-1328, 2011.
Yannis Panagakis, Jean Kossaifi, Grigorios G. Chrysos, James Oldfield. Mihalis A. Nicolaou, Anima Anandkumar, and Stefanos Zafeiriou. Tensor methods in computer vision and deep learning. Proceedings of the IEEE, (115)8863-90, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
J.-R. Poirier, O. Coulaud, and O. Kaya. Fast bem solution for 2-d scattering problems using quantized tensor-train format. IEEE Transactions on Magnelics, 56(3): -4, 2020.
Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. KiloNeRF: Speeding up neural radiance fields with thousands of tiny MLPs. In Proceedingy of the IEEE/CVE International Conference on Computer Vision (ICCV), 2021.
Alex Rogozhnikov. Einops: Clear and reliable tensor manipulations with einstein-like notation. In International Conference Learning Representations, 2022.
Yang Shi, U. N. Niranjan, Animashree Anandkumar, and Cris Cecka. Tensor contractions with extended blas kernels epu and gpu. In 2016 IEEE 23rd International Conference on High Performance Computing (HiPC), pp. 193-202, 2016.
Daniel G. Smith and Johnnie Gray. opt einsum python package for optimizing contraction order for einsum-like expressions. Journal of Open Source Software, 3(26):753, 2018.
Micheline B. Soley, Paul Bergold, and Victor S. Batista. Iterative power algorithm for global optimization with quantics tensor trains. Journal of Chemical Theory and Computation, 17(6): 3280-3291, 2021. PMID: 33956426.
12

---

Under review as a conference paper at ICLR 2023
Yannick Strümpler, Janis Postels, Ren Yang. Luc Van Gool, and Federico Tombari. Implicit neural representations for image compression, 2021.
Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast conver- gence for radiance fields reconstruction. In CVPR, 2022.
S.K. Suter, M. Makhynia, and Pajarola. Tamresh tensor approximation multiresolution hierarchy for interactive volume visualization. Computer Graphics Forum, 32(3p12):111-160 2013.
Mikhail Usvyatsov, Anastasia Makarova, Rafacl Ballester-Ripoll, Maxim Rakhuba. Andreas Krause, and Konrad Schindler. Cherry-pickinn gradients: Learning low-rank embeddings of visual data via differentiable cross-aprrxximation. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 11426-11435, 10 2021.
Mikhail Usvyatsov, Rafael Ballester- Ripoll, and Konrad Schindler. intorch: Tensor network learning with PyTorch. Journal of Machine Learning Research, 23, 2022.
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual computing and beyond. Computer Graphics Forum, 2022. ISSN 1467-8659.
Alex Yu, Sara Fridovich-Kend Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR, 2022.
Jason Zhang, Gengshan Yang, Shubham Tulsiani, and Deva Ramanan. NeRS: Neural reflectance
surfaces for sperse-view 3D reconstruction in the wild. In Advances in Neural Information Processing Systems, 2021a.
Jingyang Zhang, Yao Yao, and Long Quan. Learning signed distance field for multi-view surface reconstruction. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2021b.
Yuchen Zhou, Anru R. Zhang, Lili Zheng, and Yazhen Wang. Optimal high-order tensor svd via tensor-train orthogonal iteration. IEEE Transactions Information Theory, pp. 1-1, 2022.
13

---

Under review as a conference paper at ICLR 2023
A METHOD
A.1 RANKS AND PARAMETERIZATTONS
TT-rank Selection We clarify the procedure of choosing a TT-rank, mentioned in Sec. 4.1. To parameterize TT-NE, we need know the dimensions (modes) of the tensor M, D, the payload dimension Ro, and scalar hyperparameter defining the degree of compression. A TT-rank resulting from a TT-SVD (Oseledets, 201 1) algorithm is bounded in Holtz al. (2012,
Eq. 20), with the only difference in added RD term account for the discussed block structure:
h M. RD II M, 1=++1
D
1 RR S Rg = min II
h 1, D-1.
This means that the maximal TT-rank forms a "pyramid" of integers, where each position is assigned minimum of products of modes to the left and right if We denote Rmaax the maximum value of TT-rank and fmax as the position of the maximum value (the "peak"). As shown in Fig. 3, 7, and 8, Rmax can go high as 1024 in 2 and 32768 in 2 sizes. To achieve compression with our parameterizations, we clamp the "pyramid" at a chosen value of rank r E [1, Rmax] thus making the TT-rank "trapezoid". This range corresponds to the X-axes of the aforementioned figures.
Full TT-NF Parameterization Full parameterization assumes the allocation of learned parameters each element of TT-cores (()) i 1, D. Such parameterization compatible with v1, v2 sampling schemes, as well as tensor contraction and subsampling.
Reduced TT-NF Parameterization A TT manifold of fixed TT-rank has certain number of Degrees of Freedom (DOF) (Holtz et al., 2012), which smaller than the number learned parameters of TT-NF with v2 sampling. Practical mappings from DoF to learned parameters exist but require complex transformations (Obukhov et al., 2021), The latter work provides insights into simpler form of reduced purameterization, which becomes possible once the rank becomes large enough that some TT-cores have square "matricizations" meaning that either R = RG or R M, R. Such cores can effectively be replaced with fixed identity matrices (reshaped into the original core shapes), leading to parameter count reduction. Additionally, knowledge of such reduced parameterization suggests modification to the v2 sampling algorithm, aliased v3.
Given r, we denote S S S D indices of the first and last clamped values in TT-rank. It is easy to verify that TT-cores C() to the left of p and to the right of q should have square matricizations. Indeed, TT-core at position < p has shape R:-1 M; x R3 Rimax x M; R.max
(IT My M4 (II) MI)) thus its left matricization (merging the first two dimensions.
denoted leftmat) is square.
As shown in Obukhov et al. (2021), such cores can effectively be replaced with fixed identity matrices (reshaped into the original core shapes) without any loss of representation power of such TT-NF. The sparsity of identity matrices suggests the possibility skipping steps of the v2 sampling algorithm and changing several matrix multiplication operations (Linear layers) to single indexing operation.
Considering the left side of such TT-NF again, if C LR. then on the first step of Alg. 3 Cit where e, denotes an ort with 1 in j-th position. By induction, if v = iian. and leftmat(C(k)) = I then C/ Miti- Such index propagation rule allows us skip BIMVP computation for all such cores and index v directly along the first rank mode of the first parameterized TT-core at position p. The index propagation rule for the right side of TT-NF can derived similarly. The full algorithm for v3 sampling can be found in Alg. 4.
Notably, in the extreme case of decomposition rank r Rmaxx only a single TT-core contains learned parameters, effectively of an uncompressed tensor. In this case, sampling with v3 becomes the indexing operation along all three modes of the TT-core, leading to zero FLOPs. This corner case explains the drop of FLOPs line around the far end of the rank axis. The remaining saw-like drops occur when r increases above intermediate values, making one or two more TT-cores become reshaped identity matrices. While the absolute gain in FL.OPs is not substantial when rank is not.
14

---

Under review as a conference paper at ICLR 2023
Algorithm 3 Recap of v2 sampling explained in
Algorithm Memory- Sampling from
Sec. 4.2 (for comparison with v3 on the right).
Reduced Parametcrization of TT-NF (v3), Auto- differentiation paths are highlighted in blue. Re- fer Sec. A.1 for more details.
Require:
D number of tensor dimensions,
B number of samples,
Require:
(1, RI.... ..... Rp) TT-rank,
D number of tensor dimensions,
Mp) TT-modes,
B number of samples,
(C )(D)) TT-cores representing A,
1 p S q
parameterized cores range, (1, R1. RD) TT-rank,
(1... (1)
(B) (110.11111 ()...bb))) indices.
(M,,..., Mp) TT-modes,
Ensure:
(C(P) C(a)) TT-cores representing A,
A A (B) samples without computing the whole A.
A (),
(11 ((1)).1b1))) indices.
Ensure:
I: π + (I,..., B²
b> forward permutation
V
AA, (1)
A (B)
samples
2: a (I,
D inverse permutation
without computing the whole A.
3: v + c(1)
1ii..
B x R1
1: leett +- (0....0)
B left indices
4: for & to D do
Fright (O..... 0)
B right indices
5: is x(i))
align mode indices in, B x Rg invert 4th permutation & update forward perm. update inverse perm.
3: for k + 1 to do
6: U, Th (C
D Propagate left indices 5: Heft <- Heft M₁ + (ix - end for
7:
ak k
8:
π t (T)
(Ow) 10: end for
9:
7: for k - 1 to Do
& Propagate right indices
V (v)
recover samples order
9: bright bright (ix 1) R
12: return
> B x Rp
10: end for
11: feett +- Hlet
make start from 1
12: fright +- fright +
make start from
13: if = then
14: & Directly index in ((P)
+ cc(p) fnige-h..fngggeianee--
c(p)
15:
16: else
17:
π
t (1,...,1)
B)
D forward permutation inverse permutation
18: t (L...,
19:
+ C(P)
B x Rp
20:
for 1 + p 1 to do
Intentionally left blank
21:
i x T(i))
Dalign mode indices ug B Rş invert bth permutation t update forward perm. & update inverse perm.
22:
v. Th BIMVP(C(M)
23:
σk + n
24:
- TA (π) a 5 (okk
25:
26:
end for
27: f -v)
P recover order, Rg B Ry
28:
+- -...nghhhiggghtt
29:
end
30: return
B x RD
large, we observe a noticeable speed-up of 2-4x with v3, compared to training with v2, due to fewer sequential invocations of Linear layer functions.
The reduced parameterization is compatible with v1, v2, v3 sampling schemes, as well as tensor contraction and subsampling.
Conversion from Full to Reduced Parameterization Given Full parameterization, conversion to
Reduced can be performed in single pass over TT-cores. Indeed, starting from the first TT-ccore
15

---

Under review as a conference paper at ICLR 2023
Computations (FLOPs)
Training memory (floats)
Inference memary (foats)
108
108
18
16
100M
100M
100M
Barci
10M
1M
100k
100k
10k
1000
10k
1000
100
108
18
100M
100M
10M
10M
1M
18
100k
1006
100k
255
100
10k
1000
108
18
18235 107P9
100M
100M
1006
10M
106
1M
169
16
100k
101
1006
100
10k
1000
106
100
18
9228 93088
100M
100M
100M
100
1M
10
100k
1096
1006
1000
106
100K
1323
1000
10k
100
10K
18
100
1.6
1000
106
106
106
1225
Jin GPU RAM
1000
Contraction
100%
10M
S5
Sampling
106
Sampling v2
106
16
Sampling
1M
100g
200
400
600
800
1000
200
400
600
800
1000
200
400
600
800
1000
rank
rank
rank
Figure 7: Space-time complexity of sampling from TT-NF of size 220 with various methods, batch sizes, and ranks. We compare three sampling schemes discussed in Sec. 4, A. as well as the traditional tensor contraction scheme. As seen from the plots, the optimal choice of sampling scheme depends on the rank, problem, and batch sizes. Lower is better. Best viewed in color.
and repeating until reaching position p. we can compute the left matricization the current TT-core and absorb (through matrix multiplication) this square matrix into the right-hand-side TT-core. The
16

---

Under review as a conference paper at ICLR 2023
Computations (FLOPs)
Training memory (flcats)
Inference memory (foats)
1008
108
11
18
Batch 5:22: 64
108
100M
100M
1M
100k
10k
10k
100
1000
100T
1008
108
17
Batch size: 255
100M
100M
100M
10M
1M
1M
1M
100k
10k
10k
100
100T
17
1000
Petton 02281 1024
11
108
108
18
100M
10M
1M
1M
1001
1007
11
11
1008
108
2238 10880
108
100M
1M
1096
1M
10k
100k
100T
101
10T
17
17
1325 SOPE0
1008
1008
100
108
18
100M
18
100M
100M
10M
10M
1M
1M
100T
1T
1008
108
18
Sampling
GPU RAM
Contraction
1008
108
18
100M
9ES59 1225
100M
Sampling
Sampling v2
100M
10M
1M
5k
10k
15k
20k
25k
30k
5k
10k
15k
20k
25k
30k
5k
10k
15k
20k
25k
30k
rank
rank
rank
Figure 8: Space-time complexity of sampling from TT-NF of size 230 with various methods, batch sizes, and ranks. We compare three sampling schemes discussed in Sec. 4, A. as well as the traditional tensor contraction scheme. As seen from the plots, the optimal choice of sampling scheme depends on the rank, problem, and batch sizes. Lower is better. Best viewed in color.
current core is then replaced with an identity matrix and reshaped into the original shape of the core. similar process starting from the last TT-core until reaching position g completes the conversion.
17

---

Under review as a conference paper at ICLR 2023
A.2 NEURAL RENDERING
The core of neural rendering is comprised of two main components: (1) neural field and (2) differentiable ray marching.
Neural Field In the paper by Mildenhall et al. (2020), the neural field is represented with neural network mapping fo (xr, d) + (c, (C) where æ is a position in some scene-centric frame of reference, camera, is view-dependent color (e.g., 8-bit RGB channels),
d is viewing direction pointing and a is view-independent volume density at r.
Other works (Yu al., 2022) popularized the usage of spherical harmonics to represent view- dependent color variations, Effectively, the viewing direction d is removed from the parameterization; instead, color descriptors of sufficient size are regressed at each location along with density o. Such color descriptors can be coefficients in a pre-defined spherical harmonics basis, effectively parameterizing color as a function on sphere. Alternatively, the transformation of the learned descriptor and viewing direction can be learned with dedicated "shading" MLP, as done in Chen et al. (2022). Empirically, degree two harmonics appear sufficient to represent iieedeependent color variations with high fidelity, which translates into a per-voxel payload size of 28: one value to represent density and three groups of nine coefficients per channel.
Ray Marching Image formation with radiance fields usually accomplished through a differen- tiable ray marching procedure, which is simplistic version of ray-tracing. Given camera calibration, a ray is cast through each pixel of the output image to accumulate color at intersections with the neural field. Samples from the neural field are taken along each ray guided by prior knowledge of how the ray passes through the field and where the scene roughly is. In NeRF (Mildenhall et al., 2020), 64 samples are sampled uniformly to probe the field between hardcoded near and far planes, and then 128 more samples are obtained to increase sampling density at places with content. Voxel grid neural fields (Yu et al., 2022) alter the sampling procedure: rays are filtered based whether they intersect with the voxel grid, and points inside the grid are sampled uniformly between near and far intersection points. A soating-point coordinate sampling of voxel grids is resolved through trilinear interpolation of payload vectors. We follow the same procedure in our experiments with QTT-NF.
Samples (ca, o;) at positions r; of ray are accumulated using the following equations:
C(r) Σ
ca where T; exp
Σ
Here 8; = |rc - T-- 1122. T. represents optical transmittance of the ray position ri, and the last term denotes contribution of the position to the accumulated color.
B EXPERIMENTS
B. EFFECT OF REDUCED PARAMETERIZATION
We compare our reduced (v3) parameterization with the full one (v2) in the default QTT-NF setting with spherical harmonies and report results in Tab. 3 (top). As can be seen, the average PSNR does not change much depending on parameterization. This is contrary to the tensor denoising case described in Sec. 5.1, which we can attribute to a different (longer) training regime.
B.2 INITIALIZATION OF QTT-NF USING TT-SVD
This set of experiments demonstrates the positive effect of QTT-NF initialization from an uncom- pressed voxel grid, We start by training such uncompressed representations with the same grid configuration (2563 x 28 parameters). The training protocol is the same as for QTT-NF training. except for the usage of larger yet different learning rates for the parameters of spherical harmonics (1e-1) and density (1e1). The results can be seen in Tab. (Uncompressed); many scenes receive lower scores than training compressed representations from random initialization due to the lack
18

---

Under review as a conference paper at ICLR 2023
Table 3: Top: per-scene comparison of QTT-NF trained with v2 and v3 from random initialization (Sec. B.1); Bottom: experiments with training uncompressed voxel grid and using it as initialization for QTT-NF through TT-SVD (Sec. B.2); Unlike the tensor denoising setting, in neural rendering. both v2 and v3 parameterizations produce almost identical results. Initialization from uncompressed volume improves performance.
Chair
Drums
Ficus
Hotdog
Lego
Materials
Mic
Ship
Metric
Method (Alaoo)
0000
Avg.
0000
QTT-NF (v2)
31.88
24.95
30.72
35,30
32.17
28.23
31.58
27.84
30.33
QTT-NF (v3)
32.09
24.96
30.89
35,49
32.48
28.22
31.50
27.55
30.40
PSNR
Uncompressed
32.54
25.10
31.30
33.53
31.91
27.63
32.38
22.76
29.64
QTT-NF (init unc.)
32.26
25.09
30.93
35.53
32.76
28.58
32.18
27.93
30.66
QTT-NF (v2)
0,953
0.918
0,967
0.970
0.956
0,929
0.971
0.843
0.938
QTT-NF (V3)
0.955
0.918
0.967
0.971
0.957
0.929
0.971
0.840
0.939
SSIM f
Uncompressed
0.967
0.925
0,970
0.960
0.959
0.926
0.979
0.785
0.934
QTT-NF (init unc.)
0.958
0.921
0.968
0.972
0.961
0,934
0.976
0.845
0.942
QTT-NF (v2)
0.065
0.098
0.040
0.063
0.067
0.093
0.041
0.209
0.085
QTT-NF (V3)
0.062
0:095
0.039
0.060
0.066
0,093
0.039
0,203
0.082
LPIPS 1.
Uncompressed
0.042
0.077
0.038
0.071
0.047
0,088
0.025
0.223
0.076
QTT-NF (init unc.)
0,059
0.093
0.037
0.058
0.058
0,087
0.031
0,202
0.078
of L1 or TV regularization. Further, we apply dimensions factorization of the uncompressed voxel grid to match the decomposition scheme from Fig. 2 and perform TT-SVD with rounding to the TT-rank matching our QTT-NF. The resulting decomposition is loaded into QTT-NF instead of ran- dom initialization per Eq. 2, and the same training protocol (only with lower learning rate of 1e-3) is executed. The results of such fine-tuning are presented in Tab. 3 (QTT-NF init unc.). Evidently, TT-SVD provides a good initialization for QTT-NF, best in L2 distance between the approximation and the uncompressed volume, yet suboptimal in terms of the downstream task performance. This suboptimality is corrected through fne-tuning, and the resulting performance exceeds that of training from random initialization.
C DISCUSSION
QTT -uu-Quantzzedooo or Quantics? The concept of representing data as low-rank tensor decomposi- tions with the logarithmic number of modes and particular mode grouping pattern was first discussed in Oseledets (2009); Khoromskij (2009), the latter work coining the term "Quantics Tensor Train", used up until present days, e.g., Oseledets & Tyrtyshnikov (201 1); Soley et al. (2021). Several other works (Dolgov et al., 2012; Kazeev et al., 2017; Poirier et al., 2020) call the same concept "Quantized Tensor Train", thus making these terms interchangeable. The term means factorizing (or quantizing) space into repeating small factors ("quants"). The term "quantization" in machine learning often refers to the reduction of the dynamic range of values of learned parameters a parametric model. This exact terminology clash seen in QTTNet (Lee et al., 2021) suggests that the usage of the "Quantics" variant might lead to less ambiguity in the considered context.
Supported Parallelism All discussed sampling schemes support data parallelism: TT-NF param- eters can be replicated across multiple computing devices, and a batch of samples is split among them. This contrary to tensor contraction, where the main constraint is to be able fit the entire uncompressed tensor into memory. On the other hand, optimized contraction schemes (Smith & Gray, 2018: Rogozhnikov, 2022) followed by subsampling are expected to outperform all sampling schemes starting with some sufficiently large batch size.
Level Optimizations The proposed sampling schemes (Alg. 1,4) are designed to perform well at training time in all off-the-shelf deep learning frameworks with automatic differentiation support while keeping memory pressure low. Permutations enable the use of the standard Linear layer, which is one of the first functions undergoing heavy optimization any wew-generation computational device, along with memory operations such as permutations themselves. BLAS (Blackford et al., 2002), convolutions (LeCun al., 1989), normalization layers (loffe & Szegedy, 2015), and activation
19

---

Under review as a conference paper at ICLR 2023
functions (Nair & Hinton, 2010). It is possible, however, get rid of permutations through custom implementation of the BIMVP function (Alg. 2), thus obtaining 2x reduction in space requirements. Shi et al. (2016) propose StriddeBBttchedGem BLAS primitive, which would need to accept batch of offsets instead of scalar strides. CUTLASS (Kerr et al., 2017) provides more promising building blocks for implementing both forward and backward passes. Deploying TT-NF sampling for inference in production and edge devices can be feasible, provided the presence of Al accelerators optimized for neural network inference. An overview the current state of mobile computing is given in Ignatov et al. (2019).
Limitations In short training protocols such as experiments with tensor denoising (Sec. 5.1), v3 sampling consistently does not reach the performance of v2, despite effectively learning the same representation. We hypothesize that this effect may have an explanation in the deep linear networks literature. This effect is not pronounced in the NeRF setting with its longer training protocol, in Tab. 3. The reduced complexity of v3 can nevertheless help with saving computational resources during inference sampling via conversion from v2, explained in the last paragraph of Sec. A.1.
Future Research Directions The underlying Tensor Train representation of TT-NF permits many interesting usage scenarios yet be explored. For example, the TT-format can be used quickly compute marginals over selected modes {M;}. which could be used to compute level-of-detail (LOD) samples with QTT-NE. Representation capacity can be controlled dynamically through rank rounding (with TT-SVD or expansion (by padding TT-cores). TT-NF may be found useful in streaming setting, where the neural field continuously updated.
20


Reviews:
[{"review_id": "bG0RfnPmzx7", "reviewer": "Reviewer_woGR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes TT-NF, a low-rank representation for learning neural fields, along with a sampling method for improve the training efficiency. It applies TT-NF to a synthetic tensor denoising task and a neural radiance field rendering task, which demonstrate the performance of the approach.", "summary_of_the_review": "Although the paper has its novelty in applying TT to NF, my major concerns are the efficiency of TT (matrix multiplication vs point-wise multiplication) and its final performance, as discussed above. I think the current results are not convincing to me that TT-NF is superior than TensoRF.", "strength_and_weaknesses": " Strength:\n\n- The idea of incorporating tensor train decomposition to neural fields is exciting, albeit it is a natural way when thinking of different tensor decompositions beyond the CP/VM ones proposed in Tensorf [Chen et al. 2022]\n\n- The sampling algorithm does alleviate the computational issue with the TT formulation\n\nWeakness:\n\n- One major issue of using such TT form is that it involves many matrix multiplications instead of the element-wise multiplications used in CP/VM decomposition as in [Chen et al. 2022]. Therefore, the computational complexity of a single sampling point is O(Dr^3) compared with O(Dr) in CP, where r is the internal rank. Based on my understanding, the sampling method only aims for reducing the space complexity during the inference, and relaxes the matrix multiplication to vector-matrix multiplication. However, the vector-matrix multiplication is still **inefficient** compared with the point-wise multiplication in CP/VM as in [Chen et al. 2022]. Moreover, for back-propagation during training, I believe this would take more GPU memory compared with CP/VM.\n\n- The experimental results on Synthetic NeRF dataset is not convincing. In particular, it apparently has inferior performance than TensoRF in both SSIM and LPIPS. Even for PSNR, I don't see a clear advantage since in the original paper of TensoRF the reported quantitative performance is much higher than that of reported here. Although it is said in the paper that QTT-NF intentionally avoid voxel pruning, total variation loss, etc, I don't think this is a reasonable argument since we can also adopt the same practice here to see if QTT-NF can achieve the similar results under the same setting. Moreover, I know that VM decomposition does a great job in rendering, which is exactly the novelty of TensoRF, and thus the paper misses quantitative comparisons with this representation, making the comparisons not convincing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ebv2nGTNv9V", "reviewer": "Reviewer_wzYM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed a new method, named TT-NF, for learning neural fields in terms of tensor train networks. Technically, a storage-efficient algorithm for sampling from TT-NF was introduced: the storage complexity is reduced from O(BDR^2) to O(BDR), where R denotes the maximum bond dimension (TT-ranks). Empirically, the new method was applied to the task of neural radiance fields. The numerical results showed that the proposed methods achieve competitive performance with SOTAs.", "summary_of_the_review": "The trick on “sampling from TT-NF” is practically useful for other tensor researchers to implement their codes on computers more efficiently. The drawbacks of this paper are also obvious: (relatively) weak novelty and unconvincing experimental results. My recommendation is thus on the boundary.", "strength_and_weaknesses": "++Strength\n\nThe part of “sampling from TT-NF” introduced a practically valuable trick to reduce the storage complexity when calculating the (partial) contraction of tensor networks.\n\n— Weakness\n\n1. Although a lot of literature was overviewed in the “Related work” part, for each highlighted aspect, the overview is not sufficient, especially for the parts “tensor decomposition” and “neural network compression with tensors.”\n2. The novelty is relatively weak. In Section 4 — “Method” — the paper introduced an initialization of TT-NF in Sec 4.1 and the sampling trick from TT-NF in Sec 4.1. For the former, the variance-preserving core initilization has been discussed in (Pan Y et al., 2022); for the latter, it is practically useful (mentioned in Strength), but the trick seems trivial in my own opinion.\n\nPan, Yu, et al. \"A Unified Weight Initialization Paradigm for Tensorial Convolutional Neural Networks.\" International Conference on Machine Learning. PMLR, 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2xPyMpj9DV", "reviewer": "Reviewer_2ByK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new way to sample tensor trains via efficient linear layers in deep learning frameworks. By further leveraging automatic differentiation, they propose to fit tensor trains to data by stochastic gradient descent and backpropagation. This gives a method that is robust to noise and that can be sampled at arbitrary batches of indices. The authors apply their sampling and fitting ideas to tensor denoising where it outperforms existing tensor decomposition methods. They further experiment with novel view synthesis and show improvements over a \"bare-metal\" version of a recent tensor based framework.\n", "summary_of_the_review": "I think this is a nice paper (modulo a couple of editorial remarks) and the premise of using structured tensors to represent images and scenes is exciting. But currently I also feel that the  novelty is not quite at the ICLR level. This is compounded by some lack of transparency in the numerics.\n\n## post-rebuttal (copy)\n\nThank you for the responses to my critical remarks. \n\nAbout the first one: I don't think that suggesting that any of the reviewers advocate for a \"monoculture\" is a great move. \n\nThe comments of Ivan and Rafael adopt a better strategy which is to clarify the contributions of your paper. A good paper needs good results and good presentation. After reading your responses and Ivan's and Rafael's comments, I do think that I misjudged the novelty of your contribution. (Although I still think it is specialist and not at the level of a clear accept.)\n\nRegarding the presentation, it is certainly possible that all reviewers made comments about neural fields, initializations, experiments, ..., because they are advocates of monocultures. But it could also be because the way you structured the story puts too much emphasis on these themes. Your TL;DR is not \"We remove important bottlenecks in tensor decomposition algorithms by doing this and that\" but \"We repurpose the tensor train decomposition to learning compressed neural fields via backpropagation through samples.\" The abstract and the introduction share this emphasis.\n\nSince I do feel that I mischaracterized the novelty (thanks, again, to the remarks of Ivan and Rafael) I will increase my score to a 6. But I will also reiterate my proposal that before employing lofty high-road arguments like \"if you have it your way machine learning will be a monoculture!\", the authors may consider the possibility that a different way to tell the story might have led to different discussions.\n\nWith regard to this last point, I don't think that a revised manuscript has yet been uploaded. I hope that the authors will consider restructuring the story to better emphasize the contributions if the paper is accepted.\n\n", "strength_and_weaknesses": "The manuscript studies an important and very hot topic through the lens of efficient low-rank tensor decompositions. The authors' implementation indeed exhibits better robustness to noise than earlier decomposition algorithms, while offering more flexibility. The idea to use automatic differentiation in this task seems like the right thing to do, and the overall idea to use structured tensors to represent natural images and scenes is intriguing.\n\nOn the critical side, I am uncertain about the level of contribution. The idea to use structured tensors for radiance fields and novel view synthesis has been proposed before. The same is true for the tensor train ideas. Solving non-convex factorization problems by gradient-based optimization is also by now standard. I feel that many relevant comparisons are missing (please see below) and in the one with TensoRF they \"cripple\" the baseline to only compare the tensor-related algorithms. I can see why one may do that but it makes it unclear in what situation one should opportunely use QTT-NF. Some related questions below.\n\n### Various questions and comments\n\n- I am a bit confused by the first paragraph in the introduction: \"Since, in extreme cases, the dimensionality of such fields can exceed the memory size of a typical computer by several orders of magnitude, we look at the problem of learning such fields from the angle of stochastic methods.\"\nI suppose you mean dimensionality in some raw representation? But this depends on the chosen resolution. Since the underlying objects are continuous the resolution can be chosen as high as one desires but without introducing new information, so the notion of dimensionality is relative (as you show, these fields can be compressed). Can you back this up by concrete examples from practice?\n\n- In the second paragraph you write \"Both methods operate under the assumption of noise-free data and are not guaranteed to output sufficiently good approximations in the presence of noise.\"---this statement implies that data is regular in some predetermined sense and low-rank tensors respect this regularity. \"Noise\" would then be any excursion from this model. But there exists a great variety of models for images and other data---can you give an intuitive explanation for why TT-NF or any other tensor decomposition method should be the right choice?\n\n- On p4, first paragraph, you write that \"... $\\cdot$ denotes merged (flattened) dimensions\" but there are no $\\cdot$s in the hierarchical expression a few lines up.\n\n- What exactly does it mean that some algorithms \"don't support noise in observations\"? What happens when there _is_ noise? In particular, since this work is purely algorithmic, how would non-sub-gaussianity of noise manifest with TT-OI? (From Figure 4 it seems that TT-OI does a bit better than TT-SVD which according to your classification should not \"support\" noise.)\n\n- is there a principled way to choose R_max in practice? is it possible to dynamically increase rank (e.g., if the loss stops decreasing at a value that is too high?)\n\n- what is the influence of initialization (2) on the final result? \n\n- This is to some degree semantics, but a lot of (most?) recent work on neural fields is about representations that can be queried at a continuum of input coordinates. There are many good reasons this is convenient, one important one is that it is easy to interface them with the various PDE solvers which evaluate spatial derivatives (often adaptively). Unless I am missing something, this is not possible with TT-NF. So TT-NF are primarily useful for compression, or some sort of regularization via the implicit multiscale inductive bias (from structured low-rankness).\n\n- I find that the prose lacks clarity. A lot of it is too wordy (e.g. \"... in the presence of access to full tensor elements...\") and I had great difficulties following certain parts of the technical exposition. A case in point is the description of \"v2\" sampling scheme. (Up to this point I understood what is going on.) What does it mean to produce a batch of intermediates v? In what sense do linear layers require the samples to be packed densely? What is the alternative? Does overline over 2, D signify a range set? The statement \"... we split the inputs v according to which M_i slices...\". (After some effort I think I understand this part, but it should be written in a way that is friendlier to readers who only occassionaly dabble in tensor decompositions.)\n\n- In Section 5.2, I understand the motivation to turn off the various techniques TensoRF uses to improve results, but then I find it a bit unfair to criticise it for grid alignment artifacts, at least without proper hedging. The method was designed as a package with a reason. For a fully transparent comparison I would like to see results for TensoRF with all the bells and whistles turned on, as well as results where neither TensoRF nor QTT-NF use the \"tiny MLP\".\n\n- Further, it would be good to see comparisons with NERF or some state of the art neural-network based neural field on the rendering task. For full transparency, I would be interested in times (in addition to flops) required to fit some good neural model and QTT-NF on this task. I would also like to see comparisons like in Figure 3 for TT-{SVD, OI, cross}, perhaps also with the timings.\n\n- Related to the above, I find it strange to use the term neural field for a tensor-based method which does not use a neural network.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject