Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a new way to sample tensor trains via efficient linear layers in deep learning frameworks. By further leveraging automatic differentiation, they propose to fit tensor trains to data by stochastic gradient descent and backpropagation. This gives a method that is robust to noise and that can be sampled at arbitrary batches of indices. The authors apply their sampling and fitting ideas to tensor denoising where it outperforms existing tensor decomposition methods. They further experiment with novel view synthesis and show improvements over a "bare-metal" version of a recent tensor based framework.

strength_and_weaknesses: The manuscript studies an important and very hot topic through the lens of efficient low-rank tensor decompositions. The authors' implementation indeed exhibits better robustness to noise than earlier decomposition algorithms, while offering more flexibility. The idea to use automatic differentiation in this task seems like the right thing to do, and the overall idea to use structured tensors to represent natural images and scenes is intriguing.

On the critical side, I am uncertain about the level of contribution. The idea to use structured tensors for radiance fields and novel view synthesis has been proposed before. The same is true for the tensor train ideas. Solving non-convex factorization problems by gradient-based optimization is also by now standard. I feel that many relevant comparisons are missing (please see below) and in the one with TensoRF they "cripple" the baseline to only compare the tensor-related algorithms. I can see why one may do that but it makes it unclear in what situation one should opportunely use QTT-NF. Some related questions below.

### Various questions and comments

- I am a bit confused by the first paragraph in the introduction: "Since, in extreme cases, the dimensionality of such fields can exceed the memory size of a typical computer by several orders of magnitude, we look at the problem of learning such fields from the angle of stochastic methods."
I suppose you mean dimensionality in some raw representation? But this depends on the chosen resolution. Since the underlying objects are continuous the resolution can be chosen as high as one desires but without introducing new information, so the notion of dimensionality is relative (as you show, these fields can be compressed). Can you back this up by concrete examples from practice?

- In the second paragraph you write "Both methods operate under the assumption of noise-free data and are not guaranteed to output sufficiently good approximations in the presence of noise."---this statement implies that data is regular in some predetermined sense and low-rank tensors respect this regularity. "Noise" would then be any excursion from this model. But there exists a great variety of models for images and other data---can you give an intuitive explanation for why TT-NF or any other tensor decomposition method should be the right choice?

- On p4, first paragraph, you write that "... $\cdot$ denotes merged (flattened) dimensions" but there are no $\cdot$s in the hierarchical expression a few lines up.

- What exactly does it mean that some algorithms "don't support noise in observations"? What happens when there _is_ noise? In particular, since this work is purely algorithmic, how would non-sub-gaussianity of noise manifest with TT-OI? (From Figure 4 it seems that TT-OI does a bit better than TT-SVD which according to your classification should not "support" noise.)

- is there a principled way to choose R_max in practice? is it possible to dynamically increase rank (e.g., if the loss stops decreasing at a value that is too high?)

- what is the influence of initialization (2) on the final result? 

- This is to some degree semantics, but a lot of (most?) recent work on neural fields is about representations that can be queried at a continuum of input coordinates. There are many good reasons this is convenient, one important one is that it is easy to interface them with the various PDE solvers which evaluate spatial derivatives (often adaptively). Unless I am missing something, this is not possible with TT-NF. So TT-NF are primarily useful for compression, or some sort of regularization via the implicit multiscale inductive bias (from structured low-rankness).

- I find that the prose lacks clarity. A lot of it is too wordy (e.g. "... in the presence of access to full tensor elements...") and I had great difficulties following certain parts of the technical exposition. A case in point is the description of "v2" sampling scheme. (Up to this point I understood what is going on.) What does it mean to produce a batch of intermediates v? In what sense do linear layers require the samples to be packed densely? What is the alternative? Does overline over 2, D signify a range set? The statement "... we split the inputs v according to which M_i slices...". (After some effort I think I understand this part, but it should be written in a way that is friendlier to readers who only occassionaly dabble in tensor decompositions.)

- In Section 5.2, I understand the motivation to turn off the various techniques TensoRF uses to improve results, but then I find it a bit unfair to criticise it for grid alignment artifacts, at least without proper hedging. The method was designed as a package with a reason. For a fully transparent comparison I would like to see results for TensoRF with all the bells and whistles turned on, as well as results where neither TensoRF nor QTT-NF use the "tiny MLP".

- Further, it would be good to see comparisons with NERF or some state of the art neural-network based neural field on the rendering task. For full transparency, I would be interested in times (in addition to flops) required to fit some good neural model and QTT-NF on this task. I would also like to see comparisons like in Figure 3 for TT-{SVD, OI, cross}, perhaps also with the timings.

- Related to the above, I find it strange to use the term neural field for a tensor-based method which does not use a neural network.

clarity,_quality,_novelty_and_reproducibility: Per my comments above, I think that clarity could be improved, but the majority of the manuscript is reasonably clear. I have some reservations about novelty which I also expressed above. The results seem reproducible with the code provided in the supplement (I have not attempted to run it myself.)
summary_of_the_review: I think this is a nice paper (modulo a couple of editorial remarks) and the premise of using structured tensors to represent images and scenes is exciting. But currently I also feel that the  novelty is not quite at the ICLR level. This is compounded by some lack of transparency in the numerics.

## post-rebuttal (copy)

Thank you for the responses to my critical remarks. 

About the first one: I don't think that suggesting that any of the reviewers advocate for a "monoculture" is a great move. 

The comments of Ivan and Rafael adopt a better strategy which is to clarify the contributions of your paper. A good paper needs good results and good presentation. After reading your responses and Ivan's and Rafael's comments, I do think that I misjudged the novelty of your contribution. (Although I still think it is specialist and not at the level of a clear accept.)

Regarding the presentation, it is certainly possible that all reviewers made comments about neural fields, initializations, experiments, ..., because they are advocates of monocultures. But it could also be because the way you structured the story puts too much emphasis on these themes. Your TL;DR is not "We remove important bottlenecks in tensor decomposition algorithms by doing this and that" but "We repurpose the tensor train decomposition to learning compressed neural fields via backpropagation through samples." The abstract and the introduction share this emphasis.

Since I do feel that I mischaracterized the novelty (thanks, again, to the remarks of Ivan and Rafael) I will increase my score to a 6. But I will also reiterate my proposal that before employing lofty high-road arguments like "if you have it your way machine learning will be a monoculture!", the authors may consider the possibility that a different way to tell the story might have led to different discussions.

With regard to this last point, I don't think that a revised manuscript has yet been uploaded. I hope that the authors will consider restructuring the story to better emphasize the contributions if the paper is accepted.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes TT-NF, a low-rank representation for learning neural fields, along with a sampling method for improve the training efficiency. It applies TT-NF to a synthetic tensor denoising task and a neural radiance field rendering task, which demonstrate the performance of the approach.
strength_and_weaknesses:  Strength:

- The idea of incorporating tensor train decomposition to neural fields is exciting, albeit it is a natural way when thinking of different tensor decompositions beyond the CP/VM ones proposed in Tensorf [Chen et al. 2022]

- The sampling algorithm does alleviate the computational issue with the TT formulation

Weakness:

- One major issue of using such TT form is that it involves many matrix multiplications instead of the element-wise multiplications used in CP/VM decomposition as in [Chen et al. 2022]. Therefore, the computational complexity of a single sampling point is O(Dr^3) compared with O(Dr) in CP, where r is the internal rank. Based on my understanding, the sampling method only aims for reducing the space complexity during the inference, and relaxes the matrix multiplication to vector-matrix multiplication. However, the vector-matrix multiplication is still **inefficient** compared with the point-wise multiplication in CP/VM as in [Chen et al. 2022]. Moreover, for back-propagation during training, I believe this would take more GPU memory compared with CP/VM.

- The experimental results on Synthetic NeRF dataset is not convincing. In particular, it apparently has inferior performance than TensoRF in both SSIM and LPIPS. Even for PSNR, I don't see a clear advantage since in the original paper of TensoRF the reported quantitative performance is much higher than that of reported here. Although it is said in the paper that QTT-NF intentionally avoid voxel pruning, total variation loss, etc, I don't think this is a reasonable argument since we can also adopt the same practice here to see if QTT-NF can achieve the similar results under the same setting. Moreover, I know that VM decomposition does a great job in rendering, which is exactly the novelty of TensoRF, and thus the paper misses quantitative comparisons with this representation, making the comparisons not convincing.
clarity,_quality,_novelty_and_reproducibility: Overall, the paper is clear, and should be reproducible.
summary_of_the_review: Although the paper has its novelty in applying TT to NF, my major concerns are the efficiency of TT (matrix multiplication vs point-wise multiplication) and its final performance, as discussed above. I think the current results are not convincing to me that TT-NF is superior than TensoRF.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed a new method, named TT-NF, for learning neural fields in terms of tensor train networks. Technically, a storage-efficient algorithm for sampling from TT-NF was introduced: the storage complexity is reduced from O(BDR^2) to O(BDR), where R denotes the maximum bond dimension (TT-ranks). Empirically, the new method was applied to the task of neural radiance fields. The numerical results showed that the proposed methods achieve competitive performance with SOTAs.
strength_and_weaknesses: ++Strength

The part of “sampling from TT-NF” introduced a practically valuable trick to reduce the storage complexity when calculating the (partial) contraction of tensor networks.

— Weakness

1. Although a lot of literature was overviewed in the “Related work” part, for each highlighted aspect, the overview is not sufficient, especially for the parts “tensor decomposition” and “neural network compression with tensors.”
2. The novelty is relatively weak. In Section 4 — “Method” — the paper introduced an initialization of TT-NF in Sec 4.1 and the sampling trick from TT-NF in Sec 4.1. For the former, the variance-preserving core initilization has been discussed in (Pan Y et al., 2022); for the latter, it is practically useful (mentioned in Strength), but the trick seems trivial in my own opinion.

Pan, Yu, et al. "A Unified Weight Initialization Paradigm for Tensorial Convolutional Neural Networks." International Conference on Machine Learning. PMLR, 2022.
clarity,_quality,_novelty_and_reproducibility: **Clarity and Quality:**

1. Following Eq. (2), there should be proofs to say why $\hat{\sigma}$ guarantees the variance of the full tensor.
2. In the experiment of tensor denoising, the proposed method outperforms TT-SVD and others, but there is no discussion on where the advantage comes from. One reason, I guess, is that the TT-SVD assumes the semi-orthogonality of the TT-cores, but the proposed method does not. It makes the TT-SVD is representationally weaker than the proposed one due to the additional constraints. Therefore, I doubt if the experiment of tensor denoising in the paper can support the central claim properly.
3. For the second experiment of neural radiance fields, Tab 2 shows that the proposed method is only comparable with the existing methods. I am thus confused about the main advantages of the new methods then. If there is another advantage, such as computational efficiency or interpretability, it should be clearly mentioned in this part.

=========================
*Update*:
1. " One reason, I guess, is that the TT-SVD assumes the semi-orthogonality of the TT-cores, but the proposed method does not. It makes the TT-SVD is representationally weaker than the proposed one due to the additional constraints."

-- I am convinced that the additional semi-orthogonality will not affect the representational power of the TT model. In other words, it is fair to say the TT-SVD, TT-OI, Contraction and the proposed v2 and v3 in Fig. 4 have similar representational power, and the main differences among these methods are optimization schemes. Based on this point, it is not clearly explained why v2/3 outperforms other methods a lot in Fig. 4. Although the authors mentioned that Contraction might get stuck in saddle points, it cannot convince me due to the lack of more detailed investigation. 

**Novelty:**

Please check the “Strength and Weakness” section.

**Reproducibility**

The experimental settings were introduced clearly.
summary_of_the_review: The trick on “sampling from TT-NF” is practically useful for other tensor researchers to implement their codes on computers more efficiently. The drawbacks of this paper are also obvious: (relatively) weak novelty and unconvincing experimental results. My recommendation is thus on the boundary.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this paper, the authors introduce a method called Tensor Train Neural Fields (TT-NF) for learning neural fields on dense regular grids. They use backpropagation to train the TT-NF representation, which is a low-rank parameterization of the neural field, and minimize a non-convex objective. They show various experiments by applying it to a tensor denoising task and Neural Radiance Fields. The reviewers thought the paper studies an important and contemporary topic through the lens of efficient low-rank tensor decompositions and thought the robustness to noise experiments were interesting. The reviews however raised concerns about the novelty of using tensors for radiance fields and novel view synthesis given prior work in the area, novelty w.r.t. tensor train methods, and lack of comparison with many baselines, clarity of writing, and the need for many multiplications. One reviewer also thought "the experimental results on Synthetic NeRF dataset is not convincing". While some of the concerns were alleviated based on the authors thorough response many were not. Based on my own reading this paper is interesting and has potential but is also clear that the above issues need to be properly addressed before the paper can be accepted for publication. I therefore recommend rejection. 
justification_for_why_not_higher_score: All scores clearly below threshold
Decision: Reject