Deep Tree Tensor Networks
Authors: Chang Nie
Venue: NeurIPS 2025 poster ()
Keywords: Tree Tensor Networks
Multilinear Networks
Image Recognition
Primary area: deep_learning
TL;DR: This work proposes a tree tensor network without activation nonlinearity, and for the first time reveals its equivalence to quantum-inspired models.

Abstract: Originating in quantum physics, tensor networks (TNs) have been widely adopted as exponential machines and parametric decomposers for recognition tasks. Typical TN models, such as Matrix Product States (MPS), have not yet achieved successful application in natural image recognition. When employed, they primarily serve to compress parameters within pre-existing networks, thereby losing their distinctive capability to enhance exponential-order feature interactions. This paper introduces a novel architecture named Deep Tree Tensor Network (DTTN), which captures 
-order multiplicative interactions across features through multilinear operations, while essentially unfolding into a tree-like TN topology with the parameter-sharing property. DTTN is stacked with multiple antisymmetric interacting modules (AIMs), and this design facilitates efficient implementation. Furthermore, our theoretical analysis demonstrates the equivalence between quantum-inspired TN models and polynomial/multilinear networks under specific conditions. We posit that the DTTN could catalyze more interpretable research within this field. The proposed model is evaluated across multiple benchmarks and application domains, demonstrating superior performance compared to both peer methods and state-of-the-art architectures. Our code will be made publicly available upon publication.

Deep Tree Tensor Networks
Chang Nie Nanjing University of Science and Technology Nanjing, China
changmie@njust. edu.
Abstract
Originating in quantum physies, tensor networks (TNs) have been widely adopted exponential machines and parametric decomposers for recognition tasks. Typ- ical TN models, such as Matrix Product States (MPS), have not yet achieved successful application in natural image recognition. When employed, they pri- marily serve to compress parameters within pre-existing networks, thereby losing their distinctive capability to capture exponential-order feature interactions. This paper introduces a novel architecture named Deep Tree Tensor Network (DTTN), which captures -order multiplicative interactions across features through mul- tilinear operations, while essentially unfolding into a tree-like TN topology with the parameter-shariee property. DTTN is stacked with multiple antisymmet- ric interaction modules (AIMs), and this design facilitates efficient implemen- tation. Furthermore, our theoretical analysis demonstrates the equivalence be- tween quantum-inspiiee TN models and polynomia//mlliiines networks under specific conditions. We posit that the DTTN could catalyze more interpretable research within this field. The proposed model is evaluated across multiple benchmarks and domains, demonstrating superior performance compared to both peer methods and state-of-the-ard architectures. Our code publicly available at httpp:: htttp:://itthbb.. wttp:/co//nni/eeeee deep Sreespp__rre___eeee _eneeer_etorr__e _ettrrrreetrrrrree
Introduction
"Simplicity is the ultimate sophistication." Leonardo da Vinei
The wavefunction of quantum many-body system typically resides an extremely high-dimensione Hilbert space, with its complexity increasing exponentially the particle count grows [27 60). For example, consider a system consisting of spin- particles; the dimensionality the corresponding Hilbert space would be 2 Tensor networks (TNs) offer powerful numerical techniques for tackling the "Curse of Dimensionality" [12] By leveraging the local entanglement properties quantum states, TNs represent complex wavefunctions into multilinear representations of multiple low- now-dimensional cores, thereby significantly reducing computational and storage requirements (T]] 44 This class of methods allows for an accurate representation of quantum states while mitigating the exponential
growth complexity, making it feasible simulate large-scale quantum systems 26 44
Recently, TN-based interpretable and quantum-inspired white-bon -bx machine learning has attracted the attention of researchers. It holds the potential generate novel schemes that can run quantum hardware 25 433. Typical TN models, including Matrix Product States [13 (M Tree Tensor
Network (TTN) 5 and Multi-Scale Entanglement Renormalization Ansatz (MERA) 45
'In tensor network theory, states that satisfy area law entanglement entropy can be efficiently approximated using TNs with finite bond dimensions.
"The MPS is also referred Tensor Train 41 or Tensor Ring 61 with the periodic condition in classical machine learning.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Quantum-inspired TN model
Deep Tree Tensor Network
Parameter-d
Flatten
Tiny
Natural Image
Linear
Image
Feature Map
MPS
Output
Projection
DIIN
Output
Figure 1: Schematic diagram of the quantum-inspired inspired MPS model and DTTN towards image recognition task. The former is applied for tiny inputs and setting a small local mapping dimension d = 2 and bond dimension D s 64 in general DTTN handles complex inputs while retaining spatial locality in linear projection, and its parameter-sharing nature allows for maintaining a high bond dimension.
skillfully applied in image classification. Routine practice is to map each pixel or local patch of the image to d-dimensional vector by a local map function o(-) and then use the tensor product to obtain a joint feature map & (xx) of d' dimensionality (see Fig. left). This process can be expressed as follows:
f(z) = arg max (W", Q(x)).
Here, T represents (N + 1)-th order tensor with an output index m; f()) Ruxbxx + IR denotes multilinear function. In principle, mapping samples into exponential dimensional space to achieve linear separability instead of adopting activation functions is the essence of TNs [47 43 42 However, existing methods are limited simple tasks, e.g., MNIST and Fashion MNIST [5 43) and we reveal that this is mainly due to 1) low computational efficiency and 2) lack of feature self-interaction capability (Section 3.3 for more details). Consequently, our goal is to address these challenges by applying TNs to complex benchmarks such as ImageNet-1K, thereby bridging the gap between TNs and modern architectures.
Alternatively, TNs are popularly employed to parameterize existing network models for acceleration. For example, the variational parameters of a network can be directly decomposed into TN formats, including convolutional kernels [40], fully-connected weights [32 39], and attention blocks (34), to name few. During inference, one can choose to retain merge the TN structure as required. However, such techniques are devoid of probabilistic interpretation and feature multiplicative in- teraction. Notably, there exist intriguing and previously unexplored similarities between advanced deep learning architectures and TNs, such as MLP-Mixer (51), polynomial networks, and multilinear networks 9 10 71 which have demonstrated strong performance complex visual tasks. As shown in Fig. 2. we visualize different architectures from the TN perspective for comparison. We note that contemporary architectures exhibit several key distinctions from existing TNs, including the incorporation of nonlinear activations and instance normalization. Consequently, we can improve TNs by drawing insights from advanced architectures through rigorous equivalence analysis, thereby overcoming their limitations on complex benchmarks.
Concretely, introduce novel class of TN architectures, named Deep Tree Tensor Network (DTTN), which uses multilinear operations to capture exponential-orde -order interactions among features and predict targets without relying on activation functions attention mechanisms. DTTN is constructed by sequentially stacking multiple antisymmetric interaction modules (AIMs) and inherently unfolds into tree-like structure to reach on-par performance compared to advanced architectures, with better interpretability and understanding. Overall, the contributions of this paper are summarized as follows:
We introduce DTTN, a simple yet effective architecture constructed by sequentially stacking AIMs. DTTN captures feature 24 multiplicative interactions without activation functions and achieves state-of-the-arr performance compared to other polynomial and multilinear networks with faster convergence (see Fig. (c))
We provide a theoretical analysis of the equivalence between DTTN and other architectures under specific conditions. Without instance normalization, DTTN essentially reduces to a tree-topology TN, thereby overcoming the limitations of TN-based Born machines that excel mainly in simple tasks.

---

Ingert
Telerta
Lineet
nihee 13000, 23.351
LN
Hovel Moleet (2720 13:00
Top-3
13820 to 2023 12,30
Lisces
CELU
PICINST (202), 22.241
Miser
Badic
Lansaz
OITI Squee 12.
Hil in
X0
Epoch
G6) ML P-Mixer
(b) Nets
(c) MONet
(d) DTTN
(e) Comparison ImageNNetTKK
Figure 2: (a-d) Illustration Core Blocks for Different Architectures. The MLP-Mixer utilizes GELU activation and other networks via the Hadamard product "*" to enable the network to learn complex representations. We emphasize that instance batch normalization (IBN) and layer normalization (LN (57]) preceded before Hadamard product operations disrupt the polynomial unfolding nature of R-PolyNets [0]] and MONet 77]. In contrast, the succinctly designed AIM circumvents this issue.
The optional LN inside AIM only enhances performance but also facilitates faster convergence. (e) Comparison of Different Networks on ImageNet- When comparing various networks trained ImageNet different epochs, DTTN stands out by achieving state-of-the-arr performance compared to other multilinear networks, significantly outperforming them.
We conduct comprehensive evaluation of the proposed model's effectiveness and broader applicability across diverse benchmarks and domains, including visual recognition, rec- ommender systems, and physics-informed neural networks. Our results show that DTTN achieves performance comparable to, and certain cases even outperforms, carefully designed state-of-th offthe--rrrr art architectures.
Notations.
Throughout this paper, we use x E R X E Rlik, X E R RIXx.xXX to denote first-order vectors, second-order matrices, and N-th order tensors, respectively. The blackboard letters are employed to represent a set of objects, e.g., lR and denote real numbers and integers. In addition, *, O and denote the Hadamard product, Khatri-Rao product, and fensor product, respectively. For brevity, |K] denotes the cardinality of set IK, and KN denotes the positive integers set (1, ..,........ J. Moreover, for given tensors A 1 and B IR 1 x1,, with 1a 11 and /3 = 4s. The tensor contraction is executed by summing along the shared modes of A, B to yield new tensor C = A 2,3
B E R/xlex The entry-wise calculation can be expressed as
(hi.ioii)) = Ele =1 Elg A (biddaii) ( ((anianooo))) Refer to 29 for additional definitions.
Related Work
2.1 Quantum-Inspired TNs
Quantum-inspires tensor networks (TNs) enhance the performance of classical algorithms by mimick- ing quantum computational characteristics (25). These methods map inputs into Hilbert space with exponential dimensionality, achieving linear separability through local mappings and tensor prod- ucts, while employing multiple low-order cores to parameterize coefficients, significantly reducing computational and storage complexity [47 49 The avoidance of activation functions, alongside the theoretical underpinnings rooted in many-bodi physics, contributes to the interpretability of TNs [43) Recently, numerous studies have successfully applied TNs to tasks such as image classification [49], generation [5), and segmentation 488 These studies effectively integrate established neural network techniques like residual connections (38) multiscale structures (5). and normalization (47) into TN
frameworks. However, current TNs are predominantly suited for simpler tasks and face limitations in
terms of computational efficiency and expressive power.
Advanced Modern Networks
In the contemporary landscape of deep learning, the design of network architectures has grown
increasingly sophisticated and varied, each architecture presenting distinct advantages. Advanced models such as Convolutional Neural Networks (CNNs) renowned for efficient feature extraction 220
3

---

22
Transformers distinguished by their powerful contextual understanding capabilities [54), MLP. based architectures celebrated for their simple yet effective designs [5T], and Mamba noted for its linear complexity [18] have become pivotal across a wide array of applications. These networks leverage nonlinearities, while beneficial for model expressiveness, but limit applicability domains such as security and encryption. Notably, Leveled Fully Homomorphic Encryption schemes can only
support addition and multiplication operations
2.3 Polynomial and Multilinear Networks
Polynomial and multilinear networks employ addition and multiplication operations to construct intricate network representations TO Specifically, the pioneering Polynomial Network (PN) 9 constructs higher-order polynomial expansions of the input features in modular fashion while supporting end-to-end training, achieving notable success in both image recognition and generation tasks. In their follow-up work, Chrysos et al. []0] introduce regularization strategies-such as data augmentation, instance normalization, and higher-order feature interactions- further enhance model performance. Cheng et al. ? drew inspiration from modern architectural designs to propose MONet, aiming to narrow the gap between multilinear networks and modern architectures. is worth noting that both polynomial and multilinear networks capture exponential-ore feature interactions. However, key distinction lies in their structural unfoldability: polynomial networks maintain an unfoldable structure, whereas multilinear networks may lose this property when LN is applied, as
(Az) * (B2) = vec(= 2 @ <AA + B"). LN(Ax) (Bz) + vec(# & 2) LN(A O B²).
Here LN(.) represents layer normalization, while A and B are learnable matrices.
In this work, we aim to achieve two objectives: (1) re-establish the polynomial expansion form and analyze the differences between multilinear and quantum-inspines TNs; and (2) develop layer- normalized multilinear DTTNi that outperforms existing high high-perforrance multilinear networks (7].
3 Method
In this section, we provide detailed description the Deep Tree Tensor Network (DTTN). We aim to construct tree topology network by sequentially stacking AIM blocks, which consist solely of multilinear operations. For an input image æ, we apply a vanilla linear projection, also known as patch embedding (51), to obtain the feature map o(æ, A E W XXCC Similar approaches are used
in other methodologies. Here A E RRS represents a learnable matrix, where S, C N denote the local patch size and the number of output channels, respectively. This procedure corresponds to the local mapping illustrated in Fig. [ with its output serving as the input for the DTTN. It should be noted that batch normalization (BN) operations following the linear layer have been omitted for brevity, as these operations can be integrated with the nearest linear layer during inference through structural re-parameterization technique [15].
3.1 Antisymmetric Interaction Module
The antisymmetric interaction module (AIM) is the core of DTTN. As illustrated in Fig. dd), for the 1-th block input feature map x R @WXXHixC,, we utilize an antisymmetric two-branch structure to capture the linear interactions of the input features separately. Both parts, denoted as f. incorporate depthwise convolution layer and linear layer, but apply them in reversed order. These layers are designed to capture spatial locality and channel interactions, respectively, effectively combining the advantages of CNN and MLP architectures
[20 51). The antisymmetric design specifically targets reducing the complexity of AIM. The ratio of parameters and FLOPs between the two branches can be expressed as follows:
Teaph2 Ci + T Cp
R
Teaph? C1 12 reap C2
Teep
(3)
Temph? WWWWHHCCi + Teep WHH.C
RFiops
Teaph W,,,WHH.. + r2 WHH.C
Tezp

---

Table 1: Specifications of different DTTN variants config- uration. "Tiny", "Small", and "Large" refer to DTTN- T. DTTN-S, and DTTN-L configurations, respectively, each differing in parameter sizes. The primary distinc- tion among these variants lies the number of blocks and the hidden-sizes within their multi-stage structures. Addi- tionally, models with layer normalization (LN) layers are denoted with a 'f' such as DTTNI-S.
DTIN Architeclure
Po- Proecioo
XxANN AIM Block
Anarag Podlig
w/oLNN
w/ LN
Specification
Tiny
Small
Numbers of Blocks
34
44
56
Hidden-size
64, 1.,1228,60000
16,128,192,1999
128,192,256,888
Stages
6,6,16,6
6,6,24,8
8,8,32.8
146 Block
Expansion ratio
Parameters (M)
7.1
12.3
35.9
Figure 3: Schematic overview of the FLOPS (B)
2.3
12.3
DTTN architecture.
where Teep E N+ represents the expansion ratio in the inverted bottleneck design, typically set to 3, & E N denotes the kernel size, and Ct, Wi, Hj are the number of channels, width and height of the 1-th block input feature map, respectively. We use the hardware-friendy: Hadamard product (+) to capture the second-order interactions of the branch outputs. Following this, an optional LN and linear projection layer are sequentially applied to the computation results. Finally, shortcut connection is used to preserve the input signal and accelerate training (see Fig. 2 Overall, the AIM forward calculation can be expressed as
x x² Pro (Fx(2) B(xx)) Vi Kr,
where Pro(-) denotes the projection transformation operation. In summary, AIM captures second- order multiplicative interactions among input elements through multilinear operations without em- ploying nonlinear activations. In contrast core blocks inside other architectures, such the Basic Block in R-PolyNets [10] and Mu-layer in MONet (7), AIM employs an antisymmetric design with only one shorteut connection.
3.2 Network Architecture
Our proposed architecture constructed by stacking AIM blocks sequentially. As shown in Fig. 3] the final output is derived through average pooling and fully-connected layer. The DTTN architecture is multi-stage, in line with its peers ( 2T 7 By varying the hidden-sizes and the number of blocks in each stage, we have designed three variants: DTTN-T, DTTN-S, and DTTN-L. (see Table D. each with distinct parameters facilitate comparative analysis. high level, we assert that the DTTN exhibits the following theoretical property:
Proposition 1. The DITN has the capability to capture 2 multiplicative interactions among input elements, which can be represented in the format of Equation (1) as Φ(x) = 8² ((x, Ao). Consequently, the elements of fxx) are homogeneous polynomials of degree 2 over the feature map Al)
It is important to note, however, that this characteristic no longer holds when LN is incorporated into the network, as LN introduces second-order statistical information. This phenomenon is clearly exemplified in models like MONet and DTTNf, where the network essentially behaves as a standard multilinear model.
Unfolding Topology. As illustrated Fig. 3 the AIM is equivalent to a binary tree node in the
absence of LN. In this case, AIM forward computation can be regarded TN contraction, which
'The MONet architecture comprises a stack of two variants of Mu Mayery The second variant differs from the shown in Fig. 2) in that it does incorporate spatial shift operation.
5

---

can be formulated as
x x + B² ((A)2') (A2a²)
d' Reshape (A 2 Yr) X2 (a' o(
(5)
(ax @ xz)
Here x' vec(X") E C C' signifies third-order tensor representing structural re- parameterization with the learnable matrices Ai. Al, and B'. The AIM captures the second-order multiplicative interactions among imput elements via tensor product and structures the threc-order tensor utilizing PyTorch operators. Thus, one DTTN composed AIMs can essentially be unfolded into a tree network with 2 leaf nodes, pictured in Fig.
3.3
DTTN vs. Other Architectures
DTTN Polynomial Networks. DTTN can be expressed the same polynomial expansion form as NT-Net [9, which is given by
f(z) = Σ Will
LI...d 241 41 (o'6(z))) + 3
(6)
where 3 E Rm represents the constant term, and Will is (1 1)-th order learnable parameter tensor. Notably, equations ① and (6) become equivalent upon introducing bias term for the elements of the input vector æ. The networks exhibit different structured representations of the coefficients Will, for all E K22,, due to their distinct network blocks and computational graphs.
DTTN & Multilinear Networks. We note that networks which exclusively involve multilinear operations-such MONet, DTTN', and R-PolyNets-yy lack the polynomial expansion structure can be classified as multilinear networks.
The main advantages of DTTN over Quantum-inspired TNs are twofold. (1) The alternative of the local map function. Existing TNs employ trigonometric functions for local mapping, which leads to the absence of higher-order (2 2) terms involving input elements in the network's 's unfolded form. This limitation results in a loss of feature self-interaction capabilities. (2) Higher bond dimension induced by parameter-sharing properties. Quantum-inspire TNs facilitate the parallel contraction of shared indices among N cores within the same layer. However, due to limited memory capacity, the bond dimension must often be restricted to smaller values. The following theorem establishes the equivalence between DTTNS and TNs:
DTTN& & Quantum-inspired TNs.
Theorem 1. Given the local mapping function o (xz) = [ ..," 1F polynomial network with the expansion form of Equation (6) can be transformed into a quantuuminspiin TNs model with finite bond dimension.
Since the internal cores TN can be decomposed into a "core-diagonal factor-core" structure via Higher-order Singular Value Decomposition (HOSVD) (2) and subsequently merged with connectivity cores, we regard the structural differences between DTTN and quantum-inspired TNs negligible. We believe the above theorem not only establishes equivalence between quantum- inspired TNs and modern architectures, but also offers insights for the future development more interpretable and perforaefor TN models.
4 Experiments
In this section, we provide comprehensive assessment of the DTTN's effectiveness. Specifically, in Section 4.1, we perform experiments on a series of image classification benchmarks to validate the model's 's superiority over other multilinear and TN architectures. In Section 4.2, we show the broader impact the DTTN across other domains, including recommendation system and partial differential equation (PDE) solving. Section 4.3 presents ablation studies aimed at examining the impact of various design choices. The paper concludes with a discussion the model's limitations. Further analysis and detailed results provided in the appendix.

---

Table ImageNet-IK, classification accuracy for various network architectures, Models that can be polynomially expanded are marked in red, whereas other multilinear models that do not include acti- vation functions incorporate layer normalization are marked in green. Our DTTNf-T outperforms the previous SOTA model MONet-T by 0.9% with fewer parameters and FLOPs.
Model
Top-1($)
Purams (M)
FLOPs()
Epoch
Activation
Atteation
Reso.
CNN-based
ResNet-50 50 200
77.2
25.0
4.1
ReLU
224
NNet [
77.0
33.4
31.3
ReLU
224
AA Ra-RReNNet Net-152 12112211
79.1
61.6
23.8
100
ReLU
224
RepVGG-B2g+
79.4
55.7
11.3
200
ReLU
X
224
Transformer- and Mamba-based
VIT-B/16 16
77.9
86.0
55.0
300
GeLU
224
DeiT-S/16 33
2244
Swin-T/16 36
81.3
29.0
4.5
300
GeLU
224
224
MLP bused
MLP-Mixer-BI16
76.4
MLP
71.8
507.0 .0
44.6
300
GeLU
221
CycleMLP-1
3
81.3
28.8
4.4
Hire-MI P-Tiny
19
79.8
18.0
2.1
300
Gel.U
224
ResMLP-24
79.4
GeLU
5"MI.-
59
80.0
71.0
14.0
300
GeL.U
224
80.7
10.5
ViP-Small/14
21
80.5
30.0
6.5
300
GeL.U
224
24
Polynomial- and Multilincar--en based
11-Nets [
65.2
12.3
1.9
224
DTTN-S(ours)
7188/7.2
12.3
4.1
90
300
22/2
Hybrid L1-Nets
70.7
11.9
1.9
ReLU+Tanh
X
224
PDC 8
71.0
10.7
1.6
100
ReLU+Tanh ReLU+Tanh
221
PDC-comp
S
70.2
7.5
1.3
100
224
R-PolyNets
10
70.2
12.3
1.9
120
221
D-PolyNets
70.0
11.3
1.9
120
224
MONet-T
77.0
10.3
2.8
300
221
DITNI-TTous))
77.9
7.1
2.3
300
224
DTTN' DTTN'-Sourrs
79.4
12.3
4.1
300
224
MONet-S
7
81.3
32.9
6.8
300
X
DITN'-L(Our))
82.4
221
4.1 Visual Recognition
Setup and Training Details. A series of benchmarks with different types, scales, and resolutions employed for the experiments, including CIFAR- 10 (30), Tiny ImageNet (31), ImageNet- 100 (58), ImageNet-INe 1K (46). MNIST, and Fashion-MNIST (56). A detailed description of the benchmarks and training configurations is included in the supplement.
Experimental valida- tion was conducted comparing the DTTN-S with quantum-inspiree TNs on MNIST and Fashion- MNIST datasets. The highest per- formances, marked bold, are achieved by DTTN-S, which outperforms previous tensor net- works, including aResMPS that utilizes residual connections and ReLU activation.
Table 4:
Table 3: Experimental validation of vari- network architectures was conducted on smaller benchmarks with differing resolu- tions. The tor-perforning results are high- lighted in bold. Among these, our DTTNf-S model achieves the best performance across all benchmarks, outperforming other polyno- mial and multilinear networks.
Tiny ImageNet
ImageNet-1000
Resulation
322
612
22
Resner18
94.4
61.5
85.6
MLP-Mixer
Model
Fashion-MNNST
IL.Nets
Res-MLP
90.7
92.3
50.2
48.9
$1.4
84.8
MPS Machine
0.9880
0.8970
Hybrid II-Nets
Bayesian
PIX
PLPS LoTeNet
0.833
D-PolyNets
94.7
61.8
86.2
M
94.8
61.5
87.2
aResMPS
0.9907
0.9142
DTINI-S
DTTN vs. Polynomial Networks. Table2r the results the unfoldable networks DTTN-S and I1-Nets 19 on ImageNet-1K (light red areas), using neither activation functions instance

---

PINN (12 eencr (0006)
PINN-DITNN (12 emer -0.003)
Table 5: Validating AIM as a pluggable mod- ule for enhancing feature interaction in rec- ommendation models with consistent perfor- mance gains.
Model
Criteo
Avazu
SA-PINN (12 ence 01044 SAPFNN-DTENIE eour -0.064)
DeepFM
10.12
75.46
DeepFM+AIM
80.44
75.73+0.27
FiBiNet
80,42
76.01
FiBiNet+AIM
80.97
76.08 +0.07
DCN-V2
80.93
76.14
DCN-V2+AIM
81.15
76.52+0.38
Figure Performance of PINNs on linear and nonlinear Allen-Cahn PDEs:
L2 error and absolute error across the Spatial-Tempont domain.
Table 6: The influence of network depth and width on model perfor- mance.
Table 7: The influence of differ- ent design choices for AIM on the performance of the DTTN
Table 8: The influence of layer normalization inside AIM on the performance of the DTTN variants.
variants.
Top-1 (%)
Params(M)
Model
Top-1 (%)
Params(M)
T=8, d=256
79.2
5.6
Top-1 (%)
Params(M)
DTTN.T
85.6
6.9
2=16,d=256
85.5
10.2
SIM-Comy
86.2
9.1
DTINI-T
86.4...
L=32,d=256
d=24,
87.2
86.8
19.4
14.8
SIM Linear
DTTN' -T
86.4
84.9
4.8
6.9
DTTN'-S
DTIN-S
87.740.
87.3
12.1
12.1
I==2,,=64
63.4
Sim-Conv
87.8
15.9
DTTN-L
87.6
35.6
L=32,d=128
82.5
4.9
Sim-Linear
85.2
8.3
DTTN'-L
88.1.05
L=32,d=512
87.9
76.8
DITN'-S 87.7
12.1
normalization. DTTN-S achieves Top-1 accuracy 71.8% and 77.2% after 90 and 300 training epochs, respectively, significantly outperforming II-Nets by 5.6% and 12%, respectively. Additionally, DTTN-S maintains a significant udvantage-neare II-Nets that incorporate activation functions (which lose their unfolding properties).
10% improvement in accuracy accuracy-over over Hybrid
DTTN vs. Multilinear Networks. Fig. 2[e) illustrates the performance of various multilinear networks trained ImageNet-IK IK over different epochs, with the curve for DTTNI-S showing superior results compared to others. Tables P and 3 detail the Top-1 accuracies of multilinear architectures across range of benchmarks of varying scales, including CIFAR- 10, Tiny ImageNet, ImageNet-100 100, and ImageNet-Iee IK. Specifically, DTTN-S achieves improvements of 0.2%, 2.3%, 0.5%, and 2.4% over the previous best models at similar scales. Moreover, the unfoldable DTTN-S attains impressive 77.2% accuracy on ImageNet- 1K without instance normalization. Additionally, DTTN' surpasses the prior state-of-the-ar model MONet 7 by 0.9%, while utilizing approximately 30% fewer parameters and reducing FLOPs by 20%. These comparisons demonstrate that DTTN achieves significantly faster convergence and superior performance.
DTTN vs. Quantum-inspired TNs. DTTN outperforms other quantum-inspired TNs on smaller benchmarks like MNIST and Fashion-MNIST. Notably, these models have not yet been success- fully scaled to large benchmarks. The basclines included for comparison are MPS [49]. Bayesian TNS, PEPS (6), LoTeNet [47). and aResMPS (38 The test results are reported in Tab. 4 where DTTN-S consistently achieves the best performance. Specifically, on the Fashion-MNIIS dataset, DTTN-S outperformed the second-best model, aResMPS, by 0.96%. This superior performance can be attributed to the higher bond dimension facilitated by parameter sharing and self-interaction capabilities, as discussed in Section 3.3.
DTTN vs. Advanced Architectures. We further demonstrate the competitive performance of DTTN relative to modern architectures, marking it as the first TN model to be applied to large- scale benchmarks. Table [ reports the Top-1 accuracy of the DTTN family compared to other models, including CNN-based, Transformer-basee Mamba-based, and MLP-based architectures

---

trained on ImageNet- IK. The DTTN'-L achieves an accuracy of 82.4% with 35.9M parameters, which is competitive and outperforms models such as DeiT-S/16 (53]. ViP-Small/14 (2. and Vim- S [62 Additionally, results in Tab. 3 further illustrate that DTTN achieves the highest accuracy on small benchmarks. The ViP-Small/14 21 with an MI. architecture and MONet exhibit slower convergence on ImageNet- 100 compared to ResNet-50 (see Appendix B), which is due to their inductive bias. In contrast, DTTN demonstrates remarkable training efficiency and superior accuracy relative to its counterparts.
4.2 Broader Impact
Recommendation System. Given the critical role of feature interaction in recommendation sys- tems [33], we select two widely-used datasets, Criteo and Avazu, to evaluate the effectiveness of AIM in enhancing feature interaction modeling. In our implementation, AIM is designed as a pluggable module and seamlessly integrated into existing Click-Through Rate (CTR) prediction models, includ- ing DeepFM, FiBiNet [23], and DCN-V2 V2 [55 In this setup, AIM replaced all linear layers in the target models, with internal convolution operations removed, leaving only linear and normalization layers. Experimental validation, reported in Tab. [5] and evaluated using AUC, shows that AIM consistently enhances performance across both datasets for all tested CTR models. Specifically, FiBiNet saw a 0.55% improvement on the Criteo dataset.
DaTN-based PDE Solver. Physics-Informed Neural Networks (PINNs) 14 incorporate physical constraints of systems into the training process, enabling the model to learn governing physical laws for describing system behaviors. The nonlinear activation functions serve as the critical component for PINNs to capture complex patterns. Here, we apply DTTN to solve specific partial differential equations (PDEs), including both the linear case and the highly nonlinear Allen-Cahn equation (defined u c + v², where e controls interface width and F(u) u² represents nonlinear reaction terms. The two equations to be solved are as follows:
au u
Ut 0.0001 Uzz + 520 5u 0
u(x, t 0t 0) = (in) u (1,1 (1)
u xx, t = 0) = x² (#:cos(TT) (Tz)
u u11,1) = u/ 11-1, -1
We employ PINN 14] and Self-Adaptive PINN (SA-PINN (50) with hard boundary conditions to address these problems. For comparison, we replace the linear layers with the AIM module to construct PINN-DTTN and SA-PINN-DTTN, which maintain identical configurations to their baselines but omit the Tanh nonlinearity. Numerical results, including the average L2-error and the absolute error of the prediction in the spatial-temporal domain, are presented in PINNs achieve lower prediction errors without activation nonlinearity.
Fig. 4 -bTN-
4.3 Ablation Study
Here, we conduct ablation studies to evaluate the effectiveness of various design choices, including network depth and width, the antisymmetric design of AIM, and layer normalization. We utilize ImageNet-100-22 representative subset of ImageNet- -K-as our test benchmark, which is well-suited for model validation and hyperparameter tuning under limited computational resources. Throughout these experiments, all hyperparameters are kept consistent with those of the final model, except for the variable under investigation. Each model was trained from scratch. We report the experimental results in Tables 6 7 and
Depth and Width. To control the depth and width of DTTN, we vary the number of AIM blocks and the hidden-size denoted as and d, for brevity. For ease of comparison, we kept d fixed across different stages. As illustrated in Tab. 6 we first set L to {8, 16, 24, 32} with d = 256, and subsequently varied d to {64, 128, 256, 512} while fixing L at 32. This approach enables us to explore how network depth and width affect model performance. Notably, performance declines significantly as both L and d are reduced. Specifically, Top-1 accuracy falls by 24.5% when is reduced from 512 to 64. Furthermore, the number of model parameters grows linearly with L and quadratically with d. These observations provide indirect evidence for the effectiveness of DTTN's multi-stage design.

---

Antisymmetrical Design of AIM. We evaluate the effectiveness of the antisymmetric design of AIM on the DTTNi-T and DTTN'-S architectures. As shown in Tab. 7 we compare the impact of symmetric versus antisymmetric designs on model performance. In this context, SIM-Conv and SIM-Linear refer to the Symmetric Interaction Module (SIM), which uses prioritized convolutional and linear layers, respectively, as alternatives to AIM. We observe that DTTNI-T achieves the best results among its counterparts, whereas DTTNÍ-S performs only slightly worse than SIM-Conv (by 0)1% while using approximately 20% fewer parameters. Furthermore, the reduction in the number of parameters aligns with the theoretical predictions corresponding to Equation 3
Importance of LN. Figure 3 d) and Table highlight the essential role of the Layer Normalization (LN) layer in enhancing the network's representational capacity key technique for improving the performance of multilinear networks [1O 77.
the improvements in DTTN variants after incorporating I.N, showing gains of 1.8% 0.4%, and 0.5%, respectively. Moreover, by comparing the results in Fig. [ and Tab. 2 we observe that DTTN-S achieves a Top-1 accuracy that is 5.6% and 2.2% lower than DTTNI-S after 90 and 300 training epochs, respectively.
Moreover, by combining the data from Figure e] and Tab. 2 it can be seen that DTTN-S exhibits Top-1 accuracy that is 5.6% and 2.2% lower than DTTN -S after training for 90 and 300 epochs, respectively. This further underscores the importance of LN in boosting the performance and convergence of our proposed architecture.
Limitations. Although we have conducted an extensive evaluation of DTTN's effectiveness and advantages, computational resource limitations prevented us from performing additional experiments. These include pre-training on larger datasets like JFT-300M and evaluating model robustness. Future research will focus a theoretical analysis of the proposed model, well as investigate multilinear transformer architectures tailored for large language models (LLMs).
5 CONCLUSION
This paper introduces multilinear network architecture named DTTN, which bridges the gap be- tween quantum-inspired TNs and advanced architectures, achieving this uniquely without activation nonlinearity. Specifically, DTTN employs a modular stacking design to capture exponential interac- tions among input features, essentially unfolding into tree-structured TN. We conducted extensive experiments showcase the effectiveness of DTTNs across various benchmarks. Notably, this is the first validation of TNs' effecctiveness on large-scale benchmarks, yielding competitive results compared advanced architectures. Additionally, explored the broader applicability of DTTN in other domains, such as recommendation systems and solving PDEs. Lastly, more theoretical discussions about DTTN will be addressed in future work.
Acknowledgment
We thank Yajie Chen and Junfang Chen for their valuable feedback and suggestions for improving this paper.
References
[1] Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoe V Le. Attention augmented convolutional networks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3286-3295, 2019.
[2] Zvika Brakerski, Craig Gentry, and Vinod Vaikuntanathan. (leveled) fully homomorphic encryption without bootstrapping. ACM Transactions on Computation Theory (TOCT), 6(3):1- 36, 2014.
[3] Shoufa Chen, Enze Xie, Chongjian Ge, Runjian Chen, Ding Liang, and Ping Luo. Cyclemlp: A mlp-like architecture for dense prediction. arXiv preprint arXiv:2107 10224, 2021.
[4] Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng Yan, and Jiashi Feng. A 2-nets: Double attention networks. Advances in neural information processing systems, 31, 2018.
10

---

[5] Song Cheng, Lei Wang, Tao Xiang, and Pan Zhang. Tree tensor networks for generative modeling. Physical Review B, (9(15)115133,, 2019.
[6] Song Cheng, Lei Wang, and Pan Zhang. Supervised learning with projected entangled pair states. Physical Review B, 103(12):125117, 2021.
[7] Yixin Cheng, Grigorios G Chrysos, Markos Georgopoulos, and Volkan Cevher. Multilinear operator networks, arXiv preprint arXiv:2401.1799) 2024.
[8] Grigorios G Chrysos, Markos Georgopoulos, Jiankang Deng, Jean Kossaifi, Yannis Panagakis, and Anima Anandkumar. Augmenting deep classifiers with polynomial neural networks. In European Conference on Computer Vision, pages 692-716. Springer, 2022.
[9] Grigorios G Chrysos, Stylianos Moschoglou, Giorgos Bouritsas, Jiankang Deng, Yannis Pana- gakis, and Stefanos Zafeiriou. Deep polynomial neural networks. IEEE transactions on pattern analysis and machine intelligence, 44(8))4021-4P0 -4344, 2021.
[10] Grigorios G Chrysos, Bohan Wang, Jiankang Deng, and Volkan Cevher. Regularization of polynomial networks for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16123-1611 2023.
[11] Andrzej Cichocki. Era of big data processing: A new approach via tensor networks and tensor decompositions. arXiv preprint Ttt:::14032044. 2014.
[12] Andrzej Cichocki, Namgil Lee, Ivan Oseledets, Anh-Huy Phan, Qibin Zhao, Danilo Mandic, et al. Tensor networks for dimensionality reduction and large-scale optimization: Part low-rank tensor decompositions. Foundations and Trends@ in Machine Learning, (4-5):2499-42 429, 2016.
[13] J Ignacio Cirac, David Perez-Garcie Norbert Schuch, and Frank Verstraete. Matrix product states and projected entangled pair states: Concepts, symmetries, theorems. Reviews of Modern Physics, 93(4):045003, 2021.
[14] Salvatore Cuomo, Vincenzo Schiano Di Cola, Fabio Giampaolo, Gianluigi Rozza, Maziar Raissi, and Francesco Piccialli. Scientific machine learning through physics-infonee neural networks:
Where we are and what's next. Journal of Scientific Computing, 92(3):88, 2022.
[15] Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13732-13742, 2021.
[16] Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1929, 2020.
[17] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010.
[18) Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint 1rXiv:2312.007555 2023.
[19] Jianyuan Guo, Yehui Tang, Kai Han, Xinghao Chen, Han Wu, Chao Xu, Chang Xu, and Yunhe Wang. Hire-mlp: Vision mlp via hierarchical rearrangement. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 826-8366 836, 2022.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[21] Qibin Hou, Zihang Jiang, Li Yuan, Ming-Ming Cheng, Shuicheng Yan, and Jiashi Feng. Vision permutator: A permutable mlp-like architecture for visual recognition. IEEE transactions on pattern analysis and machine intelligence, 45(1):1328-1333 2022.
[22] Qibin Hou, Cheng-Ze Lu, Ming-Ming Cheng, and Jiashi Feng. Conv2former: A simple transformer-style convnet for visual recognition. IEFE Transactions on Puttern Analysis and Machine Intelligence, 2024.

---

[23] Tongwen Huang, Zhiqi Zhang, and Junlin Zhang, Fibinet: combining feature importance and bilinear feature interaction for click- click-through rate prediction. In Proceedings of the 13th ACM conference on recommender systems, pages 169-177, 2019.
[24] Zhipeng Huang, Zhizheng Zhang, Cuiling Lan, Zheng-Jun Jheng-Junnooe Zha, Yan Lu, and Baining Guo. Adaptive frequency filters as efficient global token mixers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6049-6059, 2023.
[25] William Huggins, Piyush Patil, Bradley Mitchell, K Birgitta Whaley, and E Miles Stoudenmire. Towards quantum machine learning with tensor networks. Quantum Science and technology. 4(2):024001, 2019.
[26] Daniel Jaschke, Simone Montangero, and Lincoln D Carr. One-dimensional many-body entan- gled open quantum systems with tensor network methods. Quantumn science and technology. 4(1):013001, 2018
[27] Hong-Chen Jiang, Zheng- Yu Weng, and Tao Xiang. Accurate determination of tensor network state of quantum lattice models in two dimensions. Physical review letters, (01(9)00,003,, 2008.
[28] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid net- works. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6399-6408, 2019.
[29] Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review, 5155)5555-000 2009.
[30] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[31] Yann Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3 2015.
[32] Heng-Chao Li, Zhi-Xin Lin, Tian- Yu Ma, Xi-Le Zhao, Antonio Plaza, and William J Emery. Hybrid fully connected tensorized compression network for hyperspectral image classification. IEEE Transactions on Geoscience and Remole Sensing, 61:1-16, 2023.
[33] Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun. xdeepfm: Combining explicit and implicit feature interactions for recommender systems. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1754-1763, 2018
[34] Yingyu Liang, Zhenmei Shi, Zhao Song, and Yufa Zhou. Tensor attention training: Provably efficient learning of higher-order transformers. arXiv preprint arXiv:2405.16411, 2024.
[35] Ding Liu, Shi-Ju Ran, Peter Wittek, Cheng Peng, Raul Blázquez García, Gang Su, and Maciej Lewenstein. Machine learning by unitary tensor network of hierarchical tree structure. New Journal of Physics, 21(7):073059, 2019.
[36] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[37] I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
[38] Ye-Ming Meng, Jing Zhang, Peng Zhang, Chao Gao, and Shi-Ju Ran. Residual matrix product state for machine learning. SciPost Physics, 14(6):142 142, 2023.
[39] Chang Nie, Huan Wang, and Lu Zhao. Stn: Scalable tensorizing networks via structure-aware training and adaptive compression. arXiv preprint 2rXiv:2205./5198, 2022.
[40] Chang Nie, Huan Wang, and Lu Zhao, Adaptive tensor networks decomposition for high-order tensor recovery and compression. Information Sciences, 629:667-684, 2023.
[41] Ivan V Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing,
33(5):2295-23177 2011.
12

---

[42] Siddhartha Patra, Saeed S Jahromi, Sukhbinder Singh, and Román Orús. Efficient tensor network simulation of ibm's largest quantum processors. Physical Review Research, 6(1):013326, 2024.
[43] Shi-Ju Ran and Gang Su. Tensor networks for interpretable and efficient quantum-inspired machine learning. Intelligent Computing, 2:0061, 2023.
[44] Shi-Ju Ran, Emanuele Tirrito, Cheng Peng, Xi Chen, Luca Tagliacozzo, Gang Su, and Maciej Lewenstein. Tensor network contractions: methods and applications quantum many-body systems. Springer Nature, 2020.
[45] Justin Reyes and Miles Stoudenmire. A multi-scale tensor network architecture for classification and regression. arXiv preprint wrXiv:2001.082660 2020.
[46] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-22 2522, 2015.
[47] Raghavendra Selvan and Erik B Dam. Tensor networks for medical image classification. In Medical imaging with deep learning, pages 721-732. PMLR, 2020.
[48] Raghavendra Selvan, Erik Dam, Søren Alexander Flensborg, and Jens Petersen. Patch- based medical image segmentation using matrix product state tensor networks. arXiv preprint arXiv:2109.071386 2021.
[49] Edwin Stoudemmire and David J Schwab. Supervised learning with tensor networks. Advances in neurul information processing systems, 29, 2016.
[50] Shashank Subramanian, Robert M Kirby, Michael W Mahoney, and Amir Gholami. Adaptive sule-supevviioon algorithms for physics-informed neural networks. In ECAI 2023, pages 2234- 2241. IOS Press, 2023.
[51] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp- mixer: An all-mlp architecture for vision. Advances in neural information processing systems, 34:24261-244 24272, 2021.
[52] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al. Resmlp: Feedforward networks for image classification with data-efficient training. IEEE transactions on pattern analysis and machine intelligence, 45(4):5314 121311--11,,, 2022.
[53] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
[54] A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
[55] Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. Den v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. In Proceedings of the web conference 2021, pages 1785-1797, 2021.
[56] H Xiao. Fashion-mmin arXiv preprint arXiv: (00::/07887777 2017.
novel image dataset for benchmarking machine learning algorithms.
[57] Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normalization. Advances in neural information processing systems, 32, 2019.
[58] Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In Proceedings of the IÉEE/CVF conference on computer vision and pattern recognition, pages 3014-3023, 2021.
[59] Tan Yu, Xu Li, Yunfeng Cai, Mingming Sun, and Ping Li. S2-mlp: Spatial-shift mlp architecture for vision. In Proceedings of the IEFF/CVF winter conference on applications of computer vision, pages 297-306, 2022.
13

---

[60] Hui-Hai Zhao, Zhi- Yuan Xie, Qihong N Chen, Zhong-Char Chas Wei, Jianwei W Cai, and Tao Xiang. Renormalization of tensor-network states. Physical Review B Condensed Matter and Materials Physics, 81(17)1174411, 2010.
[61] Qibin Zhao, Guoxu Zhou, Shengli Xie, Liqing Zhang, and Andrzej Cichocki. Tensor ring decomposition. arXiv preprint W1Xi:16006005555, 2016.
[62] Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprinl arXiv:2401.09417 2024.
14

---

Appendix
A Experimental Setup
A.1 Image Classification Benchmarks and Training Settings
A detailed description of the benchmarks and training configurations is provided below.
CIFAR-10 [0] consists of 60K color images of 32 32 resolution across 10 classes, with 50K images for training and 10K for testing. We train our model using the SGD optimizer with a batch size of 128 for 160 epochs. The MultiStepl. strategy applied to adjust the learning rate, and data augmentation settings are in accordance with [TO].
Tiny ImageNet [31], ImageNet-100 [58], and ImageNet- IK [46] contain 100k, 100k, and 1.2M color-annotated training images with resolutions of 64 x 64. 224x224, 224 224 pixels. respectively. These datasets serve as standard benchmarks for evaluating image recognition models, During training, we optimized our model using configuration consistent with
7 51 Specifically, we utilized the A optimizer [37] alongside cosine decay schedule for learning rate tuning, and applied data augmentations including label smoothing,
Cut-Mix, Mix-Up, and Autonuument. Note that we did not employ additional large-scale datasets such as JFT-300M for pre-training, nor did we use unsupervised or cupe-super methods to optimize our model. All experiments were conducted on 8 GeForce RTX 3090
GPUs using native PyTorch.
MNIST and REshion-MNIST [56] both contain 60,000 grayscale images for training and 10,000 for validation, with each image sized at 28 x 28. We conducted comparison experi- ments between DTTN and other TN models on these two benchmarks, employing training configurations consistent with those used for CIFAR-10.
A.2 Hyperparameters for Training on ImageNet-1K
Table shows the experimental hyperparameters used for training the DTTN family on the ImageNet- 1K benchmark. We utilize the timm library and ensure all settings are aligned with the comparison method MONet 77.
More Experimental Results
Image Segmentation. We employ the Semantic FPN framework [28] for performing semantic segmentation on the ADE20K dataset, which includes 20,000 training images and 2,000 validation images. DTTN'-S and DTTNN-L were initialized using pre-trained ImageNet- 1 K weights before being integrated as the backbone of the framework. Additionally, all newly added layer weights were initialized using Xavier initialization
presents the outcomes of the DTTN model trained for 12 epochs using the Adam optimizer, evaluated by Mean Intersection over Union (mloU). Some experimental data are derived from 77 Notably, DTTN'-L achieves better performance over other multilinear networks, underscoring the efficacy of our designed AIM.
[17]. Table |
Training Convergence. Figure 5 displays the Top-1 accuracy and loss curves of the proposed model alongside other architectures -such as ResNet-50 [20]. ViP-Small (2I] representing the MLP architecture, and MONet 7 all trained on ImageNet- 100 for 90 epochs. It can be observed that DTTN achieves significantly faster convergence and superior performance.
Motivation and Interpretability
The principal motivation of this work is to uncover both the potential and inherent limitations of cruntum-inspinee tensor networks (TNs) on large-scale benchmarks, by establishing their equivalence with DTTN through Theorem 1. Although TNs have been extensively studied in the fields of quantum mechanics and white-box machine learning, state-of-the-ar models are still largely restricted small datasets and limited tasks. DTTN overcomes this barrier by extending Tensor Network Networks
http://s///ttaacc/o//////aaaaaar Operator Networks
15

---

BackBone
mloU(%)
Resnet18
32.9
FCN
29.3
Top- Acc (%6
Seg-Fooe neg-Former
37.4
ResNet-S0 (85.1 Accl ViP-Small
Evol Evol Lozs
R-PDC
20.7
MONet (84.5 Accl
R-PolyNets
19.9
DTTN1-5 -5 (86.1 Acc!
MONer-S
37.5
DTTNi-S 36.9
DTTNi-L 38.6
Figure 5: Top-1 1 accuracy and loss visualiza- tion for architectures trained from scratch on ImageNet- 100.
Table 9: Experimental valida- tion of semantic segmentation ADK20K with Semantic FPN.
(TNNs) to large-scale applications through techniques such as parameter sharing and exponential interaction modeling, thereby achieving competitive performance.
Furthermore, DITNs retain high degree of interpretability that aligns with -rantruminnspied TNs,
offering intuitive deterministic lincar representations and probabilistic interpretations. For deeper understanding of the theoretical underpinnings, readers are referred to related works [43 44).
Table 10: Comparison of inference latency, throughput, and memory usage on an NVIDIA
A6000 GPU.
Model
Batch Size
Latency (ms/batch)
Throughput (sample/sec)
Peak Memory (GB)
MONet_P T
64
128.39
498.47
1.14
VIP 1mall111 (ours)
86.1
743.28
0.584
DTTN-S (ours)
64
81.94
781.08
1.099
D Latency Analysis
Table TO
presents detailed comparison of inference latency and memory usage, benchmarked an NVIDIA A6000 GPU. Compared to the previous state-of-the-arr model, MONet-T, our DTTN-S reduces latency by nearly 35% while also achieving a 2.4% performance gain on ImageNet (see Table 1). Furthermore, DTTN demonstrates faster convergence on both the ImageNet and ImageNet-100 benchmarks, as shown by the convergence curves in Fig. 2e and Fig. 5.
Table 11: Training settings for ImageNet- in Section 4.1
Item
Setting
Optimizer
AdamW
Base learning rate
1e-3
Wa
le-6
Learning rate schedule
cosine
Weight Decay
0.01 & 0.02
Batch size
320 x GPU
Label smoothing
0.1
Auto augmentation
Random erase
0.1
Cutmix
0.5
Mixup
0.5
Dropout
0.0
16

---

PINNADTIN
SA-PIN
Figure 6: Visualization of training loss and prediction results of PINNs solving PDEs across the spatial-temporal domain.
Details DTTN Resolving PDEs
In this study, use the Physics-Ifformed Neural Network (PINN) [4] and the Self-Adaptive PINN (SA-PINN) [50) as our baseline models. These are multi-layer feedforward networks with a hidden dimension of 32. Each linear layer is succeeded by tanh activation functior Subsequently, replaced these linear layers with AIM modules while removing the activation functions, yielding the novel architectures PINN-DTTN and SA-PINN-DTTN, These enhanced networks then applied to solve both the linear PDE and the nonlinear Allen-Cahn equation presented in Equation (8). As noted in reference [50], the Allen-Cahn PDE serves intriguing benchmark for testing PINNs, requiring the network to approximate solutions with sharp transitions in space and time, along with periodic boundary conditions.
Fig. O illustrates the prediction results and absolute errors for various PINN variants addressing the two equations across the spatial-temporal domain Ω T + [-1, 1] [0, 1]. For the first equation, which has an analytical solution of u e sin(xx), the PINN-DTTN exhibits lower errors both training and prediction. However, for the second equation, despite the SA-PINN-DTT CINN- achieving lower errors, there are noticeable spikes training loss. This phenomenon may be due to instability when fitting complex functions with simple polynomials.
F Implementation and Motivation of the AIM Design
In Algorithm[] [1 we provide PyTorch-style implementation of AIM. The main difference between training and inference lies in structural re-parameterizatton. Specifically, during training, BN layers are integrated with adjacent convolutional or linear layers. During inference, the AIM propagation process simplifies, matching the schematic shown in Fig. d))
The primary motivation for designing DTTN was develop a computationally efficient model, based solely on tensor operations, that is capable of scaling to large benchmarks. The core of our solution is the AIM, which designed facilitate direct multiplicative interactions. A structure emerges as natural consequence of this design: since each AIM module performs binary fusion of two feature tensors into one, stacking these modules hierarchically inherently constructs tree. Significantly, this straightforward, hierarchical construction is provably equivalent to powerful Tree Tensor Network
17

---

Algorithm Code for AIM (PyTorch-like)
H: height, width, C: channel, R: expansion ratio
xx input tensor shape (B, C, H, W)
ІНБЯИАИА Initialization нААнААеенеесс
proj_1 mn. Linear (C, C + R) # Aggregate channel information
conv_1 nn.Conv2d (C R, C R, kernel_size-3, groups-C R) Aggregate
spatial information
proj_r nn. (C R
R) Aggregate channel information
conv Cn. * R, kernel_sized groups=C) # Aggregate spatial
information
proj Conv2d(C
kernel size=1) For information fusion
In_norm nn Lay [C Rl) # Layer normalization
10 1_orm nn. Bt.BacchNorm2d(C R) A Batch normalizatin
11 r_norm nn mm.BacchNorm2d(C
R) À Batch normalization
12 res_norm norm = nn.BatchNorm2dd00 (C) # Batch normalization
scale am.aaaaeterr (torch. werorrch.oee((0 (1))
14 #*## Training stage **ААнАенесе*сE
15 def AIM (x, use_in)
16
x_1 conc (erv_](proj__ pwwpetereeeeee (0, 2, 3, 1) perputete (0, 3, 1, 2
x [ (x) wwr.ute (0, 2, 3, 11 permute (0, 3, 2)
18
if use_In:
19
out In_norm x_r)
20
else:
out out = 1 nxrmommxx 1) norm(x_F (prn(proj(c)) (out))
23
return + scale * out
24 чнрهААААААА Inference stage AAGAABEСUACCCR
25 def (x, use_ln):
x_1 conv conv_l(proj___ perperuut (0, 2, 3, 1) .eremmtte (0, 3, 1, 2)1] x_r proj_riconv_r (x) permute (0, 2, 3, 1p)-permute (0, 2)
28
out x_l*xx x_r x_r
29
if use_in:
30
out [n_norm( return + proj (out)
(TTN), bypassing the complex methods that have historically limited their application (5) Thus, while the tree topology was not an initial design constraint, it is fundamental to our model's expressive power and theoretical grounding.
Training Stability of DTTN. The training of traditional TNs is often hindered by well well-known challenges, such as gradient instability (i.e., exploding or vanishing gradients) and high sensitivity to hyperparameters. In contrast, DTTN model trains stably across all scenarios, showing signs these issues, as evidenced by the smooth loss curves in Fig. 5 This stability provides a significant advantage over traditional TNs, where long chains of sequential tensor contractions can lead to optimization problems. Although DTTN is theoretically equivalent a TTN, its hierarchical optimization process mirrors that of standard deep network, ensuring training dynamic.
robust and stable
Complexity Analysis
We now analyze the complexity of DTTN in terms of its storage requirements and computational cost. Without loss of generality, we consider the multi-stage DTTN with hidden-size d for each stage, and the total depth is L. We then consider the complexity of the main components of the network. including the local mapping, the core blocks, and the classification head separately.
In this section, we analyze the complexity of DTTN, focusing on both storage requirements and computational cost. Without loss of generality, we consider multi-stage DTTN where each stage has a hidden size of d, and the overall depth is L. We then examine the complexity associated with the main components of the network: the local mapping function, the core blocks, and the classification head.
18

---

Local Mapping. The local mapping function o(-) consists of two consecutive convolutional layers, each with a kernel size and stride of This function is applied the imput image z produce a feature map Φ(x, Aω) E RW*HxC, Here, Ag E RS* represents a learnable matrix, where S,C d E N denote the local patch size and the number of output channels, respectively. The parameters and hoating-point operations (FLOPs) involved this process are given by:
Params O(14
dd), FLOPs* (48 d WH W.H.
(8)
WH).
Core Blocks. we analyze the complexity associated with the core AIM blocks within the DTTN architecture. Assuming that each of the four stages contains equal number of blocks, the corre- sponding feature map sizes are W x H, 2 x H W. X 4 focuses on the parameters and FLOPs involved in the three linear layers and two convolutional layers of AIM. The total parameters and FLOPss can be expressed as follows:
H and W x 8 respectively. Our analysis
22 rary d+ (2 Fexp + tra raan de d) 1. 1)
Params ALM =0
=0 ((22 Fany + 2. (Pep reag)) d 1) d.d. 1):
(9)
WH
WH.
L
FLOP& ATM
=0
18 Texp d.
Texp + r de
4"
4'
765
85
(Fezp raa)) di WH d.H
128
Tap
128
Classification Head, For classification head with m classes, we receive a feature map of size x H which is then processed through an average pooling layer followed by a fully-connected layer to output an m-dimensional vector. The parameters and FLOOPs involved in this process are given by:
W. 8 8
Params" H =Odd (m + 1)),
FLOPs = = WH
(10)
d+ m d).
64
In summary, conclusions regarding the number of parameters and FLOPs for the DTTN architec- ture can be expressed as:
Params Params' + Params ATM Params".
(1I)
PLOPs = FLOP FLOPSATM + FLOPP
It can be observed that the number of parameters in the DTTN architecture remains fixed, while the computational complexity exhibits linear relationship with the input image scale. This characteristic represents significant advantage of the DTTN over modern MLP and Transformer architectures, which typically exhibit quadratic complexity.
H Proofs
Here, we derive the proofs of Proposition and Theorem from the main paper and further elucidate their significance.
The DTTN has the capability to capture - multiplicative interactions among input ele- ments, which can be represented in the format of Equation 1 as ((x)) 8² o(x,, Aa). Consequently; the elements of (xx are homogeneous polynomials of degree 2 over the feature map 6(xr Ag).
Proposition.
Proof. For each AIM block with input x' vec(22') RWHHIC, let DJ = W/ H1 C₁ and I Kz, Suppose that the left and right branches of AIM be represented as f(('') A and f3(a))) Alar', where A Al. D''DD are obtained through structured combinations of convolutional and linear layer weights. The feedforward propagation of the AIM block can then be
19

---

expressed as:
x 1+ = x' + B' ((A/2') (A2a')
x' Reshape (Ai ④ 4/2
12 (a' 2o)
(12)
2,3
1.2 x z'
(x' 2 x²).
2,3
where B' denotes the fused linear layer inside AIM, and Z' = Reshape (A @ AT) D'x x x
is a structured learnable tensor. Then each element x can calculated by
D'
D' z
(13)
Thus, each AIM module captures second-order multiplicative feature interactions of the input. By induction, the DTTN stacked with L AIMs captures 2 interactions of the input Note that the network's bias terms and shortcut connections can be eliminated by introducing an additional homo-
geneous dimension in the local mapping. Hence, we have a => Ep (( The D Therefore, the expression (a) a homogeneous polynomial of degree 2 of 6(x A which concludes our proof.
Theorem. Given the local mapping function 6f' = the expansion form of Equation (6) can be transformed into quantuan-nnspired TN model with finite bond dimension.
[=
2 , a polynomial network with
Proof. For an input image E W the computation the vanilla quantum-inspired TN model can be expressed as
f(x) TN (I).))))))))))) (R.)
(14)
where N WHC, E Kw,p E Kn E Kg, and (R, { denotes the tensor cores. TN(+) (2 + 1) ..)........ (2 +1
R
Rm represents the contraction operation that outputs an m-dimensional vector. We can further complete the contraction of 24 physical indices and transform Equation 14 into
f(x) = TN (o' (x)).) x R1, ... (I(T))))) x RN)
(15)
(2 Bx).
Since each output element of polynomial network can be expressed
wesi Fig .......-.. Digh: E 2
(16)
The above equation is equivalent to
Σ
wea An
where k; 2 0 and h .... AN = 2
(17)
Since Equation 15 encompasses each term of Equation 17 this proves claim.
The above proof indicates that by selecting an appropriate local mapping, the DTTN can be trans- formed into standard tensor network model. We re-emphasize that this is the first work demonstrat- ing that tensor networks can achieve competitive performance on large-scale benchmarks, such as attaining 77. 77.2% Top-1 accuracy ImageNet- IK. Previous TNs have been limited to much smaller tasks due to expression limitations and lower bond dimensions. We hope that this research will inspire further explorations into tensor networks.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions and scope of this paper are summarized in the abstract and detailed in the introduction. Specifically, the scope of the study is outlined in the first two paragraphs of the introduction, while the contributions highlighted in the final paragraph.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this work the last paragraph of Section 4.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer: [Yes]
Justification: Detailed proofs of the propositions and theorems presented in this paper are provided in Appendix H.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We describe the details of the experiments in the experimental section and the Appendix. We plan to make the code publicly available to ease the reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
22

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: All experimental datasets used are publicly available. The pseudocode for the AIM module provided in Appendix
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
pub public//a gwtp///uueeeee/eeg / Coweeuuoeeiieeeiieeoeoeoo www.iiisssopooooooooooooooooocee
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips cc/ httpssscccppbbcccccpccccccc crabliccguidee Ttdes//ooeSusii SeooeSuuuei.... Suwmissiomssin...........si..i.e onP ciooooiopiiyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: provide detailed experimental settings, including data split, optimizer configurations, and other parameters, in Section 4 (Experimental Section) of the main paper. Additional implementation details are provided in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Given the substantial computational resources and time required, statistical analysis is not conducted.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We describe the software and hardware platforms for the experiments in the Experiments section, with specific resource consumption consistent with regular deep model training.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have made sure that our paper conforms with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the positive impacts of our approach the Introduction and Experimental sections.
Guidelines:

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There no risk of misuse of the proposed method and the datasets used the paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the original paper or attached the link to the existing assets used in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
25

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papelpappssss wwwwhhocee com/ / /ataaaaasssssee has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy (https: //pp::////urripss httpr//ps...ccoooceeeeeee////22
LLM) for what should or should not be described.
27


Reviews:
[{"review_id": "tzxuiQDU6r", "reviewer": "Reviewer_bpqs", "summary": "The paper introduces a novel architecture called Deep Tree Tensor Network (DTTN), which leverages multilinear operations to capture exponential-order feature interactions without activation functions. DTTN is constructed by stacking antisymmetric interaction modules (AIMs) and unfolds into a tree-like tensor network topology. The authors demonstrate its superior performance on various benchmarks, including ImageNet-1K, and explore its broader applicability in recommendation systems and PDE solving. Theoretical analysis establishes connections between DTTN and quantum-inspired tensor networks (TNs), polynomial networks, and multilinear networks.", "questions": "1. **Figure 1 and Feature Interaction** :\n\n* Can the left-side MPS model in Figure 1 also capture 2^L-order multiplicative interactions? What are the key differences in how the left and right models capture feature maps?\n* What does the \"projection\" in the right-side figure refer to, and how does it compare to the operations in the left-side model? What advantages does this projection offer?\n\n2. **AIM Design Intuition** :\n\n* The AIM incorporates a depthwise convolution layer and a linear layer in reversed order. What is the intuition behind this antisymmetric design? Does it improve efficiency, expressiveness, or both? A more detailed explanation would strengthen the paper.\n\n3. **Performance Comparisons** :\n\n* Why is MONet-L's performance missing from Table 2? How does DTTN-L compare to MONet-L in terms of accuracy, parameters, and FLOPs?\n* Tables 3 and 4 only include DTTN-S. Are DTTN-T and DTTN-L unsuitable for these tasks, or were they simply not tested? If so, please provide an analysis of their limitations or advantages in such scenarios.\n\n4. **AIM in MPS Architecture** :\n\n* How would AIM blocks perform if integrated into an MPS architecture instead of a tree structure? Would the linear topology of MPS limit the model's ability to capture interactions compared to DTTN's tree structure? An ablation study or discussion on this would be valuable.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. **Innovative Architecture** : The proposed DTTN effectively bridges the gap between quantum-inspired TNs and modern architectures, achieving competitive performance on large-scale benchmarks.\n2. **Theoretical Rigor** : The paper provides a clear theoretical foundation, including proofs of DTTN's capability to capture high-order interactions and its equivalence to other network types under specific conditions.\n3. **Comprehensive Evaluation** : Extensive experiments across diverse tasks (visual recognition, recommendation systems, PDE solving) validate the model's versatility and superiority over existing methods.\n\n### Weaknesses\n\n1. **Limited Comparison with MONet-L** : The paper does not report the performance of MONet-L in Table 2, leaving a gap in understanding how DTTN compares to larger variants of existing multilinear networks.\n2. **Ablation on MPS Architecture** : The potential performance of AIM blocks in an MPS architecture is not explored, which could provide insights into the advantages of tree structures over linear topologies.\n3. **Experimental Scope** : Tables 3 and 4 only report results for DTTN-S, omitting DTTN-T and DTTN-L, which raises questions about the suitability of these variants.", "was_revised": true, "final_justification": "After carefully evaluating the authors’ detailed rebuttal, comments with other reviewers, and feedback from the Area Chair (AC), I provide the following justification for my revised score:\nResolved Issues\n1. Distinction of DTTN vs. MPS:\nThe authors clarified that the core innovation lies in computational scalability and expressive capacity. They demonstrated that DTTN avoids the prohibitive computational cost of MPS, where the number of contractions scales linearly with input size (W×H) and complexity depends on bond dimension D. This fundamentally underscores the paper’s contribution relative to MPS.\n2. AIM Design Rationale:\nThe antisymmetric architecture was rigorously justified through ablation studies (Table 7), showing 25% fewer parameters/FLOPs than symmetric variants without performance degradation. Comparisons with MONet’s branching ratios further validated the design efficiency.\n3. Missing Benchmarks & Comparisons:\nThe addition of results for DTTN-T/L (Tables 3–4) and latency comparisons (35% faster than MONet-T) addressed concerns about evaluation completeness. Clarifying that MONet-L does not exist resolved a key point of confusion. Experimental rigor now meets acceptance standards.\nUnresolved Issues\nAs noted by other reviewers, the motivation behind AIM still requires clearer exposition in the revision to aid reader comprehension. Additionally, concerns persist regarding the paper’s treatment of well-known training challenges in quantum-inspired tensor networks:\n• (i) Gradient instability (exploding/vanishing weights),\n• (ii) Sensitive weight initialization,\n• (iii) Difficult learning rate tuning.\nThe authors responded that their architecture inherently avoids gradient instability through Xavier initialization and by adopting MONet’s training strategy. However, the core issue remains whether DTTN resolves these challenges (not just mitigates them via standard methods). Finally, while the integration of tensor networks with quantum-inspired learning represents a valuable direction, the paper should more explicitly articulate how this connection is established and what specific advantages it offers over classical approaches.\nFinal Recommendation\n• Score Adjustment: Borderline Accept (raised from Borderline Reject). The rebuttal resolved critical concerns regarding scalability, efficiency, and interpretability.\n• To the Authors: This work represents a meaningful advance in making tensor networks viable for quantum machine learning. The revisions have elevated it to acceptance standards.", "rebuttal": "Dear reviewer bpqs,\n\nWe sincerely thank you for your detailed review and for acknowledging the strengths of our work, including its Innovative Architecture, Theoretical Rigor, and Comprehensive Evaluation. Your insightful questions help us clarify critical aspects of our method. We address each point below.\n\n> Q1: Can the left-side MPS model in Figure 1 also capture $2^L$-order multiplicative interactions? What are the key differences in how the left and right models capture feature maps?\n\nA1: Yes, according to Theorem 1, the MPS model can also capture $2^L$-order multiplicative interactions if an appropriate polynomial mapping function is used. . However, the fundamental difference lies in computational scalability and expressive capacity.The traditional MPS model must contract a core factor for every single input feature (e.g., each pixel), leading to a prohibitive computational cost where the number of contractions scales linearly with the input size ($W \\times H$), and the complexity of each contraction depends on the bond dimension $D$. This is the primary reason MPS models have failed to scale to large-scale benchmarks.\nIn contrast, our DTTN offers two decisive advantages:\n1.  Decreased Number of Contractions: The number of contractions in DTTN depends only on the network's depth $L$, not the input size ($W \\times H$), making it vastly more efficient.\n2.  Expanded Effective Bond Dimensions: We are not constrained by a small, explicit bond dimension (often $\\leq$64 in prior work like Lotenet). Instead, our AIM module implicitly expands the effective bond dimension to the order of $W \\times H \\times C$ (almost $10^5\\sim 10^6$), allowing it to capture far more complex correlations without incurring impossible memory costs.\nThese two advantages are the key differentiators that enable DTTN to succeed where traditional MPS models could not.\n\n> Q2: What does the \"projection\" in the right-side figure refer to, and how does it compare to the operations in the left-side model? What advantages does this projection offer?\n\nA2: The \"projection\" in the DTTN model refers to a linear MLP layer. This design choice offers two significant advantages over the local mapping function (typically a trigonometric function) used in the traditional MPS model: (1) The MPS's local map processes one pixel at a time, isolating it from its spatial context. Our initial projection layer takes the entire image feature map as input, preserving spatial locality. (2) Enabling Higher Bond Dimensions. This projection-based design is the key that allows us to implicitly achieve the massive bond dimensions mentioned earlier, enabling $2^L$-order interactions where the number of sequential contractions scales with $\\mathcal{O}(L)$, a dramatic improvement over the $\\mathcal{O}(W \\times H)$ scaling of MPS.\n\n> Q3: The AIM incorporates a depthwise convolution layer and a linear layer in reversed order. What is the intuition behind this antisymmetric design? Does it improve efficiency, expressiveness, or both? A more detailed explanation would strengthen the paper.\n\nA3: The antisymmetric design of AIM is a deliberate choice for improving computational efficiency without sacrificing expressive power. As shown in Eq. 3, the computational and parameter costs of the two branches are asymmetric (roughly a 1:3 ratio).\nWe provide a detailed ablation study on this in Appendix D (Table 7). We compared AIM with a Symmetric Interaction Module (SIM), where both branches have identical structures. The results show that our antisymmetric AIM design reduces parameters and FLOPs by approximately 25% compared to SIM, while achieving the same level of performance. This confirms that the design is a more efficient parameterization of the same core functionality.\n\n> Q4: Why is MONet-L's performance missing from Table 2? How does DTTN-L compare to MONet-L in terms of accuracy, parameters, and FLOPs?\n\nA4: Thank you for pointing this out. The original MONet paper[1] only proposed and benchmarked MONet-T and MONet-S architectures; a \"MONet-L\" variant was not provided. Therefore, we could not include it for a direct comparison in Table 2. However, to demonstrate DTTN's superior efficiency, we note that DTTN-T achieves a 0.9% accuracy gain over MONet-T while using 30% fewer parameters and 18% fewer FLOPs. Furthermore, as detailed in our response to Reviewer QNHq, our DTTN-S reduces inference latency by nearly 35% compared to MONet-T (81.9ms vs. 128.4ms on an A6000 GPU).\n\n> Q5: Tables 3 and 4 only include DTTN†-S. Are DTTN-T and DTTN-L unsuitable for these tasks, or were they simply not tested? If so, please provide an analysis of their limitations or advantages in such scenarios.\n\nA5: We apologize for this omission. We provide the results for DTTN-T and DTTN-L on the benchmarks below. As you can see, all variants exhibit stable and strong performance on these benchmarks, confirming their general applicability.\n\n| Model | CIFAR-10 | Tiny ImageNet |ImageNet-100 |  Fashion-MNIST |\n|:---|---:|---:|---:|---:|\n| DTTN†-T | 94.5 | 62.5 | 87.5 | 0.917 |\n| DTTN†-S | 95.0 | 63.8 | 87.7 | 0.924 |\n| DTTN†-L | 94.8 | 63.9 | 87.9 | 0.924 |\n\nWe hope this resolves your concern.\n\n> Q6: How would AIM blocks perform if integrated into an MPS architecture instead of a tree structure? Would the linear topology of MPS limit the model's ability to capture interactions compared to DTTN's tree structure? An ablation study or discussion on this would be valuable.\n\nA6: This is a very insightful architectural question. We did not integrate AIM into an MPS for two primary reasons: (1) Fundamental architectural mismatch. AIM is designed to process two identical input tensors. The linear topology of an MPS, however, operates on two different inputs at each step: the result of the previous contraction and the core factor of the next feature. AIM is therefore not directly pluggable into the MPS operational flow. (2) Theoretical equivalence. While MPS and DTTN have different apparent topologies (line vs. tree), they are theoretically interconvertible [2]. The performance limitations of prior TN models, as shown in our Table 4 (which includes MPS, PEPS, and LoTeNet[3,4,5]), stem not from their specific topology, but from their computational inefficiency and lack of feature self-interaction capability, as we detail in Section 3.3. DTTN's design directly solves these core bottlenecks.\n\nReferences\n\n[1] Cheng, Yixin, et al. \"Multilinear operator networks.\" arXiv preprint arXiv:2401.17992 (2024).\n\n[2] Ran, Shi-Ju, et al. Tensor network contractions: methods and applications to quantum many-body systems. Springer Nature, 2020.\n\n[3] Orús, Román. \"A practical introduction to tensor networks: Matrix product states and projected entangled pair states.\" Annals of physics 349 (2014): 117-158.\n\n[4] Cheng, Song, Lei Wang, and Pan Zhang. \"Supervised learning with projected entangled pair states.\" Physical Review B 103.12 (2021): 125117.\n\n[5] Selvan, Raghavendra, and Erik B. Dam. \"Tensor networks for medical image classification.\" Medical imaging with deep learning. PMLR, 2020.\n\n\nIf you have any remaining questions or concerns, we are happy to address them.\n\nBest,\n\nAuthors"}, {"review_id": "yu2bsPWW1X", "reviewer": "Reviewer_Zidm", "summary": "This paper presents a new Deep Tree Tensor Networks, which consists of multiple Antisymmetric Interaction Modules (AIMs). Evaluation is conducted on image classification, recommender systems, and physics-informed neural networks.", "questions": "Please address the weaknesses listed above.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper is quite well written.\n- Results on ImageNet-1K shows that the proposed method outperforms existing CNN and Transformer based models, especially with the large configuration.\n- Existing Tensor Networks are typically applied to small datasets, whereas the proposed method can scale up to more complex, larger datasets.\n- Evaluation has been conducted on multiple tasks.\n\nWeaknesses:\n- The motivation is not clear. While AIM is obviously very important, there is little description about the design motivation and why such a design should achieve the advantages as demonstrated.\n- It's also not clear why stacking AIMs would give a tree structure. And it's not clear whether it's essential to construct a tree structure in order for the model to perform well. \n- Is the main motivation to create a tree structure? Or is the tree structure just a result of stacking AIMs? Essentially, the introduction reads like the tree structure is critical but it's not reflected in the method description or the results. \n- For results in Table 2, it's not clear where the numbers of other methods come from.", "was_revised": true, "final_justification": "The rebuttal has provided answers to my questions, which however, reveal that the main contribution was not about the tree topology. This then lowers the contributions of this work. The introduction and motivation need to be revised to better reflect the contributions of this work.", "rebuttal": "Dear reviewer Zidm,\n\nThank you for your insightful and constructive review. Your questions are crucial and give us the opportunity to clarify the core contributions and design principles of our work. We address each point below.\n\n> Q1: Why is AIM obviously very important and why should such a design achieve the advantages as demonstrated?\n\nA1: Thank you for this crucial question. The AIM module is the cornerstone of our work because it was specifically engineered to solve the two fundamental barriers that have historically prevented Tensor Networks (TNs) from scaling to large-scale problems.\n\n1.  **The Scalability Barrier:** Traditional TNs like MPS are fundamentally unscalable for high-resolution inputs because the number of sequential contractions scales linearly with the number of features (e.g., $W \\times H$). This creates an insurmountable computational bottleneck[1,2]. Our AIM design shatters this barrier. By decoupling the number of contractions from the input size, the number of contractions depends only on the network's depth $L$, making it far more efficient.\n\n2.  **The Expressivity Barrier:** Prior TN models (e.g., Lotenet, aResMPS[3]) were crippled by small, explicit bond dimensions (often $\\sim$64) due to memory limitations. Our AIM circumvents this by implicitly expanding the effective bond dimension to the order of $10^5\\sim10^6$ for a standard image (via Eq. 5). This allows DTTN to capture immensely complex feature correlations without the impossible memory cost of storing such a large tensor.\n\nIn essence, AIM is the key innovation that transforms Tensor Networks from a niche theoretical tool into a powerful and practical paradigm for large-scale machine learning. We will revise the paper to state these two critical contributions more explicitly.\n\n> Q2: It's also not clear why stacking AIMs would give a tree structure. And it's not clear whether it's essential to construct a tree structure in order for the model to perform well.\n\n> Q3: Is the main motivation to create a tree structure? Or is the tree structure just a result of stacking AIMs? The introduction reads like the tree structure is critical but it's not reflected in the method description or the results.\n\nA2/A3: Thank you for these related questions, which allow us to clarify the role and origin of the tree structure in a unified way. We apologize if our introduction was misleading.\n\nTo be clear, our primary motivation was not to build a tree structure for its own sake, but to design a computationally feasible and pure tensor-based model that could scale to large benchmarks. Our solution was the AIM module, designed for direct multiplicative interactions.\nThe tree structure is the natural and powerful consequence of this design: each AIM performs a binary fusion (taking two feature tensors to produce one). Stacking these hierarchically inherently forms a tree.\nThe significance is that this simple, practical procedure is provably equivalent to a powerful Tree Tensor Network (TTN), bypassing the complex construction methods that have historically limited their use [4]. So, while not the initial goal, the resulting tree structure is essential to our model's expressive power and theoretical grounding.\n\n> Q4: For results in Table 2, it's not clear where the numbers of other methods come from.\n\nA4: Thank you for pointing out this important omission. For clarity: results for the CNN, Transformer, Mamba, and MLP models are cited from their respective original publications. The results for the Polynomial and Multilinear methods are from [5]. In the revised version, we will add a dedicated column or a comprehensive footnote to Table 2 that cites the source for each baseline result. This will ensure full transparency.\n\nReferences:\n\n[1] Glasser, Ivan, et al. \"Expressive power of tensor-network factorizations for probabilistic modeling.\"\n\n[2] Ran, Shi-Ju, et al. Tensor network contractions: methods and applications to quantum many-body systems.\n\n[3] Meng, Ye-Ming, et al. \"Residual matrix product state for machine learning.\"\n\n[4] Cheng, Song, et al. \"Tree tensor networks for generative modeling.\"\n\n[5] Cheng, Yixin, et al. \"Multilinear operator networks.\"\n\nIf you have any remaining questions or concerns, we are happy to address them.\n\nBest,\n\nAuthors"}, {"review_id": "qtl3aCaw1U", "reviewer": "Reviewer_QNHq", "summary": "The authors propose Deep Tree Tensor Network (DTTN), a novel architecture that bridges quantum-inspired tensor networks and modern deep learning for image recognition tasks. The key innovation is the Antisymmetric Interaction Module (AIM) that captures 2^L-order multiplicative interactions across features through multilinear operations, essentially unfolding into a tree-like tensor network topology with parameter-sharing properties. They also claim this is the first tensor network model to achieve competitive performance on large-scale benchmarks like ImageNet-1K, while providing theoretical analysis showing equivalence between quantum-inspired TN models and polynomial/multilinear networks under specific conditions. They demonstrate superior performance compared to other polynomial and multilinear networks across multiple domains, including visual recognition, recommendation systems, and physics-informed neural networks.", "questions": "I have several questions that could help strengthen the paper:\n\n1. Can the authors provide more insight into when DTTN is expected to work well? What properties of the input data or task characteristics make the exponential-order multiplicative interactions particularly beneficial? This would help practitioners understand when to apply the method.\n2. The parameter-sharing mechanism seems crucial for enabling higher bond dimensions, but could you provide more detailed analysis of this? How does the parameter sharing specifically enable the computational advantages over traditional TNs?\n3. How sensitive is DTTN to the choice of L (number of layers)? The authors show some ablation with depth, but more analysis of how the exponential interaction order 2^L affects different types of data would be valuable.\n4. Could the authors provide more analysis of computational trade-offs? While Table 1 shows parameter counts and FLOPs, a more detailed breakdown of training time, memory usage, and convergence behavior compared to baselines would be helpful.", "limitations": "I think the authors adequately address limitations in their discussion, particularly noting the computational resource constraints that prevented more extensive experimentation. They also acknowledge that the method is currently focused on general eigenvalue problems and would need extension for nonlinear cases. However, I think they could expand on some practical limitations - for instance, discussing scenarios where traditional architectures might be preferable, or analyzing the method's behavior on different types of visual data.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: I think the authors have tackled an important and challenging problem. Tensor networks have shown promise in quantum physics and small-scale machine learning tasks, but scaling them to complex benchmarks like ImageNet has been a significant challenge. They correctly identify that existing TN approaches suffer from low computational efficiency and lack of feature self-interaction capability.\nThe technical approach is well-motivated and appears sound. The AIM design is clever - using an antisymmetric two-branch structure to capture multiplicative interactions while maintaining computational efficiency. I appreciate the theoretical analysis, particularly Proposition 1 and Theorem 1, which establish formal connections between DTTN and both polynomial networks and quantum-inspired TNs. The parameter-sharing property that enables higher bond dimensions is a nice insight. The experimental evaluation is quite comprehensive, covering multiple domains beyond just image classification. The authors demonstrate that DTTN can serve as a pluggable module in recommendation systems and can even be applied to PDE solving, which shows the generality of the approach.\n\nWeaknesses: However, I have several concerns about the work. First, while the authors achieve competitive results on ImageNet-1K, the performance gains over modern architectures are relatively modest. The 82.4% achieved by DTTN†-L is good but not groundbreaking compared to other methods in Table 2. The authors seem to position this as the first successful application of TNs to large-scale problems, but the practical impact beyond this \"first\" achievement is questionable.\n\nSecond, I think the theoretical analysis, while formal, could provide more insight into when we expect DTTN to work well versus when it might fail. The authors show equivalence results, but there's limited discussion of the practical implications of these theoretical connections. For instance, what properties of the data or task make DTTN particularly suitable?\n\nThird, some experimental details are concerning. The authors don't provide error bars or statistical significance testing, citing computational constraints. While understandable, this makes it difficult to assess the reliability of the performance differences, especially when some improvements are relatively small.\nThe paper organization could also be improved. Key ablation studies are relegated to the appendix, and some important details about the method (like the exact parameter-sharing mechanism) could be explained more clearly in the main text. On the clarity front, the paper is generally well-written, though some sections feel dense. The connection between the quantum physics motivation and the practical ML algorithm could be explained more intuitively for readers not familiar with tensor networks.", "was_revised": true, "final_justification": "This represents solid engineering research that achieves an important milestone (scaling tensor networks to ImageNet) with reasonable theoretical grounding. The authors addressed most of my technical concerns adequately, but the fundamental limitation remains and hence why I havent increased the score: The understanding of when/why DTTN excels remains largely empirical and I understand that the high-order correlation explanation but to me it lacks formal analysis or predictive power.", "rebuttal": "Dear reviewer QNHq,\n\nWe particularly appreciate your recognition that our work tackles the key challenge of applying Tensor Networks to complex benchmarks, alongside your insightful and constructive feedback. The concerns raised are valid and provide clear directions for improving our paper. We address each point below:\n\n> Q1: The performance gains over modern architectures are relatively modest. \n\nA1: Thank you for this critical point. While gains over highly-optimized modern architectures are modest, our most meaningful comparison is against our direct peers—multilinear and exponential networks. In this context, DTTN's improvement is significant, outperforming the prior exponential network, $\\Pi$-Net[1], by 6.6%~12.0% under an identical parameter budget on ImageNet-1k.\nMore importantly, DTTN represents a foundational step forward because we are the first to demonstrate—both experimentally and theoretically—that Tensor Networks can succeed on large-scale benchmarks, a feat previously unattained by other TN variants. Therefore, our primary contribution is not a marginal performance gain, but a fundamental demonstration that Tensor Networks are a viable and scalable paradigm for complex challenges.\n\n> Q2: The authors seem to position this as the first successful application of TNs to large-scale problems, but the practical impact beyond this \"first\" achievement is questionable.\n\nA2: Thank you for this critical question, as it allows us to clarify the core long-term impact of our work, which extends far beyond a simple \"first\" achievement. The practical impact stems from DTTN being a native modeling paradigm, not just a compression tool.\n\nMost prior \"TN-inspired\" works (e.g., R-Nets[2], NORTON[3]) use tensor decomposition merely as a **post-hoc parameter compression tool** for existing architectures. These methods do not fundamentally alter the model's computational nature, as they still rely on standard non-linear activation functions (like ReLU) for their expressive power.\n\nIn stark contrast, **DTTN is a native tensor model** where the tensor network itself is the core computational engine, achieving non-linearity through pure tensor contractions. This 'purity' is not just an elegant property; it unlocks a significant, long-term practical advantage by making our model potentially executable on future quantum computers. Because DTTN is constructed from pure tensor contractions, it has a direct mapping to quantum circuits[4]. Therefore, our primary impact is not just being the 'first' to scale TNs, but proposing the first 'quantum-native' blueprint for complex recognition tasks.\n\n> Q3: What properties of the input data or task characteristics make the exponential-order multiplicative interactions particularly beneficial? \n\nA3: The exponential-order multiplicative interactions are particularly beneficial for tasks where success hinges on capturing complex, high-order correlations between features. A prime example is fine-grained visual recognition, where distinguishing between similar subclasses relies on subtle combinations of attributes. As our results in Figure 5 (supplementary) show, DTTN outperforms CNN and Transformer variants even on smaller benchmarks, which often depend on such fine-grained distinctions. This aligns with prior work [5, 6] that highlights the power of multiplicative interactions for extracting these types of bilinear or higher-order features. Similarly, in recommendation systems, modeling the conjunction of multiple user and item attributes is critical, and in PDE solving, the underlying physical laws are often inherently multiplicative.\nWhile DTTN is especially powerful in these scenarios, we also want to emphasize its robustness. Our experiments across diverse domains—including computer vision, recommendation, and PDEs—demonstrated consistent and strong performance, suggesting its native tensor structure provides a versatile foundation for a wide range of tasks, from modern vision to fields like quantum chemistry where tensor networks are a fundamental tool.\n\n> Q4:  Difficult to assess the reliability of the performance differences, especially when some improvements are relatively small. \n\nA4: Thank you for this critical point on reliability. We want to clarify why we believe our results are reliable, even when some improvements are modest. First, our reported results stem from single, principled runs to avoid any cherry-picking. This provides a fair and representative comparison. Second, the context of the domain is crucial. The seemingly small improvements, particularly in mature fields like recommendation systems, are meaningful and considered significant within those highly optimized domains. Finally, the most compelling evidence for our method's reliability is its consistent effectiveness across a wide range of diverse tasks—from computer vision to scientific computing. To fully ensure reliability and transparency, we will release our source code for independent verification.\n\n> Q5:  The paper organization could also be improved.\n\nA5: Thank you for the valuable feedback. We revise the paper's structure and clarity by moving the key ablation study to the main text, adding a more intuitive explanation for our parameter-sharing mechanism and its quantum motivation, and streamlining the denser sections in the final version of the paper.\n\n> Q6:  The parameter-sharing mechanism seems crucial for enabling higher bond dimensions, but could you provide more detailed analysis of this? How does the parameter sharing specifically enable the computational advantages over traditional TNs?\n\nA6: Compared to traditional Tensor Networks, our parameter-sharing mechanism offers two key advantages: 1) **Decreased Number of Contractions.** For a conventional MPS, the number of contractions scales with the number of input features (e.g., $W \\times H$), which is computationally prohibitive. In DTTN, the number of contractions depends only on the network's depth $L$ (see Fig. 1), making it far more efficient. 2) **Expanded Effective Bond Dimensions.** We do not explicitly define a small, fixed bond dimension, which is often limited to ~64 in prior work (like Lotenet and aResMPS[7]) due to memory constraints. Instead, we leverage Eq. 5 to implicitly expand the bond dimension to the order of $W \\times H \\times C$ (nearly $10^5-10^6$ for a $224 \\times 224$ input). We avoid storing this massive tensor by using the tensor product to learn the network parameters efficiently. We will add these crucial details to the revised version.\n\n> Q7:  How sensitive is DTTN to the choice of L (number of layers)? \n\nA7: The network depth L is a critical hyperparameter controlling the interaction order. As shown in Tables 2 and 6, its sensitivity is predictable: performance consistently improves with L before saturating, a trend similar to standard deep networks. Our specific L values were chosen to match common model sizes (e.g., S, T, L) for a fair comparison with baselines, as exhaustive tuning was computationally prohibitive.\n\n> Q8:  Could the authors provide more analysis of computational trade-offs? While Table 1 shows parameter counts and FLOPs, a more detailed breakdown of training time, memory usage, and convergence behavior compared to baselines would be helpful.\n\nA8: Thank you for this excellent suggestion. We have provided a detailed comparison of inference latency and memory usage, benchmarked on an NVIDIA A6000 GPU, in the table below. Notably, when compared to the previous state-of-the-art model, MONet-T, our DTTN-S reduces latency by nearly 35% while simultaneously achieving a 2.4% performance gain on ImageNet (as shown in Table 1). Regarding convergence behavior, DTTN also demonstrates faster convergence on both the ImageNet and ImageNet-100 benchmarks (please see Fig. 2e, Fig. 5 for convergence curves).\n\n| Model | Batch Size | Latency (ms/batch) | Throughput (samples/sec) | Peak Memory (GB) |\n|:---|---:|---:|---:|---:|\n| DTTN-S (ours) | 64 | 81.94 | 781.08 | 1.099 |\n| MONet_T | 64 | 128.39 | 498.47 | 1.14 |\n|ViP-Small/14 | 64 | 86.1 | 743.28 | 0.584 |\n\nReferences\n\n[1] Chrysos, Grigorios G., et al. \"Deep polynomial neural networks.\"\n\n[2] Wang, Wenqi, et al. \"Wide compression: Tensor ring nets.\"\n\n[3] Zniyed, Yassine,et al. \"Enhanced network compression through tensor decompositions and pruning.\"\n\n[4] Pan, Feng, and Pan Zhang. \"Simulation of quantum circuits using the big-batch tensor network method.\"\n\n[5] Martyn, John, et al. \"Entanglement and tensor networks for supervised image classification.\" \n\n[6] Zheng, Heliang, et al. \"Learning deep bilinear transformation for fine-grained image representation.\"\n\n[7] Meng, Ye-Ming, et al. \"Residual matrix product state for machine learning.\"\n\n\n\nIf you have any remaining questions or concerns, we are happy to address them.\n\nBest,\n\nAuthors"}, {"review_id": "WDwk7fvbiv", "reviewer": "Reviewer_8ZK2", "summary": "This paper introduces a new deep learning architecture based on stacked blocks comprising convolutions, linear operations, and the Hadamard product; and can be seen as a variation of recently introduced polynomial and multilinear networks, such as PolyNets [10] and MONet [7]. The proposed structure induces a tree topology network and can be interpreted as a tensor network contraction applied to patch embeddings.\n\nThe paper includes an extense experimental section on visual recognition tasks, including large-scale benchmarks like ImageNet, where the Deep Tree Tensor Network (DTTN) is evaluated against classical, multilinear, and polynomial networks, as well as Transformer- and Mamba-based models. The results demonstrate competitive performance, with DTTN outperforming related multilinear and polynomial approaches. Additionally, the DTTN model is evaluated the model on recommender systems and dynamical systems, with detailed results provided in the appendices", "questions": "1.\tCould the authors elaborate on the claimed interpretability advantages of the AIM architecture? For example, how does AIM's interpretability compare quantitatively or qualitatively to existing interpretable architectures. Are there measurable metrics demonstrating AIM's improved interpretability over alternatives that could be tested empirically?\n\n2.\tCould the authors further clarify the significance of the parameter/FLOP ratios reported for the AIM module branches in Section 3.1? Specifically: How do these branching ratios compare to those used in similar architectures? What is the observed relationship between these ratios and model performance? \n\n3.\tDoes the architecture exhibit vanishing/exploding gradients during training? If so, what mitigation strategies were employed.\n\n4.\tWere specific initialization schemes required to achieve stable training? How do these compare to standard deep learning initialization practices?\n\n5.\tHow was the learning rate tuned? Were adaptive optimizers (e.g., AdamW) or learning rate schedules found to be critical for convergence?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- The paper enables a systematic comparison of recently proposed polynomial and multilinear architectures while providing interpretability through the lens of quantum-inspired tensor network models. This perspective may inspire future improvements and theoretical analysis.\n\n- The visual recognition experiments comprehensively evaluate the proposed model against a diverse set of architectures, including CNNs, Transformers, Mamba-based models, MLPs, and polynomial/multilinear networks. Benchmarking is performed across multiple datasets, ranging from standard benchmarks like MNIST and CIFAR-10 to large-scale datasets such as ImageNet.\n\nWeaknesses:\n\nThe paper contains several clarity and technical issues that should be addressed. Below, I outline the key concerns I identified:\n\n-\tThe quantum-inspired tensor network (TN) formulation presented in Equation (1) contains several technical inaccuracies: (i) The model omits standard normalization practices - quantum-inspired models typically include a softmax output layer to produce valid probability distributions, implying at least this nonlinearity should be present; (ii) Equation (1) incorrectly uses the Hadamard product (*) where a scalar/inner product would be more appropriate for maintaining the quantum-inspired interpretation; (iii) Dimensionality error: The weight tensor W^m is claimed to have (N+1) dimensions in Line 37, when it should only have N dimensions; (iv) The paper neglects standard quantum-inspired TN preprocessing, where input nonlinearities (typically trigonometric or custom functions at pixel level) are used to induce higher-order or infinite polynomial representations\n\n-\tThe justification for the AIM architecture remains insufficient. While the authors claim it offers \"better interpretability and understanding,\" the paper lacks: (i) concrete comparative analysis with existing interpretable architectures; (ii) quantitative or qualitative evidence demonstrating improved interpretability, and; (iii) clear mechanistic explanations of how the architecture enables better understanding compared to alternatives.\n\n-\tAlthough DTTN achieves competitive performance (82.4%, Table 2) on ImageNet, the paper should acknowledge that this remains below state-of-the-art results (~90% in Zhai et al., 2021). For a more balanced perspective, the authors should: (i) clarify whether the performance gap reflects a fundamental limitation or a trade-off for other advantages (e.g., interpretability, computational efficiency), and (ii) discuss the potential for closing this gap through scaling or architectural modifications.\n\n-\tSection 2.3 claims that unfoldable structures (lost in normalized networks) offer interpretability advantages. However no concrete examples or analysis demonstrate how this property aids interpretation, thus the claimed interpretability advantage remains hypothetical.\n\n-\tSection 3.1 reports parameter/FLOP ratios between the AIM module's branches, but their significance remains unclear. The analysis would benefit from: (i) Explicit comparison with branching ratios in similar architectures; (ii) Discussion of how these ratios affect model performance (e.g., accuracy-efficiency tradeoffs); (iii) Theoretical justification for the optimal ratio range observed.\n\n-\tEquation (4) uses the operation Pro() without proper mathematical definition. Based on context, this appears to represent a projection via matrix multiplication (B), but this should be explicitly defined.\n\n-\tThe paper does not address well-known training challenges of quantum-inspired TN models, including: (i) Gradient instability (exploding/vanishing weights); (ii) Sensitive weight initialization requirements; (iii) Difficult learning rate tuning.", "was_revised": true, "final_justification": "After reading the authors' responses to my questions and the other reviews, I have increased my rating to 4, as I believe the final version of the paper will address the clarity issues I previously raised that were acknowledged and clarified by the authors responses.", "rebuttal": "Dear reviewer 8ZK2,\n\nWe are thankful to the reviewer 8ZK2 for their effort to review our paper. We address below their concerns and questions:\n\nQ1: Could the authors elaborate on the claimed interpretability advantages of the AIM architecture? \n\nA1: Thank you for this excellent and highly important question. Our model's interpretability is a key advantage, and it stems from three distinct but related aspects.\n\nFirst, DTTN is built on the established principle that activation-free networks are inherently more interpretable. As demonstrated in prior work [1, 2, 3], removing non-linear activations simplifies theoretical analysis and is critical for applications like Fully Homomorphic Encryption. In scientific domains, it is a prerequisite for recovering the underlying governing equations from data, a task impossible with standard activated networks.\n\nSecond, building on this foundation, DTTN's quantum-inspired structure unlocks a unique form of structural interpretability. For analysis, our DTTN can be recast into an MPS form by contracting its internal cores. This allows us to make a \"cut\" at any bond, partitioning the input features into two regions. We can then quantitatively measure the correlation the model has learned between these regions by calculating the entanglement entropy across the cut[4,5].\n\nFinally, this framework is not merely theoretical; it has direct, empirically testable implications, particularly in applications like our PINN experiments. For tasks with low-dimensional inputs, like the PDE problems we solve, the DTTN model can be explicitly written out as a high-order polynomial. This learned polynomial can be directly compared, term-by-term, against the analytical polynomial expansion of the ground-truth PDE solution. The \"error\" can thus be measured as the difference in coefficients, providing a concrete, quantitative validation of the model's physical correctness that is impossible for standard architectures.\n\nQ2: Could the authors further clarify the significance of the parameter/FLOP ratios reported for the AIM module branches in Section 3.1? Specifically: How do these branching ratios compare to those used in similar architectures? What is the observed relationship between these ratios and model performance?\n\nA2: Our design is conceptually related to the efficiency-driven designs in similar architectures. For example, the baseline model MONet also employs a form of asymmetry in its building block, using a single-branch low-rank design to reduce computational overhead. Their work showed that a ratio of 4 offered the best balance, saving computation with no performance loss, while higher ratios (e.g., 8 or 16) began to degrade accuracy.\n\nInspired by this principle, our AIM takes a different but related approach. Instead of a single low-rank branch, we use two asymmetric branches with a computational cost ratio of approximately 1:3 (per Eq. 3). This design choice is validated by our ablation study in Appendix D (Table 7), where we compare AIM against a Symmetric Interaction Module (SIM). The results are definitive: our antisymmetric design reduces parameters and FLOPs by approximately 25% compared to its symmetric counterpart, while maintaining the same level of performance. \n\n> Q3: Does the architecture exhibit vanishing/exploding gradients during training? If so, what mitigation strategies were employed?\n\nA3: No, our DTTN model trains stably across all scenarios without signs of vanishing/exploding gradients or training spikes, as evidenced by the smooth loss curves in Figure 5. This stability is a key advantage over traditional TNs, where the long chain of sequential contractions required to merge the core factors often leads to optimization instability. Although DTTN is theoretically equivalent to a TNN, its optimization process is structured like a standard deep network, which provides a more stable optimization process.\n\n> Q4: Were specific initialization schemes required to achieve stable training? How do these compare to standard deep learning initialization practices?\n\nA4: We employ a standard initialization scheme, Xavier Normal initialization, for all DTTN parameters. This is the same practice used in standard deep learning and is consistent with the baseline[6] to ensure a fair and direct comparison.\n\n> Q5: How was the learning rate tuned? Were adaptive optimizers (e.g., AdamW) or learning rate schedules found to be critical for convergence?\n\nA5: For visual recognition tasks, we adopted the same training strategy as MONet[6], including the AdamW optimizer and a cosine learning rate schedule (with a base LR of 1e-3 and warmup LR of 1e-5). With this standard setup, DTTN exhibits faster convergence than MONet and attention-based architectures, such as ViP.\n\nImportantly, DTTN is robust to different optimization choices. For the recommendation system and PINN tasks, we used Adam and LBFGS optimizers, respectively, with a fixed learning rate of 0.001, consistent with their respective baselines [7, 8]. In all cases, the model converged effectively, demonstrating its flexibility.\n\n> Q6: The paper should acknowledge that DTTN's performance remains below the state-of-the-art on ImageNet and (i) clarify whether the performance gap reflects a fundamental limitation or a trade-off for other advantages (e.g., interpretability, computational efficiency), and (ii) discuss the potential for closing this gap through scaling or architectural modifications.\n\nA6: The performance gap does not reflect a fundamental limitation of our approach, but rather a difference in training regimes and model scale. We emphasize that DTTN is trained solely on ImageNet-1K, whereas many state-of-the-art models leverage massive external datasets (like JFT-300M) and significantly larger model sizes, which are beyond our current computational capacity. Our results demonstrate that DTTN is competitive within its class of modern architectures, especially considering its superior performance against other polynomial and multilinear networks. Regarding computational efficiency, as detailed in our response to Reviewer QNHq, our DTTN-S reduces inference latency by nearly 35% compared to MONet-T while also improving accuracy by 2.4%. Therefore, the current results represent a favorable trade-off, delivering strong performance with high efficiency.\n\nWe are confident there is significant potential to close this gap. The path forward includes scaling up our model architecture, exploring more advanced training recipes, and leveraging large-scale pre-training. Investigating these avenues to push the performance boundaries of pure, scalable Tensor Network models will be a key direction for future research, building directly upon the foundation established by this work.\n\n> Q7: The paper neglects standard quantum-inspired TN preprocessing, where input nonlinearities (typically trigonometric or custom functions at pixel level) are used to induce higher-order or infinite polynomial representations.\n\nA7: Thank you for this point. Our approach does not neglect this concept, but rather integrates it more flexibly into our architecture. As implied by Theorem 1, DTTN is theoretically equivalent to applying a local polynomial mapping $\\phi(x_j)=[x_j^0, \\dots, x_j^{2^L}]^T$ to each input feature. This mapping, which is the key to our model's high-order interactions, is an emergent property of our end-to-end trained architecture, not a fixed pre-processing step.\n\n> Q8: Several points were raised regarding the clarity and technical accuracy of the paper.\n\nA8: We sincerely thank you for this detailed feedback and for bringing these important issues to our attention. (1) On the Pro operation in Eq. 4: We will explicitly define the Pro operation in the revised paper. (2) On the Dimensionality of weight tensor $W^m$: Thank you for questioning this. Our formulation includes an additional dimension for the class index, which is necessary for multi-class classification tasks within this framework, a practice also seen in [9]. (3) On Softmax and Normalization: We agree that a softmax is used, but we argue it is not an intrinsic part of the DTTN. The softmax function is a standard, task-specific tool applied to the final output logits for classification problems, just as it would be for any CNN or Transformer. It is not a feature of the model itself. (4) On the use of Hadamard product: We will correct this to use the appropriate tensor inner product to align with the quantum-inspired interpretation. (5) Regarding other clarity issues: We thank you for highlighting these and other areas needing improvement. We will conduct a thorough revision of the entire paper to ensure all technical details are precise and the paper is clear and easy to follow.\n\nReferences\n[1] Dubey, et al. “Scalable interpretability via polynomials”\n\n[2] Fronk and Petzold. “Interpretable polynomial neural ordinary differential equations”\n\n[3] Arora, et al. \"A convergence analysis of gradient descent for deep linear neural networks\"\n\n[4] Ran, Shi-Ju, et al. \"Tensor network contractions: methods and applications to quantum many-body systems.\n\n[5] Ran, Shi-Ju, and Gang Su. \"Tensor networks for interpretable and efficient quantum-inspired machine learning\"\n\n[6] Cheng, Yixin, et al. \"Multilinear operator networks\"\n\n[7] Jianxun, Lian, et al. \"Combining explicit and implicit feature interactions for recommender systems\"\n\n[8] Shashank Subramanian, et al., \"Adaptive self-supervision algorithms for physics-informed neural networks\"\n\n[9] Stoudenmire, Edwin, and David J. Schwab. \"Supervised learning with tensor networks\"\n\nIf you have any remaining questions or concerns, we are happy to address them.\n\nBest,\n\nAuthors"}]

Author rebuttal:

Meta-review: The paper proposes an activation-free tree tensor network without nonlinearity and demonstrates its foundational equivalence to quantum-inspired models. The proposed method represents an important milestone in scaling tensor networks to ImageNet, supported by solid technical execution. The authors’ rebuttal successfully addressed some concerns regarding core innovation, technical issues, and additional experimental evaluations. These revisions should be incorporated into the final version, including a clearer motivation for AIM and a more detailed discussion on overcoming the well-known training challenges in quantum-inspired tensor networks.
Decision: Accept (poster)