SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To illustrate the benefit of aggregating equivariant messages from local patches, we study a simple case. Let  $f_{a}(\mathbf{B},\mathbf{C}) = \mathbf{h}_{B}\cdot \mathbf{h}_{C}$  be an invariant function of  $\mathbf{B}$  and  $\mathbf{C}$  (see Fig. 2), then  $f_{a}$  can be calculated by a direction composition of scalar messages and equivariant vector messages:  $f_{a}(\mathbf{B},\mathbf{C}) = \frac{1}{2} [||\mathbf{m}_{a}||^{2} - ||\mathbf{h}_{B}||^{2} - ||\mathbf{h}_{C}||^{2}]$ , where  $\mathbf{m}_a = \mathbf{h}_B + \mathbf{h}_C$  is an equivariant vector. Note that  $\mathbf{m}_a$  follows the local equivariant aggregation formula 6, and the other vectors' norm  $\| \mathbf{h}_B\|$  and  $\| \mathbf{h}_C\|$  are obtained through local scalarization on atoms  $b$  and  $c$ . As a comparison, it's worth mentioning that  $f_{a}(\mathbf{B},\mathbf{C})$  can also be expressed by local scalarization with the additional transition matrix data  $R_{BC}$  defined by Eq. 5. Let  $\tilde{h}_B$  and  $\tilde{h}_C$  be the scalarized coordinates with respect to two local equivariant frames  $\mathcal{F}_B$  and  $\mathcal{F}_C$ . Then  $f_{a}(\mathbf{B},\mathbf{C}) = \frac{1}{2}\left[\left\| R_{BC}^{-1}\tilde{h}_B + \tilde{h}_C\right\|^2 -\left\| \tilde{h}_B\right\|^2 -\left\| \tilde{h}_C\right\|^2\right]$ . However, it requires adding the transition matrix  $R_{BC}$  for each  $(\mathbf{B},\mathbf{C})$  pair into the aggregation procedure, which is computationally expensive compared to directly implementing equivariant tensor updates.
5 Building an efficient and expressive equivariant 3D GNN
We propose to leverage the full power of LSE and FTE modules (along with a necessary tensor update module) to push the limit of efficient and expressive 3D equivariant GNNs design. Specifically, we improve a recently proposed method based on constructing local frames but without the implementation of LSE and FTE [30]. We also analyze related works following this framework detailed in Appendix G.
Figure 3: Illustrations of our modular framework for building equivariant GNNs and the realization of LEFTNet. Each interaction block contains LSE to encode local 3D structures, equivariant message passing to update both invariant (unbold letters, e.g.  $h_i$ ) and equivariant (bold letter, e.g.  $\mathbf{h}_i$ ) features, and FTE to encode frame transition.  $S_{i-j}$  is the local 3D structure of each edge  $e_{ij}$ .  $\mathcal{F}_{ij}$  and  $\mathcal{F}_i$  are the equivariant frames for each edge  $e_{ij}$  and node  $i$ .  $\odot$  indicates element-wise multiplication, and  $\| \cdot$  indicates concatenation. Note that we do not include  $\mathbf{e}_{ij}$  in the figure since, practically, they are generated based on  $\mathbf{h}_i$  and  $\mathbf{h}_j$ .
LSE Instantiation. We propose to apply edge-wise equivariant frames (following [30]) to encode the local 3D structures  $\mathbf{S}_{i-j}$ . By definition,  $\mathbf{S}_{i-j}$  contains edge  $e_{ij}$ , nodes  $i$  and  $j$ , and their common neighbors. We use the equivariant frame  $\mathcal{F}_{ij}$  built on  $e_{ij}$  (see the precise formula in Appendix F) to scalarize  $\mathbf{S}_{i-j}$ . After scalarization (8), the equivariant coordinates of all nodes in  $\mathbf{S}_{i-j}$  are transformed into invariant coordinates:  $\{\mathbf{x}_k \to \tilde{x}_k$  for  $\mathbf{x}_k \in \mathbf{S}_{i-j}\}$ . To encode these scalars sufficiently, we first weight each  $\tilde{x}_k$  by the RBF distance embedding:  $\tilde{x}_k \to \mathbf{RBF}(\|\mathbf{x}_k\|) \odot \mathrm{MLP}(\tilde{x}_k)$  for each  $\mathbf{x}_k \in \mathbf{S}_{i-j}$ . Note that to preserve the permutation symmetry, the MLP is shared among the nodes. Finally, the 3D structure weight  $A_{ij}$  is obtained by the average pooling of all node features.
FTE Instantiation. We propose to introduce equivariant tensor message passing and update function for encoding local FT information. At initialization, let  $\mathbf{NF}^l (\mathbf{x}_i,\mathbf{x}_j)$  denote the embedded tensorvalued edge feature between  $i$  and  $j$ . We split it into two parts: 1. the scalar part  $\mathrm{SF}^l (\mathbf{x}_i,\mathbf{x}_j)$  for aggregating invariant messages; 2. the higher order tensor part  $\mathbf{TF}^l (\mathbf{x}_i,\mathbf{x}_j)$  for aggregating tensor messages. To transform  $\mathbf{TF}^l (\mathbf{x}_i,\mathbf{x}_j)$ , we turn to the equivariant frame  $\mathcal{F}_{ij}$  once again. After scalarization by  $\mathcal{F}_{ij}$ ,  $\mathbf{TF}^l (\mathbf{x}_i,\mathbf{x}_j)$  becomes a tuple of scalars  $\tilde{\mathrm{TF}}^l (\mathbf{x}_i,\mathbf{x}_j)$ , which is then transformed by MLP. Finally, we output arbitrary tensor messages through equivariant tensorization 22:
$$
\tilde {\mathrm {T F}} ^ {l} \left(\mathbf {x} _ {i}, \mathbf {x} _ {j}\right) \xrightarrow [ \mathcal {F} _ {i j} ]{\text {T e n s o r i z e}} \mathbf {N F} ^ {l + 1} \left(\mathbf {x} _ {i}, \mathbf {x} _ {j}\right).
$$
Further details are provided in Appendix F. As we have discussed earlier, the node-wise update function  $\phi$  in Eq. 6 is also one of the guarantees for a powerful FTE. As a comparison,  $\phi$  is usually a standard MLP for updating node features in 2D GNNs, which is a universal approximator of invariant functions. Previous works [14, 31] updated equivariant features by taking linear combinations and calculating the invariant norm of tensors, which may suffer from information loss. Then a natural question arises: Can we design an equivariant universal approximator for tensor update? We answer this question by introducing a novel node-wise frame. Consider node  $i$  with its position  $\mathbf{x}_i$ , let  $\bar{\mathbf{x}}_i \coloneqq \frac{1}{N} \sum_{\mathbf{x}_j \in N(\mathbf{x}_i)} \mathbf{x}_j$  be the center of mass around  $\mathbf{x}_i$ 's neighborhood. Then the orthonormal equivariant frame  $\mathcal{F}_i \coloneqq (\mathbf{e}_1^i, \mathbf{e}_2^i, \mathbf{e}_3^i)$  with respect to  $\mathbf{x}_i$  is defined similar to the edge-wise frame between  $\mathbf{x}_i$  and  $\bar{\mathbf{x}}_i$  in [30], detailed in appendix B. Finally, we realize a powerful  $\phi$  by the following theorem:
Theorem 5.1. Equipped with an equivariant frame  $\mathcal{F}_i$  for each node  $i$ , the equivariant function  $\phi$  defined by the following composition is a universal approximator of tensor transformations:  $\phi : \text{Scalarization} \to \text{MLP} \to \text{Tensorization}$ .
Proof. The detailed proof is provided in Appendix F.
LEFTNet. An overview of our  $\{\mathbf{LSE},\mathbf{FTE}\}$  enhanced efficient graph neural network (LEFTNet) is depicted in Fig. 3. The detailed algorithm for LEFTNet is shown in Algorithm 1 of Appendix C. LEFTNet receives as input a collection of node embeddings  $\{v_1^0,\dots,v_N^0\}$ , which contain the atom types and 3D positions for each node:  $v_{i}^{0} = (z_{i},\mathbf{x}_{i})$ , where  $i\in \{1,\ldots ,N\}$ . For each edge  $e_{ij}\in E$
we denote the associated equivariant features consisting of tensors by  $\mathbf{e}_{ij}$ . During each messaging passing layer, the LSE module outputs the scalar weight coefficients  $A_{ij}$  as enhanced invariant edge feature and feed into the interaction module. Moreover, scalarization and tensorization as two essential blocks are used in the equivariant update module that fulfills the function of FTE. The permutation equivariance of a geometric graph is automatically guaranteed for any message passing architecture, we provide a complete proof of SE(3)-equivariance for LEFTNet in Appendix F.
SE(3) vs E(3) Equivalence. Besides explicitly fitting the  $SE(3)$  invariant molecular geometry probability distribution, modeling the energy surface of a molecule system is also a crucial task for molecule property prediction. However, the Hamiltonian energy function  $E$  of a molecule is invariant under refection transformation:  $\mathbf{Energy}(\mathbf{X}) = \mathbf{Energy}(RX)$ , for arbitrary reflection transformation  $R \in E(3)$ . In summary, there exist two different inductive biases for modeling 3D data: (1) SE(3) equivariance, e.g. chirality could turn a therapeutic drug to a killing toxin; (2) E(3) equivariance, e.g. energy remains the same under reflections.
Since we implement  $SE(3)$  equivariant frames in LEFTNet, our algorithm is naturally  $SE(3)$  equivariant. However, our method is flexible to implement  $E(3)$  equivariant tasks as well. For  $E(3)$  equivariance, we can either replace our frames to  $E(3)$  equivariant frames, or modify the scalarization block by taking the absolute value:  $\mathbf{x} \to \tilde{x} := \underbrace{(\mathbf{x} \cdot e_1, \mathbf{x} \cdot e_2, \mathbf{x} \cdot e_3)}_{SE(3)} \to \underbrace{(\mathbf{x} \cdot e_1, |\mathbf{x} \cdot e_2|, \mathbf{x} \cdot e_3)}_{E(3)}$ .
Intuitively, since the second vector  $e_2$  is a pseudo-vector, projections of any equivariant vectors along the  $e_2$  direction are not  $E(3)$  invariant until taking the absolute value.
Efficiency. To analyze the efficiency of LEFTNet, suppose 3D graph  $G$  has  $n$  vertices, and its average node degree is  $k$ . Our algorithm consists of three phases: 1. Building equivariant frames and performing local scalarization; 2. Equivariant message passing; 3. Updating node-wise tensor features through scalarization and tensorization. Let  $l$  be the number of layers, then the computational complexity for each of our three phases are: 1.  $O(nk)$  for computing the frame and local (1-hop) 3D features; 2.  $O(nkl)$  for 1-hop neighborhood message aggregation; 3.  $O(nl)$  for node-wise tensorization and feature update.
We evaluate our LEFTNet on both scalar value (e.g. energy) and vector value (e.g. forces) prediction tasks. The scalar value prediction experiment is conducted on the QM9 dataset [32] which includes  $134k$  small molecules with quantum property annotations; the vector value prediction experiment is conducted on the MD17 dataset [33] and the Revised MD17(rMD17) dataset [34] which includes the energies and forces of molecules. We compare LEFTNet with a list of state-of-the-art equivariant (invariant) GNNs including SphereNet [15], PaiNN [35], Equiformer [36], GemNet [37], etc [28, 38, 13, 39, 40, 14, 16, 41-48]. The training details, results on rMD17, etc. are listed in Appendix I.
Table 1: Mean Absolute Error for the molecular property prediction benchmark on QM9 dataset. (The best results are bolded and the second best are underlined.)
Task
Units	α
bohr3	Δε
meV	εHOMO
meV	εLUMO
meV	μ
D	Cν
cal/mol K	G
meV	H
meV	R2
bohr3	U
meV	U0
meV	ZPVE
meV
NMP	.092	69	43	38	.030	.040	19	17	.180	20	20	1.50
Cormorant	.085	61	34	38	.038	.026	20	21	.961	21	22	2.03
LieConv	.084	49	30	25	.032	.038	22	24	.800	19	19	2.28
TFN	.223	58	40	38	.064	.101	-	-	-	-	-	-
SE(3)-Tr.	.142	53	35	33	.051	.054	-	-	-	-	-	-
EGNN	.071	48	29	25	.029	.031	12	12	.106	12	11	1.55
SEGNN	.060	42	24	21	.023	.031	15	16	.660	13	15	1.62
ClofNet	.063	53	33	25	.040	.027	9	9	.610	9	8	1.23
EQGAT	.063	44	26	22	.014	.027	12	13	.257	13	13	1.50
Equiformer	.056	33	17	16	.014	.025	10	10	.227	11	10	1.32
LEFTNet (ours)	.048	40	24	18	.012	.023	7	6	.109	7	6	1.33
Schnet	.235	63	41	34	.033	.033	14	14	.073	19	14	1.70
DimeNet++	.044	33	25	20	.030	.023	8	7	.331	6	6	1.21
SphereNet	.046	32	23	18	.026	.021	8	6	.292	7	6	1.12
ClofNet	.053	49	33	25	.038	.026	9	8	.425	8	8	1.59
PaiNN	.045	46	28	20	.012	.024	7	6	.066	6	6	1.28
LEFTNet (ours)	.039	39	23	18	.011	.022	6	5	.094	5	5	1.19
Table 2: Mean Absolute Error for per-atom forces prediction (kcal/mol Å) on MD17 dataset. The best results are bolded.
WoFE=100	WoFE=1000	Others
Molecule	sGDML	SchNet	DimeNet	SphereNet	SpookyNet	LEFTNet	SphereNet	GemNet	LEFTNet	PaiNN	NewtonNet
Aspirin	0.68	1.35	0.499	0.430	0.258	0.300	0.209	0.217	0.196	0.371	0.348
Benzene	0.20	0.31	0.187	0.178	-	0.145	0.147	0.145	0.142	-	-
Ethanol	0.33	0.39	0.230	0.208	0.094	0.138	0.091	0.086	0.099	0.230	0.264
Malonaldehyde	0.41	0.66	0.383	0.340	0.167	0.209	0.172	0.155	0.142	0.319	0.323
Naphthalene	0.11	0.58	0.215	0.178	0.089	0.073	0.048	0.051	0.044	0.083	0.084
Salicylic acid	0.28	0.85	0.374	0.360	0.180	0.167	0.113	0.125	0.117	0.209	0.197
Toluene	0.14	0.57	0.210	0.155	0.087	0.084	0.054	0.060	0.049	0.102	0.088
Uracil	0.24	0.56	0.301	0.267	0.119	0.116	0.106	0.097	0.085	0.140	0.149
6.1 QM9 - scalar-valued property prediction
The QM9 dataset is a widely used dataset for predicting molecular properties. However, existing models are trained on different data splits. Specifically, Cormorant [40], EGNN [14], etc., use 100k, 18k, and 13k molecules for training, validation, and testing, while DimeNet [38], SphereNet [15], etc., split the data into 110k, 10k, and 11k. For a fair comparison with all methods, we conduct experiments using both data splits. Results are listed in Table 1. For the first data split, LEFTNet is the best on 7 out of the 12 properties and improves previous SOTA results by  $20\%$  on average. Consistently, LEFTNet is the best or second best on 10 out of the 12 properties for the second split. These results validate the effectiveness of LEFTNet. Detailed ablation in Appendix I shows that both LSE and FTE contribute to
the final performance. In addition, we compare the forward time of existing methods to show the efficiency of LEFTNet. Specifically, we use the same batch size for all methods and report the one epoch training time. Fig. 4 shows that LEFTNet can achieve the best performance using similar forward time as SchNet, ClofNet, and ComENet and is much faster than DimeNet and SphereNet. This is consistent with our efficiency analysis in Sec. 5 and Table 4.
Figure 4: Comparisons of existing methods in terms of the training time and the MAE of the property  $U_{0}$  of QM9.
6.2 MD17 - vector-valued property prediction
We evaluate LEFTNet to predict forces on the MD17 dataset. Following existing studies [28, 38, 15], we train a separate model for each of the 8 molecules. Both training and validation sets contain 1000 samples, and the rest are used for testing. Note that all baseline methods are trained on a joint loss of energies and forces, but different methods use different weights of force over energy (WoFE). For example, SchNet [28] sets WoEF as 100, while GemNet [37] uses a weight of 1000. For a fair comparison with existing studies, we conduct experiments on two widely used weights of 100 and 1000 following Liu et al. [15]. The results are summarized in Table 2. Results show that when WoFE is 100, LEFTNet
outperforms all baseline methods on 5 of the 8 molecules. In addition, LEFTNet can outperform all baseline methods on 6 of the 8 molecules when WoFE is 1000. These experimental results on MD17 demonstrate the performance of LEFTNet on vector-valued property prediction tasks. The ablation results in Table 3 demonstrate that both LSE and FTE are important to the final results. Specifically, the algorithm for LSE only is in Algorithm 1 of Appendix C. Note that the original MD17 dataset we used has numerical noise [34], and a recomputed version of MD17, called Revised MD17 (rMD17) [34] is proposed to reduce the numerical noise. In addition to the original MD17, we also conduct experiments on the rMD17, and the results are listed in Appendix I.
Table 3: Ablation study on Aspirin of the MD17 dataset. Detailed ablations on other molecules are listed in Appendix I
Method	MAE
w/o LSE and FTE	1.083
LSE only	0.451
LSE + FTE	0.300
7 Limitation and future work
In this paper, we seek a general recipe for building 3D geometric graph deep learning algorithms. Considering common prior of 2D graphs, such as permutation symmetry, has been incorporated in off-the-shelf graph neural networks, we mainly focus on the  $E(3)$  and  $SE(3)$  symmetry specific to 3D geometric graphs. Despite our framework being general for modeling geometric objects, we only conducted experiments on commonly used molecular datasets. It's worth exploring datasets in other domains in the future.
To elucidate the future design space of equivariant GNNs, we propose two directions that are worth exploring. Firstly, our current algorithms consider fixed equivariant frames for performing aggregation and node updates. Inspired by the high body-order ACE approach [49] (for modeling atom-centered potentials), it is worth investigating in the future if equivariant frames that relate to many body (e.g., the PCA frame in [50, 51]) can boost the performance of our algorithm. For example, to build the A-basis proposed in [49], we can replace our message aggregation Eq. 6 from summation to tensor product, which is also a valid pooling operation. Another direction is to explore geometric mesh graphs on manifolds  $M$ , where the local frame is defined on the tangent space of each point:  $\mathcal{F}(x) \in T_xM$ . Since our scalarization technique (crucial for realizing LSE in LEFTNet) originates from differential geometry on frame bundles [52], it is reasonable to expect that our framework also works for manifold data [53, 54].
Y.D., D.F. and C.P.G. are supported by the Eric and Wendy Schmidt AI in Science Postdoctoral Fellowship, a Schmidt Futures program; the National Science Foundation (NSF), the Air Force Office of Scientific Research (AFOSR). L.W. and S.J. are supported by NSF IIS-2243850.
[1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[2] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.
[3] Xuan Zhang, Limei Wang, Jacob Helwig, Youzhi Luo, Cong Fu, Yaochen Xie, Meng Liu, Yuchao Lin, Zhao Xu, Keqiang Yan, et al. Artificial intelligence for science in quantum, atomistic, and continuum systems. arXiv preprint arXiv:2307.08423, 2023.
[4] Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural descriptor fields: Se (3)-equivariant object representations for manipulation. In 2022 International Conference on Robotics and Automation (ICRA), pages 6394-6400. IEEE, 2022.
[5] Frank Noé, Alexandre Tkatchenko, Klaus-Robert Müller, and Cecilia Clementi. Machine learning for molecular simulation. Annual review of physical chemistry, 71:361-390, 2020.
[6] Lars Holdijk, Yuanqi Du, Priyank Jaini, Ferry Hooft, Bernd Ensing, and Max Welling. Path integral stochastic optimal control for sampling transition paths. In ICML 2022 2nd AI for Science Workshop.
[7] Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34 (4):18-42, 2017.
[8] Kenneth Atz, Francesca Grisoni, and Gisbert Schneider. Geometric deep learning on molecular representations. Nature Machine Intelligence, 3(12):1023-1032, 2021.
[9] Zhengyang Wang, Meng Liu, Youzhi Luo, Zhao Xu, Yaochen Xie, Limei Wang, Lei Cai, Qi Qi, Zhuoning Yuan, Tianbao Yang, et al. Advanced graph and sequence neural networks for molecular property prediction and drug discovery. Bioinformatics, 38(9):2579-2586, 2022.
[10] Arne Schneuing, Yuanqi Du, Charles Harris, Arian Jamasb, Ilia Igashov, Weitao Du, Tom Blundell, Pietro Lio, Carla Gomes, Max Welling, et al. Structure-based drug design with equivariant diffusion models. arXiv preprint arXiv:2210.13695, 2022.
[11] Yuanqi Du, Tianfan Fu, Jimeng Sun, and Shengchao Liu. Molgensurvey: A systematic survey in machine learning models for molecule design. arXiv preprint arXiv:2203.14500, 2022.
[63] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
[64] Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
[65] Meng Liu, Youzhi Luo, Limei Wang, Yaochen Xie, Hao Yuan, Shurui Gui, Haiyang Yu, Zhao Xu, Jingtun Zhang, Yi Liu, et al. Dig: A turnkey library for diving into graph deep learning research. The Journal of Machine Learning Research, 22(1):10873-10881, 2021.
Appendix for “A new perspective on building efficient and expressive 3D equivariant graph neural networks”
A Supplementary background 15
B Equivariant frame construction 16
C Algorithm for the proposed LEFTNet 17
D Related proofs and discussions of Section 3 18
E Related proofs and discussions of Section 4 20
F Related proofs and discussions of Section 5 23
G Extended Related Work 24
G.1 Modular overview of 3D GNN 24
G.2 Relationship to Geometric WL test 25
H Neural sheaf interpretation 26
I Additional experiments 27
A Supplementary background
In this section, we first detail the concept of scalarization and tensorization and then review the concept of (contravariant) tensor fields and their associated equivariant group representations.
Scalarization. Scalarization is a general technique that originated from differential geometry for realizing covariant operations on tensors [52]. Our method will apply a simple version of scalarization in  $\mathbf{R}^3$  to transform equivariant quantities. At the heart of its realization is the notion of equivariant orthonormal frames, which consist of three orthonormal equivariant vectors:
$$
\mathcal {F} := \left(\mathbf {e} _ {1}, \mathbf {e} _ {2}, \mathbf {e} _ {3}\right).
$$
Based on  $\mathcal{F}$ , we can build orthonormal equivariant frames for higher order tensors by taking tensor products  $\otimes$ , see Eq. 21 in Appendix. By taking the inner product between  $\mathcal{F}$  and a given equivariant vector (tensor)  $\mathbf{x}$ , we get a tuple of invariant scalars (see [30] for a proof):
$$
\mathbf {x} \rightarrow \tilde {x} := \left(\mathbf {x} \cdot \mathbf {e} _ {1}, \mathbf {x} \cdot \mathbf {e} _ {2}, \mathbf {x} \cdot \mathbf {e} _ {3}\right), \tag {8}
$$
and  $\tilde{x}$  can be seen as the scalarized' coordinates of  $\mathbf{X}$
Tensorization. Tensorization, on the other hand, is the 'reverse' process of scalarization. Given a tuple of scalars:  $(x_{1}, x_{2}, x_{3})$ , tensorization creates an equivariant vector (tensor) out of  $\mathcal{F}$ :
$$
\left(x _ {1}, x _ {2}, x _ {3}\right) \xrightarrow {\text {P a i r i n g}} \mathbf {x} := x _ {1} \mathbf {e} _ {1} + x _ {2} \mathbf {e} _ {2} + x _ {3} \mathbf {e} _ {3}. \tag {9}
$$
The same procedure is extended to higher order cases, see Eq. 22 in Appendix.
A  $s$  order (contravariant) tensor  $\mathbf{T}$  on a vector space  $\mathbf{V}$  is a multilinear map:
$$
T: \underbrace {\mathbf {V} ^ {*} \times \cdots \times \mathbf {V} ^ {*}} _ {s} \to \mathbf {R} ^ {1},
$$
where  $\mathbf{V}^*$  denotes the dual space of  $\mathbf{V}$ . In fact, there is a canonical 'multiplication' operation between two tensors. Define the tensor product  $\mathbf{S} \otimes \mathbf{T}$  of two tensors  $\mathbf{S}$  and  $\mathbf{T}$  to be a tensor of order  $r + s$ :
$$
\mathbf {S} \otimes \mathbf {T} \left(v ^ {1}, \dots , v ^ {r + s}\right) = \mathbf {S} \left(v ^ {1}, \dots , v ^ {r}\right) \mathbf {T} \left(v ^ {r + 1}, \dots , v ^ {r + s}\right). \tag {10}
$$
where  $v^i\in \mathbf{V}^*$
From now on, we assume  $V = V^{*} = \mathbf{R}^{3}$ . Note that when  $s = 1$ ,  $\mathbf{T}$  is exactly an equivariant vector. In practice, the tensor data in  $\mathbf{R}^3$  is usually given by its coefficients under a Cartesian coordinate system. Take a second-order tensor as an example, assume we are given an orthonormal frame (basis)  $(\mathbf{e}_1, \mathbf{e}_2, \mathbf{e}_3)$  and its dual frame  $(\mathbf{e}^1, \mathbf{e}^2, \mathbf{e}^3)$ , then the nine coefficients of  $T$  are given by
$$
T _ {i j} = \mathbf {T} \left(\mathbf {e} ^ {i}, \mathbf {e} ^ {j}\right), 1 \leq i, j \leq 3.
$$
In other words, we say the collection  $\{T_{ij}\}_{1\leq i,j\leq 3}$  is a faithful representation of  $\mathbf{T}$  in a fixed coordinate system:
$$
\mathbf {T} = \sum_ {i, j} T _ {i j} \mathbf {e} _ {i} \otimes \mathbf {e} _ {j}. \tag {11}
$$