SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma 35 (Simplified Talagrand's concentration inequality [42]). Let  $\mathcal{F}$  be a function class on  $\mathcal{X}$  that is separable with respect to  $L_{\infty}$ -norm, and  $\{\mathbf{x}_i\}_{i=1}^N$  be i.i.d. random variables in  $\mathcal{X}$ . Furthermore, suppose there exist constants  $V \geq$  and  $U \geq 0$  such that  $V = \sup_{f \in \mathcal{F}} \mathbb{E}[(f - \mathbb{E}[f])^2]$  and  $U = \sup_{f \in \mathcal{F}} \|f\|_{L_{\infty}}$ . Letting  $Z := \sup_{f \in \mathcal{F}} |N^{-1} \sum_{i=1}^{N} f(\mathbf{x}_i) - \mathbb{E}[f]|$ , then it holds for all  $t > 0$  that
$$
\mathbb {P} \left[ Z \geq 2 \mathbb {E} [ Z ] + \sqrt {\frac {2 V t}{N}} + \frac {2 U t}{N} \right] \leq e ^ {- t}.
$$
F.2 Some Results for Analyzing t-NNs
In this subsection, we present several fundamental statements described as lemmas for analyzing t-NNs. First, we give Lemmas 36-38, which are used in the analysis of the t-product layers.
Lemma 36. Let  $\sigma(\cdot): \mathbb{R} \to \mathbb{R}$  be a  $L_{\sigma}$ -Lipschitz function, i.e.,  $|\sigma(x) - \sigma(y)| \leq L_{\sigma}|x - y|$ ,  $\forall x, y \in \mathbb{R}$ . If it is applied element-wisely to any two real vectors  $\mathbf{x}$  and  $\mathbf{y}$ , then it holds that
$$
\left\| \sigma (\mathbf {x}) - \sigma (\mathbf {y}) \right\| _ {l _ {p}} \leq L _ {\sigma} \left\| \mathbf {x} - \mathbf {y} \right\| _ {l _ {p}}.
$$
Lemma 37. The following inequalities hold:
$$
\| \mathbf {T} \| _ {\mathrm {s p}} \leq \| \mathbf {T} \| _ {\mathrm {F}}, \quad \text {a n d} \quad \| \underline {{\mathbf {W}}} * _ {M} \underline {{\mathbf {x}}} \| _ {\mathrm {F}} \leq \| \underline {{\mathbf {W}}} \| _ {\mathrm {s p}} \| \underline {{\mathbf {x}}} \| _ {\mathrm {F}} \leq \| \underline {{\mathbf {W}}} \| _ {\mathrm {F}} \| \underline {{\mathbf {x}}} \| _ {\mathrm {F}}.
$$
Proof. According to the definition of the  $M$  transform in Eq. (1) and the orthogonality of  $\mathbf{M}$ , we have
$$
\left\| \underline {{\mathbf {T}}} \right\| _ {\mathrm {s p}} = \left\| \widetilde {\mathbf {T}} _ {M} \right\| \leq \left\| \widetilde {\mathbf {T}} _ {M} \right\| _ {\mathrm {F}} = \| M (\underline {{\mathbf {T}}}) \| _ {\mathrm {F}} = \| \underline {{\mathbf {T}}} \| _ {\mathrm {F}},
$$
$$
\| \underline {{\mathbf {W}}} * _ {M} \underline {{\mathbf {x}}} \| _ {\mathrm {F}} = \left\| \widetilde {\mathbf {W}} _ {M} \cdot \widetilde {\mathbf {x}} _ {M} \right\| _ {\mathrm {F}} \overset {(i)} {\leq} \left\| \widetilde {\mathbf {W}} _ {M} \right\| \| \widetilde {\mathbf {x}} _ {M} \| _ {\mathrm {F}} = \| \mathbf {W} \| _ {\mathrm {s p}} \| \mathbf {x} \| _ {\mathrm {F}} \leq \| \mathbf {W} \| _ {\mathrm {F}} \| \mathbf {x} \| _ {\mathrm {F}},
$$
where inequality (i) holds because  $\| \mathbf{AB}\|_{\mathrm{F}}\leq \| \mathbf{A}\| \| \mathbf{B}\|_{\mathrm{F}}$  for any matrices  $\mathbf{A},\mathbf{B}$  with appropriate dimensions.
Lemma 38 (The t-product layer is Lipschitz continuous). Suppose the activation function is  $L_{\sigma}$ -Lipschitz, then a layer of t-product layer  $h(\underline{\mathbf{x}}) = \sigma (\underline{\mathbf{W}} *_M\underline{\mathbf{x}})$  is at most  $L_{\sigma}\| \underline{\mathbf{W}}\|_{\mathrm{F}}$ -Lipschitz.
Proof. According to the Lipschitzness of the activation function, we have
$$
\begin{array}{l} \left\| h \left(\underline {{\mathbf {x}}} _ {1}\right) - h \left(\underline {{\mathbf {x}}} _ {2}\right) \right\| _ {\mathrm {F}} = \left\| \sigma \left(\underline {{\mathbf {W}}} * _ {M} \underline {{\mathbf {x}}} _ {1}\right) - \sigma \left(\underline {{\mathbf {W}}} * _ {M} \underline {{\mathbf {x}}} _ {2}\right) \right\| _ {\mathrm {F}} \\ \leq L _ {\sigma} \left\| \mathbf {W} * _ {M} \mathbf {x} _ {1} - \mathbf {W} * _ {M} \mathbf {x} _ {2} \right\| _ {\mathrm {F}} \\ = L _ {\sigma} \left\| \underline {{\mathbf {W}}} * _ {M} \left(\underline {{\mathbf {x}}} _ {1} - \underline {{\mathbf {x}}} _ {2}\right) \right\| _ {\mathrm {F}} \\ = L _ {\sigma} \left\| \underline {{\mathbf {W}}} \right\| _ {\mathrm {F}} \left\| \underline {{\mathbf {x}}} _ {1} - \underline {{\mathbf {x}}} _ {2} \right\| _ {\mathrm {F}}. \\ \end{array}
$$
We then present Lemma 39 and Lemma 40 which are used in upper bounding the input and output of t-NNs in adversarial settings.
Lemma 39. Given a fixed example  $\underline{\mathbf{x}}\in \mathbb{R}^{d\times 1\times c}$ , if an adversary  $\underline{\mathbf{x}}^{\prime}$  satisfies  $R_{\mathrm{a}}(\underline{\mathbf{x}} -\underline{\mathbf{x}}^{\prime})\leq \xi$ , then it holds that
$$
\| \underline {{\mathbf {x}}} ^ {\prime} \| _ {\mathrm {F}} \leq B _ {x} + \xi \mathrm {C} _ {R _ {\mathrm {a}}}.
$$
Lemma 40. The  $L_{\infty}$ -norm of any  $f(\underline{\mathbf{x}};\underline{\mathbf{W}})\in \mathfrak{F}^{\mathrm{adv}}$  defined on the set of input examples  $\mathcal{X}$  is upper bounded by
$$
\sup  _ {\tilde {f} \in \mathfrak {F} ^ {\mathrm {a d v}}} \left\| \tilde {f} \right\| _ {L _ {\infty}} \leq B _ {\tilde {f}} := B _ {\underline {{\mathbf {W}}}} B _ {x, R _ {\mathrm {a}}, \xi}. \tag {71}
$$
The diameter of  $\mathfrak{F}^{\mathrm{adv}}$  is can be upper bounded as follows
$$
D _ {\tilde {f}} := 2 \sup  _ {\tilde {f} \in \mathfrak {F} ^ {\mathrm {a d v}}} \left\| \tilde {f} \right\| _ {S} \leq 2 B _ {\underline {{\mathbf {W}}}} B _ {x, R _ {\mathrm {a}}, \xi}. \tag {72}
$$
Proof. For any  $f \in \mathfrak{F}$ , given an example  $(\underline{\mathbf{x}}, y) \in \mathcal{X} \times \{\pm 1\}$ , let  $\underline{\mathbf{x}}^* \in \operatorname{arginf}_{R_{\mathrm{a}}(\underline{\mathbf{x}} - \underline{\mathbf{x}}') \leq \xi} yf(\underline{\mathbf{x}}')$  be one adversarial example. Then, we have
$$
\begin{array}{l} | \tilde {f} (\underline {{\mathbf {x}}}, y) | = | \inf  _ {R _ {\mathrm {a}} (\underline {{\mathbf {x}}} - \underline {{\mathbf {x}}} ^ {\prime}) \leq \xi} y _ {i} f (\underline {{\mathbf {x}}} ^ {\prime}) | \\ = \left| f \left(\underline {{\mathbf {x}}} ^ {*}\right) \right| \\ = \left| \mathbf {w} ^ {\top} \operatorname {v e c} \left(\mathbf {h} ^ {(L)} \left(\underline {{\mathbf {x}}} ^ {*}\right)\right) \right| \\ \leq \| \mathbf {w} \| \left\| \sigma (\underline {{\mathbf {W}}} ^ {(L)} * _ {M} \mathbf {h} ^ {(L - 1)} (\underline {{\mathbf {x}}} _ {i} ^ {*})) \right\| _ {\mathrm {F}} \\ = \| \mathbf {w} \| \left\| \sigma \left(\underline {{\mathbf {W}}} ^ {(L)} * _ {M} \mathbf {h} ^ {(L - 1)} \left(\underline {{\mathbf {x}}} ^ {*}\right)\right) - \sigma (\mathbf {0}) \right\| _ {\mathrm {F}} \\ \leq \| \mathbf {w} \| \left\| \underline {{\mathbf {W}}} ^ {(L)} * _ {M} \mathbf {h} ^ {(L - 1)} (\underline {{\mathbf {x}}} ^ {*}) \right\| _ {\mathrm {F}} \\ \leq \| \mathbf {w} \| \left\| \underline {{\mathbf {W}}} ^ {(L)} \right\| _ {\mathrm {F}} \left\| \mathbf {h} ^ {(L - 1)} \left(\underline {{\mathbf {x}}} ^ {*}\right) \right\| _ {\mathrm {F}} \\ \leq \dots \\ \leq \mathbf {w} \prod_ {l = 1} ^ {L} B _ {l} \left\| \mathbf {h} ^ {(0)} \left(\underline {{\mathbf {x}}} ^ {*}\right) \right\| _ {\mathrm {F}} \\ \leq B _ {\mathbf {w}} \prod_ {l = 1} ^ {L} B _ {l} B _ {x, R _ {\mathrm {a}}, \xi} \\ =: B _ {\tilde {f}}, \\ \end{array}
$$
$$
D _ {\tilde {f}} = 2 \max  _ {\tilde {f} \in \mathfrak {F} ^ {\mathrm {a d v}}} \left\| \tilde {f} \right\| _ {S} \leq 2 B _ {\mathbf {w}} \prod_ {l = 1} ^ {L} B _ {l} B _ {x, R _ {\mathrm {a}}, \xi}.
$$
Lemma 41 helps upper bounding the F-norm of residuals after low-tubal-rank approximation of the weight tensors of t-NNs under Assumption 13.
Lemma 41. Given constants  $a > 0, \alpha > 1$ , suppose a sequence  $\{z_j\}_{j=1}^{\infty}$  satisfying polynomial decay  $z_j \leq aj^{-\alpha}$ , then for any positive integer  $n$ , we have
$$
\sum_ {j > n} z _ {j} \leq \frac {a n ^ {1 - \alpha}}{\alpha - 1}.
$$
Proof. We compute the sum of the sequence using integration as follows:
$$
\sum_ {j > n} z _ {j} \leq \int_ {n} ^ {\infty} a t ^ {- \alpha} d t = \frac {a}{\alpha - 1} t ^ {1 - \alpha} | _ {n} ^ {\infty} \leq \frac {a n ^ {1 - \alpha}}{\alpha - 1}.
$$