SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \sigma \left(\mathbf {U} \mathbf {x} _ {1}\right) ^ {\top} \mathbf {z} _ {\mathbf {x} _ {2}} \leq \sigma \left(\mathbf {U} \mathbf {x} _ {1}\right) ^ {\top} \sigma \left(\mathbf {W} \mathbf {z} _ {\mathbf {x} _ {2}}\right) + \sigma \left(\mathbf {U} \mathbf {x} _ {1}\right) ^ {\top} \sigma \left(\mathbf {U} \mathbf {x} _ {2}\right) \tag {52} \\ \leq \mu^ {2} \sigma (\mathbf {U x} _ {1}) ^ {\top} \mathbf {z} _ {\mathbf {x} _ {2}} + \sigma (\mathbf {U x} _ {1}) ^ {\top} \sigma (\mathbf {U x} _ {2}) \\ \end{array}
$$
$$
\begin{array}{l} \sigma \left(\mathbf {U} \mathbf {x} _ {1}\right) ^ {\top} \mathbf {z} _ {\mathbf {x} _ {2}} \leq \frac {\sigma \left(\mathbf {U} \mathbf {x} _ {1}\right) ^ {\top} \sigma \left(\mathbf {U} \mathbf {x} _ {2}\right)}{1 - \mu^ {2}} \tag {53} \\ \mathbf {z} _ {\mathbf {x} _ {1}} ^ {*} \top \mathbf {z} _ {\mathbf {x} _ {2}} \leq \frac {\sigma (\mathbf {U x} _ {1}) ^ {\top} \sigma (\mathbf {U x} _ {2})}{(1 - \mu^ {2}) ^ {2}} \\ \end{array}
$$
Therefore, we can conclude
$$
\kappa_ {\text {o p t e q}} \left(\mathbf {x} _ {1}, \mathbf {x} _ {2}\right) \leq \frac {\mathbb {E} \left[ U _ {i} ^ {2} \right] \left\| \mathbf {x} _ {1} \right\| \left\| \mathbf {x} _ {2} \right\| \left(\sin \theta_ {0} + (\pi - \theta_ {0}) \cos \theta_ {0}\right)}{2 \pi (1 - \mu^ {2}) ^ {2}} \tag {54}
$$
For GEQ, we set  $\beta_{\mathbf{x}} = e^{-\gamma \| \mathbf{U}\mathbf{x} - \mathbf{W}\mathbf{z}\| _2^2}$  and  $\beta_{max}:= \max_{\mathbf{x}\in \mathcal{D}}e^{-\gamma \| \mathbf{U}\mathbf{x} - \mathbf{W}\mathbf{z}\| _2^2} < 1$
$$
\begin{array}{l} \mathbf {z} _ {\mathbf {x} _ {1}} ^ {* \top} \mathbf {z} _ {\mathbf {x} _ {2}} \leq \beta_ {\max } \sigma \left(\mathbf {W} _ {s} \mathbf {z} _ {\mathbf {x} _ {1}}\right) ^ {\top} \mathbf {z} _ {\mathbf {x} _ {2}} + \beta_ {\mathbf {x}} \| \mathbf {W} \| _ {2} \sigma \left(\mathbf {U} \mathbf {x} _ {1}\right) ^ {\top} \mathbf {z} _ {\mathbf {x} _ {2}} \tag {55} \\ \leq \beta_ {\max } \mu^ {2} \sigma \left(\mathbf {z} _ {\mathbf {x} _ {1}}\right) ^ {\top} \mathbf {z} _ {\mathbf {x} _ {2}} + \beta_ {\mathbf {x}} \mu \sigma \left(\mathbf {U} \mathbf {x} _ {1}\right) ^ {\top} \mathbf {z} _ {\mathbf {x} _ {2}} \\ \end{array}
$$
Like OptEqs, we obtain the following equations for GEQ:
$$
\mathbf {z} _ {\mathbf {x} _ {1}} ^ {* \top} \mathbf {z} _ {\mathbf {x} _ {2}} \leq \frac {\mu^ {2} \beta_ {\mathbf {x} _ {1}} \beta_ {\mathbf {x} _ {2}} \sigma \left(\mathbf {U} \mathbf {x} _ {1}\right) ^ {\top} \sigma \left(\mathbf {U} \mathbf {x} _ {2}\right)}{\left(1 - \beta_ {\max } \mu^ {2}\right) ^ {2}} \tag {56}
$$
$$
\kappa_ {g e q} \left(\mathbf {x} _ {1}, \mathbf {y} _ {2}\right) \leq \frac {\mu^ {2} \beta_ {\mathbf {x} _ {1}} \beta_ {\mathbf {x} _ {2}} \mathbb {E} \left[ U _ {i} ^ {2} \right] \| \mathbf {x} _ {1} \| _ {2} \| \mathbf {x} _ {2} \| _ {2} \left(\sin \theta_ {0} + (\pi - \theta_ {0}) \cos \theta_ {0}\right)}{2 \pi \left(1 - \beta_ {\max } \mu^ {2}\right) ^ {2}} \tag {57}
$$
And we have,
$$
\begin{array}{l} \beta_ {\mathbf {x} _ {1}} \beta_ {\mathbf {x} _ {2}} = e ^ {- \gamma (\| \mathbf {W z} _ {\mathbf {x} _ {1}} - \mathbf {U x} _ {1} \| _ {2} ^ {2} + \| \mathbf {W z} _ {\mathbf {x} _ {2}} - \mathbf {U x} _ {2} \| _ {2} ^ {2})} \\ \leq e ^ {- \gamma / 2 \left(\| \mathbf {W} \left(\mathbf {z} _ {\mathbf {x} _ {1}} - \mathbf {z} _ {\mathbf {x} _ {2}}\right) - \mathbf {U} \left(\mathbf {x} _ {1} - \mathbf {x} _ {2}\right) \| _ {2} ^ {2}\right)} \tag {58} \\ \leq D e ^ {- \frac {\gamma}{4} \left(\sigma_ {\min } (\mathbf {U}) ^ {2} \| \mathbf {x} _ {1} - \mathbf {x} _ {2} \| _ {2} ^ {2}\right)}, \\ \end{array}
$$
where  $D = e^{\gamma \| \mathbf{W}\| _2^2 B}$ , the latter two inequalities is acquired by Jensen's inequality. Then we have,
$$
\kappa_ {g e q} \left(\mathbf {x} _ {1}, \mathbf {x} _ {2}\right) \leq \frac {\mu^ {2} D e ^ {- \frac {\gamma}{4} \left(\sigma_ {\min } (\mathbf {U}) ^ {2} \| \mathbf {x} _ {1} - \mathbf {x} _ {2} \| _ {2} ^ {2}\right)} \mathbb {E} \left[ U _ {i} ^ {2} \right] \| \mathbf {x} _ {1} \| _ {2} \| \mathbf {x} _ {2} \| _ {2} \left(\sin \theta_ {0} + \left(\pi - \theta_ {0}\right) \cos \theta_ {0}\right)}{2 \pi \left(1 - \beta_ {\max } \mu^ {2}\right) ^ {2}} \tag {59}
$$
A.6 Experiment Settings
A.6.1 Experiments on CIFAR
For our GEQ, we parallel 6 branches with each branch taking the scale of 32, 16, 8, 8, 4, 4 and using the average fusion method for branches' fusion. The output channels for 6 branches are all 256 or 320 but the mid-channel number(output channel for weight  $\mathbf{U}$  and  $\mathbf{W}$ ) for the six branches are 64, 128, 128, 128, 256, 256 or 80, 160, 160, 160, 320, 320 with patch size 2 and c splitting is 8. And the inner MLP inner  $\mathbf{W}_h$  output 64 hidden dimension for each patch. We use the SGD [23] optimizer with momentum and step learning rate schedule for all the models. We also use RandomAug for all the models for comparison.
A.6.2 Experiments on ImageNette and ImageNet-100
We take the input scale as 256 for all models. For our GEQ, we parallel 6 branches with each branch taking the scale of 64, 32, 16, 16, 8, 8 after two downsampling convolutions and using the average fusion method for branches' fusion. The output channels for 6 branches are all 256 or 384 but the mid-channel number(output channel for weight  $\mathbf{U}$  and  $\mathbf{W}$ ) for the six branches are 32, 64, 128, 128, 256, 256 or 48, 96, 192, 192, 384, 384 with patch size 2 and c splitting is 8. And the inner MLP inner  $\mathbf{W}_h$  output 128 hidden dimension for each patch. We use the SGD optimizer with momentum and step learning rate schedule for all the models. We also use RandomAug for all the models for comparison.