SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We use subscript  $\cdot_{-U}$  to denote all but the indices in  $U$ . The equation  $Aw = r$  implies the following two equations:
$$
A _ {U} w = r _ {U} \quad A _ {- U} w = r _ {- U}.
$$
As  $A_U$  is has full rank  $(D - 2)$ ,  $A_U w = r_U$  admits a unique solution  $\mathbf{w}_U^* \neq 0$  — because  $r_U$  has at least one non-zero element by construction. Then, it must hold that
$$
A _ {- U} \mathbf {w} _ {U} ^ {*} = r _ {- U}. \tag {44}
$$
For any row  $v^{\top} \in A_{-U}$ , eq. (44) implies that  $v^{\top}\mathbf{w}^{*}$  equals a fixed constant. As  $v$  is a vector of i.i.d standard normal random variables,  $v^{\top}\mathbf{w}^{*}$  is a gaussian random variable with mean  $\sum (\mathbf{w}_i^*)$  and variance  $\| \mathbf{w}^{*}\|^{2}$ . Then with probability 1,  $v^{\top}\mathbf{w}^{*}$  will not equal a constant. Thus, w.p.1  $A_{-U}\mathbf{w}_U^* = r_{-U}$  is not satisfied, which means w.p.1 there are no solutions to  $A\mathbf{w} = r$ .
Case where  $(b - w_{y}^{*})\mathbf{1} - w_{z}^{*}Y\cdot Z$  is zero element-wise As  $X$  has rank  $D - 2$ $X_{\delta}\mathbf{w}_{\delta}^{*} = 0$  only when  $\mathbf{w}_{\delta}^{*} = 0$
Each element in  $(b - w_{y}^{*})\mathbf{1} - w_{z}^{*}Y\cdot Z$  is either  $b - w_{y}^{*} + w_{z}^{*}$  or  $b - w_{y}^{*} - w_{z}^{*}$ . Thus,
$$
\left(b - w _ {y} ^ {*}\right) \mathbf {1} - w _ {z} ^ {*} Y \cdot Z = 0 \quad \Longrightarrow \left\{ \begin{array}{l l} b - w _ {y} ^ {*} + w _ {z} ^ {*} = 0, \\ b - w _ {y} ^ {*} - w _ {z} ^ {*} = 0 \end{array} \right. \tag {45}
$$
Adding and subtracting the two equations on the right gives
$$
2 (b - w _ {y} ^ {*}) = 0 \qquad \text {a n d} \qquad 2 w _ {z} ^ {*} = 0.
$$
Thus,  $\mathbf{w}_{\delta}^{*} = 0, w_{z}^{*} = 0, b = w_{y}^{*}$ .
B Appendix: further experimental details and results
B.1 Default-ERM with  $\ell_2$ -regularization.
In section 3, we show default-ERM achieves zero training loss by using the shortcut to classify the shortcut group and noise to classify the leftover group, meaning the leftover group is overfit. The usual way to mitigate overfitting is via  $\ell_2$ -regularization, which, one can posit, may encourage models to rely on the perfect stable feature instead of the imperfect shortcut and noise.
We train the linear model from section 3 with default-ERM and  $\ell_2$ -regularization — implemented as weight decay in the AdamW optimizer [Loshchilov and Hutter, 2019] — on data from eq. (1) with
Figure 4: Default-ERM with  $\ell_2$ -regularization with a penalty coefficient of  $\lambda = 10^{-8}$  achieves a test accuracy of  $\approx 50\%$ , outperforming default-ERM. The right panel shows that  $\ell_2$ -regularization leads to lower test loss on the minority group, meaning that the regularization does mitigate some overfitting. However, the difference between the shortcut and leftover test losses shows that the model still relies on the shortcut.
Figure 5: Comparing log-loss with MARG-CTRL as functions of the margin. Each MARG-CTRL loss has a "bump" which characterizes the loss function's transition from a decreasing function of the margin to an increasing one. These bumps push models to have uniform margins because the loss function's derivative after the bump is negative which discourages large margins. The hyperparameters (temperature in  $\sigma$ -damp or function output target in MARG-LOG.) affect the location of the bump and the slopes of the function on either side of the bump.
$d = 800$ ,  $B = 10$ ,  $n = 1000$ . Figure 4 plots accuracy and losses for the  $\ell_2$ -regularized default-ERM with the penalty coefficient set to  $10^{-8}$ ; it shows that  $\ell_2$ -regularization leads default-ERM to build models that only achieve  $\approx 50\%$  test accuracy.
For smaller penalty coefficients, default-ERM performs similar to how it does without regularization, and for larger ones, the test accuracy gets worse than default-ERM without regularization. We give an intuitive reason for why larger  $\ell_2$  penalties may lead to larger reliance on the shortcut feature. Due to the scaling factor  $B = 10$  in the synthetic experiment, for a fixed norm budget, the model achieves lower loss when using the shortcut and noise compared to using the stable feature. In turn, heavy  $\ell_2$ -regularization forces the model to rely more on the shortcut to avoid the cost of larger weight needed by the model to rely on the stable feature and the noise.
B.2 Margin control (MARG-CTRL)
In fig. 5, we plot the different MARG-CTRL losses along with log-loss. Each MARG-CTRL loss has a "bump" which characterizes the loss function's transition from a decreasing function of the margin to an increasing one. These bumps push models to have uniform margins because the loss function's derivative after the bump is negative which discourages large margins. The hyperparameters — like temperature in  $\sigma$ -damp or function output target in MARG-LOG — affect the location of the bump and the slopes of the function on either side of the bump.
B.3 MARG-CTRL on a linear model
In fig. 6, we compare default-ERM to  $\sigma$ -stitch. In fig. 7 and fig. 8, compare SD and MARG-LOG respectively to default-ERM. The left panel of all figures shows that MARG-CTRL achieves better test accuracy than default-ERM, while the right most panel shows that the test loss is better on the leftover group using MARG-CTRL. Finally, the middle panel shows the effect of controlling margins
Figure 6: A linear trained with  $\sigma$ -stitch depend on the perfect stable feature to achieve perfect test accuracy, unlike default-ERM. The middle panel shows that  $\sigma$ -stitch does not let the loss on the training shortcut group to go to zero, unlike default-ERM, and the right panel shows the test leftover group loss is better.
Figure 7: A linear model trained with SD depend on the perfect stable feature to achieve perfect test accuracy whereas default-ERM performs worse than random chance. The middle panel shows that SD does not let the loss on the training shortcut group to go to zero, unlike vanilla default-ERM, and the right panel shows the test-loss is better for the leftover group.
in training; namely, the margins on the training data do not go to  $\infty$ , evidenced by the training loss being bounded away from 0. Depending on the shortcut feature leads to different margins and therefore test losses between the shortcut and leftover groups; the right panel in each plot shows that the test losses on both groups reach similar values, meaning MARG-CTRL mitigates dependence on the shortcut. While default-ERM fails to perform better than chance (50%) even after 100,000 epochs (see fig. 1), MARG-CTRL mitigates shortcut learning within 5000 epochs and achieves 100% test accuracy.
B.4 MARG-CTRL vs. default-ERM with a neural network
With  $d = 100$  and  $B = 10$  in eq. (1), we train a two layer neural network on 3000 samples from the training distribution. The two layer neural network has a 200 unit hidden layer that outputs a scalar. Figure 9 shows that a neural network trained via default-ERM fails to cross  $50\%$  test accuracy even after 40,000 epochs, while achieving less than  $10^{-10}$  in training loss.
Figure 8: A linear model trained with MARG-LOG depend on the perfect stable feature to achieve perfect test accuracy whereas default-ERM performs worse than random chance. The middle panel shows that MARG-LOG does not let the loss on the training shortcut group to go to zero, unlike default-ERM, and the right panel shows the test-loss is better for the leftover group.
(a) Average accuracy and loss curves.
(b) Accuracy and loss on shortcut and leftover groups.
Figure 9: Training a two-layer neural network with default-ERM on data from eq. (1). The model achieves  $100\%$  train accuracy but  $< 40\%$  test accuracy even after 40,000 epochs. The plot below zooms in on the first 4000 epochs and shows that the model drives down loss on the test shortcut groups but not on the test leftover group. This shows that the model uses the shortcut to classify the shortcut group and noise for the leftover.
In fig. 10, we compare default-ERM to  $\sigma$ -stitch. In fig. 12 and fig. 13, compare SD and MARG-LOG respectively to default-ERM. The left panel of all figures shows that MARG-CTRL achieves better test accuracy than default-ERM, while the right most panel shows that the test loss is better on the leftover group using MARG-CTRL. Finally, the middle panel shows the effect of controlling margins in training; namely, the margins on the training data do not go to  $\infty$ , evidenced by the training loss being bounded away from 0.
Figure 10: A neural network trained with  $\sigma$ -stitch depend on the perfect stable feature to achieve perfect test accuracy, unlike default-ERM. The middle panel shows that  $\sigma$ -stitch does not let the loss on the training shortcut group to go to zero, unlike default-ERM, and the right panel shows the test leftover group loss is better.
B.5 Spectral decoupling for a linear model on the linear DGP in eq. (1).
We first show that a linear classifier trained with SD achieves  $100\%$  test accuracy while default-ERM performs worse than chance on the test data; so, SD builds models with more dependence on the stable perfect feature, compared to Empirical Risk minimization (ERM). Next, we outline the assumptions for the gradient starvation (GS) regime from Pezeshki et al. [2021] and then instantiate it for a linear model under the data generating process in eq. (1), showing that the assumptions for the GS-regime are violated.
Figure 7 shows the results of training a linear model with SD on training data of size 1000 sampled as per eq. (1) from  $p_{\rho = 0.9}$  with  $d = 300$ ; the test data also has a 1000 samples but comes from  $p_{\rho = 0.1}$ .
Figure 11: A neural network trained with  $\sigma$ -damp depends on the perfect stable feature to achieve perfect test accuracy whereas default-ERM performs worse than random chance. The middle panel shows that  $\sigma$ -damp does not let the loss on the training shortcut group go to zero, unlike vanilla default-ERM, and the right panel shows the test-loss is better for the leftover group.
Figure 12: A neural network trained with SD depend on the perfect stable feature to achieve perfect test accuracy whereas default-ERM performs worse than random chance. The middle panel shows that SD does not let the loss on the training shortcut group to go to zero, unlike vanilla default-ERM, and the right panel shows the test-loss is better for the leftover group.
Figure 7 shows that SD builds models with improved dependence on the perfect stable feature, as compared to ERM, to achieve  $100\%$  test accuracy.
B.5.1 The linear example in Equation (1) violates the gradient starvation regime.
Background on Pezeshki et al. [2021]. With the aim of explaining why ERM-trained neural networks depend more on one feature over a more informative one, Pezeshki et al. [2021] derive solutions to  $\ell_2$ -regularized logistic regression in the NTK; they let the regularization coefficient be small enough for the regularized solution to be similar in direction to the unregularized solution. Given  $n$  samples  $\mathbf{y}^i, \mathbf{x}^i$ , let  $\mathbf{Y}$  be a diagonal matrix with the labels on its diagonal,  $\mathbf{X}$  be a matrix with  $\mathbf{x}^i$  as its rows, and  $\hat{\mathbf{y}}(\mathbf{X}, \theta) = f_{\theta}(\mathbf{X})$  be the  $n$ -dimensional vector of function outputs where each element is  $\hat{\mathbf{y}}^i = f_{\theta}(\mathbf{x}^i)$ . In gradient-based training in the NTK regime, the vector of function outputs of the network with parameters  $\theta$  can be approximated as  $\hat{\mathbf{y}} = \Phi_0 \theta$ , where  $\Phi_0$  is the neural-tangent-random-feature (NTRF) matrix at initialization:
$$
\Phi_ {0} = \frac {\partial \hat {\mathbf {y}} (\mathbf {X} , \theta_ {0})}{\partial \theta_ {0}}
$$
To define the features, the strength (margin) of each feature, and how features appear in each sample, Pezeshki et al. [2021] compute the singular value decomposition (SVD) of the NTRF  $\Phi_0$  multiplied by the diagonal-label matrix  $\mathbf{Y}$ :
$$
\mathbf {Y} \Phi_ {0} = \mathbf {U S V} ^ {\top}. \tag {46}
$$
The rows of  $\mathbf{V}$  are features, the diagonal elements of  $\mathbf{S}$  are the strengths of each feature and the  $i$ th row of  $\mathbf{U}$  denotes how each feature appears in the NTRF representation of the  $i$ th sample.
To study issues with the solution to  $\ell_2$ -regularized logistic regression, Pezeshki et al. [2021] define the gradient starvation (GS) regime. Under the GS regime, they assume  $\mathbf{U}$  is a perturbed identity matrix that is also unitary: for a small constant  $\delta < 1$ , such a matrix has all diagonal elements  $\sqrt{1 - \delta^2}$  and the rest of the elements are of the order  $\delta$  such that the rows have unit  $\ell_2$ -norm.
Figure 13: A neural network trained with MARG-LOG depend on the perfect stable feature to achieve perfect test accuracy whereas default-ERM performs worse than random chance. The middle panel shows that MARG-LOG does not let the loss on the training shortcut group to go to zero, unlike default-ERM, and the right panel shows the test-loss is better for the leftover group.
The GS regime is violated in eq. (1). When  $f_{\theta}$  is linear,  $f_{\theta}(\mathbf{x}) = \theta^{\top}\mathbf{x}$ , the NTRF matrix is
$$
\frac {\partial \hat {\mathbf {y}} (\mathbf {X} , \boldsymbol {\theta} _ {0})}{\partial \boldsymbol {\theta} _ {0}} = \frac {\partial \mathbf {X} \boldsymbol {\theta} _ {0}}{\partial \boldsymbol {\theta} _ {0}} = \mathbf {X}.
$$
In this case, let us look at an implication of  $\mathbf{U}$  being a perturbed identity matrix that is also unitary, as Pezeshki et al. [2021] assume. With  $(\mathbf{u}^i)^\top$  as the  $i$ th row of  $\mathbf{U}$ , the transpose of  $i$ th sample can be written as  $(\mathbf{x}^i)^\top = (\mathbf{u}^i)^\top \mathbf{S}\mathbf{V}$ . Pezeshki et al. [2021] assume that  $\delta < 1$  in that the off-diagonal terms of  $\mathbf{U}$  are small perturbations such that off-diagonal terms of  $\mathbf{U}(\mathbf{S}^2 + \lambda \mathbf{I})\mathbf{U}^\top$  have magnitude much smaller than 1, meaning that the terms  $|(\mathbf{u}^i)^\top \mathbf{S}^2 (\mathbf{u}^j) + \lambda | < 1$  for  $i \neq j$  and positive and small  $\lambda < 1$ .
$$
\begin{array}{l} \left| \mathbf {y} ^ {i} \mathbf {y} ^ {j} \left(\mathbf {x} ^ {i}\right) ^ {\top} \mathbf {x} ^ {j} \right| = \left| \left(\mathbf {x} ^ {i}\right) ^ {\top} \mathbf {x} ^ {j} \right| (47) \\ = \left| \left(\mathbf {u} ^ {i}\right) ^ {\top} \mathbf {S V} ^ {\top} \mathbf {V S u} ^ {j} \right| (48) \\ = \left| \left(\mathbf {u} ^ {i}\right) ^ {\top} \mathbf {S} ^ {2} \mathbf {u} ^ {j} \right| (49) \\ <   <   1 (50) \\ \end{array}
$$
In words, this means that any two samples  $\mathbf{x}^i, \mathbf{x}^j$  are nearly orthogonal. Now, for samples from eq. (1), for any  $i, j$  such that  $\mathbf{z}^j = \mathbf{z}^i$  and  $\mathbf{y}^i = \mathbf{y}^j$ ,
$$
\left| \left(\mathbf {x} ^ {i}\right) ^ {\top} \mathbf {x} ^ {j} \right| = \left| B ^ {2} \mathbf {z} ^ {i} \mathbf {z} ^ {j} + \mathbf {y} ^ {i} \mathbf {y} ^ {j} + \left(\boldsymbol {\delta} ^ {i}\right) ^ {\top} \boldsymbol {\delta} ^ {j} \right| \geq \left| 1 0 0 + 1 + \left(\boldsymbol {\delta} ^ {i}\right) ^ {\top} \boldsymbol {\delta} ^ {j} \right| \tag {51}
$$
As  $\delta$  are isotropic Gaussian vectors, around half the pairs  $i,j$  will have  $(\delta^i)^\top \delta^j > 0$  meaning  $\left|\left(\mathbf{x}^i\right)^\top \mathbf{x}^j\right| > 101$ . This lower bound implies that  $\mathbf{U}$  is not a perturbed identity matrix for samples from eq. (1). This violates the setup of the gradient starvation regime from [Pezeshki et al., 2021].
Thus, the linear DGP in eq. (1) does not satisfy the conditions for the GS regime that is proposed in [Pezeshki et al., 2021]. The GS regime blames the coupled learning dynamics for the different features as the cause for default-ERM-trained models depending more on the less informative feature. Pezeshki et al. [2021] derive spectral decoupling (SD) to avoid coupling the training dynamics, which in turn can improve a model's dependence on the perfect feature. SD adds a penalty to the function outputs which Pezeshki et al. [2021] show decouples training dynamics for the different features as defined by the NTRF matrix:
$$
\ell_ {\mathrm {S D}} (\mathbf {y}, f _ {\theta} (\mathbf {x})) = \log (1 + \exp (\mathbf {y} f _ {\theta})) + \lambda | f _ {\theta} (\mathbf {x}) | ^ {2}
$$
As eq. (1) lies outside the GS regime, the success of SD on data from eq. (1) cannot be explained as a consequence of avoiding the coupled training dynamics in the GS regime Pezeshki et al. [2021]. However, looking at SD as MARG-CTRL, the success of SD, as in fig. 7, is explained as a consequence encouraging uniform margins.
An example of perfect test accuracy even with dependence on the shortcut. In fig. 14, we train a linear model with default-ERM on data from eq. (1), showing that even when shortcut dependence
Figure 14: With  $d = 200$  and  $n = 1000$ , a linear classifier can still depend on the shortcut feature and achieve  $100\%$  test accuracy. Nagarajan et al. [2021] consider linearly separable data and formalize geometric properties of the data that make max-margin classifiers give non-zero weight to the shortcut feature  $(\mathbf{w}_z > 0)$ . In their example, it is unclear when  $\mathbf{w}_z > 0$  leads to poor accuracy in the leftover group because Nagarajan et al. [2021] do not separate the model's dependence on the stable feature from the dependence on noise. The example here gives an example where  $\mathbf{w}_z > 0$  but test accuracy is  $100\%$ . demonstrating that guarantees on test leftover group error require comparing  $\mathbf{w}_y$  and  $\mathbf{w}_z$ ; the condition  $\mathbf{w}_z > 0$  alone is insufficient.
is non-zero, test leftover group accuracy can be  $100\%$ . Nagarajan et al. [2021] consider linearly separable data and formalize geometric properties of the data that make max-margin classifiers give non-zero weight to the shortcut feature  $(\mathbf{w}_z > 0)$ . In their example, it is unclear when  $\mathbf{w}_z > 0$  leads to poor accuracy in the leftover group because Nagarajan et al. [2021] do not separate the model's dependence on the stable feature from the dependence on noise. The example in fig. 14 gives an example where  $\mathbf{w}_z > 0$  but test accuracy is  $100\%$ , demonstrating that guarantees on test leftover group error require comparing  $\mathbf{w}_y$  and  $\mathbf{w}_z$ ; the condition  $\mathbf{w}_z > 0$  alone is insufficient. In contrast, theorem 1 characterizes cases where leftover group accuracy is worse than random even without overparameterization.
B.6 Experimental details
B.6.1 Background on Just Train Twice (JTT) and Correct-n-Contrast (CNC)
JTT Liu et al. [2021] develop JTT with the aim of building models robust to subgroup shift, where the mass of disjoint subgroups of the data changes between training and test times. To work without training group annotations, JTT assumes ERM builds models with high worst-group error. With this assumption, JTT first builds an "identification" model via ERM to pick out samples that are misclassified due to model's dependence on the shortcut. Then, JTT trains a second model again via ERM on the same training data with the loss for the misclassified samples upweighted (by constant  $\lambda$ ). As Liu et al. [2021] point out, the number of epochs to train the identification model and the weighting constant are hyperparameters that require tuning using group annotations. As Liu et al. [2021], Zhang et al. [2022] show that JTT and CNC outperforms LFF and other two-stage shortcut-mitigating methods ([Zhang et al., 2022]), so we do not compare against them.
Correct-n-Contrast (CNC) In a fashion similar to JTT, the first stage of CNC is to train a model with regularized ERM to predict based on spurious attributes, i.e. shortcut features. Zhang et al. [2022] develop a contrastive loss to force the model to have similar representations across samples that share a label but come from different groups (approximately inferred by the first-stage ERM model). Formally, the first-stage model is used to approximate the spurious attributes in one of two ways: 1) predict the label with the model, 2) cluster the representations into as many clusters as there are classes, and then use the cluster identity. The latter technique was first proposed in [Sohoni et al., 2020]. For an anchor sample  $(\mathbf{y}^i,\mathbf{x}^i)$  of label  $\mathbf{y} = y$ , positive samples  $P_{i}$  are those that have the same label but have the predicted spurious attribute is a different value:  $\hat{z}\neq y$ . Negatives  $N_{i}$  are those that have a different label but the spurious attribution is the same:  $\hat{z} = y$ . For a temperature parameter  $\tau$  and representation function  $r_\theta$ , the per-sample contrastive loss for CNC is:
$$
\ell_ {c o n t} (r _ {\theta}, i) = \mathbb {E} _ {\mathbf {x} ^ {p} \sim P _ {i}} \left[ - \log \frac {\exp \big (r _ {\theta} (\mathbf {x} ^ {i}) ^ {\top} r _ {\theta} (\mathbf {x} ^ {p}) / \tau \big)}{\sum_ {n \in N _ {i}} \exp \big (r _ {\theta} (\mathbf {x} ^ {i}) ^ {\top} r _ {\theta} (\mathbf {x} ^ {n}) / \tau \big) + \sum_ {p \in P _ {i}} \exp \big (r _ {\theta} (\mathbf {x} ^ {i}) ^ {\top} r _ {\theta} (\mathbf {x} ^ {p}) / \tau \big)} \right].
$$
The samples  $i$  are called anchors. For a scalar  $\lambda$  to trade off between contrastive and predictive loss, the overall per-sample loss in the second-stage in CNC is
$$
\lambda \ell_ {c o n t} (r _ {\theta}, i) + (1 - \lambda) \ell_ {l o g - l o s s} (\mathbf {y} ^ {i} w ^ {\top} r _ {\theta} (\mathbf {x} ^ {i})).
$$
CNC uses hyperparameters informed by dataset-specific empirical results from prior work. The original implementation of CNC from Zhang et al. [2022] uses specific values of first-stage hyperparameters like weight decay and early stopping epoch for each dataset by using empirical results from prior work [Sagawa et al., 2020a, Liu et al., 2021]. The prior work finds weight-decay and early stopping epoch which lead default-ERM models to achieve low test worst-group accuracy, implying that the model depends on the spurious attribute. This means the first-stage models built in CNC are pre-selected to pay attention to the spurious attributes. For example, [Zhang et al., 2022] point out that the first-stage model they use for Waterbirds predicts the spurious feature with an accuracy of  $94.7\%$ .
Without using dataset-specific empirical results from prior work, choosing LR and WD requires validating through the whole CNC procedure. We let CNC use the same LR and WD for both stages and then validate the choice using validation performance of the second-stage model. This choice of hyperparameter validation leads to a similar number of validation queries for all methods that mitigate shortcuts.
B.6.2 Training details
Variants of MARG-CTRL to handle label imbalance. The three datasets that we use in our experiments — Waterbirds, CelebA, and Civilcomments — all have an imbalanced (non-uniform) marginal distribution over the label; for each dataset,
$$
\max_{\substack{\text{class}\in \{-1,1\}}}p(\mathbf{y} = \text{class}) > 0.75.
$$
When there is sufficiently large imbalance, restricting the margins on all samples could bias the training to reduce loss on samples in the most-frequent class first and overfit on the rest of the samples. This could force a model to predict the most frequent class for all samples, resulting in high worst-group error.
To prevent such a failure mode, we follow [Pezeshki et al., 2021] and define variants of  $\sigma$ -damp, MARG-LOG, and  $\sigma$ -stitch that have either 1) different maximum margins for different classes or 2) different per-class loss values for the same margin value. Mechanically, these variants encourage uniform margins within each class, thus encouraging the model to rely less on the shortcut feature. We give the variants here for labels taking values in  $\{-1, 1\}$ :
1. With per-class temperatures  $T_{-1}$ ,  $T_{1} > 0$  the variant of  $\sigma$ -damp is
with  $f_{\theta} = w_{f}^{\top}r_{\theta}(\mathbf{x})$