SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 4: LtrRNN separates learning- from condition-related variability in motor neural data. a. We apply ltrRNN to recordings of the motor and premotor cortex during learning of a perturbed reach task. (Schematic adapted from [16]). b. We hold out for testing blocks of trials and time points (100 ms by 50 trials). This reduces temporal correlations between the train and test sets as compared to holding out individual tensor entries. c. State of the ltrRNN ( $R = 5$ ) 100 ms after the go cue. LtrRNN captures the topology of the task. Furthermore, different initial states seem to emerge during the perturbation period. d. PCA directly applied to the neural data. e. Activity of the ltrRNN projected on the column vectors of the first three components. f. Vector field along pairs of column vectors. We constrain the activity to the columns of the tensor such that  $\mathbf{x} = q_r \cdot \mathbf{a}_r + q_{r'} \cdot \mathbf{a}_{r'}$  for  $q_r, q_{r'} \in [-2, 2]$  and we compute the vector field  $\mathbf{a}_r(\phi(\mathbf{x}) \cdot \mathbf{b}_r) + \mathbf{a}_{r'}(\phi(\mathbf{x}) \cdot \mathbf{b}_{r'})$ . g. Eigenspectrum of  $W^{(k)}$ . Saturation gradient indicates trial. h. The trial factors  $\mathbf{c}_r$  can be seen as a latent variable of learning, such that they describe the evolution of the weights in a low-dimensional subspace of the weight space spanned by  $\mathbf{a}_r \otimes \mathbf{b}_r$ . i. Trial factors. The black line is computed using  $\sigma = 0$ . j. LtrRNN activity projected onto each column factor (ordered as in i). In particular the activity along the top two components seem to be more specific to reach angle.
5 Low tensor rank learning in task-trained RNNs
We next decided to test the low-tensor-rank learning hypothesis in a model in which we had direct access to the ground truth weight tensor. Towards this end, we trained an RNN with unconstrained weight structure to perform the same motor task.
Task-trained RNN model. At any given trial  $k$ , the RNN linearly controls the force applied to the hand:
$$
\ddot {\mathbf {y}} ^ {(k)} = D \phi (\mathbf {x} ^ {(k)} (t)) + \boldsymbol {\kappa} ^ {(k)} (t)
$$
where  $\kappa^{(k)}(t) \in \mathbb{R}^2$  is an Ornstein-Uhlenbeck process representing noise in the execution of movement [36]. To create a purely ballistic model, we provide the RNN the target position and a hold cue. For the objective we use the integrated hand to target position so that the RNN simply has to push the hand as fast as possible towards the target (Fig. 5a).
After first pre-training the RNN to move the hand to the goal position at the go onset, we then probe motor learning following the same protocol as the neural data. Specifically, the RNN first performs 50 trials in the baseline condition, after which we introduce a force perturbation orthogonal and proportional to the velocity of the hand for 100 trials (Fig. 5b):
$$
\ddot {\mathbf {y}} ^ {(k)} = D \phi (\mathbf {x} ^ {(k)}) (t) + c R \dot {\mathbf {y}} ^ {(k)} (t) + \boldsymbol {\kappa} ^ {(k)} (t)
$$
where  $R \in \mathbb{R}^{2 \times 2}$  is a  $90^{\circ}$  clockwise rotation matrix, and  $c \in \mathbb{R}$  is the coefficient of perturbation. Finally, the RNN performs the original unperturbed task for another 100 trials (washout). Throughout the perturbation and washout periods, the weights are updated following SGD.
b
c
Perturbed - late
g
Figure 5: Learning-induced weight changes in task-trained RNNs are low tensor rank. a. RNN model. b. Average MSE between hand and target positions integrated throughout the trial. c. Left: Variance explained of the weights resulting from pre-training  $W^{*}$ . The original task training results in a rank-3 RNN. Right: Variance explained of the tensor of updates  $\mathbf{W} - W^{*} \otimes \mathbf{1}$  due to retraining. The retraining procedure results in a rank-2 tensor. We further found that the subspaces spanned by the pre-training columns (resp. rows) and retraining columns (resp. rows) were different, suggesting a tensor of rank at most 5. d. Hand movements during various periods of learning, where "early" and "late" describe respectively the first and last trial during which the perturbation is introduced or removed. e. Using the same cross-validation as in Fig. 4 uncovers the low tensor rank structure. f. Eigenspectrum of the weights  $W^{(k)}$  over learning. Top: Ground truth weights of the task-trained RNN. Bottom: Weights of the ltrRNN fit to the task-trained RNN activity. Imaginary eigenvalues emerge to counter the rotational effect of the perturbation, and are uncovered by ltrRNN. g. Activity projected on PCs. The rotational activity during perturbed trials is uncovered by ltrRNN. h. Example of trial factor uncovered by ltrRNN which correlates with learning at the level of the behaviour.
We next analyzed how the structure of the weight matrix changed over the baseline, perturbation, and washout blocks. The change in weights as a result of the pre-training  $W^{*} - W_{0}$ , where  $W_{0}$  is the random initialization, and  $W^{*}$  the weights after pre-training, is of matrix rank 3 (Fig. 5c, left). Since the weight updates resulting from learning the perturbation are small (consistent with the literature [17]), we cannot simply apply PARAFAC on the weights recorded during perturbation learning, as only the 3 rank-1 terms from pre-training will be visible. However, running PARAFAC on the updates  $\mathbf{W} - W^{*} \otimes \mathbf{1}$  (where  $\mathbf{1}$  is the vector of ones), reveals tensor rank 2 updates (Fig. 5c, right). Furthermore, the subspace spanned by the columns (or rows) resulting from pretraining is different than those resulting from perturbation learning (Supplementary Material C).
LtrRNN application. We next ask whether ltrRNN can uncover this low-tensor-rank structure in the weights, especially the small changes due to perturbation learning. Using our cross-validation procedure, we find that the variance of the activity of the task-trained RNN that is unexplained by the fitted ltrRNN plateaus at tensor rank 5 (Fig. 5e), consistent with our analysis of the ground truth weights. Furthermore, the weights of the ltrRNN share similar spectral properties, and a similar evolution over learning, compared to ground truth (Fig. 5f). In particular, in both cases, imaginary eigenvalues, corresponding to rotational activity, emerge over learning, consistent with the behaviour being rotational post-learning to counter the force field (5d). The emergence of rotational activity is also visible at the level of the neural activity (Fig. 5g).
Therefore, consistent with the results found through ltrRNN fit to neural data, we found that the weight updates in an RNN trained on a perturbed ballistic reach task had low-tensor-rank structure. Furthermore, ltrRNN was able to uncover this structure in the weights of the task-trained RNN, and its evolution over learning.
6 Gradient-based learning constrains the tensor rank of weight updates
We have so far observed that learning leads to low-tensor-rank weight changes in both biological data and task-trained RNNs. To gain deeper insight into why this is the case, we next present a set of mathematical results regarding the tensor rank of gradient-based learning in RNNs. Towards this end, we use the method of the adjoint [37, 38] to derive a dynamical system whose solution is the gradient of a loss functional with respect to the RNN weights [39].
The adjoint. For a dynamical system  $\dot{\mathbf{x}} = f(\mathbf{x},\pmb {\theta})$  , the state adjoint of the loss functional  $L:\mathbb{R}^n\to \mathbb{R}$  at a particular time point  $t$  is defined as  $\mathbf{a}_{\mathbf{x}}(t) = \frac{dL(\mathbf{x}(T))}{d\mathbf{x}(t)}$  where  $T$  is the time of loss evaluation (but see Supplementary Material E for the case of a loss functional that is integrated over time). From the dynamics of  $\mathbf{x}$  , the state adjoint dynamics can be derived:
$$
\dot {\mathbf {a}} _ {\mathbf {x}} (t) = \left(\frac {d f (\mathbf {x} (t))}{d \mathbf {x} (t)}\right) ^ {T} \mathbf {a} _ {\mathbf {x}} (t)
$$
However, to understand gradient-based learning, we require the parameter adjoint:  $\mathbf{a}_{\pmb{\theta}}(t) = \frac{dL(\mathbf{x}(T))}{d\pmb{\theta}(t)}$ . This can be accomplished by concatenating the original dynamical system by its parameters  $\mathbf{z}(t) = [\mathbf{x}(t),\pmb{\theta}(t)]$  (noting that  $\dot{\pmb{\theta}}(t) = 0$ ) to define the augmented adjoint  $\mathbf{a}_{\mathbf{z}}(t) = [\mathbf{a}_{\mathbf{x}}(t),\mathbf{a}_{\pmb{\theta}}(t)]$ , whose dynamics can be shown to follow
$$
\dot {\mathbf {a}} _ {\mathbf {z}} (t) = \left[ \left(\frac {d f (\mathbf {x} (t))}{\mathbf {x} (t)}\right) ^ {T} \mathbf {a} _ {\mathbf {x}} (t), \left(\frac {d f (\mathbf {x} (t))}{d \boldsymbol {\theta}}\right) ^ {T} \mathbf {a} _ {\mathbf {x}} (t) \right]
$$
with terminal condition  $\mathbf{a}_z(T) = \left[\frac{dL(\mathbf{x}(T))}{d\mathbf{x}(T)},0\right]$ . The gradient of the loss with respect to the parameters can then be found by integrating the parameter adjoint dynamics backwards through time to get  $\mathbf{a}_{\theta}(0)$ . In Supplementary Material E we provide a short derivation of the adjoint adapted from [38].
Main results. While the adjoint method is extensively used as a numerical tool in autodifferentiation packages [38, 40], it can also provide analytical insight into the gradient dynamics of dynamical systems. Towards this end, we now return to the case of an RNN, with the parameter of interest being the weight matrix. We demonstrate several mathematical results with the aid of the adjoint.
Lemma 1. Consider the RNN  $\dot{\mathbf{x}} = W\phi (\mathbf{x}) - \mathbf{x} + B\mathbf{u}(t)$ , where  $\mathbf{x}(t)\in \mathbb{R}^n$ ,  $\mathbf{u}(t)\in \mathbb{R}^m$ ,  $W\in \mathbb{R}^{n\times n}$ ,  $B\in \mathbb{R}^{n\times m}$ , and  $\phi :\mathbb{R}^n\to \mathbb{R}^n$  an element-wise nonlinearity, and define a loss functional of the linearly decoded RNN state,  $L(D\phi (\mathbf{x}(T));\mathbf{y}(T))$  for  $\mathbf{y}\in \mathbb{R}^d$ . Furthermore, let  $W = \sum_{r}^{R}\pmb{\alpha}_{r}\otimes \pmb{\beta}_{r}$ . The adjoint dynamics are then given by,
$$
\dot {\mathbf {a}} _ {\mathbf {x}} = \left(\sum_ {r} ^ {R} \boldsymbol {\alpha} _ {r} \otimes (\boldsymbol {\beta} _ {r} \odot \phi^ {\prime} (\mathbf {x} (t)))\right) ^ {T} \mathbf {a} _ {\mathbf {x}} - \mathbf {a} _ {\mathbf {x}}, \qquad \dot {\mathbf {a}} _ {W} = \mathbf {a} _ {\mathbf {x}} \otimes \phi (\mathbf {x} (t))
$$
with terminal conditions  $\mathbf{a}_{\mathbf{x}}(T) = \phi^{\prime}(\mathbf{x}(T))\odot \sum_{r^{\prime}}^{d}L^{\prime}(D_{r^{\prime}}\cdot \phi (\mathbf{x}(T));\mathbf{y}_{r^{\prime}})D_{r^{\prime}}$  and  $\mathbf{a}_W(T) = 0$
2Where  $\odot$  denotes the Hadamard product and  $\phi^\prime$  the element-wise derivative of  $\phi$
From the adjoint dynamics, it can be seen that the singular values of the gradient of the loss with respect to the weights of the RNN  $(\nabla_W L = \mathbf{a}_W(0))$  depend on the dimensionality of the subspaces over which  $\phi(\mathbf{x}(t))$  and  $\mathbf{a}_{\mathbf{x}}$  evolve. This can be used to show that the gradient's singular values are bounded by the singular values of both the activity and the adjoint, which we formalize in the following theorem.
Theorem 1. Consider an RNN be defined as in Lemma 1. Then the singular values of its gradient can be bounded as:
$$
\max \left\{\sigma_ {n} ^ {\phi (\mathbf {x})} \sigma_ {r} ^ {\mathbf {a _ {x}}}, \sigma_ {n} ^ {\mathbf {a _ {x}}} \sigma_ {r} ^ {\phi (\mathbf {x})} \right\} \leq \sigma_ {r} ^ {\nabla_ {W} L} \leq \min \left\{\sigma_ {1} ^ {\phi (\mathbf {x})} \sigma_ {r} ^ {\mathbf {a _ {x}}}, \sigma_ {1} ^ {\mathbf {a _ {x}}} \sigma_ {r} ^ {\phi (\mathbf {x})} \right\},
$$
where  $\sigma_r^{\mathbf{v}}$  denotes the  $r$ th singular value of  $\mathbf{v}$ .
This theorem demonstrates that there is a natural bound to the numerical rank [41] of gradient-based learning in RNNs. Numerically, we find that the singular value spectrum of both  $\phi (\mathbf{x})$  and  $\mathbf{a}_{\mathbf{x}}$  tend to decay exponentially, even in the case of a chaotic RNN (Fig. 6). In Supplementary Material E we repeat the same analysis with various activation functions, initial weight variances and ranks: overall, we find that smooth activation functions (such as the most commonly used tanh) tend to lead to the fastest decaying singular value spectrum of the adjoint.
So far, we haven't made any assumption on the architecture of the RNN such as constraints on  $W$  or  $\phi$ . Under such constraints, stronger and more explicit bounds on both the matrix and tensor ranks can be obtained.
Theorem 2. Consider an RNN defined as in Lemma 1 with  $\phi = id$  and  $W^{(0)}$  of rank  $R$ . Furthermore suppose  $x^{(k)}(0)$  is constrained to the subspace spanned by the columns of  $W^{(k)}$  and  $B$ . Consider the weight tensor  $\mathbf{W} = [W^{(0)}, W^{(1)}, \ldots]$  where  $W^{(k+1)} = W^{(0)} + \alpha \sum_{j=1}^{k} \nabla_W L(D\mathbf{x}^{(j)}(T); \mathbf{y}^{(j)})$ , with  $\mathbf{y}^{(j)} \in \mathbb{R}^d$  where  $\mathbf{x}^{(j)}$  denotes the activity of the RNN after the  $j$ th weight update. Then,
1. The rank of the gradient at the first step is at most  $\mathrm{rank}(\nabla_W L^{(0)}) \leq R + 1$ .
2. The rank of the trial slices of the weight tensor is at most  $\mathrm{rank}(W^{(k)})\leq 2R + m + d$
3. The tensor rank of the weight tensor is at most  $\mathrm{rank}(\mathbf{W})\leq (2R + m + d)^2$
We note that these bounds are tight, in the sense that there exists networks for which they are equalities; therefore, they cannot be improved upon without restricting the set of architectures considered (e.g. to normal weight matrices). We also point out that they are non-trivial as the current best upper bound on the max rank of a  $(\mathbb{R}^n)^{\otimes 3}$  tensor is  $n^2 - n - 1$  [42]. In particular, in the case of  $W^{(0)} = 0$  and  $m = d = 1$  considered by [14], the tensor rank is at most 4. For arbitrary weight initializations, the matrix and tensor mathematical ranks can be high. Nevertheless, in most cases the matrix and tensor numerical ranks will fall vastly below these bounds due to Theorem 1.
We illustrate this result with  $R = 3$ ,  $d = 2$ ,  $m = 2$  in Fig. 6. We find that the true ranks fall within these bounds – strictly below due to limited machine precision – and that the numerical ranks are extremely low. Intuitively, decoding the firing rate of the RNN into a low-dimensional space produces weight updates that push the RNN and adjoint activity to lie in a low-dimensional subspace of the state space. This, in turn, further pushes weight updates to lie in a low-dimensional subspace of the weight space.
The proofs of Lemma 1 and Theorems 1 and 2 are provided in Supplementary Material E. We also discuss two additional cases, namely that of a loss integrated over some period of time, and the gradient of the loss w.r.t. to other parameters of the RNN. Finally, we show that momentum-based optimization methods such as ADAM have the same property. Together, these analytical results provide insight as to why gradient dynamics bound the matrix rank of the weight updates in generic RNNs, as well as the tensor rank of learning dynamics in the linear case.
Summary. In the present work, we explored the tensor rank of learning in artificial and biological neural networks. We showed that learning leads to low-tensor-rank weight updates, which can be exploited to uncover smooth changes in dynamics along with a principled choice of low-dimensional
Figure 6: The singular value spectrum of the gradient of random RNNs with random objective. Top: Full rank RNN with tanh activation function in a chaotic regime. Bottom: Low-rank  $(R = 3)$  linear RNN. In both cases  $m = d = 2$ . The networks are trained to map a time-varying input (parameterized as an LDS) to a given target output. Error shade represents the standard deviation over 5 random initialization. a. Loss over training. b. Example of network activity at one iteration. Inset: activity of two neurons over learning, red is post-learning. c. Singular value spectrum of the activity and of the adjoint. d. Singular value spectrum of the weights, gradient, and bound we derive. d. Tensor rank of the weight tensor minus initial weights. Variance is computed per trial slice.
factors over which weights and activity evolve throughout the entire course of learning. Finally, we derived upper bounds on the singular values of gradient dynamics of nonlinear RNNs, and on the matrix and tensor ranks in the linear case.
Modelling limitations and future work. Inferring connectivity from neural recordings is in general an ill-posed problem [19]: given any observed pattern of activity, it is always possible that the data could be explained entirely in terms of a external input to a set of unconnected neurons. In our application to neural data, we resolved such ambiguities by assuming that inputs were fixed on each trial of a given condition, forcing the changes in neural activity across trials of the same condition to be captured by (smoothly varying) changes in weights. We note that, in principle, our framework and code could also implement residual trial-to-trial variability in the inputs.
In addition to learning- and condition-specific changes, neural recordings show substantial variability across consecutive trials, which are thought to reflect a combination of i) unmeasured covariates such as behavioural or intrinsic state, ii) stochasticity in the neural system itself, and iii) changes in the initial state at trial onset. Future work could incorporate behavioural covariates, allow for trial-specific initial states, and model the data using a stochastic dynamical system within each trial.
Here, we focused on gradient-based learning, motivated by recent evidence that it is able to explain many features of motor learning in neural data [17, 18], as well as by more general support for an optimization-based framework to understand neural learning [43]. The tensor rank of learning in RNNs with local synaptic rules (e.g., Hebbian) remains an open question. Towards this end, theoretical work has established links between Hebbian and gradient-based learning [44, 45], opening the possibility of an extension of our mathematical results to biologically plausible learning rules.
Our analytical results provide intuition for our observation that the rank of learning dynamics is limited by task complexity. This supports previous findings of low (matrix) rank weight changes in RNNs [14] and in deep networks [46, 47, 48]. Our results on the numerical matrix rank also has interesting ties to work that uses rank compression for more efficient training in deep networks [46], and for numerical solutions to systems with time-varying dynamics [49].
Broader impact. We introduce a novel framework for understanding learning in the brain and artificial neural networks. Our mathematical results on the rank of RNN gradients have broad relevance to the machine learning community, while our results on motor neural data could drive future applications to brain computer interfaces. Overall, our work makes novel contributions towards understanding the emergence of computations through learning in neural systems.
We thank Francesca Mastrogiuseppe, Friedrich Schuessler, and Sina Tootoonian for feedback on the manuscript, and Matthias Hennig, Matt Nolan, and Srdjan Ostojic for helpful discussions. We are particularly grateful to Matthew Perich for sharing his data. This work was supported by the Agence Nationale de la Recherche (ANR-20-CE37-0004; ANR-17-EURE-0017).
[40] Xuechen Li, Ting-Kam Leonard Wong, Ricky T. Q. Chen, and David Duvenaud. Scalable gradients for stochastic differential equations. International Conference on Artificial Intelligence and Statistics, 2020.
[41] Per Christian Hansen. Rank-deficient and discrete ill-posed problems: numerical aspects of linear inversion. SIAM, 1998.
[42] Jarosław Buczyński and Joseph M Landsberg. Ranks of tensors and a generalization of secant varieties. Linear Algebra and its Applications, 438(2):668-689, 2013.
[43] Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al. A deep learning framework for neuroscience. Nature neuroscience, 22(11):1761-1770, 2019.
[44] Cengiz Pehlevan, Tao Hu, and Dmitri B Chklovskii. A hebbian/anti-hebbian neural network for linear subspace learning: A derivation from multidimensional scaling of streaming data. Neural computation, 27(7):1461-1495, 2015.
[45] Shanshan Qin, Shiva Farashahi, David Lipshutz, Anirvan M Sengupta, Dmitri B Chklovskii, and Cengiz Pehlevan. Coordinated drift of receptive fields in hebbian/anti-hebbian network models during noisy representation learning. Nature Neuroscience, pages 1-11, 2023.
[46] Steffen Schotthöfer, Emanuele Zangrando, Jonas Kusch, Gianluca Ceruti, and Francesco Tudisco. Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations. arXiv preprint arXiv:2205.13571, 2022.
[47] Charles H Martin and Michael W Mahoney. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. The Journal of Machine Learning Research, 22(1):7479–7551, 2021.
[48] Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32, 2019.
[49] Othmar Koch and Christian Lubich. Dynamical low-rank approximation. SIAM Journal on Matrix Analysis and Applications, 29(2):434-454, 2007.
[50] John A Rhodes. A concise proof of kruskal's theorem on tensor decomposition. Linear Algebra and its Applications, 432(7):1818-1824, 2010.
[51] Tamara G Kolda. Orthogonal tensor decompositions. SIAM Journal on Matrix Analysis and Applications, 23(1):243-255, 2001.
[52] Grigorios A Pavliotis. Stochastic processes and applications: diffusion processes, the Fokker-Planck and Langevin equations, volume 60. Springer, 2014.
[53] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[54] David Zoltowski, Jonathan Pillow, and Scott Linderman. A general recurrent state space framework for modeling neural dynamics during decision-making. In International Conference on Machine Learning, pages 11680-11691. PMLR, 2020.
[55] Walter Rudin. Functional analysis. McGraw-Hill, 1973.
[56] Roger A Horn and Charles R Johnson. Topics in matrix analysis: Cambridge university press. Cambridge, UK, 1991.
[57] Rudolf Emil Kalman. Mathematical description of linear dynamical systems. Journal of the Society for Industrial and Applied Mathematics, Series A: Control, 1(2):152-192, 1963.
A Low tensor rank recurrent neural networks
Low tensor rank weights. In order to probe for the tensor rank of learning in neural data, we introduce an RNN architecture that captures the evolution of neural activity over slow timescales. We first recall the description of the architecture of the main text,  $\mathbf{W} \in \mathbb{R}^{N \times N \times K}$ ,
$$
\mathbf {W} = \sum_ {r = 1} ^ {R} \mathbf {a} _ {r} \otimes \mathbf {b} _ {r} \otimes \mathbf {c} _ {r}.
$$
In particular, at trial  $k$ , the dynamics of the RNN can be described as,
$$
\tau \dot {\mathbf {x}} = W ^ {(k)} \phi (\mathbf {x}) - \mathbf {x} + B \mathbf {u} ^ {(k)} (t) = \sum_ {r = 1} ^ {R} c _ {r} ^ {(k)} \left(\mathbf {a} _ {r} \otimes \mathbf {b} _ {r}\right) \phi (\mathbf {x}) - \mathbf {x} + B \mathbf {u} ^ {(k)} (t)
$$
for  $B \in \mathbb{R}^{N \times N_{\mathrm{input}}}$ ,  $\mathbf{u}^{(k)}(t) \in \mathbb{R}^{N_{\mathrm{input}}}$ , so that the RNN is a low rank RNN [15]. When training, we initialize  $\mathbf{a}_r \sim \mathcal{N}(\mathbf{0}, I)$ ,  $\mathbf{b}_r = \mathbf{a}_r$ . Furthermore, the weights are parameterized such that  $\|\mathbf{a}_r\| = \|\mathbf{b}_r\| = 1$ , so that the magnitude of a component is captured by  $\mathbf{c}_r$ .