SlowGuess/NeurIPS_2023_OCR huggingface dataset:
All models grow from a small seed architecture to the full-sized one in 9 stages, each trained for  $\{8,9,11,13,16,19,23,28,33\}$  epochs (160 total) on CIFAR-10, and  $\{10,12,14,17,20,24,29,35,39\}$  (200 total) on CIFAR-100. Net2Net follows the design of growing by splitting via simple neuron replication, hence achieving the same training efficiency as our gradient-free method under the same growing schedule. Splitting and Firely require additional training effort for their neuron selection schemes and allocate extra GPU memory for auxiliary variables during the local optimization stage. This is computationally expensive, especially when growing a large model.
ResNet-20, VGG-11, and MobileNetV1 on CIFAR-10. Table 2 shows results in terms of test accuracy and training cost calculated based on overall FLOPs. For ResNet-20, Splitting and Firefly achieve better test accuracy than Net2Net, which suggests the additional optimization benefits neuron selection at the cost of training efficiency. Our method requires only  $54.9\%$  of the baseline training cost and outperforms all competing methods, while yielding only  $0.09p.p$  (percentage points) performance degradation compared to the static baseline. Moreover, our method even outperforms the large fixed-size VGG-11 by  $0.20p.p$  test accuracy, while taking only  $52.91\%$  of its training cost. For MobileNetV1, our method also achieves the best trade-off between training efficiency and test accuracy among all competitors.
ResNet-18, VGG-19, and MobileNetV1 on CIFAR-100. We also evaluate all methods on CIFAR-100 using different network architectures. Results in Table 3 show that Firely consistently achieves better test accuracy than Firefly-split, suggesting that adding new neurons provides more flexibility for exploration than merely splitting. Both Firely and our method achieve better performance than the original VGG-19, suggesting that network growing might have an additional regularizing effect. Our method yields the best accuracy and largest training cost reduction.
Table 4: ResNet-50 and MobileNetV1 on ImageNet.
Method	ResNet-50	MobileNet-v1
Train Cost(%) ↓	Test Acc.(%)	Train Cost(%) ↓	Test Acc.(%)
Large	100	76.72 ± 0.18	100	70.80 ± 0.19
Net2Net	60.12	74.89 ± 0.21	63.72	66.19 ± 0.20
FireFly	71.20	75.01 ± 0.11	86.67	66.40 ± 0.14
GradMax	-	-	86.67	68.60 ± 0.20
Ours	60.12	75.90 ± 0.14	63.72	69.91 ± 0.16
Table 5: Transformer on IWSLT'14.
Method	Transformer
Train Cost(%) ↓	BLEU↑
Large	100	32.82 ± 0.21
Net2Net	64.64	30.97 ± 0.35
Ours-w/o buffer	64.64	31.44 ± 0.18
Ours-w buffer	64.64	31.62 ± 0.16
Ours-w buffer-RA	64.64	32.01 ± 0.16
4.2 ImageNet Results
We first grow ResNet-50 on ImageNet and compare the performance of our method to Net2Net and FireFly under the same epoch schedule:  $\{4, 4, 5, 6, 8, 9, 11, 14, 29\}$  (90 total) with 9 growing phases. We also grow MobileNetV1 from a small seed architecture, which is more challenging than ResNet-50. We train Net2Net and our method uses the same scheduler as for ResNet-50. We also compare with Firefly-Opt (a variant of FireFly) and GradMax and report their best results from [10]. Note that both methods not only adopt additional local optimization but also train with a less efficient growing scheduler: the final full-sized architecture needs to be trained for a  $75\%$  fraction while ours only requires  $32.2\%$ . Table 4 shows that our method dominates all competing approaches.
4.3 IWSLT14 De-En Results
We grow a Transformer from  $d_{model} = 64$  to  $d_{model} = 512$  within 4 stages, each trained with 5 epochs using Adam. Applying gradient-based growing methods to the Transformer architecture is nontrivial due to their domain specific design of local optimization. As such, we only compare with Net2Net. We also design variants of our method for self-comparison, based on the adaptation rules for Adam in Appendix C. As shown in Table 5, our method generalizes well to the Transformer architecture.
4.4 Analysis
Ablation Study. We show the effects of turning on/off each of our modifications to the baseline optimization process of Net2Net (1) Growing: adding neurons with functionality preservation. (2) Growing+VT: only applies variance transfer. (3) Growing+RA: only applies LR rate adaptation. (4) Full method. We conduct experiments using both ResNet-20 on CIFAR-10 and ResNet-18 on CIFAR-100. As shown in Table 6, different variants of our growing method not only outperform Net2Net consistently but also reduce the randomness (std. over 3 runs) caused by random replication. We also see that, both RA and VT boost the baseline growing method. Both components are designed and woven to accomplish the empirical leap. Our full method bests the test accuracy.
Variant	Res-20 on C-10 (%)	Res-18 on C-100 (%)
Net2Net	91.60 ± 0.21(+0.00)	76.48 ± 0.20(+0.00)
Growing	91.62 ± 0.12(+0.02)	76.82 ± 0.17(+0.34)
Growing+VT	92.00 ± 0.10(+0.40)	77.27 ± 0.14(+0.79)
Growing+RA	92.24 ± 0.11(+0.64)	77.74 ± 0.16(+1.26)
Full	92.53 ± 0.11(+0.93)	78.12 ± 0.15(+1.64)
Table 6: Ablation study on VT and RA components.
(a) Train Loss
Figure 3:Baselines of 4-layer simple CNN.
(b) Test Accuracy
(a) Test Accuracy
(b) Global LR
Figure 4: (a) Performance with Var. Transfer and Rate Adaptation growing ResNet-20. (b) and (c) visualize the gradients for different sub-components along training in the last two stages.
(c) Rate Adaptation
Justification for Variance Transfer. We train a simple neural network with 4 convolutional layers on CIFAR-10. The network consists of 4 resolution-preserving convolutional layers; each convolution has 64, 128, 256 and 512 channels, a  $3 \times 3$  kernel, and is followed by BatchNorm and ReLU activations. Max-pooling is applied to each layer for a resolution-downsampling of 4, 2, 2, and 2. These CNN layers are then followed by a linear layer for classification. We first alternate this network into four variants, given by combinations of training epochs  $\in \{13(1 \times), 30(2.3 \times)\}$  and initialization methods  $\in \{\text{standard}, \mu \text{transfer} [41]\}$ . We also grow from a thin architecture within 3 stages, where the channel number of each layer starts with only  $1/4$  of the original one, i.e.,  $\{16, 32, 64, 128\} \to \{32, 64, 128, 256\} \to \{64, 128, 256, 512\}$ , each stage is trained for 10 epochs.
For network growing, we compare the baselines with standard initialization and variance transfer. We train all baselines using SGD, with weight decay set as 0 and learning rates sweeping over  $\{0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 0.8, 1.0, 1.2, 1.5, 2.0\}$ . In Figure 3(b), growing with Var. Transfer (blue) achieves overall better test accuracy than standard initialization (orange). Large baselines with merely  $\mu$  transfer in initialization consistently underperform standard ones, which validate that the compensation from the LR rescaling is necessary in [41]. We also observe, in both Figure 3(a) and 3(b), all growing baselines outperform fixed-size ones under the same training cost, demonstrating positive regularization effects. We also show the effect of our initialization scheme by comparing test performance on standard ResNet-20 on CIFAR-10. As shown in Figure 4(a), compared with standard initialization, our variance transfer not only achieves better final test accuracy but also appears more stable. See Appendix F for a fully-connected network example.
Justification for Learning Rate Adaptation. We investigate the value of our proposed stage-wise learning rate adaptation as an optimizer for growing networks. As shown in the red curve in Figure 3, rate adaptation not only bests the train loss and test accuracy among all baselines, but also appears to be more robust over different learning rates. In Figure 4(a), rate adaptation further improves final test accuracy for ResNet-20 on CIFAR-10, under the same initialization scheme.
Figure 4(b) and 4(c) visualize the gradients of different sub-components for the 17-th convolutional layer of ResNet-20 during last two growing phases of standard SGD and rate adaptation, respectively. Our rate adaptation mechanism rebalances subcomponents' gradient contributions to appear in lower divergence than global LR, when components are added at different stages and trained for different durations. In Figure 5, we observe that the LR for newly added Subnet-8 (red) in last stage starts around  $1.8 \times$  the base LR, then quick adapts to a smoother level. This demonstrates that our method is able to automatically adapt the updates applied to new weights, without any additional local optimization costs [39, 10]. All above show our method has a positive effect in terms of stabilizing training dynamics, which is lost if one attempts to train different subcomponents using a global LR scheduler. Appendix D provides more analysis.
Flexible Growing Scheduler. Our growing scheduler gains the flexibility to explore the best trade-offs between training budgets and test performance in a unified configuration scheme (Eq. 6 and Eq. 7). We compare the exponential epoch scheduler  $(p_t \in \{0.2, 0.25, 0.3, 0.35\})$  to a linear one  $(p_t = 0)$  in ResNet-20 growing on CIFAR-10, denoted as 'Exp.' and 'Linear' baselines in Figure 6. Both baselines use the architectural schedulers with  $p_c \in \{0.2, 0.25, 0.3, 0.35\}$ , each generates trade-offs between train costs and test accuracy by alternating  $T_0$ . The exponential scheduler yields better
Figure 5: Visualization of our adaptive LR.
Figure 6: Comparison of growing schedules.
(a) GPU memory allocations
(b) Training time
Figure 7: Track of GPU memory and wall clock training time for each growing phase of ResNet-18.
overall trade-offs than the linear one with the same  $p_c$ . In addition to different growing schedulers, we also plot a baseline for fixed-size training with different models. Growing methods with both schedulers consistently outperform the fixed-size baselines, demonstrating that the regularization effect of network growth benefits generalization performance.
Wall-clock Training Speedup. We benchmark GPU memory consumption and wall-clock training time on CIFAR-100 for each stage during training on single NVIDIA 2080Ti GPU. The large baseline ResNet-18 trains for 140 minutes to achieve  $78.36\%$  accuracy. As shown in the green bar of Figure 7(b), the growing method only achieves marginal wall-clock acceleration, under the same fixed batch size. As such, the growing ResNet-18 takes 120 minutes to achieve  $78.12\%$  accuracy. The low GPU utilization in the green bar in Figure 7(a) hinders FLOPs savings from translating into real-world training acceleration. In contrast, the red bar of Figure 7 shows our batch size adaptation results in a large proportion of wall clock acceleration by filling the GPU memory, and corresponding parallel execution resources, while maintaining test accuracy. ResNet-18 trains for 84 minutes  $(1.7\times$  speedup) and achieves  $78.01\%$  accuracy.
We tackle a set of optimization challenges in network growing and invent a corresponding set of techniques, including initialization with functionality preservation, variance transfer and learning rate adaptation to address these challenges. Each of these techniques can be viewed as 'upgrading' an original part for training static networks into a corresponding one that accounts for dynamic growing. There is a one-to-one mapping of these replacements and a guiding principle governing the formulation of each replacement. Together, they accelerate training without impairing model accuracy - a result that uniquely separates our approach from competitors. Applications to widely-used architectures on image classification and machine translation tasks demonstrate that our method bests the accuracy of competitors while saving considerable training cost.
This work was supported by the National Science Foundation under grant CNS-1956180 and the University of Chicago CERES Center.
[26] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E. Reed, Cheng-Yang Fu, and Alexander C. Berg. SSD: Single shot multibox detector. In ECCV, 2016.
[27] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
[28] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, 2002.
[29] Valerio Perrone, Rodolphe Jenatton, Matthias W. Seeger, and Cedric Archambeau. Scalable hyperparameter transfer learning. In NeurIPS, 2018.
[30] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classifier architecture search. In AAAI, 2019.
[31] Pedro Savarese, David McAllester, Sudarshan Babu, and Michael Maire. Domain-independent dominance of adaptive methods. In CVPR, 2021.
[32] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
[33] Tijmen Tieleman, Geoffrey Hinton, et al. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 2012.
[34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
[35] Chengcheng Wan, Henry Hoffmann, Shan Lu, and Michael Maire. Orthogonalized SGD and nested architectures for anytime neural networks. In ICML, 2020.
[36] Tao Wei, Changhu Wang, Yong Rui, and Chang Wen Chen. Network morphism. In ICML, 2016.
[37] Wei Wen, Feng Yan, Yiran Chen, and Hai Li. AutoGrow: Automatic layer growing in deep convolutional networks. In KDD, 2020.
[38] Lemeng Wu, Bo Liu, Peter Stone, and Qiang Liu. Firefly neural architecture descent: a general approach for growing neural networks. In NeurIPS, 2020.
[39] Lemeng Wu, Mao Ye, Qi Lei, Jason D Lee, and Qiang Liu. Steepest descent neural architecture optimization: Escaping local optimum with signed neural splitting. arXiv:2003.10392, 2020.
[40] Greg Yang and Edward J. Hu. Tensor Programs IV: Feature learning in infinite-width neural networks. In ICML, 2021.
[41] Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. In NeurIPS, 2021.
[42] Dani Yogatama and Gideon Mann. Efficient transfer learning method for automatic hyperparameter tuning. In AISTATS, 2014.
[43] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training BERT in 76 minutes. In ICLR, 2020.
[44] Xin Yuan, Pedro Henrique Pamplona Savarese, and Michael Maire. Growing efficient deep networks by structured continuous sparsification. In ICLR, 2021.
[45] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv:1605.07146, 2016.
[46] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In CVPR, 2022.
A More Analysis on Variance Transfer
Our fixed rescaling formulation, regarding relative network width, is an extension to the principled zero-shot HP transfer method [40, 41], based on the stability assumption, denoted as VT. A dynamic rescaling based on actual old weight values is an alternative plausible implementation choice, denoted as VT-constraint.
Theorem A.1. Suppose the goal is to enforce the unit variance feature, then the scaling factor of an input layer with weights  $\mathbf{W}^x$  with input shape  $C^x$  is  $\sqrt{\frac{1}{C^x\mathbb{V}[\mathbf{W}^x]}},$  while for a hidden layer with weights  $\mathbf{W}^u$  and input shape  $C^u$ , it is  $\sqrt{\frac{1}{C^u\mathbb{V}[\mathbf{W}^u]}}.$
Proof. Consider a hidden layer that computes  $\boldsymbol{u} = \boldsymbol{W}^x\boldsymbol{x}$  followed by another layer that computes  $\boldsymbol{h} = \boldsymbol{W}^u\boldsymbol{u}$  (ignoring activations for simplicity). At a growth step, the first layer's outputs change from
$$
\boldsymbol {u} _ {t} = \boldsymbol {W} _ {t} ^ {x} \boldsymbol {x} \tag {9}
$$
$$
\boldsymbol {u} _ {t + 1} = \left[ \boldsymbol {u} _ {t} ^ {\prime}, \boldsymbol {u} _ {t} ^ {\prime \prime}, \boldsymbol {u} _ {t} ^ {\prime \prime} \right] = \left[ s ^ {x} \boldsymbol {W} _ {t} ^ {x} \boldsymbol {x}, \boldsymbol {V} _ {t} ^ {x} \boldsymbol {x}, \boldsymbol {V} _ {t} ^ {x} \boldsymbol {x} \right]. \tag {10}
$$
where  $s^x$  denotes the scaling factor applied to  $W^x$ . The second layer's outputs change from
$$
\boldsymbol {h} _ {t} = \boldsymbol {W} _ {t} ^ {u} \boldsymbol {u} _ {t} \tag {11}
$$
$$
\boldsymbol {h} _ {t + 1} = \left[ \boldsymbol {h} _ {t} ^ {\prime}, \boldsymbol {h} _ {t} ^ {\prime \prime}, \boldsymbol {h} _ {t} ^ {\prime \prime} \right] = \left[ s ^ {u} \boldsymbol {W} _ {t} ^ {u} \boldsymbol {u} _ {t} ^ {\prime}, \boldsymbol {V} _ {t} ^ {u} \boldsymbol {u} _ {t + 1}, \boldsymbol {V} _ {t} ^ {u} \boldsymbol {u} _ {t + 1} \right]. \tag {12}
$$
where  $s^u$  denotes the scaling factor applied to  $W^u$ .
The variance of the features after the growth step are:
$$
\mathbb {V} \left[ \boldsymbol {u} _ {t} ^ {\prime} \right] = \left(s ^ {x}\right) ^ {2} C ^ {x} \mathbb {V} \left[ \boldsymbol {W} _ {t} ^ {x} \right] \tag {13}
$$
$$
\mathbb {V} \left[ \boldsymbol {u} _ {t} ^ {\prime \prime} \right] = C ^ {x} \mathbb {V} \left[ \boldsymbol {V} _ {t} ^ {x} \right] \tag {14}
$$
$$
\mathbb {V} \left[ \boldsymbol {h} _ {t} ^ {\prime} \right] = \left(s ^ {u}\right) ^ {2} C _ {t} ^ {u} \mathbb {V} \left[ \boldsymbol {W} _ {t} ^ {u} \right] \mathbb {V} \left[ \boldsymbol {u} _ {t} ^ {\prime} \right] \tag {15}
$$
$$
\mathbb {V} \left[ \boldsymbol {h} _ {t} ^ {\prime \prime} \right] = \mathbb {V} \left[ \boldsymbol {V} _ {t} ^ {u} \right] \left(C _ {t} ^ {u} \mathbb {V} \left[ \boldsymbol {u} _ {t} ^ {\prime} \right] + \left(C _ {t + 1} ^ {u} - C _ {t} ^ {u}\right) \mathbb {V} \left[ \boldsymbol {u} _ {t} ^ {\prime \prime} \right]\right) \tag {16}
$$
Given the goal of enforcing unit-variance features for across all four vectors, we get:
$$
s ^ {x} = \sqrt {\frac {1}{C ^ {x} \mathbb {V} \left[ \boldsymbol {W} _ {t} ^ {x} \right]}} \Rightarrow \mathbb {V} \left[ \boldsymbol {u} _ {t} ^ {\prime} \right] = 1 \tag {17}
$$
$$
s ^ {u} = \sqrt {\frac {1}{C _ {t} ^ {u} \mathbb {V} \left[ \boldsymbol {W} _ {t} ^ {u} \right]}} \Longrightarrow \mathbb {V} \left[ \boldsymbol {h} _ {t} ^ {\prime} \right] = \mathbb {V} \left[ \boldsymbol {u} _ {t} ^ {\prime} \right] = 1 \tag {18}
$$
$$
\mathbb {V} \left[ \boldsymbol {V} _ {t} ^ {x} \right] = \frac {1}{C ^ {x}} \Rightarrow \mathbb {V} \left[ \boldsymbol {u} _ {t} ^ {\prime \prime} \right] = 1 \tag {19}
$$
$$
\mathbb {V} \left[ V _ {t} ^ {u} \right] = \frac {1}{C _ {t + 1} ^ {u}} \Rightarrow \mathbb {V} \left[ \boldsymbol {h} _ {t} ^ {\prime \prime} \right] = \frac {1}{C _ {t + 1} ^ {u}} \left(C _ {t} ^ {u} \mathbb {V} \left[ \boldsymbol {u} _ {t} ^ {\prime} \right] + \left(C _ {t + 1} ^ {u} - C _ {t} ^ {u}\right) \mathbb {V} \left[ \boldsymbol {u} _ {t} ^ {\prime \prime} \right]\right) = 1. \tag {20}
$$
This differs from the default VT formulation in Section 3.1, which corresponds to scaling factors of  $s^x = 1$  and
$$
s ^ {u} = \sqrt {\frac {C _ {t} ^ {u}}{C _ {t + 1} ^ {u}}}
$$
We compare the default VT with VT-constraint by growing ResNet-20 on CIFAR-10. As shown in Table 7, both VT and VT-constraint outperform the standard baseline, which suggests standard initialization is a suboptimal design in network growing. We also note that involving the weight statistics is not better than our simpler design, which suggests that enforcing the old and the new features to have the same variance is not a good choice.
Variants	Test Acc. (%)
Standard	91.62 ± 0.12
VT-constraint	91.93 ± 0.12
VT	92.00 ± 0.10
Table 7: Comparisons among standard initialization, VT-constraint (Theorem A.1) and the default VT (Section 3.1) for growing ResNet-20 on CIFAR-10.
(a) Growing CNN hidden layer
(b) Growing residual block
Figure 8: Illustration for growing other layers.
B General Network Growing in Other Layers
We have shown network growing for 3-layer fully connected networks as a motivating example in Section 3.1. We now show how to generalize network growing with K  $(>3)$  layers, with conv layers, residual connections.
Generalization to  $\mathbf{K} (> 3)$  Layers. In our network-width growing formulation, layers may be expanded in 3 patterns. (1) Input layer: output channels only; (2) Hidden layer: Both input (due to expansion of the previous layer) and output channels. (3) Output layer: input channels only. As such, the 3-layer case is sufficient to serve as a motivating example without loss of generality. For  $\mathbf{K} (> 3)$  layer networks, the 2nd to  $(K - 1)$ th layers simply follow the hidden layer case defined in Section 3.1.
Generalization to Convolutional layers. In network width growth, we only need to consider the expansion along input and output channel dimensions no matter for fully connected networks or CNNs. Equations 1-4 still hold for CNN layers. Specifically, when generalizing to CNNs (from 2-d  $C_{out} \times C_{in}$  weight matrices to 4-d  $C_{out} \times C_{in} \times k \times k$  ones), we only consider the matrix operations on the first two dimensions since we do not change the kernel size  $k$ . For example, in Figure 2(b), newly added weights  $+Z_{t}^{u}$  in linear layer can be indexed as  $W[0:C_t^h,C_t^u:C_t^u +\frac{C_{t + 1}^u - C_t^u}{2}]$ . In CNN layer, it is simply  $W[0:C_t^h,C_t^u:C_t^u +\frac{C_{t + 1}^u - C_t^u}{2},\therefore]$ .
Residual Connections. We note that differently from plain CNNs like VGG, networks with residual connections do require that the dimension of an activation and its residual match since these will be added together. Our method handles this by growing all layers at the same rate. Hence, at each growth step, the shape of the two tensors to be added by the skip connections always matches.
Concrete Example of Adding Units to a Network. In model width growth, our common practice is to first determine the output dimension expansion from the growth scheduler. If a previous layer's output channels are to be grown, then the input dimension of the current layer should be also expanded to accommodate that change.
Let's consider a concrete example for a CNN with more layers ( $>3$ ) and residual blocks. Without loss of generality, we omit the kernel size  $k$  and denote each convolutional layer channel shape as  $(C_{in}, C_{out})$ . We denote the input feature with 1 output dimension as  $x$ . Initially, we have the input layer  $W^{1}$  with channel dimensions of  $(1,2)$  to generate  $h_{1}$  (2-dimensional), followed by a 2-layer residual block (identity mapping for residual connection) with weights  $W^{(2)}(2,2)$  and  $W^{(3)}(2,2)$ , followed by an output layer with weights  $W^{(4)}(2,1)$ .
The corresponding computation graph (omitting BN and ReLU for simplification) is
$$
y = W ^ {(4)} \left(W ^ {(3)} W ^ {(2)} W ^ {(1)} x + W ^ {(1)} x\right),
$$
In more detail, we rewrite the computation in the matrix formulation: