SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[53] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8769-8778, 2018.
[54] Haoran Wang, Weitang Liu, Alex Bocchieri, and Yixuan Li. Can multi-label classification networks know what they don't know? Advances in Neural Information Processing Systems, 2021.
[55] Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2097-2106, 2017.
[56] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-3492. IEEE, 2010.
[57] Mingyu Xu, Zheng Lian, Bin Liu, and Jianhua Tao. Vra: Variational rectified activation for out-of-distribution detection, 2023.
[58] Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv preprint arXiv:1504.06755, 2015.
[59] Huanrui Yang, Minxue Tang, Wei Wen, Feng Yan, Daniel Hu, Ang Li, Hai Li, and Yiran Chen. Learning low-rank deep neural networks via singular vector orthogonality regularization and singular value sparsification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 678-679, 2020.
[60] Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:2110.11334, 2021.
[61] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015.
[62] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3): 107-115, 2021.
[63] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6):1452-1464, 2017.
[64] Helen Zhou, Cheng Cheng, Zachary C Lipton, George H Chen, and Jeremy C Weiss. Mortality risk score for critically ill patients with viral or unspecified pneumonia: Assisting clinicians with Covid-19 ECMO planning. In International Conference on Artificial Intelligence in Medicine, pages 336–347. Springer, 2020.
[65] Feng Zhu, Hongsheng Li, Wanli Ouyang, Nenghai Yu, and Xiaogang Wang. Learning spatial regularization with image-level supervisions for multi-label image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5513-5522, 2017.
The objective of this research project is to enhance the dependability and safety of contemporary machine learning models. The outcomes of our investigation possess the potential for significant advantages and societal effects, particularly in the realm of safety-critical domains like autonomous driving. It is important to note that our research does not involve any human subjects or breach legal compliance. We do not foresee any possible adverse ramifications resulting from our work. By conducting this study, our intention is to foster increased research efforts and raise awareness within both the research community and society at large regarding the issue of out-of-distribution detection in practical, real-world scenarios.
OOD detection methods may not always detect out-of-distribution samples accurately. As it is presented in our experiments, most OOD detection methods are not able to recognize OOD data across different OOD datasets. They may provide superior performance on some OOD data while catastrophic performance on some other variations of OOD data. Though our method, Gradorth, presents comparable performance to state-of-the-art methods, it still requires more investigation to achieve full stable performance across all OOD datasets.
The Appendix section is organized as follows:
- Section A presents the relationship between stochastic gradient descent (SGD) updates and the span of input data points in convolutional layers.
- Section B presents an experimental study that evaluates the role of leveraging the last-layer gradient of the network in our method (GradOrth).
- Section C discusses the results of the experimental analysis on the computation of the ID subspace leveraging different numbers of ID samples per class.
- Section D includes an explanation of the cross-entropy loss and its derivative.
- Section E presents proof showing that stochastic gradient descent (SGD) updates lie in the span of input data points in a mini-batch setting.
- Section F offers a brief explanation of singular value decomposition (SVD).
- Section G presents an ablation study that evaluates the impact of SVD on the GradOrth out-of-distribution (OOD) detection performance.
- Section H offers more explanation regarding  $k$ -rank matrix approximation.
- Section I examines the impact of threshold  $\epsilon_{th}$  on GradOrth performance.
- Section J provides further information about our experiments, including networks, datasets, and the scoring functions of the baselines.
A Input and Gradient Spaces in Convolutional Layers
Our algorithm exploits the observation that updates in stochastic gradient descent (SGD) reside within the subspace spanned by the input data points [62], as discussed in section 2.1 and this appendix, section D. In this section, we aim to establish this relationship specifically for convolutional layers. The analysis presented herein possesses general applicability to any layer within a network, regardless of the task.
In contrast to the weights in a fully connected (FC) layer, filters within a convolutional (Conv) layer operate differently on the input. Let us consider a convolutional layer comprising the input tensor  $\mathcal{X} \in \mathbb{R}^{C_i \times h_i \times w_i}$  and filters  $\pmb{\theta} \in \mathbb{R}^{C_o \times C_i \times k \times k}$ . Their convolution, denoted as  $\langle \mathcal{X}, \pmb{\theta}, * \rangle$ , yields the output feature map  $\mathcal{O} \in \mathbb{R}^{C_o \times h_o \times w_o}$  [35]. Here,  $C_i(C_o)$  represents the number of input (output) channels in the Conv layer, while  $h_i, w_i(h_o, w_o)$  correspond to the height and width of the input (output) feature maps, and  $k$  denotes the kernel size of the filters. Figure 4(a) provides a visual representation of this process.
If we reshape  $\mathcal{X}$  into a  $(h_o\times w_o)\times (C_i\times k\times k)$  matrix denoted as  $\pmb{x}$ , and reshape  $\pmb{\theta}$  into a  $(C_i\times k\times k)\times C_o$  matrix denoted as  $\pmb{\theta}$ , the convolution can be expressed as a matrix multiplication between  $\pmb{x}$  and  $\pmb{\theta}$ , yielding  $\mathbf{O} = \pmb {x}\pmb {\theta}$ , where  $\mathbf{O}\in \mathbb{R}^{(h_0\times w_0)\times C_o}$ .
Formulating the convolution in terms of matrix multiplication provides an intuitive depiction of gradient computation during the back-propagation process. Similar to the fully connected layer scenario, in the convolutional layer, during the backward pass, an error matrix  $\Omega$  of size  $(h_0 \times w_0) \times C_o$  (equivalent to the size of  $\mathbf{O}$ ) is obtained from the subsequent layer. As illustrated in figure 4(b), the gradient of the loss with respect to the filter weights is computed as follows:
$$
\nabla_ {\boldsymbol {\theta}} L = \boldsymbol {x} ^ {T} \boldsymbol {\Omega}, \tag {8}
$$
where  $\nabla_{\pmb{\theta}}L$  possesses a shape of  $(C_i\times k\times k)\times C_o$  (matching the size of  $\pmb{\theta}$ ). Considering that the columns of  $\pmb{x}^T$  correspond to the input.
Figure 4: The convolution operation in matrix multiplication format during the forward Pass (a) and backward pass (b).
B GradOrth Considering All The Network Layers
This experimental study presents that the gradients obtained from the final layer contain substantial and informative content. In this experimental study, we aim to investigate the content and significance of gradients obtained from the final layer in a neural network. We explore an alternative variation of the GradOrth method, focusing on the extraction of gradients from all layers of the network. The objective is to analyze the gradients of all trainable parameters across the layers and assess their informativeness. In this paper, we present this version of GradOrth as "GradOrth-All layers" and compare it with the original GradOrth, which utilizes the gradient space from the last layer. To assess the performance of the various gradient spaces, we perform a gradient projection for each layer of the neural network. Subsequently, we calculate the average of these gradient projections across all layers. This procedure allows us to derive an OOD detection score, which we refer to as the OODness score. The experiment is conducted over five random subsets of the ID subspaces. Each subspace is computed by utilizing ten random samples per class in the ImageNet benchmark and five random samples per class in CIFAR benchmarks. We report the average results obtained from these five runs. The outcomes of both GradOrth-All layers and GradOrth-Last layer are presented in tables 6, 7, and 8.
table 6 provides a comparison of the OOD detection performance using the gradient spaces from different layers (all network layers and the last network layer) on two pre-trained network architectures, ResNet and MobileNet, on the ImageNet dataset as ID data. The evaluation metrics used are the False Positive Rate at  $95\%$  True Positive Rate (FPR95) and the Area Under the Receiver Operating Characteristic curve (AUROC). These metrics are averaged across four OOD datasets.
Our findings demonstrate that gradients from the last layer consistently outperform gradients from all layers. On ResNet and MobileNet, leveraging the last-layer gradient space results in a  $1.57\%$  and  $1.52\%$  improvement in FPR95 (on average), respectively, compared to GradOrth utilizing gradients from all network layers.
Model	Methods	OOD Datasets	Average
iNaturalist	SUN	Places	Textures	FPR95 ↓ AUROC ↑	FPR95 ↓ AUROC ↓ AUROC ↑	FPR95 ↓ AUROC ↑
FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑
ResNet	GradOrth-All layers	13.23	96.52	21.05	95.21	35.58	92.49	11.56	96.72	20.35	95.23
GradOrth-Last layer	11.04	98.00	19.61	95.76	33.67	91.78	11.19	98.06	18.57	96.31
MobileNet	GradOrth-All layers	26.14	93.83	33.28	91.38	43.71	85.37	13.61	96.87	29.17	91.86
GradOrth-Last layer	26.81	93.17	30.82	93.18	40.27	89.12	12.69	97.52	27.65	93.25
Table 6: OOD detection results with ImageNet-1k as ID. Effect of leveraging all-layers and last-layer gradients space in GradOrth. The OODness score derived from the last layer yields better OOD detection performance mostly. The ResNet and MobileNet models are pre-trained solely with ID data from the ImageNet-1k dataset. We use  $\uparrow$  to denote that larger values are preferable, and  $\downarrow$  to denote that smaller values are preferable. All values are presented as percentages.
In the pursuit of an extensive investigation, the CIFAR benchmark is taken into account to examine the impact of various network gradient spaces, namely the last network layer and all network layers, on the performance of GradOrth. The findings, depicted in tables 7 and 8, reveal that utilizing last layer gradients in GradOrth yields superior results compared to employing gradients from all network layers, with improvements of  $0.82\%$  and  $1.49\%$  in FPR95 on average, respectively.
This observed outcome is highly advantageous, as gradients calculated with respect to deeper layers demonstrate computational efficiency when compared to utilizing gradients from all layers. Remarkably, the GradOrth variant derived from the final linear layer exhibits the most favorable outcomes. From a practical perspective, it is only necessary to perform back-propagation with respect
to the last linear layer, resulting in minimal computational overhead. Therefore, our primary findings are predicated on the utilization of the last fully connected (FC) layer within the neural network.
Method	OOD Datasets	Average
SVHN	LSUN-c	LSUN-r	ISUN	Textures	Places365
FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC
↓	↑	↓	↑	↓	↑	↓	↑	↓	↑	↓	↑	↓	↑
GradOrth-All layers	7.32	98.24	1.04	99.68	4.11	98.47	4.10	98.36	24.52	93.06	37.91	92.11	13.16	96.65
GradOrth-Last layer	5.84	98.72	0.81	99.78	2.33	98.71	4.25	98.32	20.63	94.77	38.22	91.64	12.34	96.99
Table 7: GradOrth leveraging last-layer network's gradient space outperforms Gradorth leveraging all network layers gradient space on CIFAR10 dataset. Detailed results on six common OOD benchmark datasets with CIFAR-10 as ID: Textures [8], SVHN [40], Places365 [63], LSUN-Crop [61], LSUN-Resize [61], and iSUN [58]. For each ID dataset, we use the same DenseNet pre-trained on CIFAR-10. ↑ indicates larger values are better and ↓ indicates smaller values are better.
Method	OOD Datasets	Average
SVHN	LSUN-c	LSUN-r	iSUN	Textures	Places365
FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑
GradOrth-All layers	27.61	92.84	4.32	97.63	49.27	89.48	44.07	90.17	34.86	91.25	48.92	88.83	34.84	91.70
GradOrth-Last layer	24.27	93.37	3.71	99.07	48.09	91.26	42.73	91.48	32.71	92.62	48.61	89.03	33.35	92.82
C Analysis of the Number of ID Samples
The initial step in our proposed method involves computing the subspace of the pre-trained neural network using the ID data. To accomplish this, we utilize a small number of data samples and pass them through the forward pass of the pre-trained network, without altering the learned parameters. Subsequently, we compute the subspace based on the last layer of the network.
To ensure a comprehensive study, we conduct an empirical investigation and compute variations of subspaces by considering different numbers of samples per class. Specifically, we vary the number of samples from 5 to 40 and 10 to 30 in the CIFAR and ImageNet1K benchmarks, respectively. For each variation, we compute five random subspaces and report the average results obtained from these five subspaces. The experimental results obtained from our study are presented in tables 9 and 10. Specifically, we introduce a notation to describe the experiments using GradOrth on the pre-trained network subspace computed based on a specific number of samples per class. We denote this notation as  $\mathrm{GradOrth - S}_n$ , where  $n$  represents the number of samples per class used to compute the subspace.
Table 8: GradOrth leveraging the last-layer network's gradient space outperforms Gradorth leveraging all network layers' gradient space on the CIFAR100 dataset. Detailed results on six common OOD benchmark datasets with CIFAR-100 as ID: Textures [8], SVHN [40], Places365 [63], LSUN-Crop [61], LSUN-Resize [61], and iSUN [58]. For each ID dataset, we use the same DenseNet pre-trained on CIFAR-100.  $\uparrow$  indicates larger values are better and  $\downarrow$  indicates smaller values are better.
Model	OOD Datasets	Average
SVHN	LSUN-c	LSUN-r	iSUN	Textures	Places365	FPR95	AUROC
FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC
↓	↑	↓	↑	↓	↑	↓	↑	↓	↑	↓	↑	↓	↑
GradOrth-S5	5.84	98.72	0.81	99.78	2.33	98.71	4.25	98.32	20.63	94.77	38.22	91.64	12.34	96.99
GradOrth-S10	5.80	98.74	0.75	99.79	2.34	98.69	4.11	98.41	20.37	94.82	38.13	92.01	11.86	97.07
GradOrth-S20	5.61	98.80	0.69	99.81	2.27	98.74	4.17	98.35	20.59	94.79	38.09	91.71	11.90	97.03
GradOrth-S40	5.62	98.79	0.69	99.83	2.24	98.75	4.16	98.38	20.55	94.84	38.11	91.74	11.90	97.05
Table 9: Detailed results on six common OOD benchmark datasets considering different numbers of ID samples (per class) in subspace computation. For each ID dataset, we use the same DenseNet pre-trained on CIFAR-10. ↑ indicates larger values are better and ↓ indicates smaller values are better.
Model	OOD Datasets	Average
SVHN	LSUN-c	LSUN-r	iSUN	Textures	Places365
FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC
↓	↑	↓	↑	↓	↑	↓	↑	↓	↑	↓	↑	↓	↑
GradOrth-S5	24.27	93.37	3.71	99.07	48.09	91.26	42.73	91.48	32.71	92.62	48.61	89.03	33.35	92.82
GradOrth-S10	24.25	93.38	3.73	99.09	48.08	91.29	42.70	91.52	32.65	92.66	48.63	89.04	33.34	92.83
GradOrth-S20	24.21	93.40	3.70	99.10	48.04	91.33	42.69	91.50	32.64	92.65	48.54	89.05	33.30	92.83
GradOrth-S40	24.09	93.46	3.67	99.11	47.86	91.38	42.53	91.73	32.49	92.72	46.91	90.12	32.92	93.09
Table 10: Detailed results on six common OOD benchmark datasets considering different numbers of ID samples (per class) in subspace computation. For each ID dataset, we use the same DenseNet pre-trained on CIFAR-100.  $\uparrow$  indicates larger values are better and  $\downarrow$  indicates smaller values are better.
The experimental results displayed in tables 9 and 10 indicate that increasing the number of samples per class during the computation of the subspace for the pre-trained network does not have a significant impact on the overall performance of OOD detection. This observation suggests that leveraging a pre-trained network, which has already learned the data well, diminishes the influence of the number of samples per class in the subspace computation.
Model	Methods	OOD Datasets	Average
iNaturalist	SUN	Places	Textures	FPR95	AUROC
FPR95↓	AUROC↑	FPR95↓	AUROC↑	FPR95↓	AUROC↑	FPR95↓	AUROC↑	FPR95↓	AUROC↑
ResNet	GradOrth-S10	11.04±0.23	98.00±0.09	19.61±1.26	95.76±0.49	33.67±0.18	91.78±0.22	11.19±0.20	98.06±0.28	18.88±0.47	95.90±0.27
GradOrth-S20	10.98±0.21	98.08±0.14	19.63±1.11	95.78±0.32	33.63±0.26	91.80±0.18	11.20±0.22	98.05±0.25	18.86±0.45	95.92±0.22
GradOrth-S30	11.00±0.27	98.04±0.12	19.66±0.51	95.78±0.27	33.68±0.19	91.83±0.23	11.17±0.25	98.08±0.19	18.88±0.31	95.93±0.26
MobileNet	GradOrth-S10	26.81±1.19	93.17±0.21	30.82±0.94	93.18±0.46	40.27±1.33	89.12±0.84	12.69±0.21	97.52±0.12	27.65±0.92	93.25±0.41
GradOrth-S20	26.78±0.83	93.20±0.24	30.76±0.70	93.20±0.39	40.22±1.16	89.13±0.69	12.65±0.24	97.55±0.17	27.60±0.73	93.27±0.37
GradOrth-S30	26.75±0.91	93.23±0.20	30.75±0.82	93.22±0.45	40.28±1.21	89.10±0.72	12.66±0.19	97.54±0.20	27.61±0.78	93.27±0.39
Table 11: GradOrth performance is not impacted by the number of ID samples significantly. Detailed results on four common OOD benchmark datasets considering different numbers of ID samples (per class) in subspace computation. We use ResNet and MobileNet pre-trained on ImageNet-1k. In our  $s_n$  notation,  $n$  represents the number of ID samples per class.
D Cross-Entropy Loss and its Derivative
In this section, we provide a detailed explanation of the cross-entropy loss and its derivative.
Consider a single-layer linear neural network in a supervised learning setting, where each training data pair  $(\pmb{x},\pmb{y})$  is drawn from a training dataset  $\mathbb{D}$ . Here,  $\pmb{x} \in \mathbb{R}^n$  represents the input vector,  $\pmb{y} \in \mathbb{R}^m$  represents the label vector in the dataset, and  $\pmb{\theta} \in \mathbb{R}^{m \times n}$  represents the learning parameters (weights) of the network. The model's prediction on input  $\pmb{x}$  is denoted by  $f(\pmb{x};\pmb{\theta})$ . For classification problems,  $f(\pmb{x};\pmb{\theta}) = \pmb{\theta}\pmb{x}$ , where  $f_k(\pmb{x};\pmb{\theta})$  represents the  $k$ -th logit associated with the  $k$ -th class.
The total loss on the training set (empirical risk) is denoted by:
$$
L _ {\mathbb {D}} (\boldsymbol {\theta}) = \sum_ {(\boldsymbol {x}, \boldsymbol {y}) \in \mathbb {D}} L _ {(\boldsymbol {x}, \boldsymbol {y})} (\boldsymbol {\theta}), \tag {9}
$$
where the per-example loss is defined as:
$$
L _ {(\boldsymbol {x}, \boldsymbol {y})} (\boldsymbol {\theta}) = \ell (\boldsymbol {y}, f (\boldsymbol {x}; \boldsymbol {\theta})), \tag {10}
$$
and  $\ell (\cdot ,\cdot)$  represents a differentiable non-negative loss function.
For classification problems, the softmax cross-entropy loss is commonly used, given by:
$$
\ell (\boldsymbol {y}, f (\boldsymbol {x}; \boldsymbol {\theta})) = - \sum_ {k = 1} ^ {m} y _ {k} \log a _ {k}, \tag {11}
$$
where  $a_{k} = \exp (f_{k}(\pmb {x};\pmb {\theta})) / \sum_{k}\exp (f_{k}(\pmb {x};\pmb {\theta}))$  represents the softmax output for the  $k$  -th class.
Using the chain rule, the gradient of the loss can be expressed as:
$$
\nabla L _ {(\boldsymbol {x}, \boldsymbol {y})} (\boldsymbol {\theta}) = \nabla f (\boldsymbol {x}; \boldsymbol {\theta}) \ell^ {\prime} (\boldsymbol {y}, f (\boldsymbol {x}; \boldsymbol {\theta})), \tag {12}
$$
where  $\ell'(\cdot, \cdot) \in \mathbb{R}^m$  denotes the derivative of  $\ell(\cdot, \cdot)$  with respect to its second argument, and  $\nabla f(\pmb{x}; \pmb{\theta})$  represents the gradient of the model  $f$  with respect to its second argument (i.e., the parameters). For the classification problem with cross-entropy softmax loss, we have:
$$
\nabla f (\boldsymbol {x}; \boldsymbol {\theta}) = \left[ \nabla f _ {1} (\boldsymbol {x}; \boldsymbol {\theta}); \dots ; \nabla f _ {m} (\boldsymbol {x}; \boldsymbol {\theta}) \right], \tag {13}
$$
where  $\nabla f_{k}(\pmb {x};\pmb {\theta})\in \mathbb{R}^{m}$  represents the gradient of the  $k$  -th logit with respect to the parameters. For simplicity, let's write equation 13 as:
$$
\nabla f (\boldsymbol {x}; \boldsymbol {\theta}) = \boldsymbol {x} ^ {T}. \tag {14}
$$
Considering that the derivative of the loss is given by:
$$
\ell^ {\prime} (\boldsymbol {y}, f (\boldsymbol {x}; \boldsymbol {\theta})) = \left[ a _ {1} - y _ {1}, \dots , a _ {m} - y _ {m} \right] ^ {\top} = \rho , \tag {15}
$$
Here,  $\pmb{\rho}$  denotes the error vector. Considering equations 12, 14, and 15, we can write equation 12 as:
$$
\nabla L _ {(\boldsymbol {x}, \boldsymbol {y})} (\boldsymbol {\theta}) = \rho \boldsymbol {x} ^ {T}, \tag {16}
$$
As a result, the gradient update will be confined within the input span  $(\pmb{x})$ , where the elements in  $\rho$  display heterogeneous magnitudes, thereby impacting the scaling of  $\pmb{x}$  correspondingly.
E Gradient Span Proof
Considering that the batch loss is the summation of losses incurred by individual examples, the overall batch loss for  $n$  samples can be represented as:
$$
L _ {\text {b a t c h}} = \sum_ {i = 1} ^ {n} L _ {i}, \tag {17}
$$
where  $L_{i}$  represents the loss of sample  $(x_{i},y_{i})$
When employing the mean-squared error loss function, the loss of a batch is calculated as the sum of the losses of individual samples, which can be expressed as:
$$
L _ {\text {b a t c h}} = \sum_ {i = 1} ^ {n} L _ {i} = \sum_ {i = 1} ^ {n} \frac {1}{2} \left\| \boldsymbol {\theta} \boldsymbol {x} _ {i} - \boldsymbol {y} _ {i} \right\| _ {2} ^ {2}. \tag {18}
$$
Following stochastic gradient optimization, we can present the gradient of this loss (per sample) with respect to weights as:
$$
\nabla_ {\boldsymbol {\theta}} \mathcal {L} = (\boldsymbol {\theta} \boldsymbol {x} - \boldsymbol {y}) \boldsymbol {x} ^ {T} = \boldsymbol {\Omega} \boldsymbol {x} ^ {T}, \tag {19}
$$
Here,  $\Omega \in \mathbb{R}^m$  denotes the error vector. Therefore, the gradient of the batch loss with respect to the weights can be expressed as:
$$
\nabla_ {\boldsymbol {\theta}} L _ {\text {b a t c h}} = \boldsymbol {\Omega} _ {1} \boldsymbol {x} _ {1} ^ {T} + \boldsymbol {\Omega} _ {2} \boldsymbol {x} _ {2} ^ {T} + \dots + \boldsymbol {\Omega} _ {n} \boldsymbol {x} _ {n} ^ {T}. \tag {20}
$$
It is noteworthy that the gradient update remains confined within the subspace spanned by the  $n$  input examples.
Considering cross-entropy (CE) loss as the desired loss, the batch loss would be the sum of the losses of individual samples as follows:
$$
L _ {\text {b a t c h}} = \sum_ {i = 1} ^ {n} L _ {i} = \sum_ {i = 1} ^ {n} \sum_ {k = 1} ^ {m} - y _ {k} \log a _ {k}, \tag {21}
$$