Supervised Batch Normalization

## Abstract

Abstract Batch Normalization (BN), a widely-used technique in neural networks, enhances generalization and expedites training by normalizing each mini-batch to the same mean and variance. However, its effectiveness diminishes when confronted with diverse data distributions.
To address this challenge, we propose Supervised Batch Normalization (SBN), a pioneering approach. We expand normalization beyond traditional single mean and variance parameters, enabling the identification of data modes prior to training. This ensures effective normalization for samples sharing common features. We define contexts as modes, categorizing data with similar characteristics. These contexts are explicitly defined, such as domains in domain adaptation or modalities in multimodal systems, or implicitly defined through clustering algorithms based on data similarity. We illustrate the superiority of our approach over BN and other commonly employed normalization techniques through various experiments on both single and multi-task datasets. Integrating SBN with Vision Transformer results in a remarkable 15.13 % accuracy enhancement on CIFAR-100. Additionally, in domain adaptation scenarios, employing AdaMatch demonstrates an impressive 22.25 % accuracy improvement on MNIST and SVHN compared to BN.

## I Introduction

In the realm of deep learning, input normalization is essential for optimizing the training process of deep neural networks (DNNs) by addressing the variations in feature magnitudes. This method has been shown to accelerate convergence in neural networks with a single hidden layer, as highlighted by LeCun et al. []. However, its efficacy diminishes in more complex architectures with multiple hidden layers. This decline is due to the progressive transformation of data through successive layers, which causes activations to diverge from the properties of the initially normalized inputs. To address this challenge, normalizing activations during training has become a critical approach. By ensuring that the statistical properties of activations remain consistent across all layers, this strategy facilitates stable and efficient training of deep neural networks. Consequently, this practice not only enhances the convergence rate but also significantly improves the overall performance of the model.
Batch Normalization (BN) [], a popular activation normalization technique, stabilizes the optimization process by normalizing feature statistics within a batch. Despite its widespread success, Batch Normalization (BN) has notable drawbacks due to its reliance on mini-batch statistics. While the variability in batch statistics can enhance robustness and generalization, it also leads to issues when the mean and variance estimates are inaccurate. This is particularly problematic with heterogeneous data and small batch sizes, which can cause BN to fail in effectively normalizing activations. In such cases, BN struggles to normalize activations using a single mean and variance [, , ].
To overcome these limitations, we introduce Supervised Batch Normalization (SBN). SBN assigns samples in a mini-batch to different modes using predefined groups called contexts, then normalizes each sample based on the statistics of its corresponding context. Instead of relying on random mini-batches, SBN utilizes contexts that group similar samples through domain knowledge or clustering algorithms. The proposed method can be seamlessly integrated as layers in standard deep learning libraries. We evaluated SBN on various classification tasks and demonstrated that it consistently outperforms BN and other widely used normalization techniques.

## II Related Work

### II-A Normalization methods

Batch normalization (BN) [] is the most common normalization technique in cutting-edge classification architectures. Recently, new alternatives have emerged to broaden its applicability and enhance its generalizability. Batch Renormalization [] is an extension of BN that addresses the issue of varying mini-batch statistics during training. Weight Normalization [] reparameterizes the weight vectors in a neural network by separating their magnitude and direction. This technique simplifies the optimization process and often results in faster convergence during training. It introduces additional parameters to stabilize training by aligning the statistics of the current mini-batch with the moving averages of the training data. Layer Normalization [] is a technique that normalizes samples across the features for each individual example, rather than across the min-batch. This approach helps stabilize the hidden states in recurrent neural networks and improves training efficiency by eliminating the dependency on mini-batch size. Instance Normalization [] normalizes samples across each feature map for individual examples, making it particularly effective for style transfer tasks. By focusing on the statistics of single instances, it helps preserve stylistic details and achieve more consistent visual outputs. Group Normalization [] divides the channels of each layer into smaller groups and normalizes the features within each group. This method provides stable training benefits similar to BN but is less sensitive to mini-batch size, making it suitable for tasks with small mini-batch sizes. Mode Normalization [] adjusts the normalization process based on the mode of the feature distributions instead of their mean. This method aims to better handle skewed data distributions, resulting in improved training stability and model performance. Mixture Normalization [] addresses the limitations of BN in capturing the complex variations present in deep neural network activations. By leveraging Gaussian Mixture Models to assign samples to components and normalize based on multiple means and standard deviations, MN adapts to the diverse modes of variation inherent in the data distribution. RMSNorm [] extends Layer Normalization by utilizing the root mean square (RMS) of the activations within each layer. This method aims to stabilize training by normalizing activations based on their magnitudes, providing a robust normalization technique for deep neural networks. Unsupervised Batch Normalization [] (UBN) leverages unlabeled examples to compute mini-batch statistics, addressing the challenge of bias on small datasets and offering regularization benefits from data manifold exploration. UBN demonstrates efficacy in tasks like monocular depth estimation, particularly beneficial where obtaining dense labeled data is challenging and costly.
While all these variants enhance the usability and stability of BN, our approach appears to be the first to extend BN by incorporating contexts, predefined groups of samples with shared characteristics, for normalization purposes.

### II-B Incorporating Multiple Modes for Effective Normalization

BN has been widely adopted in deep learning architectures to improve training stability and convergence. However, BN’s assumption that the entire mini-batch should be normalized with the same mean and variance poses challenges, especially in the face of diverse data distributions. This assumption can lead to suboptimal performance, particularly on datasets with varying characteristics. Recent research has highlighted the limitations of this assumption, emphasizing the importance of accommodating multiple modes of variation within the data distribution. Approaches such as Mixture Normalization [], which employs Gaussian Mixture Models to capture multiple means and variances associated with different modes of variation, have been proposed to address this issue. Similarly, studies like Luo et al. [] have underscored the necessity of considering diverse data distributions and employing multiple mean and variance estimates for effective normalization. These insights emphasize the importance of moving beyond the simplistic assumptions of BN to better accommodate the complexities of real-world datasets.

## III Method

We begin by examining the formulations of BN with a single mode in Section III-A, followed by an exploration of BN with multiple modes in Section III-B. Finally, we present our method in Section III-C.

### III-A Batch Normalization with Single Mode

Given an input mini-batch of height $H$ and width $W$ with $N$ samples and $C$ channels, represented as $x\in\mathbb{R}^{N\times C\times H\times W}$, BN normalizes each sample along the channel dimensions as follow:

$$ $\hat{x}_{n}=\gamma(\frac{x_{n}-\mu}{\sqrt{\sigma^{2}+\epsilon}})+\beta,$ (1) $$

where $\mu$ and $\sigma^{2}$ represent the mean and variance respectively. Parameters $\gamma$ and $\beta$ are $C$-dimensional vectors aimed at learning an affine transformation along the channel dimensions, thereby preserving the representative capacity of each layer. while $\epsilon>0$ serves as a small value to mitigate numerical instability.
The moving average of the mean $\bar{\mu}$ and variance $\bar{\sigma}^{2}$ are updated using a momentum rate $\alpha$ during training and used to normalize feature maps during inference:

$$ $\bar{\mu}=\alpha\bar{\mu}+(1-\alpha)\mu$ (2) $$

$$ $\bar{\sigma}^{2}=\alpha\bar{\sigma}^{2}+(1-\alpha)\sigma^{2}$ (3) $$

When the samples within the mini-batch are drawn from the same distribution, the operation outlined in Equation 1 results in a distribution characterized by a mean of zero and a variance of one. This requirement for zero mean and unit variance acts to stabilize the activation distribution, thereby facilitating the training process. However, in scenarios where the samples stem from diverse distributions, a single mean and variance may prove insufficient, necessitating the adoption of strategies involving multiple modes (i.e., employing multiple means and variances) to achieve optimal results [, ].

### III-B Batch Normalization with Multiple Modes

The heterogeneous nature of complex datasets necessitates extending BN to multiple modes, enabling a more flexible and effective approach to normalization. A popular method that facilitates this is Mixture Normalization (MN) []. MN approaches BN from the perspective of Fisher kernels, derived from generative probability models. Instead of computing a single mean and variance across all samples within a mini-batch, MN employs a Gaussian Mixture Model (GMM) to assign each sample in the mini-batch to a component, then normalizes using multiple means and variances associated with different modes of variation in the underlying data distribution. Considering $K$ components, MN is implemented in two stages:

- •
Estimation of the mixture model’s parameters $\theta=\{\lambda_{k},\mu_{k},\sigma^{2}_{k}:k=1,\ldots,K\}$ using the Expectation-Maximization (EM) algorithm [].
- •
Normalization of each sample based on the estimated parameters and aggregation using posterior probabilities.

For a given input mini-batch $x\in\mathbb{R}^{N\times C\times H\times W}$, each sample $x_{n}$ is normalized along the channel dimensions as follows:

$$ $\hat{x}_{n}=\gamma(\sum_{k=1}^{K}\frac{p(k|x_{n})}{\sqrt{\lambda_{k}}}.\frac{x _{n}-\mu_{k}}{\sqrt{\sigma_{k}^{2}+\epsilon}})+\beta,$ (4) $$

where $p(k|x_{n})=\frac{\lambda_{k}p(x_{n}|k)}{\sum_{j=1}^{K}\lambda_{j}p(x_{n}|j)}$ represents the probability that $x_{n}$ has been generated by the $k^{th}$ Gaussian component, with $p(x_{n}|k)$ and $\lambda_{k}$ denoting the density function of the Gaussian distribution and the mixture coefficient, respectively. The estimators for the mean $\mu_{k}$ and variance $\sigma_{k}^{2}$ are computed by weighting the contributions of $x_{n}$ ($\frac{p(k|x_{n})}{\sum_{j}p(j|x_{n})}$) with respect to the mini-batch when estimating the statistical measures of the $k$-th Gaussian component. Specifically, the $k$-th mean and variance are estimated from the mini-batch as follows:

$$ $\mu_{k}=\sum_{n}\frac{p(k|x_{n})}{\sum_{j}p(j|x_{n})}\cdot x_{n}$ (5) $$

$$ $\sigma^{2}_{k}=\sum_{n}\frac{p(k|x_{n})}{\sum_{j}p(j|x_{n})}\cdot(x_{n}-\mu_{k })^{2}$ (6) $$

Multiple modes normalization methods extend Batch Normalization (BN) to heterogeneous complex datasets and often yield superior performance in supervised learning tasks. However, they are frequently computationally expensive due to tasks such as estimating different modes, such as the EM algorithm in Mixture Normalization (MN), and employing mixtures of experts [, ] in Mode Normalization.
To address the challenge of multiple modes and reduce computational costs compared to existing methods, we propose an approach that leverages prior knowledge to construct modes. This method significantly reduces costs while maintaining or even enhancing performance.

### III-C Supervised Batch Normalization

Our proposed method, SBN, introduces a novel approach to enhance neural network training efficiency. SBN operates by initially grouping samples into $K$ distinct contexts prior to training. Subsequently, during the training process, samples belonging to the same context $k$ within a given mini-batch are normalized using identical parameters $\mu_{k}$ and $\sigma_{k}^{2}$. By leveraging these predefined contexts, each comprising samples with similar characteristics, SBN effectively introduces multiple modes without incurring the computational overhead associated with estimating them during neural network training. This approach streamlines the normalization process and significantly reduces computational costs, thereby enhancing training efficiency and overall model performance.

#### III-C 1 Understanding Context: Definition and Construction Methods

Context serves as the foundational element within SBN, representing groups of samples sharing similar characteristics. Our approach offers diverse methods for context construction:

- •
For domain adaptation tasks [, , ], each domain is treated as a distinct context.
- •
In datasets featuring additional hierarchical structures, such as CIFAR-100 [] or the Oxford-IIIT Pet dataset [], we designate each superclass as a separate context.
- •
For datasets lacking predefined contextual structures, we employ clustering algorithms like k-means [] to partition samples into clusters, with each cluster forming an individual context.

This multifaceted approach ensures flexible and comprehensive context formation, vital for the effective implementation of SBN across various domains and datasets.

#### III-C 2 Training and Inference with Supervised Batch Normalized Networks

Consider $x\in\mathbb{R}^{N\times C\times H\times W}$ as a given input mini-batch and $K$ as the number of defined contexts. To normalize $x$, we first partition the samples in $x$ into $K$ groups based on their contexts, with each group $x^{(k)}$ containing samples that belong to context $k$. Each sample $x_{n}$ in $x^{(k)}$ is normalized using the same mean $\mu_{k}$ and variance $\sigma_{k}^{2}$ as given by Equation 4. Since each $x_{n}$ belongs to a single known context, $p(k|x_{n})=1$ if $x_{n}$ is in context $k$ and $p(k|x_{n})=0$ otherwise. Consequently, Equation 4 simplifies to:

$$ $\hat{x}_{n}=\gamma(\frac{1}{\sqrt{\lambda_{k}}}.\frac{x_{n}-\mu_{k}}{\sqrt{ \sigma_{k}^{2}+\epsilon}})+\beta,$ (7) $$

where $\lambda_{k}$ represents the proportion of samples in the dataset belonging to context $k$. The mean and variance are then defined as follows:

$$ $\mu_{k}=\frac{1}{N_{k}}\cdot\sum_{n=1}^{N_{k}}x_{n}$ (8) $$

$$ $\sigma^{2}_{k}=\frac{1}{N_{k}}\cdot\sum_{n=1}^{N_{k}}(x_{n}-\mu_{k})^{2}$ (9) $$

where $N_{k}$ is the number of samples in the mini-batch that belong to context $k$.
The moving averages of the mean $\bar{\mu}$ and variance $\bar{\sigma}^{2}$ are updated with a momentum rate $\alpha$ during training. These updated values are then utilized to normalize feature maps during inference:

$$ $\bar{\mu}_{k}=\alpha\bar{\mu}_{k}+(1-\alpha)\mu_{k}$ (10) $$

$$ $\bar{\sigma}^{2}_{k}=\alpha\bar{\sigma}^{2}_{k}+(1-\alpha)\sigma^{2}_{k}\newline$ (11) $$

In the case where $K=1$, it can be noted that SBN is equivalent to BN with a single mode.

During inference, for a given sample $x_{n}$, there are two possible normalization approaches. If the context of $x_{n}$ is known and identified as $k$, we normalize it using Equation 7 with the context-specific mean $\bar{\mu}_{k}$ and variance $\bar{\sigma}^{2}_{k}$. On the other hand, if the context of $x_{n}$ is unknown, we normalize it using Equation 4, which aggregates the normalization parameters across all $K$ contexts. This ensures that the sample is appropriately normalized regardless of whether its specific context is known.

Figure: Algorithm 1 Supervised Batch Normalization, training and inference phases

The detailed steps for the training and inference phases of SBN are provided in Algorithm 1. This algorithm meticulously outlines the procedures for both phases, demonstrating how SBN normalizes mini-batches by leveraging context-specific grouping.

SBN extends BN to multiple modes without added cost by leveraging pre-defined contexts before training. Experiments on small datasets and classification tasks show improved convergence and performance compared to BN and other multi-mode normalization methods.

## IV Analyzing SBN in a Simplified Scenario

To demonstrate the principles behind SBN and its distinctions from BN, we conduct an experiment using a toy example. We train a simple 4-layer convolutional network with BN layers on the CIFAR-10 dataset []. This dataset’s simplicity allows for a deeper analysis, which would be challenging with a more complex task. For comparison, we create another model by replacing BN layers with SBN layers. To construct contexts for SBN, we use k-means clustering and vary the number of contexts across $K=\{2,4,6,8\}$. Training is conducted on 50,000 data points with a fixed mini-batch size of 256. All models are trained for 100 epochs using the AdamW optimizer [, ], with a weight decay parameter set to $10^{-4}$.

**TABLE I: Test set accuracy rates (%) of batch normalization (BN) and supervised batch normalization (SBN) on the CIFAR-100 dataset. SBN-$k$ denotes SBN with $k$ contexts.**
| model | 25 epochs | 50 epochs | 75 epochs | 100 epochs |
| --- | --- | --- | --- | --- |
| BN | 84,34 | 86,49 | 86,41 | 86,90 |
| SBN-2 | 85.56 | 87.62 | 87.70 | 87.70 |
| SBN-4 | 86.78 | 87.94 | 87.94 | 88.02 |
| SBN-6 | 86.79 | 88.00 | 88.48 | 88.56 |
| SBN-8 | 87.01 | 87.90 | 88.90 | 89.06 |

Table I demonstrates that SBN outperforms standard BN, indicating that incorporating multiple contexts is an effective method for normalizing intermediate features, even when the data is not heterogeneous.

Increasing the number of contexts $K$ does not affect performance, unlike other normalization methods with multiple modes where increasing the number of modes can decrease performance. This is likely due to finite estimation, where estimates are computed from increasingly smaller batch partitions, a known issue in traditional BN.

## V Experiments

We evaluate our methods in two experimental settings: (i) multi-task (heterogeneous dataset) and (ii) single task. To contrast with our proposed method SBN, we will utilize Batch Normalization (BN), Layer Normalization (LN), Instance Normalization (IN), Mixture Normalization (MN), and Mode Normalization (ModeN).

### V-A Multi-task: Utilize each domain as a context

In this experiment, we demonstrate how SBN can significantly enhance domain adaptation by improving local representations. Domain adaptation involves leveraging knowledge from a related domain, where labeled data is abundant, to enhance model performance in a target domain with limited labeled data. We use two contexts ($K=2$): the ”source domain” and the ”target domain”. We apply normalization methods with AdaMatch, which combines unsupervised domain adaptation (UDA), semi-supervised learning (SSL), and semi-supervised domain adaptation (SSDA). In UDA, we use labeled data from the source domain and unlabeled data from the target domain to train a model that generalizes effectively to the target dataset. Notably, the source and target datasets have different distributions, with MNIST as the source dataset and SVHN as the target dataset, encompassing various factors of variation such as texture, viewpoint, and appearance.

A model, referred to as AdaMatch [] (using BN layers), is trained from the ground up using wide residual networks [] on pairs of datasets, serving as the baseline model. The training of this model involves utilizing the Adam optimizer [] with a cosine decay schedule, gradually reducing the initial learning rate initialized at $0.03$. For comparison purposes, we substitute BN layers with LN, IN, MN, ModeN, and SBN. In MN, we employ $K=2$ Gaussian components, and for ModeN, we utilize $K=2$ modes.

**TABLE II: Test set performance rates (%) for BN, LN, IN, MN, ModeN, and SBN on multi-task with heterogeneous dataset SVHN+MNIST for domain adaptation.**
| MNIST (source domain) |  |  |  |  |
| --- | --- | --- | --- | --- |
| model | accuracy | precision | recall | f1-score |
| BN | 97.36 | 87.33 | 79.39 | 78.09 |
| LN | 96.23 | 88.26 | 76.20 | 81.70 |
| IN | 99.41 | 99.41 | 99.41 | 99.41 |
| MN | 98.90 | 98.45 | 98.89 | 98.93 |
| ModeN | 98.93 | 98.3 | 98.36 | 98.90 |
| SBN (ours) | 99.17 | 99.17 | 99.17 | 99.17 |

Table II presents the test set performance rates (%) for various normalization methods in a multi-task setting with the heterogeneous SVHN+MNIST dataset for domain adaptation. Notably, our proposed method, SBN, demonstrates significant improvements, particularly in the challenging SVHN target domain. Compared to BN, SBN achieves a remarkable gain in accuracy, with a 22.25% increase. This highlights the efficacy of SBN in adapting to diverse datasets, even outperforming other normalization methods like MN and ModeN, which are based on multiple modes assumption. These results underscore the effectiveness of SBN in enhancing model performance across heterogeneous domains, making it a promising choice for domain adaptation tasks.

### V-B Single task: Utilise each superclass as a context.

This experiment’s main focus is on leveraging CIFAR-100 superclasses as contexts ($K=20$) to predict the dataset’s 100 classes, particularly with SBN. We utilize the base Vision Transformer model [] obtained from Keras [] as our baseline. To conduct comparisons, we modify this baseline by substituting different normalization layers. The training process includes early stopping based on validation performance, and image preprocessing involves normalization with respect to the dataset’s mean and standard deviation. Additionally, data augmentation techniques such as horizontal flipping and random cropping are applied to enrich the dataset. To optimize model parameters and prevent overfitting, we employ the AdamW optimizer with a learning rate of $10^{-3}$ and a weight decay of $10^{-4}$ [, ]. Training is carried out for 100 epochs.

For Mixture Normalization (MN) and Mode Normalization (ModeN), determining the appropriate number of components and modes respectively involves conducting multiple tests. We save the best results (ref. Table III) achieved with $K=5$ for MN and $K=3$ for ModeN.

**TABLE III: Test set performance rates (%) for BN, LN, IN, MN, ModeN, and SBN on a single-task classification task using the CIFAR-100 dataset.**
| model | accuracy | precision | recall | f1-score |
| --- | --- | --- | --- | --- |
| BN | 55.63 | 8.96 | 90.09 | 54.24 |
| LN | 54.05 | 11.82 | 85.05 | 53.82 |
| IN | 54.85 | 11.63 | 86.05 | 54.71 |
| MN | 53.2 | 11.20 | 87.10 | 54.23 |
| ModeN | 54.10 | 12.12 | 87.23 | 54.98 |
| SBN (ours) | 70.76 | 27.59 | 98.60 | 70.70 |

Figure: (a) Training Error
Refer to caption: extracted/5623235/images/train.png

Table III highlights the significant performance gains achieved by SBN compared to other normalization techniques (BN, LN, IN, MN, and ModeN). SBN shows a remarkable accuracy improvement of approximately 15.113% over BN. It’s worth noting that multiple modes normalization methods (MN, ModeN) do not perform well in this single-task scenario. However, by leveraging superclasses as contexts and normalizing accordingly, SBN outperforms all known ViT models trained from scratch on CIFAR-100. Figure 1 shows that SBN accelerates learning. These results indicate that SBN stabilizes data distributions, mitigates internal covariate shift, and significantly reduces training time for better outcomes.

## VI Conclusion

Our study introduces a groundbreaking normalization technique called Supervised Batch Normalization (SBN), which extends the capabilities of traditional Batch Normalization (BN) to effectively handle heterogeneous datasets characterized by diverse data distributions. Unlike BN, which normalizes each mini-batch using a single mean and variance, SBN addresses the challenge posed by varied data distributions within a mini-batch by normalizing based on grouped data with similar characteristics, referred to as contexts. We present three methods to accurately define these contexts.

Experimental results from both multi-task scenarios with heterogeneous datasets and single-task scenarios with homogeneous datasets demonstrate that SBN consistently outperforms BN and its variants, including methods based on multiple modes such as Mixture Normalization and Mode Normalization. SBN offers ease of implementation and versatility, serving as a powerful layer in neural networks to enhance performance and accelerate convergence.

Looking ahead, our future research will delve into exploring the robustness of SBN in multimodal systems, such as those involving text, image, audio, and other modalities, where contexts are well-defined and critical for effective normalization strategies.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: * The process for constructing contexts is unclear and lacks detailed methodologies.
* The underlying principles behind SBN are insufficiently explained, making the approach less convincing.
* The novelty of SBN may be limited, as it could be perceived as a specific case of Mixture Normalization (MN) or Mode Normalization (ModeN).
soundness: 2
strengths: * SBN effectively addresses limitations of traditional BN in handling heterogeneous data, offering a solution applicable across domains with varied data distributions.
* Experiments illustrate substantial performance gains, with SBN consistently outperforming other normalization methods on classification and domain adaptation tasks.
* The method integrates seamlessly with standard deep learning frameworks, making it accessible for practical implementation in various tasks.
summary: The paper proposes Supervised Batch Normalization (SBN), a novel normalization technique aimed at enhancing the effectiveness of Batch Normalization (BN) for diverse data distributions. Traditional BN methods normalize each mini-batch using a single mean and variance, which may be suboptimal for heterogeneous data. SBN introduces “contexts”, which are predefined groups of similar samples, to normalize different subsets of data using multiple means and variances. The three methods for defining contexts in SBN are: by using predefined domains in domain adaptation tasks, grouping by superclasses, and clustering with algorithms like k-means for datasets without clear context divisions. Through experiments on both single-task (CIFAR-100) and multi-task (MNIST-SVHN) scenarios, SBN demonstrates significant improvements over BN and other normalization methods, achieving 15.13% and 22.25% accuracy improvement respectively.
weaknesses: * The process for constructing contexts is unclear and lacks detailed methodologies.
* The underlying principles behind SBN are insufficiently explained, making the approach less convincing.
* Using k-means clustering to define contexts in large-scale datasets demands substantial computation, potentially prolonging training time, and lacks experimental support in this paper.
* The novelty of SBN may be limited, as it could be perceived as a specific case of Mixture Normalization (MN) or Mode Normalization (ModeN).
* It is not clear how $\lambda$ and $p$ are calculated during inference.
* The paper contains several textual errors, such as “then normalizes each sample based onin neural networks, enhances generalization and expedites train- the statistics of its corresponding context” in Introduction section, issues with punctuation in Table 1, an incorrect f1-score for SBN in Table 2. Tables should not be inserted in the middle of a sentence.
* The cited literature lacks references from recent years, and SBN lacks comparison with recent methods.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: - Verify the effectiveness on large-scale datasets and compare with strong baseline methods.
- Prove the effectiveness of context group generation.
- Compare the influence between training/testing inconsistency and context group inconsistency.
soundness: 2
strengths: - The issue of batch normalization under diverse data distributions is an important problem.
- The idea of context group for batch normalization is interesting. While the way to get the context group seems trival.
summary: This paper studies the problem of batch normalization under diverse data distributions. A supervised batch normalization method is proposed to solve this problem, where the data modes are identified before training, and the mean and variances are used within the same context group. Experimental results on small datasets such as CIFAR, MNIST, and SVHN show the effectiveness of the proposed method.
weaknesses: - During inference, when the contexts are not known, the mean value of statistics from all context groups is used, which is inconsistent with the training process. The inconsistency between training and testing should also be considered and might be more important than the gap among different context groups.

- In the experiments on CIFAR, k-means clustering is used to obtain the context group. While k-means cannot perform highly accurate classification, it is unclear if the quality of the context group obtained in this way is meaningful or not.

- The performance is only evaluated on small datasets such as CIFAR and MNIST. Experiments on large-scale datasets such as ImageNet should be conducted to verify the effectiveness of the proposed method.

- The domain adaptation ability of the proposed method is evaluated. While adapting the variance and mean of norms is widely used in domain adaptation methods, this method does not compare with those methods and is only evaluated on two toy datasets.

- For single-task evaluation, although there are many context-adaptive normalization methods, this method only compares with a few basic normalization schemes on small datasets. The effectiveness of the proposed method is unclear.

- The writing should be improved:
  - "then normalizes each sample based onin neural networks" — what does "onin neural networks" mean?

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Please see the weakness part.
soundness: 2
strengths: 1. Clear presentation
2. Significant improvement on baselines
summary: This paper proposes a supervised batch normalization (SBN) method. This method splits various modes using the dataset's contexts.
Then, it builds a multi-mode BN to replace the traditional BN in the model.
The proposed method can be seamlessly integrated as layers in standard deep learning libraries.
And the authors have evaluated that the SBN outperforms BN and other widely used normalization techniques on various classification tasks.
weaknesses: 1. Technical novelty is limited. The core algorithm of the SBN has been discussed in mixture BN. The innovation of this article lies only in using prior information to replace the original clustering center.

2. The experiments are unfair. For the supervised classification task, e.g., cifar100, SBN provides the superclass classification but the compared methods do not.

3. Some conclusions have not been fully substantiated. Like in Section 4, the authors claim "increasing the number of contexts K does not affect performance", however, this phenomenon is only present in one toy experiment, and no analytical evidence is provided.

4. The principle behind the effectiveness of this method has not been explained clearly. That is, why is SBN better than MN or ModeN with the context obtained from unsupervised clustering?

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: 1. Would the method show similar improvements on larger and more complex datasets like ImageNet?
2. How does the performance compare with Batch Renormalization or other alternatives in dynamic domain adaptation settings?
3. Can the method handle ambiguous or overlapping contexts effectively? How is normalization affected when contexts are not well-defined?
soundness: 2
strengths: 1. Originality: The idea of introducing predefined contexts within batch normalization offers a creative extension of BN.
2. Performance Gains: It achieves solid improvements on the datasets considered, particularly in multi-task domain adaptation.
3. Efficient Integration: The approach reduces computational overhead by leveraging predefined modes, avoiding costly estimations like those in Mixture Normalization.
summary: The paper “Supervised Batch Normalization” (SBN) introduces a novel extension to traditional Batch Normalization (BN). It aims to address BN’s limitations by grouping samples into predefined contexts based on shared characteristics (e.g., domains or clusters) and applying normalization specific to those contexts. This multi-mode normalization strategy improves performance in various tasks, such as classification on CIFAR-100 and domain adaptation between MNIST and SVHN. Notably, it reports 15.13% improvement on CIFAR-100 and a 22.25% increase in accuracy on SVHN compared to BN.
weaknesses: 1. Lack of Large-Scale Validation: The experiments on small datasets (MNIST, SVHN) cannot convincingly demonstrate the method’s efficacy. Larger datasets like ImageNet are crucial to validate whether SBN scales effectively in real-world applications.
2. Insufficient Baseline Comparisons: While SBN shows improvements over BN, comparisons with other advanced methods like Batch Renormalization are missing. Moreover, the results rely on limited normalization tasks, which weakens the generalizability of the findings.
3. Limited Exploration of Complex Contexts: The method assumes clear and well-separated contexts, but real-world scenarios often involve overlapping contexts (e.g., blurry domain boundaries in domain adaptation). The paper does not adequately address how SBN performs under these conditions. You should find or build a similar dataset to prove your contribution.
4. Experiments now I don't think will convince everyone in 2024 that your work is sufficiently productive and novel.

Meta-review: This paper proposed a supervised batch normalization, which actually has limited novelty. Similar supervision in batch normalization has been explored in [R1] Haifeng Xia, and Zhengming Ding. Cross-Domain Collaborative Normalization via Structural Knowledge. Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI), 2022.

The comparisons are not enough. The authors did not reply to the reviewers' comments.

Decision: Reject