Scale Equalization for Multi-Level Feature Fusion
Authors: Bum Jun Kim
Sang Woo Kim
Venue: Accepted by TMLR ()

Abstract: Deep neural networks have exhibited remarkable performance in a variety of computer vision fields, especially in semantic segmentation tasks. Their success is often attributed to multi-level feature fusion, which enables them to understand both global and local information from an image. However, multi-level features from parallel branches exhibits different scales, which is a universal and unwanted flaw that leads to detrimental gradient descent, thereby degrading performance in semantic segmentation. We discover that scale disequilibrium is caused by bilinear upsampling, which is supported by both theoretical and empirical evidence. Based on this observation, we propose injecting scale equalizers to achieve scale equilibrium across multi-level features after bilinear upsampling. Our proposed scale equalizers are easy to implement, applicable to any architecture, hyperparameter-free, implementable without requiring extra computational cost, and guarantee scale equilibrium for any dataset. Experiments showed that adopting scale equalizers consistently improved the mIoU index across various target datasets, including ADE20K, PASCAL VOC 2012, and Cityscapes, as well as various decoder choices, including UPerHead, PSPHead, ASPPHead, SepASPPHead, and FCNHead.

Published in Transactions on Machine Learning Research (11/2024)
Scale Equalization for Multi-Level Feature Fusion
Bum Jun Kim
kmbrjjn @posttchhhhptcccccco eduu
Department of Electrical Engineering
Pohang University of Science and Technology
Sang Woo Kim
sukim edu
Department Electrical Engineering
Pohang University of Science and Technology
Reviewed on OpenReview: Attps: // openreview. net/ formf id- 0S4SKVKA7S
Abstract
Deep neural networks have exhibited remarkable performance a variety of computer vi- sion fields, especially semantic segmentation tasks. Their success is often attributed to multi-level feature fusion, which enables them to understand both global and local in- formation from an image. However, multi-level features from parallel branches exhibits different scales, which is universal and unwanted flaw that leads to detrimental gradient descent, thereby degrading performance in semantic segmentation. We discover that scale disequilibrium is caused by bilinear upsampling, which is supported by both theoretical and empirical evidence. Based on this observation, we propose injecting scale equalizers to achieve scale equilibrium across multi-level features after bilinear upsampling, Our proposed scale equalizers are easy to implement, applicable to any architecture, hyperparameter- ffeee, implementable without requiring extra computational cost, and guarantee scale equilibrium for any dataset. Experiments showed that adopting scale equalizers consistently improved the mloU index across various target datasets, including ADE20K, PASCAL VOC 2012, and Cityscapes, as well as various decoder choices, including UPerHead, PSPHead, ASPPHead, SepASPPHead, and FCNHead.
1 Introduction
Deep neural networks have shown remarkable performance, especially in the computer vision field. Their substantial modeling capability has enabled us develop significantly accurate models with rich image features for a wide range of vision tasks, including object detection and semantic segmentation.
One challenge in computer vision tasks is understanding both the global and local contexts of an image (Reddi et al. 2018 Tu et al. 2022). Indeed, the cascade architecture of deep neural network faces difficulty in understanding multiple contexts of an image owing to the single-level feature it uses. To address this problem, modern vision networks have employed parallel architecture that aggregates multi-level features in different spatial sizes to extract both global and local information from an image. For semantic segmentation as an example, multi-level feature fusion has been adopted in numerous models such as UPerNet (Xiao et al. 2018), PSPNet (Zhao et al. 2017), DecpLabV3 (Chen al. 2017). DecpLabV3+ Chen al. 2018b), FCN [ongg al., 2015), and U-Net (Ronneberger al. 2015).
Although the parallel architecture builds multiple fastlanes to facilitate multi-level features to contribute to output, if certain features are not involved in the fusion, they simply waste computational resources. Initially, all feature branches should be exploited to explore their potential usefulness, and after training, their optimal combination should be obtained. Thus, the underlying assumption of multi-level feature fusion is that, least in an initialized state, all multi-level features will participate producing a fused feature. However, claim that existing architectural design for multi-level feature fusion in semantic segmentation has potential problem of scale disequilibrium, which yields unwanted bias that diminishes the contribution

---

Published in Transactions on Machine Learning Research (11/2024)
of several features. Specifically, the multi-level features exhibit different scales at initialization, which leads to different contributions and gradient scales, thereby hindering training with gradient descent. We identify the cause of the scale disequilibrium bilinear upsampling, which is used to enlarge multi-level features to the same spatial size. Demonstration of the scale discquilibrium caused by bilincar upsampling is provided theoretically and empirically.
To solve the scale disequilibrium problem, this study proposes injecting scale equalizers into multi-level feature fusion. The scale equalizer normalizes each feature using the global mean and standard deviation, which guarantees scale equilibrium across multi-level features. Because the proposed scale equalizer is simply global normalization, which uses empirical values for subtraction and division, its implementation is easy and hyppeparaaeteer-free, requires little extra computation that is actually free, and assures scale equilibrium for any datasets and architectures. Experiments on semantic segmentation tasks showed that applying scale equalizers for multi-level feature fusion consistently improved the mIoU index across extensive experimental sctups, including datasets and backbones.
2 Background
Formulation This study considers the standard framework for supervised learning of semantic segmenta- tion networks because it is a representative task using multi-level feature fusion (Fig. D. Let I R#xxxC be an input image to a semantic segmentation network, where H, W is the size of the image and C' repre- sents the number of image channels. The objective of semantic segmentation is to generate a semantic mask Y x that classifies cach pixel in the image I into one of the N₂ categorics. A deep ncural network, which comprises an encoder and decoder, is employed as a semant segmentation model that outputs Y from the input image I. The encoder is a backbone network with several stages where the input image first goes through. The decoder-also referred to the head head-uses set of intermediate feature maps {C;], from the encoder to produce the segmentation output Y. To quantify a difference between the prediction Y and the ground truth Y, loss function such as pixel-wise cross-entropy is used. With gradient descent optimization for the loss function, the encoder and decoder are trained together on an image-label pair dataset by the fine-tuning strategy, where the encoder begins with a pretrained weight whereas the decoder is trained from scratch.
2.1 Multi-Stage Feature Fusion
The last fcature map of the encoder contains rich, high-level information on the image (Chen ct al. 2018a) and is included in the set of fealure maps used by the decoder. However, each stage of the encoder yields a downsampled feature map. Thus, the last feature map exhibits severe downsampling rate, losing fine details in the image (Chen et al. 2017 To address this problem, modern decoders have used multiple feature maps from several stages to aggregate information with various spatial sizes (Kirillov et al. 2019 Zheng et al., 2021 We refer to this practice as mulli-slage feature fusion. For multi-stage feature fusion, the common choice on the set of features is to use the encoder outputs of the second to fifth stages, i.e., (C2, Ca, Ca, Cs [Lin et al., 2017). The use of the first stage output commonly avoided bocause it requires large GPU memory. For convolution-based residual networks (He et. al. 2016) and certain vision transformers such Swin (Liu et al. ) the downsampling ratios of the four encoder features are {4, 8, 16, 32}. Others, such vanilla vision transformers (Dosovitskiy et al.) 2021), keep the same spatial size for the four encoder features. Despite the promising results of the latter, in general vision tasks, progressive downsampling is critical to encouraging heterogeneous characteristics in multiple feature maps, which advocates the former networks. This study targets the former and describes a problem regarding feature fusion using different downsampling ratios. The remainder of this section reviews the detailed mechanism of a modern decoder with multi-stage feature fusion.
UPerHead UPerHead, the head deployed in UPerNet (Xiao et al. 2018 is a prime example of a decoder designed for multi-stage feature fusion. Recent vision transformers have preferred the UPerHead for semantic segmentation tasks (Dosovitskiy et al. 2021 Bao et al. 2022 He al.] 2022), and their remarkable perfor- mance let it be one of the most widely used decoders in the current state. The UPerHead comprises different
2

---

Published in Transactions on Machine Learning Research (11/2024)
Table 1: List of notations used this study.
Notation
Meaning
I
An input image to a semantic segmentation network.
H, C,W.C
Height, width, and the number of channels for the input image. Predicted result for semantic segmentation.
Y
Ne
The number of classes to be classified in the semantic segmentation task. A set of intermediate feature maps {C:} from the encoder.
{C.)
[L)
Laterals UPerHead.
(F.)
Outputs of the top-down pathway the FPN.
(P.)
Outputs of the FPN, which is concatenation subjects with optional bilinear upsampling. TX blinear upsampling.
UP,
Upsampling ratio in bilinear upsampling.
h
A convolutional unit block.
Z
Fused feature after multi-level feature fusion. Output stride, i.e. downsampling ratio at the last stage. The number of branches multi-level feature fusion. ith concatenation subject in multi-level feature fusion. ith weight of the lincar layer in multi-level feature fusion. Bias in the linear layer in multi-level feature fusion. The fused feature Partial derivative of y with respect to ws. Mean of feature
8
mb
rz
b
Σ. w + b in multi-level feature fusion.
av.
[[x
Var[x]
Variance of feature Loss function such as the pixel-wise cross-entropy loss function. Learning rate used in gradient descent. Scale term in batch normalization, which is initialized to one. Shift term in batch normalization, which is initialized to zero. Weight matrix.
L
B
W
Feature vector.
ReLU
Rectified Linear Unit ReLU(x)= max(0, Batch normalization operation.
BatchNorm
Wfuse
Weight matrix of the convolutional layer in the convolutional unit block of fusion. An intermodiate result that is obtained after the convolutional layer fusion with Wiuse Arbitrary feature.
fuse
X
((,,02)
Normal distribution with mean μ and variance d².
modules, such as the pyramid pooling module (PPM) (Zhao et al. 2017 feature pyramid network (FPN) [Lin et al. (017), and convolutional unit block, which is composed of convolution, batch normalization, and ReLU operation. Firstly, each of the three feature maps (C2, C3, C4} is subjected to a convolutional unit block to yield laterals (L2, Lg, L,}. Additionally, the last lateral Lş is produced from the last feature map C₁ using the PPM module that is described in Section 2.2 Now, a subnetwork called FPN performs the top-down pathway to laterals to obtain its output F; = L, + (P,41.+1) for i (2, 4} and F; = Ls, where bilinear upsampling (Appendix A Subsequently, FPN applies a convolu-
the operation UP, denotes tional unit block hy to each result to yield its output P: h, for i E (, 4,5). The set of FPN output (P2, Ps, P4, Ps} has the same spatial size encoder features (C2, C3, C4, Cs). keeping their downsampling ratios (4, 8, 16, 32). Thus, feature fusion for FPN outputs requires 2'-2x x bilinear upsampling for each P; to ensure that they share the same spatial size of H/4 x W/4. Finally, they can be concatenated together with respect to channel dimension and fused with a convolutional unit block as
Z = h( (P2(P3) UP.(P.) UPs(P:))
(1)
3

---

Published in Transactions on Machine Learning Research (11/2024)
Image
Prediction
I
Input to the backbone
Generate a segmentation result
C2
P2
C3
P3
Upsample, Concat,
Z
C4
P4
Conv,
C5
P5
Specific pipeline depending on decoder
Encoder
FPN
Fusion
Figure An overview of semantic segmentation network. Input image fed the backbone yield encoder features (C,)- They used to obtaín FPN outputs (P:), which are fused through upsampling, concatenation, convolution, etc. Finally, segmentation result Y is generated using the fused feature Z.
The fused feature Z is then subjected to a x convolution and a 4x bilinear upsampling to yield a predicted semantic mask Y, which has the size of H W N..
2.2 Single-Stage Feature Fusion
Although multi-stage feature fusion uses a set encoder features from several stages, certain decoders such as PSPHead (Zhao et al. 2017) or ASPPHead (Chen et al.) 2017) only use a single feature map from the last stage Cş. They modily the encoder to exhibit a downsampling ratio of 8 or at the last stage, which is referred
the output stride. Denoting the output stride as s, the spatial size of the last feature map Cs is (H/s) x (W/s). These decoder heads perform dissimilar feature fusion: From single-stage feature, multiple feature maps with various sizes are produced, which are then fused. We refer to this type of feature fusion as single-stage feature fusion. In similar but different way, single-stage feature fusion enables the decoder to extract both global and local contexts from the targeted feature map. The remainder of this section reviews the detailed mochanisms of modern decoders with singlo-stage fcature fusion.
PSPHead PSPHead refers to the head deployed in PSPNet (Zhao al. 2017 Its underlying mechanism is to extract global and local contexts from fcature map using multiple branches, which is referred to a pyramid pooling module (PPM). Targeting the last feature map Cs, it performs four average poolings in parallel, which yields features with spatial sizes x 1, 2x2, 3x3, and 6x6. Subsequently, a convolutional unit block is applied cach branch, and then each result upsampled to fit the size of C₁- Along with the feature map Cs, the results from the four branches are concatenated together with respect channel dimension. Finally, convolutional unit block h is applied to fuse them. Denoting the outputs of convolutional unit
4

---

Published in Transactions on Machine Learning Research (11/2024)
C2
C2
GaP
Pear
Pa
Px
Pg
115
(P::UPg(p)): (P,)) (P.(P))
[C (P)
(Par))(P Py: Paa]
[Up, (Pa) UP,
Z
Z
(a)
(b)
(c)
(d)
Figure 2: Visualization of the architecture of modern decoders: (a) UPerHead, (b) PSPHead, (c) ASPPHead and SepASPPHead, and (d) their general form.
blocks in parallel branches as (P1, P2, P3, Po), fusing them is represented as
Z h(CS;: UP UP H/s (Pi); UP (P2); UP H/36 (P3): UP (/
(2)
where H W is assumed for notational simplicity. Similarly, the fused feature Z is then subjected to x 1 convolution and a 4x bilinear upsampling to yield a predicted semantic mask Y, which has the size of H x W x Ne-
ASPPHead and Others ASPPHead refers to the head deployed in DecpLabV3 (Chen et al. 2017p. It uses atrous convolution (Yu & Koltun 2016 Chen al. 2018a), which generates empty space between each element of the convolutional kernel. To extract both global and local information from feature map, the ASPPHead adopts multiple atrous convolutions with various atrous rates in parallel. For the last feature map Cs. the first branch applies series of global average pooling (GAP), convolutional unit block, and bilinear upsampling to restore the spatial size prior to the GAP. Each of the other four branches applies convolutional unit block whose convolutional operation adopts an atrous rate {1, a, 2a, 3a), where a = 96/s. The results from the five branches are concatenated together with respect to channel dimension, and then a convolutional unit block & is applied to fuse them. Denoting the outputs of convolutional unit blocks in parallel branches as (PGAP, P1, Pa: P2a; P3a), fusing them represented as
(/UUPH/ (PGA)); Pa: P2ai P3.)).
(3)
Similarly, the fused feature Z is then subjected to 1 x convolution and a bilinear upsampling to vield predicted semantic mask Y, which has the size of H x W x No. In DeepLabV3+ (Chen et al. 2018b), variant called SepASPPHead is developed using depthwise separable convolutions instead, while keeping the same decoder architecture. This single-stage feature fusion has also been used in other segmentation networks such as FCN [Long et al. 2015) and U-Net (Ronneberger et al. 2015 which progrossively repeats fusion for two features with upsampling al each time.
Summary and Generalization As reviewed above, modern decoders of segmentation networks perform multi- or single-stage feature fusion, which we collectively refer to multi-level feature fusion. Although each decoder has distinct architecture, their feature fusions share similar design pattern (Fig. 2. Using single or multiple encoder fcatures, certain operations are applied in parallel branches, and the convolutional unit block in the ith branch generates the ith feature map P, for i E {1,..., nn,) for the number of branches ny. Because the spatial size of each feature map P; differs, optional r,x bilinear upsampling is needed to assure the same spatial size. For notational simplicity, 1 x bilincar upsampling defined as the identity operation. Because the set of encoder features for fusion includes feature map that does not require bilinear upsampling, least one branch exhibits the upsampling ratio ri 1, whereas others use >> 1. The fused.
5

---

Published in Transactions on Machine Learning Research (11/2024)
Encoder
Concat Conv
We,
Fused Feature
SE
Encoder
FPN
Concat Conv
Upsampling
Equalizar
P.
P2
Fused Feature
Encoder
FPN
Concat & Conv
Wy. Fused Feature
Figure 3: Overview of the problem statement and the proposed solution. This illustration depicts fusion by UPerHead for two features for simplicity, but nonetheless, the common fusion scheme uses four features. (Top) Existing multi-level feature fusion concatenates features after bilinear upsampling. The variances of concatenation subjects, represented as chroma in this figure, exhibit disequilibrium because bilinear upsam- pling decreases variance. this fusion, P1 dominates in the fused feature as a red color, which diminishes the contribution P2 and causes slower training on w2- (Middle) Our proposed multi-level feature fusion with scale equalizers guarantees consistent variance across subjects of concatenation. In this scheme, suitably fused feature a purple color is produced with alive gradients with respect to both wy and W2- (Bottom) Efficient implementation of our proposed method, where scale equalizers are replaced by applying auxiliary initialization for w1 and W2.
feature now obtainable by concatenation with respect to channel dimension and a convolutional unit block h as
Z = h(UUP,, (Pi): ;UP, (P))))).
(4)
Below, we investigate the concatenation subjects UPr, (P.). Although we introduced the encoder features {C,) for detailed descriptions of decoders, they will not be further used in our analysis.
3 Scale Disequilibrium
3.1 Problem Statement
As reviewed above, the docoder of the segmentation network includes module to fuse features of varied sizes. Here, we claim that multi-level feature fusion requires explicit scale equalization because they exhibit different scales, which causes scale disequilibrium gradients (Fig. 3
To understand feature scale, this study uses feature variance. Other measures such as the norm depend on the size of the feature, whereas variance provides a suitably scaled result with respect to its size. Owing to the effectiveness variance in understanding feature scales, has been adopted in several pieces of literature Glorot Bengio 2010 He et al. 2015 Klambauer al. 2017). We also employ the mean of feature understand its representative value as occasion arises. Using variance, we describe the scale disequilibrium
follows.

---

Published in Transactions on Machine Learning Research (11/2024)
W2
W1
X1
2
W1 W2
y
W2
W1
X1 X2
W1 W2
y
Figure 4: Illustration of scale discquilibrium. (Top) When Var[zı] Var(12), we obtain Var Varlawy }> Varl dy ): whose landscape is difficult to optimize through gradient descent. (Bottom) Achieving scale equilibrium Var[z1] = Var[z2] stabilizes the landscape and corresponding gradient descent optimization with respect to wi and u2.
dy
Seng
Proposition 3.1. Consider multi-level feature fusion, where a concatenated feature [x1; subjected to linear layer with weight [wi, w2] and bias b to yield the Jused feature y == w1 + W2222 b. When the two features 21 and 22 on different scales, i.c., Var[z1] * Var[zz], the gradients of the fused feature with respect to the corresponding weight exhibit scale disequilibrium, i.e., Varl
By + Varl Jy
The proof is straightforward because Oy Z₁- From the chain rule, the gradient of a loss function L with respect to weight w; is ØL Bw: =y dy Bh:
ow:
AL By and thus the gradicnt scale is affected by the scale of the
corresponding feature d4. The term linear layer indicates a fully connected layer convolutional layer.
For example, consider scale disequilibrim for concatenation subjects where Var[rı] = 10Var[z2). Then we obtain Varl by 10Var[ 1 and thus gradient descent on W2 on a ten times smaller scale than w1. which slows down the training on W2 (Fig. 4). However, gradient descent optimizers inherently assume scale åL equilibrium on gradients (Zeiler 2012: : For gradient descent wi + w, 1 77w. the weight initializer sets the scale of initial weight Var w.] for weights within the same lincar layer, and common gradient descent 17 without scale discrimination, which leads to difficulty in capturing different
uses single learning Note that existing literature (Glorot & Bengio 2010 He et al. 2015 Klambauer gradient scales Var 8L [ et al. 2017 [Bachlecherr et al., (2021) have discussed matching gradient scales across inter- layers to ensure stable gradient descent without poor training dynamics such as vanishing or exploding gradients. On top of inter-layer gradient scales, we claim to equalize intra-layer gradient scales. For the feature fusion scenario, matching the intra-layer gradient scales Varl,er Varl ( requires scale equalization for the subjects of concatenation: Var[an] Var[#z]. Achieving scale equilibrium eliminates the hidden factor that causes degradation in gradient descent optimization, which enhances the training of the segmentation network well as the performance of semantic segmentation.
Furthermore, the gradient scale indicates the amount of contribution: A smaller scale on the gradient or feature indicates less contribution to the predicted mask. For example, when the last feature map that contains rich, high-level image information contributes little the predicted mask, the quality of the seg- mentation result would degrade. To use the last feature map while supplementing its deficient information using multi-level features, is desirable to ensure balanced contribution from the multi-level features. Note

---

Published in Transactions on Machine Learning Research (11/2024)
that not saying that the amount of feature contributions should be precisely controlled to be optimal at initialization; rather, we would like to equalize feature contributions at the initial state and then let them change to be optimal during training. Our claim is that unwanted imbalances in gradient scales should be avoided at initialization. This claim is supported by the above existing literature on matching inter-layer gradient scales, which have approached it this way and emphasized avoiding unwanted imbalances such as vanishing or exploding gradients at the initial state; thereafter, gradient scales are allowed to change during training. Considering this objective, our argument can be interpreted as establishing a valid initialization to achieve scale equilibrium gradients with respect to intra-layer weights.
These arguments can be extended to match the mean of gradients, which requires the same mean for the subjects of concatenation: [[<]] = E[ Based on this claim, we inspect the scale of concatenation subjects in the modern decoder of the segmentation network.
Batch Normalization Partially Equalizes Scale Fortunately, the use of batch normalization results in normalized feature and thus concatenation of several features from the output of batch normalization is allowable. Moreover, batch normalization allows the use of convolution with arbitrary weights W and ReLU operation without causing scale disequilibrium. This is because the output of a convolutional unit yields a fixed mean and variance without requiring any
block with the pipeline (Conv-BatchNorm- specific conditions on the weight. W and feature x:
|ReeLU(BatchNorml (Wo))
(5)
V2
Ver|ReLUBBatchNon (Btt///WaaacNNoWW)) π
(6)
2m
This property also implies that any architecture can be freely chosen before the input of the convolutional unit block. Furthermore, batch normalization guarantees consistent mcan and variance for each channel (Toffe
& Szegedy, 2015). This channel-wise normalization is preferable because the output features from multiple branches are concatenated with respect to channel dimension. These characteristics batch normalization explain why it is still preferred for the decoder of segmentation networks, despite the existence of several alternatives, such as layer normalization, which does not perform channel-wist normalization (Ba et al., 2016). In summary, batch normalization provides feature with a consistent scale, which allows the concatenation of several features from convolutional unit blocks.
Bilinear Upsampling Breaks Scale Equilibrium However, even with batch normalization, feature scales exhibit disequilibrium when subsequently using bilinear upsampling. Consider a multi-level feature fusion for [P1, P2), where each fcature is an output of a convolutional unit block, and the latter P2 needs rx bilinear upsampling with r > to become the same spatial size as Pl. Fusing them requires computing
z fuse
Wfuse Pr: UP:(),
(7)
which is an intermediate result after convolutional layer of fusion with Wiuse Here, we investigate the scale of concatenation subjects. Although convolutional unit blocks on parallel branches assure consistent scales for (Pi, P2), scales of concatenation subjects (Pi, (P)) are not guaranteed to be equal. Indeed, we claim that scale discquilibrium occurs during this feature fusion due to bilinear upsampling. Specifically, we prove that bilinear upsampling decreases feature variance:
Theorem 3.2. Bilinear upsampling decreases feature variance, i.e., Var[UP, < Var X for upsampling ratio r > 1 and feature X that is not a constant feature.
The constant feature here indicates a vector with the same constant elements. Note that bilinear upsampling conserves feature mcan-but not fcature variance. Furthermore, variance provides a suitably scaled result with respect. to its size, which ensures that the decreased variance is not caused by the increased size due to upsampling. The decreased variance is caused by the linear interpolation function used in bilinear upsampling, which docs not conserve the second moment that is included in the variance. See the Appendix A for detailed proof and further discussion.
For batch normalization 72 8, the initial state where 7 = and B = provides normalized feature
8

---

Published in Transactions on Machine Learning Research (11/2024)
Var(P]
(PP
(P)
0.8
(Pr(UP.(PP)
(T 1)/2x
0.6
Variance
0.4
0.2
0.2
0.4
0.6
0.8
Figure 5: Empirical observation on decreased variance after bilincar upsampling. The black dotted line (T - 1)/2x corresponds to the case when the output a convolutional unit block subjected to bilinear upsampling.
Section 2 we reviewed multi-level feature fusioon in modern decoders and found that, as a general rule, at least one branch uses the upsampling ratio r; 1, whereas others show r, > 1. Therefore, Theorem 3.2 indicates that modern decoders with multi-level feature fusion exhibit scale disequilibrium. The fatal problem is that the last. feature map always requires bilinear upsampling, which reduces its feature and gradient image. This problem arises even when using batch
scales, obstructing the use of its rich information normalization: Because bilinear upsampling is applicd after cach convolutional unit block, the equalized scales subsequently change.
Note that bilinear upsampling the de facto standard for semantic segmentation, and several studies have explicitly mentioned using bilincar upsampling in their papers (Zhao et al. 2017 Chen al.) 2017 Xiao et al.) 2018). In consideration of this practice, our study targets upsampling with bilinear interpolation. Nevertheless, the decreased variance can also be observed for other interpolation methods such bicubic, and our analysis and solution seamlessly apply to those upsampling methods.
Empirical Observation Now, empirically demonstrate decreased variance after bilinear upsampling, Considering a practical feature fusion scenario, generated artificial random normal data P sampled from N(1 o²), which corresponds to a feature after a convolutional unit block but before bilinear upsampling. The feature P set to have width 128, height 128, number of channels 256, and mini-batch size 16. Then we applied rx bilinear upsampling to with r f2, 4, 8) and measured the variance of cach outcome. Figure 5 summarizes the results across various choices of (0, 1). We observed that bilinear upsampling decreased feature variance in all simulations.
3.2 Proposed Solution: Scale Equalizer
Our claim is that we should modify the existing feature fusion method to achieve scale equilibrium for concatenation subjects-the the output of each branch that ends with bilinear upsampling. This objective may be accomplished in several ways. The naive approach is to place batch normalization after bilinear upsampling,
9

---

Published in Transactions on Machine Learning Research (11/2024)
Hi ot
M]
Figure 6: Scale equalizer with simple implementation (left) and efficient implementation (right). For efficient implementation,, pre-compuled global mean and std are applied to the weight and bias of the fusion layer in advance, as depicted by the gray dotted line. Here, the main training requires only black dotted lines, which maintains the same computational cost compared with the case without scale equalizers. Furthermore, we can remove the bias correction due to the subsequent batch normalization.
changing the pipeline from Monv-Baachhoone Renun-PP to -ouy-ReLU-UP ReLU--UPBatthNonnl BatchNNom). This pipeline yields a normalized fcature with a consistent scale but requires extra computational cost. Becanse batch normalization computes the mean and standard deviation (std) of the current incoming feature map across the mini-batch, its computational complexity increases with the larger size of the feature (Huang et al. 2018). The computational complexity of the backward operation further increases with the larger size of the feature map because the derivative for batch normalization is much more complicated (Yao et al.) 2021) Consequently, applying batch normalization to an upsampled feature causes a significantly more expensive computation compared with that of a non-upsampled feature. Considering this problem, we alternatively explore a computationally efficient solution to acquire scale equilibrium.
Here, we propose scale equalizer, module to be injected after bilinear upsampling but before concatenation. To achieve scale cquilibrium at minimal cost, we design the scale equalizer simple possible. Specif- ically, our proposed scale equalizer normalizes target feature using global mean μ and global std ScaleEqualizer(() = (x - 4)/a. The global mean and std are scalars computed from the target feature x across the training dataset, which can be performed before training. Once the global mean and std are obtained, they can be set as fixed constants during training, which simplifies forward and backward op- erations for the scale equalizer. By contrast, mean and std are not constants for common normalization operations such as batch normalization or layer normalization because they use a mean and std of a current incoming feature. Thus, compared with existing normalization operations, the proposed scale equalizer be implemented with little extra cost.
Scale Equalizers Equalize Scales Now consider multi-level feature fusion with scale equalizers, where the scale equalizer is applied after bilincar upsampling of each branch but before concatenation. The con- catenation subject Scalelqqaliierr(UPP equilibrium. Because the scale equalizer uses empirically measured values of the global mean and std, the scale equilibrium does not require architectural restrictions or specific conditions on weight. In other words, scale equilibrium is always guaranteed for any dataset and any architecture of segmentation network.
exhibits zero mean and unit variance, which assures scale
Efficient Implementation via Initialization In fact, matching intra-layer gradient scales via scale equalizers be interpreted establishing valid initialization. For multi-level feature fusion y = Σ; w;x;
10

---

Published in Transactions on Machine Learning Research (11/2024)
Algorithm 1 Efficient Implementation via Initialization
Input:
set of training images S, encoder e, decoder d. Using pretrained weights 6, initialize encoder e.
Using preferred initializers, initialize decoder d into weight Ω2 including {w;}- Set 11, = M22 0 for E ;
for E 5 do
Extract FPN outputs (P,) from I using encoder ee and subnetwork of decoder da-
for i to n do
my, m..; + EUPP r [.)
m2, = m2, + EUUP, r, P..)..
end for end for
for i= to ni do
Obtain the global mean μ4 m1.i/s
Obtain the global std o = //,..///5 m.
Update wy in Q using auxiliary initializer w¹ wi/0,-
end for
Using the updated decoder weight D', run the main training for encoder ea and decoder der
b, after replacing v; with ScaleLqualizer, (x:) = (x - mi)/o;, we obtain
y Σ 0j
+
b Σ
Wifl
(8)
Thus, injecting scale equalizers is equivalent to adopting an auxiliary initializer with w/ = wi/o, and V/ Σ; uauu/o. This auxiliary initializer means calibrating the weights and bias in the linear layer of fusion in advance using expected fcature scales (Figure 6) Furthermore, because batch normalization follows subsequently (Section 2 the latter for bias correction is actually not needed, whereas the former for weight calibration is still needed to control the scales of concatenation subjects. For UPerHead as a concrete example, weights in the convolutional layer of fusion are partitioned into four groups with respect to channel dimension, and the weights in each group W, are re-scaled via the global std og. In summary, after primary initialization of the decoder, we compute the global mean and std for each target feature, apply the auxiliary initializer weights, and then proceed with the main training (Algorithm [. This implementation requires additional computational cost. during main training, which enables to achieve scale equilibrium for free.
As mentioned carlier, there may be other ways to achieve scale equilibrium by introducing complicated operations. Nevertheless, to remonstrate the effectiveness of scale equilibrium under the same computational cost, we opt for injecting scale equalizers and their efficient implementation through auxiliary initialization, which achieves scale equilibrium for free.
Notes on Advantage of Scale Equalization Even though the initial state exhibited scale disequilibrium for existing multi-level feature fusion, neural network parameters are well optimized during training, it may achieve scale equilibrium after training. Nevertheless, our claim is that it would be better achieve scale equilibrium from the initial state to avoid poor training dynamics. Additionally, note that the proposed scale equalizer does not introduce new learnable parameters in the neural network; rather, it behaves constant scaling with fixed values. Therefore, even though scale equalizers are injected, the model's represcntation ability remains the same. While maintaining the same representation ability, the advantage of scale equalizers comes from the easier optimization in gradient descent by avoiding poor training dynamics (Section 3.1]. This behavior would be rather similar to the initialization method of a neural network. For example, applying Xavier or He initialization Glorot & Bengio 2010 He et al.] 2015) facilitates optimization in gradient descent, but they do not introduce learnable parameters.
11

---

Published in Transactions on Machine Learning Research (11/2024)
Table 2: Summarization of mloU (%) from semantic segmentation experiments with multi-stage feature fusion using UPerHead. "Scale EQ" indicates the scale equalizers, and "Diff" indicates the mloU difference after injecting the scale equalizers.
Dataset
ADE20K
PASCAL VOC 2012 AUG
Encoder
w/o Scale EQ
w/ Scale EQ
Diff
w/o Scale EQ
w/ Scale EQ
Diff
Swin-T [Lu et al. 2021
43.384
43.576
+0.192
78.750
78.996
+0.246
Swin-S
47.298
47.486
+0.188
81.940
82.138
+0.198
Swin-B
47.490
47.648
+0.158
82,200
82.378
+0.178
Twins-SVT-S Chu et al.
2021
44.914
45.018
+0.104
80.448
80.732
+0.284
Swins--VTT-B
47.224
47.500
+0.276
82.048
82.524
+0.476
Swins--VVT-L
48.648
48.894
+0.246
82. 16.
82.404
+0.236
NeXt-T Liu et al. 2022
45.024
45.300
+0.276
80.668
80.932
+0.264
Cen-NeXt-S
47.736
47.866
+0.130
82.472
82.650
+0.178
NeX-B
48.376
48.684
+0.308
82.934
83.038
+0.104
4 Experiments
4.1 Multi-Stage Stage Feature Fusion
Objective So far, we have discussed the need for scale equalizers for multi-level feature fusion. The objective here is to compare the segmentation performance before and after injecting scale equalizers into multi-stage feature fusion. We considered extensive sctups, such as the choice of backbone and target dataset. For the backbone network, we employed recent vision transformers that achieved state-of-the- art performance. Nine backbones of Swin-(T. S, B} (Liu et al.] (021), Twins-SVT-(S, B, L. (Chu et al. 2021), and Con-NeXt-(T, S, B} (Liu et al. 20222) pretrained on ImageNet-IK (Deng et al. 2009) were examined, where T, S, B, and L stand for tiny, small, basc, and large models, respectively. These encoders require bilinear upsampling for multi-stage feature fusion. Targeting multi-stage feature fusion, we employed UPerHead (iiao al. 2018). Two datasets were examined, including the ADE20K (Zhou et al. Everingham et al. 2015
2019) and
PASCAL VOC 2012
Hyperparameters To follow common practice for semantic segmentation, training recipes from MMSegmentation CaroiiionCCCooiiiii (Contributors 2020) were employed. For training with Swin and Twins encoders, Adamm optimizer (Loshchilov & Hutter 2019 with weight decay 20 betas B1 = 0.9, B2 = 0.999, and learning rate 10-5 with polynomial docay of the 160K scheduler after linear warmup were used. For training with ConvNeXt encoders, AdamW optimizer with weight decay 5 x 10-2, betas 31 0.9, 32 0.999, learning rate 10 with polynomial decay of the 160K scheduler after linear warmup, and mixed precision training (Mi- cikevicius et al. 2018) were used. The training conducted on 4x GPU machine, and SyncBNN (Zhang ct al. 2018) was used for distributed training. We measured the mean intersection over union (mIoU) and reported the average of five runs with different random seeds.
Datascts The ADE20K datasct contains sceno-centric images along with the corresponding segmentation labels. crop size of 512 x 512 pixels was used, which was obtained after applying mean-sid normalization and random resize operation using a size of 2048 512 pixels with a ratio range of 0.5 to 2.0. Furthermore, random flipping with probability of 0.5 and the photometric distortions were applied. The objective to classify each pixel into one of the 150 categories and train the segmentation network using the pixel-wise cross-entropy loss. The same goes for the PASCAL VOC 2012 dataset with 21 categories, and we followed the augmented PASCAL VOC 2012 dataset.
Results We observed that injecting scale equalizers into multi-stage feature fusion improved the mloU index compared with the same models without scale equalization (Table 2 The mIoU increases of about +0.1 to +0.4 were consistently observed across all setups of nine backbones and two datasets. Note that the scale disequilibrium arises within the decoder at the concatenation layer after upsampling. Therefore,
12

---

Published in Transactions on Machine Learning Research (11/2024)
Table 3: Summarization of mloU (%) from semantic segmentation experiments with single-stage feature fusion using various heads.
Dataset
Cityscapes
ADE20K
Decoder
w/o Scale EQ
w/ Scale EQ
Diff
w/o Scale EQ
w/ Scale EQ
Diff
FCNHead Long et al.
2015
76.578
76.972
+0.394
39.780
39.958
+0.178
PSPHead
Zhao al.
2017
79.394
79.858
+0.464
43,970
44.228
+0.258
ASPPHead
(Chen et al.
79.312
79.720
+0.408
14.854
45.004
+0.150
SepASPPHead (Chen et
2018b
80.448
80.592
+0.144
45.144
45.486
+0.342
this problem is related to the architectural design of the decoder. Using a larger encoder network, alboit having much expressivily, cannot solve this problem. This explains why scale equalization matters from tiny to large backbone models. Furthermore, cost and performance. The proposed method does not introduce additional layers; while keeping the same architecture and expressive power of the deep neural network, the performance gain of the proposed method can be obtained in actually free (Section 3.2 Tn other words, scale equalization provides a free performance gain without incurring additional computational expenses.
method goes beyond the trade-off between computational
4.2 Stagle-Stage Feature Fusion
Now we examine single-stage feature fusion. The target encoder was ResNet-101 (He et al. 2016), which was modified to exhibit output stride s = 8 and was pretrained on ImageNet-IK. We targeted most standard and popular decoders, including FCNHead (Long et al. 2015). PSPHead (Zhao et al., 20177). ASPPHead (Chen al. 2017), and SepASPPHead (Chen al. 2018b The target datasets were the Cityscapes (orrtt et al. 2016 and ADE20K datasots.
Objective
Hyperparameters Similar to Section 4.1 training recipes from MMSegmentation were employed. For training on the Cityscapes dataset, stochastic gradient descent with momentum 0.9, weight decay 5 x 10 10-4, and learning rate 10-2 with polynomial decay of the 80K scheduler were used. The same goes for training the ADE20K dataset while using the 160K scheduler instead. The training was conducted on a 4x GPU machine, and SyncBN was used for distributed training. We mcasured the mIoU and reported the average of five runs with different random seeds.
Datasets The Cityscapes dataset contains images of urban street scenes along with the corresponding segmentation labels. A crop size of 1024 x 512 pixels was used, which was obtained after applying mean-std normalization and random resize operation using size of 2048 x 1024 pixels with ratio range of 0.5 to 2.0. Furthermore, random flipping with probability of 0.5 and the photometric distortions were applied. The objective was to classify each pixel into one of the 19 categories and train the segmentation network using the pixel-wisc cross-entropy loss. Experiments on the ADE20K followed the description in Section
Results Similarly, injocting scale equalizers consistently improved the mIoU index across all sctups of four decoder heads and two datasets (Table 3. which verifies the effectiveness of scale equalizers for any choice of architecture. See the Appendix for experimental results.
5 Discussion
5.1
Comparison With Other Normalization Layers
Consider the pipeline used for generating cach concatenation subject in multi-level feature fusion. The existing pipcline of Bar--BahhhhmmHeHHe to the last upsampling. Here, when modifying into Monv-RReL.U-UR RP-.U--UP-BatchNorni BatchNorm), outputs a normalized
yields concatenation subjects with different variances due
13

---

Published in Transactions on Machine Learning Research (11/2024)
Com--ReLU-JP-BatchNNom
300
Com--RLLU-UP-GrouyNoom
Conv-ReLU-UP-LLyyrloom
Con--ReLUUUP-nnstaaneeooo
250
Conv-BacchNom-RRLLUUUP
Computation Time (s)
200
150
100
50
0
5M
10M
15M
20M
25M
30M
size (N*C"H"W)
Figure 7: When modifying the existing pipeline from Hann-BacchNormRRoe RcLUUUPP into others such as [Conv BatchNorm ReLU UPJ, computation time significantly increases.
Table 4:
Experimental results on different pipelines for the ADE20K dataset using Swin-T.
Pipcline
mIoU (%)
Conv Bon--BatchNNom-RRLLLU-
43.384
Conv Bony-BBachh-oom--BBBU------ ScaleF.
43.576
39.998
Conv ReLU UP GroupNorm
43.212
Layerror
42.870
feature, which achieves equalized scale across concatenation subjects. This behavior can also be achieved with other normalization layers, such GroupNorm and LayerNorm.
Although the use of a normalization layer after upsampling can be another solution to achieve scale equi- librium, it causes increased computational costs due to the enlarged size of the feature map. Because mean and std are computed for the incoming feature map, the larger size of the feature map requires much com- putational cost for computing the mean and std. Furthermore, the computed mcan and std are used for normalization of each element, which leads to increased complexity overall.
This behavior be verified through simulation. Using N for N 16 and C' 128, we simulated 8x bilincar upsampling and compared the computation time required for the four pipelines: [Conv- BatchNorm ReLU-UP), [Conv ReLU UP Barch] [Cony-ReLU ReLU UP GroupNorm), and [Conv- CU--UP-LayerNNon F summarizes the results. We observed that the existing Sauu-BatchNomm ReLU-UP] pipcline consistently exhibited faster computation, whereas modified pipclines showed signifi- cantly slower computation. Note that the computational cost case of injecting scale equalizers is equal to the existing pipeline of Batn-Bam ReLU-UPI because it be implemented without extra cost
14

---

Published in Transactions on Machine Learning Research (11/2024)
Table 5: Experimental results on the monocular depth estimation task.
Model
< 1.25
8x 1,25"1
6 < 1.25"
Abs Rel 4
RMSE
log10
RMSE log, L
SILog 4
Sq Rel
GEDepth-Vaninn Vanilla
0.9763
0.9971
0.99994
0.0498
2,0416
0.0218
0.0766
7.0109
0.1429
GeDeplh-aanilaa Vanilla w/ Scale EQ
0.9768
0.9972
0.9994
0.0492
2.0180
0.0215
0.0760
6.9702
0.1400
GEDepth-Adaptic
0,9751
0.9970
0.9993
0.0196
2,0909
0.0218
0.0776
7.1189
0.1479
GEDeptl-Adaptar w/ Scale EQ
0.9757
0.9971
0.9993
0.0194
2,0083
0.0217
0.0771
7.0763
0.1457
Algorithm 1 therefore, in terms of computational complexity, we claim that our proposed method is superior compared with the use of normalization layer.
(Section 3.2
In addition to computational costs, we compared segmentation performance when using modified pipelines (Table 4. We empirically observed that modifying the existing ordering of CConv-BatciNoom-RR..-- has a side effect of degraded segmentation performance, which outweighs possible advantages. This phe- nomenon was consistently confirmed for BatchNorm, GroupNorm, and LayerNorm. Although Group.Nrrm LayerNorm might yield improved performance compared with BatchNorm, their performances were even below the baseline performance of the existing pipeline of [Conv- Soov-BatchNNom- -ee---Nooo---R----- Purthermore, ap- plying normalization layer after upsampled feature requires much computational cost. Considering both computational cost and segmentation performance, the best pipeline is our proposed pipeline of [Conv BatchNorm
Batch--ormReLLUUUPP CeLU-UP-SccllEBS In consideration of this, we opt for keeping the existing pipeline without modification its ordering while injecting a scale equalizer at the end of the pipeline.
5.2 Scale Equalization for Other Tasks
We find that the use of upsampling and multi-level feature fusion is prevalent in the machine learning community. Although we focused on the multi-level feature fusion in semantic segmentation networks as a prime example, our scale equalization matiers for other multi-level feature fusion tasks. Specifically, scale equalization generally matters for modern encoder-decoder networks. For example, monocular depth estimation networks have used the encoder-decoder architecture, whose multi-level feature fusion exhibits scale disequilibrium similar to the semantic segmentation networks.
In consideration of this, we additionally verified scale equalization for a monocular depth estimation network. The target model was GEDepth (Yang et al.) 2023). where its feature fusion module concatenates upsampled and non-upsampled feature maps. Using the KITTI dataset /Giieer el al., 2013), we trained the model with and without scale equalizers in the feature fusion module (Table 5). We observed that injecting scale equalizers improved the performance of the monocular depth estimation task, which connotes that scale equalization similarly matters for other tasks where encoder-decode architecture is deployed.
5.3 Qualitative Analysis
Figure g provides segmentation examples for the ADE20K dataset. We find that injecting scale equalizers leads to better understanding of the global context of images. Specifically, with scale equalizers, the global layout is better captured for large parts. Indeed, our analysis says that existing multi-level feature fusion suffers from scale disequilibrium, which lowers the contribution of the last feature map that contains rich global information. Here, injecting scale equalizers facilitates the last feature map to be involved in muvti-level feature fusion, which leads to a better understanding of the global context of the image.
6 Conclusion
This study discussed the scale disequilibrium in multi-level feature fusion for semantic segmentation tasks. First, we reviewed the mechanisms existing segmentation networks, which perform multi- or single-stage feature fusion. We demonstrated that the existing multi-level feature fusion exhibits scale discquilibrium due bilinear upsampling, which causes degraded gradient descent optimization. To address this problem, injecting scale equalizers is proposed to guarantee scale equilibrium for multi-level feature fusion. Experi-
15

---

Published in Transactions on Machine Learning Research (11/2024)
CO
FIJ
Figure 8: Segientttiion examples without (Left) and with (Right) scale equalizers.
16

---

Published in Transactions on Machine Learning Research (11/2024)
ments showed that the use of scale equalizers consistently increased the mloU index by about +0.1 to +0.4 across numerous datasets and architectures. We hope that our proposed problem and solution will facilitate the research community in developing an improved multi-level feature fusion and segmentation network.
Acknowledgments
This work was supported by Samsung Electronics Co., Ltd (10201210-08019900)..
References
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer Normalization. CoRR, abs/ bbs//660.004500, 2016.
Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Gary Cottrell, and Julian McAuley. ReZero is all you need: fast convergence at large depth. In UAI, volume 161, pp. 1352-1361, 2021.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BETT: BERT Pre-Training of Image Transformers, In ICLR, 2022.
Liang-Chiol Chen, George Papandrcou, Florian Schroff, and Hartwig Adam. Rethinking Atrous Convolution for Semantic Image Segmentation. CoRR, abs/1706.055887, 2017.
Liang-Chies Chieh Chen, George Papandreou, Jasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. IEEE Trans. Pattern Anal. Mach. Intell., 40(4):834-848, 2018a.
Stang-Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- Encoder-Decodor with Atrous Separable Convolution for Semantic Image Segmentation. In ECCV (7); volume 11211, pp. 833 03--81, 2018b.
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the Design of Spatial Attention in Vision Transformers. In NewrIPS, pp. 9355- 9366, 2021.
MMSSemeenattion Contributors. MMMSegnentation: MMLa Semantic Segmentation Toolbox and Benchmark. Attps://github.commmorg
www.mmmwwaaaaaa.eeeeeem..emmaeme http://oppoaaamaaaaaaaa/aeeeaaap 2020.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schicle. The Cityscapos Datasct for Semantic Urban Scone Under- standing. In CVPR, pp. 3213 -223,, 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, pp. 248 255, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR, 2021.
Mark Everingham, S. M. Ali Eslami, Lue Van Gool, Christopher K. Williams, John M. Winn, and Andrew 7isserman. The Pascal Visual Object Classes Challenge: A Retrospective. Tra. J. Comput. Vis., 111(1): 98 136, 2015.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The KITTI dataset. Int. J. Robotics Res., (2(11))1331-1233 2013.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, volume 9, pp. 249 200,256,455 2010.
17

---

Published in Transactions on Machine Learning Research (11/2024)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving Deep into Rectifiers: Surpassing Human- Level Performance on ImageNet Classification. In ICCV, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, pp. 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked Autoencoders Are Scalable Vision Learners. In CVPR, pp. 15979-15988, 2022.
Lei Huang, Dawei Yang, Bo Lang, and Jia Deng. Decorrelated Batch Normalization. In CVPR, pp. 791 800, 2018.
Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In ICML, volume 37, pp. 448-456, 2015.
Alexander Kirillov, Ross B. Girshick, Kaiming He, and Piotr Dollár. Panoptic Feature Pyramid Networks. In CVPR, pp. 6399-6408, 2019.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-Normalizing Neural Networks. In NIPS, pp. 971-980, 2017.
Tsung-Y Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie. Feature Pyramid Networks for Object Detection. In CVPR, pp. 936 944, 2017.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hicrarchical Vision Transformer using Shifted Windows. In ICCV, pp. 9992-10002, 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A
ConvNet for the 2020s. In CVPR, pp. 11966-11976, 2022.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In CVPR, pp.
2015.
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In ICLR, 2019.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed Precision Training. In ICLR, 2018.
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the Convergence Adam and Beyond. ICLR, 2018.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. In MICCAI (3), volume 9351, pp. 234-241, 2015.
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan C. Bovik, and Yinxiao Li. MAXTM: Multi-Axis MLP for Image Processing. In CVPR, pp. 5759-5770, 2022.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified Perceptual Parsing for Scene Understanding. In ECCV (5), volume 11209, pp. 432-448, 2018.
Xiaodong Yang, Zhuang Ma, Zhiyu Ji, and Zhe Ren. GEDepth: Ground Embedding for Monocular Depth Estimation. In ICCV, pp. 22673-12681, 2023.
Zhuliang Yao, Yue Cao, Shuxin Zheng, Gao Huang, and Stephen Lin. Cross-Iteration Batch Normalization. In CVPR, pp. 12331-12340, 2021.
Fisher Yu and Vladlen Koltun. Multi-Scale Context Aggregation by Dilated Convolutions. In ICLR, 2016.
Matthew D. Zeiler. ADADELTA: An Adaptive Learning Rate Method. CoRR, abs/1212.5701, 2012.
18

---

Published in Transactions on Machine Learning Research (11/2024)
Hang Zhang, Kristin J. Dana, Jianping Shi, Zhongyue Zhang, Xiaogang Wang, Ambrish Tyagi, and Amit Agrawal. Context Encoding for Semantic Segmentation. In CVPR, pp- 7151-7160) 2018.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid Scene Parsing Network. In CVPR, pp.
2017.
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H. Torr, and Li Zhang, Rethinking Semantic Segmentation From a Sequence- to-Sequence Perspective With Transformers. In CVPR, pp. 6881-6890, 2021.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic Understanding of Scenes Through the ADE20K Dataset. Inl. J. Compul. Vis., 127(3):302 321, 2019.
19

---

Published in Transactions on Machine Learning Research (11/2024)
A Proof of Theorem 3.2
Notation In this section, prove Theorem 3.2 Here, consider 1D bilinear upsampling, because 2D bilinear sampling is straightforward exlension of it. For given sequence X= {X .x,) of size n, applying rx bilinear upsampling yields sequence (X,(X) of size rn. Because these sequences are discrete, bilinear upsampling can be thought of as a coordinate transformation, which transforms coordinate We follow the common option of bilinear upsampling to set aaigg_corneeeeeeeeeeeeeeeeeeeeee corners=False, where each coordinate is regularly spaced and shares center. Within pt and Pc+1 for i i n-1] 1}, the upsampled coordinate has r regularly spaced points as gr(1-0.5++) p.+ (2j- 1)f for == P-
{PI..... .r,) into coordinate (9,,... ""..". 9rn).
2r and j E {1. .... Because bilinear upsampling is piece-wise linear interpolation, we represent the wicco-wise linear function as f(x) where 1) for coordinate point ps, we define f(;)) = X; for E (1. n), 2) following the behavior of align_ corners=False, we define f((x) == X, on left outer interval (p+,))+)), we define f(x) X and as a linear line f(x) = a, + b, connecting two points (ps. X;) and (pi+1, X,+1) where a, = P-11PP
(-oopp)
and f(z) = Xn on right outer interval x (Pa. +oo), and 3) on ith interval
X Using this notation, we represent bilinear upsampling as a transformation from the b. Pi+1
original data (f(r)).... ((w)) } to its upsampled data (f(m))... f(gn)), Finally, represent the mean of sequence by sampling as [xwp]f(x)) EE= (pr) and the samne goes for variance.
91
92
43
Q15
P1
P2
P3
P4
X
X1 a₂x + b₁
a2x + b2
a3x + b3
a4x + b
Figure Example of two coordinates and pieco-wise linear function for 4x bilinear upsampling.
Objective Firstly, we know that bilinear upsampling conserves feature mean, i.e., EUP,, E[X] Using the above-mentioned notation, this property be represented Ex>p[f(x)] = Ez~g [= In fact, mean-conseevatton holds by the definition of the two coordinates because they share the center. Now, we investigate feature variance before and after bilinear upsampling. Because Var[X] E[X - (E[X))? we inspect the second moment [] for the two coordinates. In other words, we compare [f)))(((z)) and (x)))
Main Proof Firstly, we write
Eamp(((())) = Σ (f(p)))²
(9)
i=1
(f(pi)) ((f(pi)) + (f(p2))²) ..... ((f(ph-1)) +(f()))) (f(p)))
(
(10)
2n
Exzag [[(())) (((g1))² .... +
(11)
20

---

Published in Transactions on Machine Learning Research (11/2024)
This expression enables to compare sub-terms of the above for two coordinates each interval. For E (1, ; 1), we define their sub-terms as
P. = (f(pi))² (f(p++1))
(12)
2n
Q (f((r(i-03)++1)) +...+ (f(ar(+0.s)))
(13)
m
Now we investigate P. Qi- Note that 27 (f(::))) (9)) which be interpreted repeating (f(p))) in times. Using this representation, obtain
rn(P: Q:) -(((pi)) +... (F(P.)) + {(f(p++1)) +... (f(p++1))
(14)
r/2
T/2
-(f(94-000))1))) +.... +.... +((x))) ((f(9rit))) ((((()(444)))
(15)
r/2 terms
r/2 terms
=(((())) (f((ma +((([)))) (((6):))- (f(1))))} + {((((i+))) +))+ +..++ )) -
(16)
Thus, the first r/2 terms indicate the squared difference between two coordinates on the left half area, whereas the last r/2 terms similarly come from the coordinates on the right half area.
0.5r points
0.5r points
9r(i-0.5)+1
9ri
4ri+1
ar(i+0.5)
Pi
9r(i-b.5)+
9r(i+0.5)+1-)
Pi+1
x
(2j - 1)l
(2j - 1)l
2rl
Figure 10: Illustration of two coordinates on the ith interval of (pi, Pe41).
For (r(i-0.5)+5 the left half area with j E [1,...,,//22 we have p, -r((--.5)++ -(2j 1)l. Because fxx) is defined as fx) a, + by on the ith interval x E (ps. P(+1), we know that f(x1) f(xz) = (2 x22)
and ((x) + f(x2) = ax + 22) + 2b,. Thus, we have
(f(p:))² (f(0)(-000)4))) (f(p:) f(ar-- + f(a)((-.))+)))}
(17)
=a,(p; 9r(i (r(1-0.6)4)) 2b,}
(18)
=-a,1(2j - 1)(a((. 9 26.).
(19)
21

---

Published in Transactions on Machine Learning Research (11/2024)
Similarly, on the right half area, we consider (1(1+055)+1)) with j E {1.... ,r/2), which is in reversed order.
We have p: 9r(1+0,5)-1-) (2j - 1), and the right half area shares f(x) ar + by on the left half area. Therefore, we obtain
(f(p:)) ) (f(p))
((9((+000)11)))) (f = a,,p; (n(40.0)+1- + (((4+006++15)) + 2b,} (2) - 1)(a(Pii + (1(405))111))) +2bb].
(20)
(21)
(22)
Sum of Eqs.
19 and
22
yields
((((p:))2 - (0(((((055))))))) (UV0)))((((((-0))))))+++ + (000:--0))>|(0000--U----/->>>>>
= =d((22- - 1)((1++ p:) + (1-(4005)+11-- (((-05))))))) 2 O.
(23)
Equality holds if a; 0, which requires X++1 X, In summary, Eq. 16
can be written the sum of jth
terms for j E f1,... rr/2,, where cach jth term is always non-negative. Thus, we conclude that P. - Qi 2 0 for E (1, .. .n 11--1 1).
So far, we have compared the sub-terms of Eqs. 10 and [1 for two coordinates on each of the ith interval
for E f1....... 1). We are left with two sub-terms on the left outer interval and the right outer interval. Fortunately, by the definition of bilinear upsampling on outer intervals, we have
(((()))) + (((P))) - (((((p)))) + .... (f(p1))²) + {((())) + ... + (J(Pm))))) 2n
(24)
rn
r/2 terms
r/2 torms
1 [((((g))) + (f((0.s))) + (()()) (.fg))()))))))))))) (25)
rn
Thus, the outer intervals, sub-terms of Eqs.
10 and 11 are identical.
Finally, from Eqs. 10 if = 0, de., Xi+1 X, for all i {I. - 1}, which is only satisfied if the original feature is a constant feature. However, because we consider original data that is not a sequence of constants, we know that at least one a, exhibits a; * 0, which yields [x=p([[(x))]] > [[ This inequality leads Varz=pif(z)) > Varn=g[[(=)] Therefore, we conclude that applying bilinear upsampling decreases variance,
11
23 and 25 we obtain Ex (x)))) 2 [==))(((=)))] Note that equality holds
B Mean and Variance of Convolutional Unit Block
In the main text, we wrote
E R =
(26)
V2T
UeBtttLL/ (Wx))]
(27)
2n
Here, we derive these properties. Because we consider a decoder in an initialized state where 7 and 3 = 0, batch normalization provides normalized feature [O0 1), where its probability density function is p(z) = -/2 exp ) Hence, we examine the mean and variance after ReLU, ie., E[ReLU(=)) and Var[ReLU(>)]] In the case of the mean, we have
E[ReLU(=)]] = ReLU(2)p(z)dz=
(28)
zp(2)dz
(29)
1
(30)
V2T"
22

---

Published in Transactions on Machine Learning Research (11/2024)
where the last equation holds owing to the properties of half-normal or truncated normal distribution. In the case of the second moment, we derive
[[(R)LU(2)))1 = (R=L)()))'p()dd>>
(31)
>²(x)dz
(32)
p2x())dz
(33)
F[
(34)
(35)
2
where the third equation holds by even symmetry. Thus, we obtain
Var(ReLU(2)] = B(((cLU(=)))1 (E])c))(()))
(36)
1
(37)
2 2m
1
(38)
2m
which concludes the proof.
C Python Code for Fig. 5
import torch
import torch. un functional as F import math
N, H, w, c 16, 128, 128, 256
for in (1n 100):
100.0
0.8989 torch (N,, c, H, W)
std math. sqr
torch. nornal (mean=mean, std=atd) c)
13
A_1
14
print (A_1. v (ubbiasseddFa))ls)) iten ())
16
del A_1
16
17
A_2 (x, size=(2 * H, W), mode-"bilinear", aliga_corners=Falae
18
print (A_2. vubbbaaaaaadd))))l)))) iten ()) del A_2
19
20
21
A_4 - F. (x, size-(4 + H, 4 W), =ode-"bilineer", allga_cornere-Falee print print(A_4... (A_4. (ol.dd.wr(uubbiseddeaaeee iten()) del A_4
23
A_8 F. (xterpolateex, size-(8 H, 8 W), mode=" align_ corners-eersss print (A_8. var var (aabbeeeeeeFFa)le)) iten del
27
print ("")
30
Listing 1: Python code for Fig.
23

---

Published in Transactions on Machine Learning Research (11/2024)
D Additional Experimental Results
We additionally provide experimental results on single-stage feature fusson using ResNct-50.
Table 6: Summarization of mloU (%) from semantic segmentation experiments with single-stage feature
fusion using various heads.
Dataset
ADE20K
Decoder
w/o Scale EQ
w/ Scale EQ
Diff
FCNHead
36.544
36.998
+0.454
PSPHead
41.666
41.806
+0.140
ASPPHead
42.934
43.220
+0.286
ScpASPPHead
43.910
44.056
+0.146
24


Reviews:
[{"review_id": "YEH2Gzd1JZ", "reviewer": "Reviewer_B3q3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The proposed equalizer is a normalization and therefore is easy and architecture independent.\n- The paper provides theoretical justification for the proposed approach.\n\nWeaknesses:\n- The paper fails to compare to other normalizations.\n- It doesn't clearly discuss why this work is needed on top of batch norm,  specifically, in pager 10 where it tries to mention it, use of former and latter, completely confuses the reader. \n- Empirical improvement is limited and is for special case of image segmentation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper looks into the scale disequilibrium issue which is a problem with parallel deep neural networks that hurts their feature fusion performance. It impacts training performance with backpropagation (gradient descent). \n\nThe authors claim that scale disequilibrium is caused by bilinear upsampling. They show this is supported by both theoretical and empirical evidence. \n\nBased on this observation, they propose injecting scale equalizers to achieve scale equilibrium across multi-level features after bilinear upsampling. Their proposed scale equalizers are essentially just normalization and therefore are easy to implement. \nThey empirically show that scale equalizers work across some datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the following questions:\n\n* How does this method differ from normalizations used in practice? like group norm and layer norm seem to solve all issues that come up in training.\n\n* The formulation looks just like batch norm. Could you please clarify, in a network with batch norm, why your method is needed and exactly how to use it?\n\n* What are other scenarios other than empirical examples in this paper where scale equalization matters?How can you justify the importance of your work in the community? \n\n* Would scale equalization be an issue in big networks such as ViT?\n\n* What's the trade off of computation versus performance gain when using your method?", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hEGphCop4i", "reviewer": "Reviewer_2ERv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. **Innovative Problem Identification:** The paper identifies and thoroughly investigates a subtle but impactful problem in multi-level feature fusion, which has not been addressed in previous research.\n\n2. **Comprehensive Solution:** The proposed scale equalizer can be easily integrated into existing architectures without additional computational costs, making it practical for real-world applications.\n\n3. **Robust Theoretical and Empirical Support:** The paper provides strong theoretical justification for the proposed method and supports its claims with empirical results across multiple settings.\n\n## Weaknesses\n\n1. **Limited Performance Gain:** The performance improvements reported in the paper are relatively modest, with increases in mIoU typically ranging from +0.1 to +0.4. While any improvement is beneficial, the incremental nature of these gains may not be compelling enough to justify the adoption of the proposed method in all scenarios.\n\n2. **Lack of Comparison with Existing Normalization Techniques:** The paper does not provide a direct comparison between the proposed scale equalizers and other existing normalization techniques, such as batch normalization, layer normalization, or instance normalization. This omission makes it difficult to evaluate the relative effectiveness and advantages of the proposed method.\n\n3. **No Analysis of Computational Time:** Although the paper claims that the proposed scale equalizers are computationally inexpensive at runtime and faster than traditional normalization methods, it does not include any detailed analysis of preprocessing to other normalization methods at runtime to support these claims. A thorough comparison of the computational time and resource usage would strengthen the argument for the method's efficiency compared to the performance gain.\n\n4. **Limited Insights and Analysis:** The insights provided in the paper are relatively limited. Only the results of mIoU on different datasets and decoders are provided in the paper. The authors could enhance the depth of their analysis by providing more visualization and exploring the underlying reasons for the observed performance improvements, potential limitations, and scenarios where the proposed method might not be effective. This would provide a more nuanced understanding of the method's strengths and weaknesses.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses the problem of scale disequilibrium in multi-level feature fusion in semantic segmentation networks. The authors identify that bilinear upsampling, a common technique used in feature fusion, causes scale disequilibrium, leading to suboptimal training and performance. To address this, they propose using scale equalizers, which normalize feature scales across multiple levels after bilinear upsampling. The key contributions of this paper are:\n\n1. **Identification of Scale Disequilibrium:** The paper highlights a universal issue in multi-level feature fusion due to bilinear upsampling, supported by theoretical and empirical evidence.\n\n2. **Introduction of Scale Equalizers:** A novel solution is proposed to address the identified problem. The scale equalizers are hyperparameter-free, computationally inexpensive, and easy to implement.\n\n3. **Experimental Validation:** The effectiveness of the proposed scale equalizers is validated across various datasets (ADE20K, PASCAL VOC 2012, and Cityscapes) and decoder architectures (UPerHead, PSPHead, ASPPHead, SepASPPHead, and FCNHead).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. **Comparison with Existing Normalization Techniques:** Include a detailed performance and computational overhead comparison between the proposed scale equalizers and other standard normalization techniques (e.g., batch normalization, layer normalization, instance normalization).\n\n2. **Comprehensive Computational Time Analysis:** Conduct and present a thorough analysis of the computational time and resource usage associated with the proposed scale equalizers. Compare these findings with existing normalization techniques to substantiate claims about the method's efficiency.\n\n3. **In-Depth Insights and Analysis:** Provide more visualization results and analysis into the reasons behind the performance improvements observed with the proposed method. Discuss potential limitations and scenarios where the method might be less effective. This will offer a more nuanced understanding of the method's strengths and weaknesses and guide future research directions.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WvafAUPaZM", "reviewer": "Reviewer_7Qfk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Questions:\nThe author highlights that combining features of varying scales can cause a scale disequilibrium issue within the model, a pervasive and undesirable defect that adversely affects gradient descent. Could you conduct additional experiments on other computer vision tasks? For instance, exploring detection tasks, classification tasks, and temporal forecasting tasks (such as weather forecasting), which also depend on multi-level feature fusion, could provide further insights. \nGiven that neural network parameters are updated during model training, do you believe that a well trained model parameter could potentially mitigate or even resolve this disequilibrium issue? \nDo you think these kind of disequilibrium could increase the mode’s representation ability?\nThe author highlight bilinear upsampling will cause degraded gradient descent optimization. Will nearest-neighbor interpolation, transpose Conv suffer from the same problem or not?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This study discussed the scale disequilibrium in multi-level feature fusion. We demonstrated that the existing multi-level feature fusion exhibits scale disequilibrium due to bilinear upsampling, which causes degraded gradient descent optimization. To address this problem, injecting scale equalizers is proposed to guarantee scale equilibrium for multi-level feature fusion. Experiments showed that the use of scale equalizers consistently increased the mIoU index by about +0.1 to +0.4 across numerous datasets and architectures.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "No", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: please see above in claims section or the required edits.
Decision: Accept (poster)