Learnable Polyphase Sampling for Shift Invariant and Equivariant Convolutional Networks
Authors: Renan A. Rojas Gomez
Teck-Yian Lim
Alex Schwing
Minh N. Do
Raymond A. Yeh
Venue: NeurIPS 2022 Accept (2022)
Keywords: Convolutional Neural Networks
Shift equivariance
Shift invariance
Polyphase Decomposition

Abstract: We propose learnable polyphase sampling (LPS), a pair of learnable down/upsampling layers that enable truly shift-invariant and equivariant convolutional networks. LPS can be trained end-to-end from data and generalizes existing handcrafted downsampling layers. It is widely applicable as it can be integrated into any convolutional network by replacing down/upsampling layers. We evaluate LPS on image classification and semantic segmentation. Experiments show that LPS is on-par with or outperforms existing methods in both performance and shift consistency. For the first time, we achieve true shift-equivariance on semantic segmentation (PASCAL VOC), i.e., 100% shift consistency, outperforming baselines by an absolute 3.3%.

Learnable Polyphase Sampling for Shift Invariant and Equivariant Convolutional Networks
Renan A. Rojas-Gomez"
Tick-Yian Lim"
Alexander G. Schwing
Minh N. Do Raymond A. Yeh
Department of Electrical Engineering, University of Illinois Trbana-Champaig 'Department of Computer Science, Purdue University
Abstract
We propose learnable polyphase sampling (LPS), a pair of learnable down/up- sampling layers that enable truly shift-invariant and equivariant convolutional networks. IPS can be trained end-to-end from data and generalizes existing hand- crafted downsampling layers. It is widely applicable as can be integrated into any convolutional network by replacing down/upsampling layers. We evaluate LPS on image classification and semantic segmentation. Experiments show that LPS is on-par with or outperforms existing methods in both performance and shift consistency. For the first time, we achieve true shift-equivariance on se- mantic segmentation (PASCAL VOC), i.e., 100% shift consistency, outperform- ing baselines by an absolute 3.3%. Our project page and code are available at https://raymont attps://raymondyeho0/ github io http///abbbee.plllaaa.eo..eeee saapllingsmm/////////
Introduction
For tasks like image classification, shifts of an object do not change the corresponding object label, i.e., the task is shift-invariant. This shift-invariance property has been incorporated into deep-nets yielding convolutional neural nets (CNN). Seminal works on CNNs [15, 24] directly attribute the model design to shift-invariance For example, Fukushima [15] states "the network has an ability of position position-invariint pattern recognition" and LeCun al. [24] motivate CNNs by stating that they "ensure some degree of shift invariance."
CNNs have evolved since their conception. Modern deep-nets contain more layers, different non-linearities and pooling layers. Re-examining these modern architectures, Zhang [56] surprisingly finds that modern deep-nets not shift-invariant. To address this, Zhang [56] and Zou et al. [57] propose to perform anti-aliasing before each downsampling layer, and found it to improve the degree of invariance. More recently, Chaman and Dokmanic [5] show that deep-nets can be "truly shift-invariant," i.e., a model's ' output is identical for given shifted inputs. For this, they replace all downsampling layers with their adaptive polyphase sampling (APS) layer.
While APS achieves true shift-invariance by selecting the crafted downsampling scheme). important question arises: are there more effective downsampling schemes that can achieve true fhitt-invariance? Consider an extreme case, a handcrafted deep-net that always outputs zeros is truly shift-invariant, but does not accomplish any task. This motivates study how truly shift-invariant downsampling schemes can be learned from data.
polyphase component (a hand-
For this propose Learnable Polyphase Sampling (LPS), pair of down/upsampling layers that yield truly hhif-innarrianteqqiivaiann deep-nets and can be trained an end-to-end manner. For
Equal contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Pulyisho {0
G.ollteli 18.5
Fille
10,0,
Fuse
2
O
y Polviah H
Polsieha m
[[Mayitai 50.5
diccllz abift
crecalar skit
equiaazinnt
Pub
Ty
A
Filne
Rus 0,0,
42
Rase
Tye
{O.D)
Argrewd
Poly(Eis
Polstsh (O.A)
16.5
Figure 1. Left: lowpass filtering (LPF) [56] averages the inputs before downsampling to encourage shift- equivariance. The outputs y and are "more similar" due to the average filter. Despite improving shift consistency, the model still shift variant by definition. Right: adaptive polyphase sampling (APS) selects the component with the max eo-norm. This hand-crafted rule is shift equivariant as it selects the square and pentagon in both ý and y.
downsampling, LPS can be easily integrated into existing deep-net architectures by swapping out the pooling/striding layers. Theoretically, LPS generalizes APS to downsampling schemes that cannot be represented by APS. Hence, LPS's s ideal performance is never worse than that of APS. For upsampling, LPS guarantees architectures that are truly shift-equivariant, i.e., the output shifts accordingly when the input shifts. This is desirable for tasks like semantic image segmentation.
To validate the proposed LPS, conduct extensive experiments: (a) image classification on CI- FARIO [21] and ImageNet [12]: (b) semantic segmentation on PASCAL VOC [13]. We observe that the proposed approach outperforms APS and further improves anti- methods both model performance and shift consistency. Our contributions are as follows:
We propose learnable polyphase sampling (LPS), a pair of novel down/upsampling layers, and prove that they yield truly ieattt-iivviaann-eeeeee equivariant deep-nets. Different from prior works, our sampling scheme is trained from end-to-end and not handcrafted.
We theoretically prove that LPS (downsampling) is a generalization of APS. Hence, in theory, LPS improves upon APS.
We conduct extensive experiments demonstrating the effectiveness of LPS on image classification and segmentation over three datasets comparing to APS and anti-aliasing approaches.
2 Related Work
In this section, we briefly discuss related work, including shift invariant/equivariint deep- nees and pooling layers. Additional necessary concepts are reviewed in Sec. 3.
Shift Invariant/equivariint convolutional networks. Modern convolutional networks striding or pooling to reduce the amount of memory and computation in the model [17, 22, 40, 45). As pointed out by Azulay and Weiss [1] and Zhang [56], these pooling/striding layers break the shift- invariance property of deep-nets. To address this issue. Zhang [56] proposed to perform anti-aliasing, i..., lowpass filtering (LPF) before each downsampling, canonical signal processing technique for multi-rate systems [47]. We illustrate this approach in Fig. (left). Zou et al. [57] further improved the LPF technique by using adaptive filters which better preserve edge information.
While anti-aliasing filters are effective, Chaman and Dokmanic [5] show that true thift-invariance, i.e., 100% shift consistency, can be achieved without anti-aliasing Specifically, they propose Adaptive Polyphase Sampling (APS) which selects the downsampling indices, i.e., polyphase components, based on the f,-norm of the polyphase components; a handcrafted rule, as illustrated in Fig. (right). In a follow up technical report [4], APS is extended to upsampling using unpooling layers [2, 55), where the downsampling indices are saved to place values back to their corresponding spatial location during upsampling. Our work presents a novel pair of caatttinvaaiant/cquivaaa layers which are trainable, in contrast to APS's handcrafted selection rule.
down/upsampling
We note that generalizations of equivariance beyond shifts have also been studied [3, 37, 43, 46 48, 52] and applied to various domains, e.g., sets [16, 31, 35, 38, 50, 53), graphs [10, 11, 19, 27, 28, 30, 32, 44, 51], spherical images [8, 9, 20]. volumetric data [49], etc. In this work, we focus solely on shift-zquivariance for images with CNNs.
Pooling layers. Many designs for better downsampling or pooling layers have been proposed. Popular choices are vverage-Pooling [23] and Max-Pooling [36]. Other generalizations also exists, e.g., Lp-

---

Pooling [42] which generalizes pooling to use different norms. The effectiveness of different pooling layers has also been studied by Scherer et al. [41]. More similar to our work is Stochastic-Poole Pooling [54] and Mixed Max-Average Pooling [25]. Stochastic-Poolle Pooling constructs probability distribution by normalizing activations within a window and sampling during training. In our work, we present a novel design which learns the sampling distribution. Mixed Max-Average Pooling learns single scalar to permit a soft-choice between Max- and Average-Pooling guarantees while being end-to-end trainable.
contrast, our LPS has shift-equivariance
3 Preliminaries
We provide brief review on equivariant and invariant functions establish the notation. For readability, we use one-dimensional data to illustrate these ideas. In practice, these concepts are generalized to multiple channels and two-dimensional data.
Shift invariance and equivariance. The concept of equivariance, a generalization of invariance, describes how function's 's output is transformed given that the input is transformed in predefined way. For example, shift equivariance describes how the output is shifted given that the input is also shifted: think of image segmentation, if an object in the image is shifted then its corresponding mask also shifted.
A function f :RR + RM is IN, {TM, -eqeuiaariin (shift (quivvaiiui if and only if (iff)
3E E (TM, I} s.t. f(TN=) = (a)) Va E ERR.
where Tv 2 (m++1) mod N] Vn E Z denotes a circular shift, [:] denotes the indexing operator, and I denotes the identity function. This definition of equivariance handles the ambiguity that arises when shifting by one and downsampling by two. Ideally, a shift by one at the input should result in a 0.5 shift in the downsampled sigmal, which is not achievable on the integer grid. Hence, this definition considers either shift by one or no shift at the output equivariant.
Following the equivariance definition, invariance can be viewed special case where the trans- formation at the output is an identity function, I. Concretely, function f R + RM M is TN, {T}- equivariant (hhfft-ivvariant iff
xTTxz) f(x) Vo E R.
(2)
To obtain shift-invariance from shift shift-equivariant functions is common use global pooling. Observe that
f(Txx) ((
shift-invariant f shift-equivariant, as summation is an orderless operation. Note that the composition of shift-equivariant functions maintains shift-equivariance. Hence, f can be stack of equivariant layers, e.g., a composition of convolution layers.
While existing deep-nets [17, 26, 40] do use global spatial pooling, these architectures are not shift- invariant. This due pooling and downsampling layers, which are not shift-equivariant as we review next.
Downsampling and pooling layers. A downsampling- by-ttiongn----oooooo layer D R.N. + RLN/2] is defined as
[n] = x [2[2]] Vn E Z,
(4)
which returns the even indices of the input æ. As a shift operator makes the odd indices even, downsampling layer
hhift-equivarianniivvaaaant
Commonly used average or max pooling can be viewed as an average or max filter followed by downsampling, hence pooling is also not shift-quuivrian/invaaiant To address this issue, Chaman and Dokmanic [5] propose adaptive polyphase sampling (APS) which is an input dependent (adaptive) selection of the odd/even indices.
Adaptive polyphase sampling. Proposed by Chaman and Dokmanic [5], adaptive polyphase sam- pling (APS) returns whether the odd or even indices, i.e., the polyphase components, based on their norms. Formally, APS R + RLN/2] is defined as:
APS(2:) Po)y())) if ||Poly())ol > ||Pol(())]I|
(5)
Poly(2)) otherwise
3

---

Poly(ar)a 10
A
Po
D(ar)
Poly(#))
(0, A O
Po
Ai argmax
Poly(2))
Po (4
[())
0.7
cirolarr shift
logit permatationn
stift ergiiseiion
Txx (O, O IAA)
Poly( {O
Po(k
0
0.7
LPS(#) {
Poly($
P8
argmax
Poly(*) (O.4)
Po(k 1
0.3
Figure 2. Illustration of our proposed LPD design. Inspired by the polyphase permutation property. Lemmaa 1, the likelihood of selecting polyphase component is learned with thift-permutatinn equivariant CNN model po- This enforces components between the input and its shifted version to have consistent logits, which leads a ahift--quivariant downsampling layer.
where a E RNN is the input and Poly(æ); denotes the polyphase components, i.e.,
Poly(z)o in] x22n] and Poly(2)) [n] x<2n 1].
While this handerafted selection rule achieves consistent selection of the polyphase components, not the only way achieve it, e-g., returning the polyphase component with the smaller norm. In this work, we study a family of shift-equivariant sampling layers and propose how to learn them in data-driven manner.
4 Approach
Our goal is to design learnable down/upsampling layer that is hhft-invariantéquiivriant We formulate down/upsampling by modeling the conditional probability of selecting each polyphase component given input. For this we use small neural network. This enables the sampling scheme to be trained end-to-end from data, hence the name learnable polyphase sampling (LPS).
In Sec. 4.1, we introduce learnable polyphase downsampling (LPD), discuss how to train it end-to-end, and show that generalizes APS. In Sec. 4.2, we propose a practical layer design of LPD. Lastly, in Sec. 4.3, we discuss how to perform LPS for upsampling, namely, learnable polyphase upsampling (LPU). For readability, we present the approach using one dimensional data, i.e., row in an image.
4.1 Learnable Polyphase Downsampling
We propose learnable polyphase downsampling (LPD) to learn a shift-equivariant downsampling layer. Given input feature map CRCN L.DD spatially downsamples the input to produce output in [ via
LPD(z))e, = [[c, , 2n + km] A Poly(x)
where k* arg maxke 10.1) ps Ps(k (k k|(²) and (oly( denotes the k* polyphase component. We model a conditional probability pe (k)x) for selecting polyphase components, i.e.. k denotes the random variable of the polyphase indices. For ID data, there are only two polyphase components.
Critically, not all po lead to equivariant downsampling layer. For example, po pokk 0) results in the standard down-sampling which always returns values on even indices for 1D signals. We will next examine which family of po achieves shift-equivariant downsampling layer.
Shift-permutatiine equivariance of ps- Consider example in Fig. 2. We can see that a circular shift in the spatial domain induces permutation in the polyphase components. Observe that the top-row the polyphase component containing the blue circle orange square are permuted to second row when the input is circularly shifted. We now state this formally.
Lemma Polyphase shift- ciftt-permutation property
(Poly(a)) (TyPoly(2)))
if k 0 1:
Poly(Twæ))
(8)

---

Proof. By definition, Poly (oly(TNz))+(1)
= Tya: k) mod NJ =2(2n + k ++11 mod N] [2[ + +1 mod N] = Poll((t) [1(((( + 1)) mod NJ = (arPoly(a))
(9)
k: = 0
(10)
if k ==
From Lemma 1, we observe that to achieve equivariant downsampling layer spatially shifted input should lead permutation of the selection probability (Claim We note that pa said to be shift- perrmutation-equiviae if
Po (k = (k) TT) = po = k) k))z)
(11)
where n denotes a permutation on the polyphase indices, i.e., "swap" of indices characterized by T (k:), Le., (0)) = 1 and (11)= =.
Claim 1. po tttii--remaaatineequuvvieee equivariant, defined Eq. (11), then LPD defined Eq. (7) is shift-equivariant downsampling layer.
Proof. Let &e Tyæ be shifted version of B. Recall LPD(x) and LPD(a) are defined as:
LPD(x) & Poly(2)) k" arg max po (k kl(z), (E(0,1)
(12)
(+D & pol F-- = arg max po = k|(²). kE(0.1)
(13)
From Lemma 1, LPD(TTa)) can be expressed as:
(Poly(=)) TosPol(())
if 1 =0
LPD(TNæ)
(14)
if =
As Po is the shift-pemmuatioo-cqqivvainn,
i = = k
(15)
Finally, combining Eq. (14) and Eq. (15),
(x)
if k k- =0 - k + k'' 1) LLPD(x),
LPD(Tyx) Por)oly()))
(16)
showing that LPD satisfies the shift-equivariance definition reviewed in Eq. (1).
Here, we parameterize pa with a small neural network. The exact construction of shift-permutation equivariant deep-net architecture is deferred to Sec. 4.2. We next discuss how to train the distribution parameters 0 in LPD.
End-to-end training of LPD. At training time, to incorporate stochasticity and compute gradients, parameterize po using Gumbel Softmax [18, 29]. To backpropagate gradients to relax the selection of polyphase components as a convex combination, i.e.,
y Σ Z Poly())
po(k)))
(17)
where z corresponds to selection variable, i.e., Σ. F5 = 1 and 2 [0, 1]. Note the slight abuse of notation pe pk(kx)) denotes probability over polyphase indices represented in one-ho hot format. We further encourage the Gumbel Softmax to behave more like an argmax by decaying its temperature during training recommended by Jang et al. [18].
LPD generalizes APS. key advantage of LPS over APS is that can learn from data, potentially leading to better sampling scheme than handcrafted one. Here, we show that APS is a special case of LPD. Therefore, LPD should perform least well APS if parameters are trained well.

---

Paly(arjo 2OXM
Cone
ReLi
x0'xar
Flatiom
ROON
Chobal
76 0(a)) e (0,
Poiyphase
Deompuoitio
Combee Saftwox
abazed
py(k 1/xr) e [0.
Poly(att
(o(Poly(2))]] R
CR
Coor
Hell
Platee
Figure Proposed shift peremttetttttonn equivariant model. By sharing weights across components and computing logits via global pooling, this neural network is naft-perrutation equivariant. To introduce stochasticity during training. adopt the Gumbel Softmax for sampling.
Claim 2. APS a special case of LPD, i.., LPD can represent APS's selection rule.
Proof. Consider a parametrization of pe as follows,
exp (|)oy(())))l))
pe (k k|z) Σ, (xl(()
(18)
As the exponential is a strictly increasing function we have
arg magmaxuu po k (x)= = arg max (|ool()))*ll
(19)
Eq. (18) is softmax with input ||oll( (z) as such a function exists, LPD generalizes APS.
4.2 Practical LPD Design
We aim for a conditional distribution pe that is shift-permutation equivariant to obtain a shift- equivariant pooling layer. Let the conditional probability be modeled as:
exp]fe
Po (k k[(=)
(20)
Σ exp (Poly(2)))
where fo CBCC'''xW' -> R is small network that extracts features from polyphase component
Poly(2) (x)). We first show that po is shift-permutationn equivariant if fo is shift invariant.
Claim 3. In Eq. (20), if fo is shift invariant then po is shift-permutation equivariant (Eq. (11)).
Proof. Denote feature map and its shifted version & 2 TNw. By definition.
(xp(Jo(Poly(Tvaaaaa P x (Na) E3 exp((o(Poll(TTxa)))
(21)
With shift-invariant fo and using Lemma
So (Po))(TT=))(xs) (T(TM = So (Poly((z)))
(22)
exp(Poly(x)4)
(k (k= T [k)]TT=)) E, exp(k Poly(x)))) Po = k((=)
Based the result in Claim 3, shift- sefft-permutation property. The general design principle: share parameters across polyphase indices, just convolution achieves shift equivariance by sharing parameters, plus averaging over the spatial domain. An illustration the proposed meta-architecture shown in Fig. 3.
present convolution based aectterrciee that satisfies the
Fully convolutional model. Logits are extracted from the polyphase components via fully- convolutional operations followed by averaging along the channel and the spatial domain. Following fromy is denoted as:
this, f
CODY (Poly(x)))
iS (Poly(m)x)[c,m))
(23)
CM
where feomy RR CxM + CCM is a CNN model (without pooling layers) and M = [N/2]. The shift equivariance property of feomy guarantees that foonr is thift-invarian due to the global pooling.

---

4.3 Learnable Polyphase Upsampling (LPU)
Beyond shift invariant models, we extend the theory from downsampling to upsampling, which permits to design shift-equivarian models. The main idea is to place the features obtained after down- sampling back to their original spatial location. Given a feature map y E CRCxN//2 downsampled via LPD from æ, the upsampling layer outputs u E _CXN are defined as follows:
(oly(u) y, k- arg (0 Po = kt(z))
(24)
0,
otherwise.
We name this layer learnable polyphase upsampling (LPU), i.e., LPU Po) _ u. We now show that LPU and LPD achieve shift-equivariance.
Claim 4. If pa is shift-permutation equivariant, as defined in Eg. (11), then LPU o LPD is shiff-equivariant.
Proof. We prove this claim following definitions of LPU, LPD and Lemma The complete proof is deferred to Appendix Sec. A1.
End-to-end training of LPU. As in downsampling, we also incorporate stochasticity via Gumbel- Softmax. To backpropagate gradients to pe. we relax the hard selection into a convex combination, i.e.
Poly( (u) ) Zk y,
pa (k)
(25)
Anti-aliasing for upsampling. While LPU provides a shift-equivariant upsampling scheme, it intro- duces zeros the output which results feghffreuuency components. This is known as aliasing in a multirate system [47]. To resolve this, following the classical solution, we apply low-pass filter scaled by the upsampling factor after each LPU.
Experiments
We conduct experiments image classifica- tion following prior works. We report on the
Table 1. ResNet-18 (CIFARIO) top-1 accuracy and circular shift consistency. LPS oulperforms all alternative pooling and anti-aliasing methods.
same architectures and training setup. We
port both the circular shift setup in APS [5] and the standard shift setup in LPF [56].
Method Anti-Alias
Acc.
C-Cons. 1
Bascline
91.44 #
89.64 0.39
We also evaluate on semantic segmentation, considering the circular shift, inspired by APS, and the standard shift setup following DDAC [57]. For circular shift settings, the
APS
94.07 0.28
100 0.0
LPS (Ours)
94.45 0.05
100 0.0
LPF
Rect-2
93.1
1 94.75
APS
Rect-2
94.38 0.25
100 0.0
(Ours)
Rect-2
94.69
100 +.0
theory exactly matches the experiment hence true equivariance is achieved. To our knowl- edge, this the first truly shift equivariant model reported PASCAL VOC.
LPF
Tri-3
93.97 0.18
96.74
APS
Tri-3
94.36 0.117
100 0.0
LPS (Ours)
Tri-3
94.8 - 0.14
100 1 0.0
LPF
Bin-5
91.43
98.34 + 0.15
APS
Bin-5
94.44
100 + 0.0
LPS (Ours)
Bin-5
94.49
100 + 0.0
5.1 Image Classification (Circular Shift)
DDAC
Adapt-3
93.17 0.19
95.13 0.15
APS
Adapt-3
94.42 + 0.13
100 0.0
Experiment & implementation details.
LPS (Ours)
Adapt-3
94.57 + 0.12
100 + 0.0
Following APS, all the evaluated pooling and anti-aliasing models use the ResNet-18 [17] architecture with circular padding on CIFAR 10 [21] and ImageNet [12]. Anti-alias filters are applied after each
downsampling layer following LPF [56] and DDAC [57]. We also replace downsampling layers with
APS [5] and our proposed LPS layer. We provide more experimental details in Appendix Sec. A4.
Evaluation metrics. We report classification accuracy to quantify the model performance on the original dataset without any shifts. To evaluate shift-invariance, following APS, report circular- consistency (C-Cons.) which computes the average percentage of predicted labels that are equal under two different circular shifts, i.e.
(Circ. Shifts1, w1 (4)) (CCric. Shift,
(26)
7

---

where ((T denotes the predicted label for an input image I and h1, w1, h2, w2 are uniformly sampled from 0 32. We report the average over five random seeds.
CIFARIO results. Tab. shows the classification accuracy and circular consistency on CIFARIO IO. We report the mean and standard deviation the ResNet-18 model. We observe that the proposed LPS improves classification accuracy over all baselines while achieving 100% circular consisteney. In addition to attaining perfect shift consistency, we observe that combining anti-aliasing with LPS further improves performance.
five runs with different random initialization of
ImageNet results. We conduct experiments ImageNet with circular shift using ResNet- 18. In Tab. 2, we compare with APS's best model using a box filter (Rectangle-2), as reported by Chaman and Dokmanic [5]. While both APS and LPS achieve 100% circular consistency, proposed LPS improves on classification accuracy in all scenarios, highlighting its advantages.
Table 2. ResNet- 18 top-1 I classification accuracy and circular consistency ImageNet.
Table 3. ResNet-50/101 top-1 classification accuracy and shift consistency ImageNet.
Method
Anti-Alias
Acc. f
C-Cons. f
Method Anti-Alias
Model
Acc. f S-Cons. f
Baseline
64.88
80.39
Baseline
ResNet-50
76.16
89.2
APS
67.05
100
LPF
Tri-3
ResNet-50
76.83
90.91
LPS (Ours)
67 39
100
LPS (Ours)
Tri-3
ResNet-50
77.14
91.43
LPF
Rect-2
67.03
84.35
Baseline
ResNet-101
77.7
90.6
APS
Rect-2
67.6
100
LPF
Tri-3
ResNet-100
78.4
91.6
LPS (Ours)
Rect-2
68.45
100
LPS
Tri-3
ResNet-101
78.51
91.69
DDAC
Adapt-3
67.6
77.23
DDAC
Adapt-3
ResNet-
79.0
91.8
APS
A
69.02
100
DDAC*
Adapt-3
ResNet-101
78.64
91.83
LPS (Ours)
Adapt-3
69.11
100
LPS (Ours)
Adapt-3
ResNet-101
78.8
92.4
5.2 Image Classification (Standard Shift)
Experiment & implementation details. To directly compare with results from LPF and DDAC, we conduct experiments on ImageNet using the ResNet-50 and ResNet-101 architectures following their setting, i.e., training with standard shifts augmentation and using convolution layers with zero-padding.
Evaluation metrics. Shift consistency (S-Cons.) computes the average percentage of
1 (f)) = (Shhifta,.wz24)))
(27)
where h1, w1, h2, w2 are uniformly sampled from the interval (0,...,32] To avoid padding at the boundary, following LPF [56], we perform shift on an image then crop its center 224 224 region. We note that, due to the change in content at the boundary, perfect shift consistency is not guaranteed.
ImageNet results. In Tab.
compare the best anti-aliasing result reported in LPF, DDAC and DDAC" which is trained from the authors' released code using hyperparameters specified in the repository. Note, in standard shift setting LPS longer achieves true shift-invariance due padding at the boundaries. Despite this gap from the theory, LPS achieves improvements in both performance and coniiteenns over the basclines. When compared to LPF, both ResNet-50 and ResNet-101 architecture achieved improved classification accuracy and shift-consisteney. When compared to DDAC, LPS achieves comparable accuracy with higher shift-consistency.
5.3 Trainable Parameters and Inference Time
While LPD is a data-driven downsampling layer, we show that the additional trainable parameters introduced by it are marginal with respect to the classification architecture. Tab. 4 shows the number of trainable parameters required by the ResNet-101 models.
For each method, we report the absolute number trainable parameters, which includes both classifier and learnable pooling weights. We also include the relative number of trainable parameters, which only considers the learnable pooling weights and the percentage it represents with respect to the default ResNet-101 architecture weights.
For comparison purposes, we also include the inference time required by each model to evaluate their computational overhead. Mean and standard deviation of the inference time is computed for

---

Table 4. Number of trainable parameters and inference time required by ResNet- 101 using our proposed L.D
layer and alternative methods. Absolute corresponds to the total number of trainable parameters, including
classifier and pooling layers. Relative corresponds to the parameters the pooling layers only. Inference time statistics computed on 100 batches with 32 images of size 224 x 224 x 3 each.
Pooling
Anti-alias
Trainable Parameters
Inference Time
Absolute
Relative
Mcan (ms)
Std (ms)
LPD LPD ResNet-101 (Default)
LPF (Tri-3)
42, 966, 730
446, 080 (1.05%)
83.61
0.19
DDAC (Adapt-3)
44, 141,5,000 734
2. 230, 384 (5.24%)
130.69
0.2
DDAC (Adapt-3)
44,304, 954
784, 304 (4.2%)
124.69
0.25
APS
LPF (Tri-3)
42, 520, 650
0
77.68
0.22
ResNet-101 (Default)
LPF (Tni-3
42, 520, 650
68.72
0.4
Inpot 8-
Input 2 2
c-
Figure 4. ResNet-50 LPS activation maps (ImageNet). We show all four polyphase components of the fourth layer (first 8 channels). The component selected by LPS is boxed in blue, while that with the largest E--norm boxed red. This indicates LPS did not learn to reproduce APS.
each method on 100 batches of size 32, Following ImageNet default settings, the image dimensions corrrespond to 224 x 224 3.
Results show our proposed LPD method introduces approximately 1% additional trainable parameters on the ResNet- 101 architecture, and increases the inference time roughly by 14.89 ms over the LPF anti-aliasing method (the less computationally expensive of the evaluated techniques). On the other hand, most of the overhead comes from DDAC, which increases the number of trainable parameters by approximately 4% and the inference time by approximately 55.97 ms. Overall, our comparison shows that, by equipping a classifier with LPD layers, the computational overhead is almost trivial.
Despite increasing the number of trainable parameters, empirically show that our LPD approach outperforms classifiers with significantly more parameters. Please refer to Sec. A4.3 for additional experiments comparing the performance of our ResNet- + LPD model against the much larger ResNet- 152 classifier.
LPD learns sampling schemes different from APS. To further analyze LPD, we replace all the LPD layers with APS for a ResNet- 101 model trained on ImageNet. We observe critical drop in top-1 classification accuracy from 78.8% to 0.1 %, indicating that LPD did not learn downsampling scheme equivalent to APS. We also counted how many times (across all layers) LPD selects the max-norm. On the ImageNet validation set, LPD selects the max norm polyphase component only 20.57% of the time. These show LPD learned selection rule that differs from the handcrafted APS.
Qualitative study on LPD. In Fig. 4 we show the selected activations, at the fourth layer, of ResNet-50 model with LPD. Each column describes the first channels of the four possible polyphase components & E (O.....3), 33. The component selected by LPD, denoted as k", is boxed in blue. For comparison purposes, we also boxed the component that maximizes the form in red. We observe that LPD is distinct from APS as they select a different set of polyphase components. However, we did not observe a specific pattern that can explain LPD's selection rule.
5.4 Semantic Segmentation (Circular Shift)
Experiment & implementation details. We evaluate LPS's down/upsampling layers on semantic segmentation. As in DDAC [57], we evaluate using the PASCAL VOC [13) dataset. Following DDAC, we use DeepLabb33 [6] as our baseline model. We use the ResNet- 18 backbone pre-trained on the ImageNet (circular shift) reported in Sec. 5. We experiment with using only the LPD backbone and the full LPS, i..., both LPD and LPU.

---

Table 5. Semantic segmentation with circular shifts (ResNet-18 with De)plabbV+)) on PASCAL VOC.
Table Semantic segmentation with standard shifts (ResNet- 101 with DeepLabV3+) on PASCAL VOC. Results above the line are from DDAC's paper.
Method
Anli-Alias
mloUt
mASCC1
DeepLabV3+
70.03
95.42
Method
Anti-Alias
mloU 1
mASSC
LPF
Rect-2
71.02
96.03
DeepLabV3+
78.5
95.5
DDAC
Adapt-3
72.28
95.98
LPF
Tri-3
79.4
APS
Adapt-3
72.37
96.70
DDAC
Adapt-3
80.3
LPD only
Adapt-8
72.47
96.23
DDAC
Adapt-3
80.31
97.83
LPS (Ours)
Adapt-3
72.37
100
LPS (Ours)
Adapt-3
80.43
97.98
We also evaluate the performance using APS, which corresponds hand-crafted downsampling scheme, in combination with the default bilinear interpolation strategy from BeepLabV3++ Note that, while our LPS approach consists of both shift equivariant down and upsampling schemes (LPD and LPU, respectively), APS only operates on the downsampling process. Thus, the latter does not guarantee circularly shift equivariant segmentation.
Evaluation metric. We report mean intersection over union (mloU) to evaluate segmentation perfor- mance. To quantify sircular-quivarrianc, we report mean Average Segmentation Circular Consistency (mASCC) which computes the average percentage of predicted (per-pixel) labels that remained the same under two different circular shifts. Iee., shifted image is passed to model to make segmen- tation prediction. This prediction is then "unshifted" for comparison. We report five random shift pairs for each image.
Results. We report the results for PASCAL VOC in Tab. 5. Overall, we observe that LPD only and LPS achieve comparable results to DDAC and APS in mloU. Notably, LPS achieves 100% mASCC, matching the theory. This confirms that both the proposed LPD and LPU layers are necessary and are able to learn effective down/up sampling schemes for semantic segmentation.
5.5 Semantic Segmentation (Standard Shift)
Experiment & implementation details. For the standard shift setting, we directly follow the ex- perimental setup from DDAC. We use DeepLabV3+ with a ResNet-101 backbone pre-trained on ImageNet as reported in Sec. 5.2.
Evaluation metric. To quantify the shift-equivariaace, following DDAC, we report the mean Average Semantic Segmentation Consistency (mASSC) which linear-shift version of mASCC described in Sec. 5.4 except boundary pixels are ignored.
Results. In Tab. 6, compare mloU and mASSC of LPS various baselines. We observe that LPS achieves improvements mloU and consistency when compared to DDAC". We note that DDAC [57] did not release their code for mASSC. For fair comparison, report the performance of their released checkpoint using our implementation of mASSC, indicated with DDAC". Despite the gap in theory and practice due non-cireular padding at the boundary, our experiments show LPS remains an effective approach to improve both shift consistency and model performance.
Conclusion
We propose learnable polyphase sampling (LPSS,, pair of shift-equivariant down/upsampling lay- ers. LPS's design theoretically guarantees circular shift-invariance and equivariance while being end-to-end trainable. Additionally, LPS retains superior consistency standard shifts where the- oretical assumptions are broken at image boundaries. Finally, LPS captures a richer family of shaft-invarianteqqiivraaat functions than APS. Through extensive experiments image classifica- tion and semantic segmentation, we demonstrate that LPS on-par with/exceeds APS, LPF and DDAC in terms of model performance and consistency.
Ackmowledgments: We thank Greg Shakhnarovich & PALS TTI-Chicago for the thoughtful discussions and computation resources. This work is supported in part by NSF under Grants 1718221, 2008387, 2045586, 2106825, MRI 1725729, NIFA award 2020-67021-32777 -32999, funding by PPG Industries, Inc. We thank NVIDIA for providing GPU.
10

---

References
[1] A. Azulay and Y. Weiss. Why do deep convolutional networks generalize so poorly to small image transformations? JMLR, 2019.
[2] V. Badrinarayanan. A. Kendall, and R. Cipolla. SegNet: deep convolutional encoder-decoded architecture for image segmentation. IEEE TPAMI, 2017.
[3] M. M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE SPM, 2017. 2
[4] A. Chaman and I. Dokmanic. Truly shift-equivariant convolutional neural networks with adaptive polyphase upsampling. arXiv preprint arXiv:2105.04040, 2021. 2
[5] A. Chaman and I. Dokmanic. Truly shift-invariant convolutional neural networks, Proc. CVPR, 2021. 1, 2, 3, 7, 24
[6] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proc. ECCV, 2018. 9
[7] T. Cohen and M. Welling. Group equivariant convolutional networks. In Proc. ICML, 2016.
[8] T. Cohen, M. Weiler, B. Kicanaoglu, and M. Welling. Gauge equivariant convolutional networks and the icosahedral CNN. In Proc. ICML, 2019.
[9] T. S. Cohen, M. Geiger, J. Köhler, and M. Welling. Spherical CNNs. In Proc. ICLR, 2018. 2
[10] P. de Haan, M. Weiler, T. Cohen, and M. Welling. Gauge equivariant mesh CNNs: Anisotropic convolutions on geometric graphs. In Proc. ICLR, 2021. 2
[11] M. Defferrard, X. Bresson, and Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Proc. NeurlPS, 2016. 2
[12] Deng, W. Dong, R. Socher, L.-J. Li. K. Li, and L. Fei-Fei. ImageNet: large-scale hierarchical image database. In Proc. CVPR, 2009. 2,
[13] M. Everingham, S. A. Eslami, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The PASCAL visual object classes challenge: A retrospective. IJCV, 2015. 2,9
[14] W. Falcon and The PyTorch Lightning team. PyTorch Lightning, 2019. URL htLpttt:///tthbbb com
[15] K. Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 1980.
[16] Hartford, D. Graham, K. Leyton-Brown and S. Ravanbakhsh. Deep models of interactions across sets. In Proc. ICML, 2018. 2
[17] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proc. CVPR, 2016. 2, 2,3,
[18] E. Jang, S. Gu, and B. Poole. Categorical reparameterization with Gumbel-Softmax. In Proc. ICLR, 2017. 5
[19] N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In Proc. ICLR, 2017. 2
[20] R. Kondor, Z. Lin, and S. Trivedi. Clebsch-Gordan Nets: a fully Fourier space spherical convolutional neural network. In Proc. NeurIPS, 2018. 2
[21] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009. 2,
7
[22] A. Krizhevsky, I. Sutskever, and E. Hinton. ImageNet classification with deep convolutional neural networks. In Proc. NeurIPS, 2012. 2

---

[23] Y. LeCun, B. Boser, J. Denker, D. Henderson, R. Howard, W. Hubbard, and L. Jackel. Hand- written digit recognition with back proppppopggttion network. Proc. NeurIPS, 1989.
[24] Y. LeCun, P. Haffner, L. Bottou, and Y. Bengio. Object recognition with gradient- based learning. In Shape, contour and grouping computer vision, 1999. 1
[25] C.-Y. W. Gallagher, and Z. Tu. Generalizing pooling functions in convolutional neural networks: Mixed, gated, and tree. In Proc. AISTATS, 2016. 3
[26] M. Lin, Q. Chen, and S. Yan. Network in network. arXiv preprint arXiv:/312.4400/ 20133. 3
[27] I.-J. Liu, R. Yeh, and A. Schwing. Pic: permutation invariant critic for multi-agent deep reinforcement learning. In Proc. CORI, 2020. 2
[28] I.-J. Liu, Z. Ren, R. A. Yeh, and A. G. Schwing. Semantic tracklets: An object-centric represen- tation for visual multi-agent reinforcement learning. In Proc. IROS, 2021. 2
[29] C. J. Maddison, A. Mnih, and Y. W. Teh. The concrete distribution: A continuous relaxation of discrete random variables. In Proc. ICLR, 2017. 5
[30] H. Maron, H. Ben-Hamu, N. Shamir, and Y. Lipman. Invariant and equivariant graph networks.
In Proc. ICLR, 2019.
[31] H. Maron, O. Litany, G. Chechik, and E. Fetaya. On learning sets of symmetric elements. In Proc. ICML, 2020.
[32] C. Morris, G. Rattan, S. Kiefer, and S. Ravanbakhsh. SetqNets: Sparsity-aware permutation- equivariant graph networks. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proc. ICML, 2022.
[33] Neptune team. Neptune client, 2019.
URL
neptune-client. 19
[34] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang. Z. DeVito, M. Raison, A. Te- jani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, figh-performance deep learning library. In Proc. NeurIPS, 2019, 19
[35] C. R. Qi, H. Su, K. Mo, and L. Guibas. PointNet: Deep learning on point sets for 3D classification and segmentation. In Proc. CVPR, 2017. 2
[36] M. Ranzato, Y.-L. Bourcau, Y. LeCun, al. Sparse feature learning for deep belief networks. Proc. NeurIPS, 20: 185-1192, 2007. 2
[37] S. Ravanbakhsh, J. Schneider, and B. Póczos. Equivariance through parameter-sharing. In Proc. ICML, 2017. 2
[38] S. Ravanbakhsh, J. Schneider, and B. Poczos. Deep learning with sets and point clouds. In Proc. ICLR workshop. 2017.
[39] D. Romero, E. Bekkers, Tomezak, and M. Hoogendoorn. Attentive group equivariant convolutional networks. In Proc. ICML, 2020. 2
[40] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. MobileNetV2: Inverted residuals and linear bottlenecks. In Proc. CVPR, 2018. 2,3
[41] D. Scherer, A. Müller, and S. Behnke. Evaluation of pooling operations in convolutional architectures for object recognition. In Proc. ICANN, 2010. 3
[42] P. Sermanet, S. Chintala, and Y. LeCun. Convolutional neural networks applied to house numbers digit classification. In Proc. ICPR, 2012. 3
[43] M. Shakerinava and S. Ravanbakhsh. Equivariant networks for pixelized spheres. Proc. ICML, 2021. 2
12

---

[44] D. I. Shuman, S. K. Narang, P. Frossard, A. Ortega, and P. Vandergheynst. The emerging field of signal processing on graphs: Extending high- nigh-dimensional data analysis networks and other irregular domains. IEEE SPM, 2013. 2
[45] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image
recognition. In Proc. ICLR, 2015. 2
[46] S. R. Venkataraman, S. Balasubramanian, and R. R. Sarma. Building deep equivariant capsule networks. In Proc. ICLR, 2020. 2
[47] M. Vetterli, J. Kovačević, and K. Goyal. Foundations of signal processing. Cambridge University Press, 2014. 7,7
[48] M. Weiler and G. Cesa. General Equ)-quuiv steerable CNNs. In Proc. NeurlPS, 2019. 2
[49] M. Weiler, M. Geiger, M. Welling, W. Boomsma, and T. S. Cohen. 3D steerable CNNs: Learning rotationally equivariant features volumetric data. In Pruc. NeurIPS, 2018.
[50] R. A. Yeh, Y.TT. Hu, and A. Schwing. Chirality nets for human pose regression. Proc. NeurlPS, 2019. 2
[51] R. A. Yeh, A. G. Schwing, J. Huang, and K. Murphy. Diverse generation for multi-agent sports games. In Proc. CVPR, 2019. 2
[52] R. A. Yeh, Y.-T. Hu, M. Hasegawa-Johnson, and A. Schwing. Equivariance discovery by learned parameter-sharing, In Proc. AISTATS. 2022. 2
[53] M. Zaheer, Kottur, Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. Smola. Deep sets. In Proc. NeurIPS, 2017. 2
[54] M. Zeiler and R. Fergus. Stochastic pooling for regularization of deep convolutional neural networks. Proc. ICLR, 2013. 3
[55] M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In Proc. ECCV, 2014.
[56] R. Zhang. Making convolutional networks shift-invariant again. In Proc. ICML, 2019. 1,
8, 24
[57] X. Zou, F. Xiao, Z. Yu, and J. Lee. Delving deeper into anti-aliasing in convnets. In Proc. BMVC, 2020. 1, 2, 7, 9, 10,
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] All our theoretical claims are backed by proofs and are included in the main paper or appendix. Empirical claims are supported by experiments. (b) Did you describe the limitations of your work? [Yes] Our theoretical analysis focuses on circular-shift equivariance and invariance. In our experiments we also evaluate on linear-shifts which no longer achieve true equivariance due to image boundaries. See discussion at Sec. 5.2.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] In this paper, we develop general upsampling and downsampling layers. We evaluate on fairly established tasks within the computer vision community. We foresce no potential negative societal impact from this work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] All the authors have read the ethics guidelines.
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] A detailed description of our model assumptions is included in Sec. 4.
(b) Did you include complete proofs of all theoretical results? [Yes] Proofs to our theoreti- cal analysis are included in the main paper the appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or a URL)? [Yes] Please refer to the Python code included with the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Training setup and hyperparameter details are included See. A4.4 and the attached code.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We report mean and standard deviation values across random seeds in Sec. 5 when it is feasible with our computational resource. Certain experiments are simply too expensive for us to train over multiple runs.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] The computational details are included in the appendix. Please refer to Sec. A4.4.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cited the papers on datasets (CIFAR10 10, ImageNet, PASCAL VOC) and prior works.
(b) Did you mention the license of the assets? (Yss] These assets are all publicly availiable.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes]
We have attached our code in the supplemental materials.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "LUyt23r5xNB", "reviewer": "Reviewer_bbM7", "summary": "The paper tackles the issue of shift invariance/equivariance in CNNs. Prior work (APS) suggested, when downsampling, to select the index for whose value has the largest norm. This paper suggests a generalization which consists of learning the index-selection rule instead. Experiments on classification (invariance: CIFAR, ImageNet) and segmentation (equivariance ; PascalVOC) show convincing results.\n\nLearning this selection is implemented via a carefully crafted small NN and the gumbel-softmax trick.", "questions": "See above.", "limitations": "None.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The paper clearly motivates the problem.\n- The paper gives just the right amount of background knowledge and prior work.\n- The evaluations are to the point.\n\n## Weaknesses\n\nThe experiments show convincing results: shift invariance matching theory, and small but consistent improvement in \"regular\" accuracy measures. However, I wonder whether the small but consistent improvements may not be simply due to the fact of introducing more learnable parameters. This is almost addressed in Appendix A7, but what I am missing is the results of either (a) resnet + LPS but with slightly reduced ResNet size to match plain resnet params/speed, or (b) plain resnet with slightly increased size to match resnet + LPS params/speed.\n\nAlso, this should be mentioned in the main paper, with reference to the appendix. Actually, most of the appendix should be mentioned in the main paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IhA4m8k0Hc", "reviewer": "Reviewer_3D4j", "summary": "This paper presents a  new framework to enable the design of shift invariant and equivalence convolutional neural networks. The key idea in this paper is introducing a  learnable polyphase sampling scheme which can be used in both down/up sampling layers.  Compared to existing methods, the proposed LPS is trained from data instead of manually.  The LPS is introduced with detailed  analysis and the implementation is also shown in both down/up sampling layers.  Experiments on image classification and semantic segmentation valid its effectiveness in practices. \n ", "questions": "Q1.In general LPS has similar performance to APS in Resnet 18 setting. Is the result consistent in Resnet 50 or 101? \n\n\nQ2.  Table 4 shows that  LPD only works better than LPS, while there is no ablation shows that the importance of LPD and LPU in the neural network when keep replacing LPD LPU in downsampling and upsampling layers.   Will is possible LPD + a few LPU layers  can have even better performance?\n\n", "limitations": "The limitation in this paper is the marginal performance improvement over APS, which leads the bias for justifying the importance of the motivation in this paper.   This is because APS is a special case of  LPS, the contribution of this paper can only be valid through theoretical analysis.   ", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\nThe main contribution in this paper is extending the manually designed adaptive polyphase sampling layers to learnable polyphase sampling layers with theoretically prove.  Especially, the introduced conditional probability directly learn from the data while the LPS keep the property of APS.   Besides the LPS, both downsampling and upsampling LPD and LPU have been showed with implementation and detailed analysis. \n\n Cons:\nThe main idea of this paper is generalized the existing work APS in a learnable manner.  Compared to the significance of APS the contribution is limited.  Also it is not clear how LPU and LPD work in real cases. For example, in Semantic Segmentation cases, LPD  only seems works better than LPD and LPU while it is not clear how LPU works and what the reason behind that.  \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Inadequate Data and Algorithm Evaluation"]}, "extra_text": {}}, {"review_id": "FDHIjRfNlL", "reviewer": "Reviewer_MjxQ", "summary": "This paper generalizes polyphase resampling and applies learnable selection of the polyphase components, instead of based on signal magnitudes. It achieved better accuracy and shift consistency on classification and segmentation.", "questions": "Maybe I missed something, as APS is a specicial case of LPU, and why is LPU more shift-invariant than APS theoretically, and on segmentation empirically? As LPU is data dependent, so it could be approximate APS on some bad test case?\n\nIn figure 4 caption \"This indicates LPS did not learn to reproduce APS\" I can infer this from the algorithm, but is this good from visual meaning perspective? Difficult to see.\n\nThe running time, computation complexity comparison with APS, and other baseline should be included.", "limitations": "yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: The paper writes with professionism in notations and arangement. The authors did both experiment in high-level classification and low-level segmentation.\n\nWeakness: My personal feeling here is that: the idea of polyphase resampling is being derived to maybe too complicated, to lose its original meaning in signal processing, and now it becomes a fancy way of 2x2 pooling. Even APS can be used for backpropagation actually, it is just another non-smooth operator adding to some existing non-smooth operators, Relu, maxpooling, so no big trouble. So I am conservative about the motivation to be the polyphase resampling parameterized. The computation added by this layer cannot be neglet, considering the shiting doubles of workload of whatever layer follows it.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept