DCT-DiffStride: Differentiable Strides with Real-Valued Data
Authors: Clayton Harper
Mitchell Thornton
Eric Larson
Venue: Submitted to ICLR 2023 (2023)
Keywords: strides
decimation
deep learning
discrete cosine transform

Abstract: Reducing the size of intermediate feature maps within various neural network architectures is critical for generalization performance, and memory and computational complexity. Until recently, most methods required downsampling rates (i.e., decimation) to be predefined and static during training, with optimal downsampling rates requiring a vast hyper-parameter search. Recent work has proposed a novel and differentiable method for learning strides named DiffStride which uses the discrete Fourier transform (DFT) to learn strides for decimation. However, in many cases the DFT does not capture signal properties as efficiently as the discrete cosine transform (DCT). Therefore, we propose an alternative method for learning decimation strides,  DCT-DiffStride, as well as new regularization methods to reduce model complexity.  Our work employs the DCT and its inverse as a low-pass filter in the frequency domain to reduce feature map dimensionality.  Leveraging the well-known energy compaction properties of the DCT for natural signals, we evaluate DCT-DiffStride with its competitors on image and audio datasets demonstrating a favorable tradeoff in model performance and model complexity compared to competing methods.  Additionally, we show DCT-DiffStride and DiffStride can be applied to data outside the natural signal domain, increasing the general applications of such methods.

Under review as a conference paper at ICLR 2023
DCT-DIEESTRIDE: DiFFERENTIABLE STRIDES WITH REAL-VALUED DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Reducing the size of intermediate feature maps within various neural network architectures is critical for generalization performance, and memory and compu- tational complexity. Until recently, most methods required downsampling rates (i.e., decimation) be predefined and static during training, with optimal down- sampling rates requiring vast hyper- hyper-paramenn parameter search. Recent work has proposed novel and differentiable method for learning strides named DiffSttide which uses the discrete Fourier transform (DFT) to learn strides for decimation. However, in many cases the DFT does not capture signal properties as efficiently as the dis- crete cosine transform (DCT). Therefore, we propose an alternative method for learning decimation strides, Deet-iifStrrdd, as well as new regularization meth- ods to reduce model complexity. Our work employs the DCT and its inverse a low-pass filter in the frequency domain to reduce feature map dimensionality. Leveraging the well-known energy compaction properties of the DCT for natu- ral signals, we evaluate DCT-DifiStride with its competitors on image and audio datasets demonstrating favorable tradeoff in model performance and model com- plexity compared to competing methods. Additionally, we show DCT-DiffStride and DiffStride can be applied data outside the natural signal domain, increasing the general applications of such methods.
INTRODUCTION
Dimensionality reduction is necessary computation nearly all modern data and signal analy- sis. In convolutional networks, this reduction is often achieved through strided convolution that decimates a high-resolution input (e.g., images) into a lower-resolution space. Prediction in this lower-resolution space can enable the network to focus relevant classification features while ig- noring irrelevant or redundant features. This reduction is vital to the information distillation pipeline in neural networks, where data is transformed to be increasingly sparse and abstract (Chollet 1221)) On image data, decimation comes in the form of reducing spatial dimensions (e.g., width and height). On time-series data, decimation is performed along the temporal axis.
Decimation can be performed in a variety of ways including statistical aggregation, pooling methods (e.g., max and average pooling), and strided convolutions. By reducing the size of intermediate fea- ture maps, computational and memory complexity of the architecture are reduced. Fewer operations, such multiplies and accumulates (MACs), are needed produce feature maps there are there are fewer values for the network to operate on and store. In convolutional neural networks (CNNs), decimation allows for an increase in the receptive field as subsequent kernels have access to the downsampled values that span several frames. Decimation also allows for increased generalization performance because lower-resolution feature maps allow the network to resilient to redundancy and ignore spurious activations and outliers that would otherwise not be filtered out.
Most decimation methods, however, still require massive hyper-paranen mal window sizes and strides which to operate. This puts the onus on post-processing and cross- validation to find optimal values which may not be time or computationally feasible. To address this concern Riad et al. (2022) proposed DiffStride, which learns cropping mask in the Fourier domain enabling a differentiable method to find optimal downsampling rates. Thus the decimation is learned through gradient-basee based methods, rather than requiring massive hyper- param--r searches.
searches to find the opti-

---

Under review as a conference paper at ICLR 2023
Importantly, (Riad et al. 2022) utilized the discrete Fourier transform (DFT), learning cropping size in the Fourier frequency domain. However, in many applications the DFT can often be outperformed by the discrete cosine transform (DCT) (Ahmed et al. 1974) due to differing periodicity assumptions and better energy compaction. We argue the use of the DCT has several advantages over the DFT, particularly for real-valued data and many natural signals.
We introduce DCT-DiffStrde, which leverages the advantages of the DCT to learn decimation rates in CNN. Our contributions in this work are summarized follows: first, we leverage the improved energy compaction properties of the DCT over the DFT, enabling smaller feature maps in CNNs without substantial loss in model performance. Second, we examine the tradeoff in model com- plexity and model performance for both DiffStride and DCT-DiffStride across range of datasets including audio, images, and communications signals. In all tested situations, we conclude that DCT-DiffStride is superior or comparable to DiffStride. Third, we show that these methods can be applied outside the natural signal domain even though the motivation behind using the DCT/DFT as dimensionality reduction techniques is founded in natural signals (the use of low-pass filters), increasing the potential applications of such methods.
While better performance at lower model complexities is useful in many cases, it is also important that energy components are highly concentrated in the lower frequencies for the implementations of DCT-DiffStride and DifiStride. Because the learned strides are cutoff frequencies creating low-pass filter, we hypothesize that there is less energy being cropped from the signal in the higher frequencies and thus lower information loss. A property of many signals is that energy is not spread uniformly throughout the spectrum but is highly concentrated in the lower frequeecies.. For many data sources, including natural signals, this property could make the DCT preferable to the DFT.
2 FREQUENCY-BASER ANALYSIS
Naturally occurring signals are often defined as signals that humans have evolved to recognize like speech (Singh & Theunissen (003) and natural landscapes (Torralba & Oliva 2003 Ruder- man 1994). These natural signals typically have more low frequency content than high frequency content that is, signals are typically smooth and change gradually. Taking advantage of this ob- servation, Rippel et al. (015) implement low-pass filters in the Fourier domain enabling fractional decimation rates, which helps improve the pooling operation and ensure resolution not reduced too early in the network. This allows for most of the content of the signal to remain present in subsequent layers while still providing a method to reduce dimensionality.
Building on the idea of utilizing Fourier domain learning, Pratt et al. (2017) introduced Fourier or Spectral Convolutional Networks, a method for optimizing filter weights using the DFT. This approach optimized convolutional filters in the Fourier domain without conversion to spatial rep- resentations. However, without an efficient decimation scheme, these networks exploded in the number of trainable parameters. [Lin er al.] (2019) leveraged the Fourier Convolution Theorem to increase the computational efficiency of pre-trained CNNs, adapting convolutional operations to use the DFT. Chi al. (2020) expanded this idea cross scale DFTs that employ Fourier units in multiple branches.
Wood & Larson (202] used learned functions in the Fourier domain to filter signals dynamically. The parametric spectral functions often worked to preserve low frequency content, although spe- cific cropping mechanism was not employed; therefore, reduction in computational complexity was not investigated. Due to the concentration of information in low frequency content, Rippel er al.) (2013) introduced spectral pooling enabling fractional decimation rates and mitigates information loss compared to other pooling methods like max-pooling. A fractional decimation methodology was similarly proposed using Winograd algorithms for acceleration (Pan & Chen [202T). These important works led to the innovations in DiffStrrde Riad et al. (2022) and our proposed work for DCT-DifIStride.
In signal analysis and signal classification, it is often desirable to use linear transformation that tends to compact a large fraction of a signal's energy into just a few transform (or "spectral") co- efficients. Let us first define an N-dimensional real-valued discrete signal components of the vector, x E R.W. The optimal línear transformation matrix, T, for energy compaction is comprised of column vectors that are the eigenvectors of the covariance matrix of with itself, which is the
2

---

Under review as a conference paper at ICLR 2023
autocorrelation matrix, R [x }x This result has been known since at least the 1933 publication by Hotelling (1933) and T is commonly referred to as the Karhumen-Loève Transform (KLT) (Loeve 1945| Karhunen 1946).
In terms of comprising independent energy components, the DFT is often used due to the Wiener- Khinchine theorem (Ziemer & Trantor 1985) that states the power spectral density of a physical signal is equivalent to the Fourier transform of the signal's 's autocorrelation function. The DFT is optimal whenever is comprised of periodic signals. Thus, the DFT is equivalent the KLT for a periodic signal because it is the optimal transformation matrix with respect to energy compaction (Pearl 1973). As signal loses its periodic structure, the DFT becomes less optimal in terms of energy compaction because additional basis vectors have significant energies when representing x. This introduces limitations on the rate of decimation for non-periodic signals. Additionally, the spectral coefficients comprising the DFT transform are complex, even when the transformed signal x is real-valued. These properties of the DFT can be problematic when incorporated with CNNs because they can increase computational complexity and memory footprints. In an effort to further reduce computational complexity, there is interest in the use of linear transformations that maximize energy compaction yet yield real-valued spectra.
Naturally occurring signals are often non- non-periodic signals and are more amenable to being estimated stationary first-order Markov processes, or, more generally, uuto-regressive models of order one, AR(1). For this reason, we propose the use of the discrete cosine transform (DCT) over the DFT for efficient energy compaction in natural signals. In fact, the commonly used compression algorithm 1991> also makes use of the DCT for this reason.
JPEG (Wallace
The DCT uses real-valued cosine functions mapping an input sequence from RV RV For an AR(1) process, the DCT is the optimal KLT basis (Unser 1984 Torun & Akansu 2013) (also see appendix). This enables the DCT to express AR(1) processes in fewer components than the DFT. In turn, this increases the energy compaction properties of the transform, allowing for a higher decimation rate while preserving similar amount of signal content.
Because the DCT (and DFT) operate on discrete signals, there must be definition for the boundary conditions for both the left and right boundaries of the repeated sequence along with the point at which the function is defined as even. This gives rise to various definitions of the DCT total of 8 In this work, we refer to the orthonormal DCT type-Il as "the DCT." Boundary conditions (implied periodicity) for the DCT-II are even at n 2 and = N and can be seen in Figure | The single dimension orthonormal DCT-II is given by Equation I and is straightforward to extend to multiple dimensions:
N
X
Fa
COS
a N
n
+
k
for == 0, N
(1)
2
n=0
3
Similarly the architectures in (Rippel al. 2015 Riad al. 2022 DCT-DiffStride, depicted in Figure utilizes a cropping mask on the rreueencydooaal representation of the input signal. This then followed by the inverse transform on the cropped frequency signal transform back into the spatial domains. For example, an input image x E x is transformed by the DCT resulting in X Hx X is then cropped which is equivalent decimation. Thus by adjusting the size of the crop, the stride of the filtering operation is also adjusted. Because the cropping operation itself is not differentiable, a stop gradient operator (Bengio et al. (003)) is used so that the DCT-DiffStrice layer is still differentiable with respect to the learnable decimation rates, S.
The output of the DCT not necessarily symmetric, unlike the DFT for a real-valued input. Con- sequently, the learned cropping mask does not need to be symmetric to reconstruct the signal. To produce the windowed mask, we use similar implementation to (Riad et al.) (022)) based on the adaptive attention span proposed by Sukhbaatar et al. (2019); however, we account for the differ- ences in symmetry. DCT-DifiStride is able to produce a smaller minimum sized feature map than DiffStride because this symmetry does not need to be kept. The minimum output shape for DCT- DiffStride layer is given by + R and is 2 + 2R for DiffStride where R is defined smoothing factor for the cropping mask. Although this result may be inconsequential for high dimensional x,
3

---

Under review as a conference paper at ICLR 2023
H
DCT Trarsfam
Elementwis Murtplicaioon
DCT Transform
Outsut
Crop
Grocient
77
Periodicity implied DCT-II (DCT-DifStridde
Stop
Periodicity impled by DFT (DiffStride)
Window
34
Learned Decimation Rates (S)
Figure 1: An overview of DCT-DiffStride greyscale image from ImageNet (Deng al.
2009
the percentage of compression for lower-dimensional signals may be largely impacted. For example, if we use smoothness of / 4 with E R in many image datasets, DiffStrrie can compress to approximately 30% whereas DCT-DiffStride can compress to approximately 15% the original size for single layer.
3.1 REGULARIZATION
Riad al. (0022) proposed a novel regularization method to promote better usage of time and mem- ory complexity given by EL_ I Si There are few drawbacks of this method. First, the model could learn to increase the striding factor therefore decreasing the regularization loss, but reducing the feature map size. This is because of the max and min operations that keep the feature dimensionality from collapsing and retaining the smoothness factor. In other words, strides can be unconstrained and have the network decrease loss without any functional value. Second, although the regularization loss is proportional model complexity, there not direct interpretation of the regularization value since the strides are unconstrained. While increasing the weight of the regular- ization term gives the network incentive reduce memory and time complexity, the regularization term lacks semantic value. We propose novel regularization term that alleviates these concerns by expressing model complexity as percentage.
reduces the feature map size along the spatial/temporal axes; however, it does not change the number of channels or the number of parameters in neighboring layers. As such, we define the model complexity, C, as function of the spatial/temporal feature map dimensionality defined as = Σト I1 Er.t L is the number of convolutional layers, D is the dimensionality of the spatial/temporal axes, and E is the expanse (e.g., height or width for an image). maximum possible expanse (i.e., decimation is performed in the network). We note that this does not include complexity of fully connected layers as they are also kept constant in the network.
M,,a
the
The output shape of each DCT-DiffStride layer is function of the learned stride for each particu- lar dimension, the employed smoothness factor, and the minimum allowed feature map size. The minimum feature map size for DCT- is because the output of the DCT is not symmetric. For DiffStride, the minimum size is to account for the symmetry in the DFT for real-valued, even- length signals. The output shape for single dimension of DCT-DiffStride layer is represented the learned stride, and R is the smoothness factor. The output shape calculation for DiffStrrde is simi- EL. lar, but symmetry for real-valued inputs is taken into account giving mmxx Si.d 2)] + 2R. In our regularization term, normalize each output shape by the maximum shape that the output could
Ei, 1)] + where is the expanse, represents the layer, the dimension, as max( S
4

---

Under review as a conference paper at ICLR 2023
be, . (i.e., no decimation is performed). This allows each regularization term to be interpreted as the percentage of total complexity for given layer and dimension on normalized range [O, 1].
We propose novel regularization method in this work with two variants that alleviate issues of interpretability and introduce implicit constraints keeping the loss bounded. In practice, there may be computational or memory limitations on the device that the model will be deployed. By specifying the maximum desired percentage model complexity é, Equation (2) can be used to incentivize the network to not exceed the desired complexity while still allowing the network to learn the optimal rate of decimation for each layer. L is the total number of DCT-DiffStride layers. This allows the network to learn a less complex model than è (i.e., learn larger decimation rates) and optimize other loss terms such as cross entropy. In traditional decimation methods, the size of each layer would be designed apriori to keep the model within é complexity, but the proposed method allows the network to learn smaller model while determining which layers should have higher/lower decimation rates through gradient descent. This method can also be generalized specify percentage complexity to specific layers if overall model complexity is not sufficient.
Lmax complexity max (C é L, 0.0)
While Equation (2) is likely the method to employ in real-world situations, this work aims to il- lustrate an advantageous tradeoff in model complexity and model performance for the use of DCT- DiffStride against DiffStride. In order fairly compare the methods, each method needs to be evaluated with similar model complexity values. We propose a slight modification to Equation (2) such that the models will have very similar complexity values seen in Equation (3). Here, the net- work is penalized for having larger or smaller overall complexity than é rather than just creating an upper bound seen in Equation (2). It is notable that the network is still able learn decimation rates for each DCT-DiffStride layer, but the overall model complexity needs to match value close to é This purely to illustrate the tradeoff in model complexity and model performance.
Lug complexity = =C - غ L)²
(3)
For the remainder of this work, we use Equation to ensure a fair comparison between DCT- DiffSttrie and DiffSttrde. The overall loss function for all experiments in this work is L Laoss entropy + >Ls complexity. We set À = 100 for all experiments to make the regularization the dominant term in the overall loss calculation. This helps to ensure both DCT-DiffStride and Diff- stride have a similar model complexity for ease of comparison.
4 EXPERIMENTS
We compare DCT-DiffStride and Diff'Stride on five classification tasks. We include both image and audio datasets that include naturally occurring signals, as well as a modulation classification dataset representing non-natural signals. Providing wide variety of dataset types allows exploration into advantages that DCT-DiffStride and DiffStride may possess as well as expanding the the general number of applications for both methods. To ensure stability during training, all methods utilize unit no-morm global gradient clipping. In this work, we apply DCT-DiffStride between convolutional layers, but it can be applied between other architectural layers (e.g., transformer layers) with minimal modification.
While Riad et al. (022) showed that DiffStride is resilient different stride initializations, work aims to illustrate the tradeoff for DCT-DiffStride in terms of model complexity and model performance when compared to DiffStride. All models are trained with categorical cross-entropy loss along with the appropriate regularization penalty for each model complexity, é.
4.1 IMAGE DATA
Experimental Setup, CIEAR We compare OCT-DiffStrrde and DiffSttrde on various image clas- sification tasks including CIFAR1O, CIFARI00, and ImageNet. Each comparison uses the ResNet18 (He et al. (0166) architecture with a residual block using the same definition as (Riad et al. 2022). Each image experiment applies mixup (Zhang al. 2018 with
0.2, which regularizes the
5

---

Under review as a conference paper at ICLR 2023
CNN using convex combinations of training instances. Convolutional layers weight decay of 5e-3, and decimation rates are initialized to = 21.
Due to their smaller size, first benchmark DCT-DiffStride the CIFAR datasets (Krizhevsky er al. 2009) with an increased number of values in order to better characterize the complexity tradeoff. For each CIFAR experiment, we utiliize stochastic gradient descent (SGD) (Saad 1999)
with an initial learning rate of 0.1, momentum of 0.99 (Qian 1999). and a batch size 32. We train the models for total of 80 epochs with learning rate decay factor of 0.1 at epochs 20, 40, and 60
with a final learning rate of le-4. We use é [0.1, 0.2, terms of model complexity and model performance.
0.8] to develop a tradeoff curve in
We use the same model definition as (Riad er al. 2022), and we report the highest performing model from their work using traditional strided convolutions to establish a baseline compare to traditional methods.
0.85
0.00
OCT Sestine orbaraa
OCT.or99ric petennss
Hodel BCT-4-95adde
0.55
Complesity
(h)
(c)
Figure 2: Accuracy over various model complexities with (a) showing CIFARIO, (b) showing CI- 10R results, and (c) showing ImageNet results.
Results, CIFAR Figure 2 shows the results for both CIFAR datasets, showing top-1 accuracy versus the model complexity. As expected, results are similar as CIFAR100 uses the same images as CIFARIO, but with more detailed labeling. As model complexity decreases, DCT-DiffStride out- performs DiffStride. The smaller the complexity, the larger the gap performance indicating that DCT-DifiStride is able to utilize higher decimation rate than DiffStride. For resource- seouurcecoosttrineed devices, this result would enable higher model performance without the need of upgrading memory capacity. Additionally, DCT-DifSStride obtained the best performing model with model complexity of approximately 40%. Because DCT-DiffStride and DiffStride both utilize low-pass filters to re- duce complexity, this result could indicate the DCT needs fewer coefficients in the lower frequencies to represent image signals, compared to the DFT. Because the CIFAR datasets include many exam- ples of naturally occurring images, this supports our hypothesis that the DCT's energy compaction property can better represent the energies in natural images. DCT-DifiStride also outperforms the baseline method with similar model complexities. This indicates that DCT-DiffStride is able to learn more optimal fractional strides at different layers.
Experimental Setup, ImageNet We perform similar experiments on the ImageNet dataset (Deng et al. 2009) containing 1,000 classes. We use the official training split with I 1.28M training images and report results on the 50K validation images (50 images per class). We use the 64 x 64 resized dataset provided by [Chaaaazzzz et al. (2017) to ensure consistent image sizes. On ImageNet, we keep the same training procedure as the CIFAR datasets. but train the models for 45 epochs with an initial learning rate of 0.1 with a learning rate decay factor of 0.1 at 15 epoch intervals. Observing a large gap in performance between DCT-DiffSSride and DiffStride at low model complexities, we use [0.1, 0.2, 0.3) for ImageNet to see if similar result occurs larger dataset well as dataset consisting of larger sized images.
Results, ImageNet A similar result was found ImageNet as the CIFAR datasets. DCT- DiffStride achieved a much higher evaluation accuracy of 48% compared to that of 43.5% using DiffStride at 10% model complexity, and the gap decreases as more intermediate activations retained. This reaffirms that DCT-Diffstride is able to use larger decimation rates than DiffStride larger images, while maintaining strong performance.

---

Under review as a conference paper at ICLR 2023
4.2 AUDIO DATA
Experimental Setup In addition to image data, we evaluate DCT-DiffStride on the speech commands dataset (Warden 2018). Raw audio data is sampled at 16KHz and processed into 64- dimensional log mel-scaled spectrograms using a window size of 25ms and 10ms overlap (Stevens) el. 1937). All spectrograms are mean- neramnnorallized for each mel bin. In order leverage batch traming audio is randomly cropped or padded to 1s in length. We use the adaptive momentum (AdaM) (Kingma & Ba 2015) optimizer with an initial learning rate of 0.1 and a decay factor of 0.1 at epochs 20 and 40 with a batch size of 32 for a total of 60 epochs. To establish baseline to com- pare against traditional methods, we report the accuracy from (Riad et al. 2022) using traditional strided convolutions.
Similarly to (Riad el al. 2022). a 2D-CNN based on (Tagliasacchi et al. 2019) is employed where convolutional blocks consist of three convolutions. First, a (3 x 1) kernel is applied over time followed by a (1 3) kernel over frequency and a residual convolution. These convolutional blocks illustrated in Figure 3a The architecture consists of 5 convolutional blocks each followed by a decimation layer. The architecture then employs global average pooling and single linear layer to perform classification. A decimation layer is also performed the input to the network. The number of channels for each convolutional block are [128, 256, 256, 512, 512] and decimation rates initialized to S [2, 2, 1,2, 2,
The accuracy versus model complexity is shown in Figure 3b While both DCT-DiffStride and DiffStride perform comparably, DiffStride was found have a slight advantage over DCT- DiftStride for the speech commands dataset. Both methods were able to use large decimation rates without loss in performance indicating redundant activation values. Performance was found to de- crease at high model complexity values. Generalization performance may decrease without large amount of decimation possibly due to spurious activations influencing predictions. Both methods outperformed the baseline method at a similar model complexity further demonstrating the ability for the network to learn superior decimation rates.
Results
A possible explanation for the similar performances is that speech is comprised of many near pe- riodic (e.g., vowel sounds and semivowels) and non-periodic sources (e.g., voiceless fricatives and plosives). Thus, no clear advantages in periodicity or boundary conditions exists between the DCT and DFT. Moreover, the energy concentrations in speech are more dispersed among phonemes, which may explain why the energy compaction of the DCT does not provide distinct advantage.
soquroce
DVB Medel
Model bemaiie
tevee AJB
6:12
600
Complesity
Complesity
(a)
(b)
(c)
Figure 3: Parts (a) and (b) illustrate the speech commands experiment with (a) showing a convolu- tional block for audio data where BN stands for batch normalization and (b) showing top-1 accuracy over various model complexities for the speech commands dataset. (c) demonstrates accuracy over various model complexities for the RadioML 2018.01A dataset for signals in [14 14]dB SNR.
4.3 NON-NATURAL SIGNALS
Due to low-pass filtering in Fourier-based decimation methods such as DiffSttrde and spectral pool- ing, these methods are traditionally applied to natural signals as previously explained. However, low-pass frequency- decimation methods can be applied outside the domain of natural signals. For example, even though signals may contain large amount of high frequency content, successive non-linear transformations may be able to project the signal into a latent space that can more easily be represented by low frequency content.
7

---

Under review as a conference paper at ICLR 2023
Experimental Setup To investigate this, evaluate DCT-DiffStride on the RadioML 2018.01A (0 Shea et al, 2018 dataset with the task of modulation scheme classification. There are 24 differ- ent classes with total of 2.56M signals S(T), each represented
2- 2-dimensional vector consisting of in-phase (I) and quadrature (Q) components where S(T) = T(T) + jQ(7). Observations range from 20dB to +30dB signal to noise ratio (SNR) in 2dB increments for total of 26 different signal-to-noise ratio (SNR) values. The dataset is balanced across modulation type and SNR value with a total of 4,096 observations per (modulation type, SNR} pair. The 24 candidate modulation schemes fall under the following representative groups: (1) Amplitude, {OOK, 4/8ASK, AM-SSB- SC, AM-SSB-VE WC, AM-DSB- WC, AM-DSB-SSS )C) (2) Phase, (BPSK, QPSK, 8/16/32PSK, OQPSK) (3) Amplitude and Phase, (16/32/64/128APSK 1633//6/1228260QAN and (4) Frequency, {FM, GMSK3.
While there is not an official training split, we follow a similar procedure as in O'SShe et al. 2018
Harper ef al.) 2020; 202T) where 1M observations are randomly selected for training and .56M
observations are chosen for evaluation. Observed in D'SSaa et al. 2018 Harper et al. 2020 2021
classification performance plateaus approximately below - 14dB and above 14dB SNR. In this work, subset the training and evaluation datasets only include signals in the range [-14, 14]dB SNR resulting in approximately 575K training observations and 900K evaluation observations.
We utilize ID CNN convolving over the temporal axis with ID DCT-DiffSttiid and DiffStride layers. We base our model on (Harper el al.) 2020) utilizing mean and variance pooling after convo- lutional layers, common technique used by x-vector architectures (Snyder er (218)). We train a baseline model which uses an equivalent architecture to the model in (Harper et al. 2020), but trained on signals in the specified dB range. To compare DCT-DiffStride and DiffStride, the dec- imation methods are applied following each convolutional layer. We use the adaptive momentum (AdaM) (Kingma Ba 2015) optimizer with an initial learning rate of 0.01 and a decay factor of 0.1 at epochs 15, 30, and 45 with batch size of 128 for total of 60 epochs. All decimation rates are initialized to one.
Table 1: RadioML 2018.01A results for each modulation category. Best performing models for each model complexity and modulation grouping are shown in bold, To make the table more compact, DCT and DFT represent DCT-DifiStride and DiffStride respectively.
Modulation
è =
0.1
é =
0.2
0.5
0.8
Baseline
Group
DCT
DFT
DCT
DFT
DCT
DFT
DCT
DFT
Frequency
0.75
0.00
0.75
0.67
0.74
0.65
0.65
0.67
0.62
Amplitude
0.57
0.00
0.56
0.45
0.61
0.59
0.57
0.58
0.56
Phase
0.38
0.00
0.40
0.31
0.51
0.40
0.39
0.44
0.36
Amplitude and
0.19
0.11
0.22
0.16
0.26
0.32
0.25
0.28
0.29
Phase
Results Although modulation data is non-natural, Figure 3c and Table [ clearly illustrate that DCT-DifIStride and DiffStride are able classify modulation schemes even at low model com- plexities. Interestingly, both architectures outperformed the baseline architecture that does not use decimation. DCT-DiffStride was found outperform DiftStride when the decimation rate was in- creased (i.e., lower model complexity) conssitent with our image dataset results. DCT-DiffStre produced the best overall performing model with a model complexity of approximately 50%.
It is interesting that DCT-DifiStride and DiffStride are able to perform well with modulation data, particularly at small model complexities, because it well known that modulation data contains a number of high frequency components that are important for classification. One possible explanation for this relates to the concept of neural collapse (Papyan et al. 2020 Han et al. 2022 Zhang et al. 2021 Belkin et al. 2019), which has shown that models trained with categorical cross-entropy tend to produce activations very close to the mean class value in later layers. Since in spectral analysis the zeroth entry in the DFT or DCT output is equivalent to the mean value of the signal, it may be that DCT-DiffStrial and DiffStride can leverage neural collapse. More specifically, as networks trained the within-class variability decreases under neural collapse which, in turn, decreases the
8

---

Under review as a conference paper at ICLR 2023
high frequency content in later layer activations and increases low frequency content. Because DCT- DiffStride and DiftStride operate as low-pass filters, they may able to exploit this phenomenon.
DECIMATION RATE AND AR(1) RELATIONSHIP
In this section, we investigate the relationship between how well AR(1) processes the given datasets and intermediate activations along with learned decimation values. Because the DCT is the optimal KLT basis for an AR(1) process (see appendix), we would expect the trained networks to have larger decimation rates when the intermediate activations for given layer better modeled by an AR(1) process for DCT-DiffStride. To investigate this hypothesis, we take 100 random sam- ples from each dataset, compute the activations for each layer for the 10% complexity model, fit an AR(1) process for each channel, and store the absolute value of the residuals between the activations and the AR(I) predicted activations for each sample and layer. We the statsmodels package (Seabold & Perktold, 2010) to perform the AR(1) fitting. For 2-D data, the appendix describes the unraveling process to unpack the data into 1-D series that can be modeled by an AR(1) process. The distribution of absolute residuals can be seen in the box plots and the feature map shape fol- lowing decimation for the given layer are shown in the dotted lines in Figure Feature map shape is defined as height width for 2-D data and length for I-D data (RadioML 2018.01A dataset). For the speech commands dataset, decimation performed directly on the input whereas decimation is not performed until after convolutional layers in the remaining datasets. For this reason, the residual box plots for the input and the first layer are the same for the speech commands dataset.
1000
1000
4000
000
2000
400
400
Absolute Resicual
200
Featura Map Shape
000
DFT Absolute Residual
DFT Feature Map Shape
300
DCT Absolute Residual
DCT Feature Map Shape
200
Decimation Stage
Figure 4: Absolute residuals (box plots) and learned decimation shapes for each layer. The top row, left to right: CIFAR10, CIFAR100, and ImageNet. The second row, left to right: speech commands and RadioMI. 2018.01A.
A consistent trend was seen across the datasets for DCT-DiffStride, When the residuals were small for successive layers, the feature map was more aggressively decimated. When the residuals creased, decimation rate stabilized. This supports the hypothesis that DCT-DiffStride leverages the DCT for larger decimation rates when the data better fits an AR(I) process. This behavior provides further evidence to why DCT-DiffStridd outperforms DiffStride, particularly for low-complexity models.
CONCLUSION
We propose DCT-DiffStride, a method to perform differentiable decimation leveraging the energy- compaction properties of the discrete cosine transform. DCT-DiffStride is able outperform com- petitors on various classification tasks, particularly when large decimation rates are used. While low-pass filter methods traditionally applied to natural signals. show that DCT-DiffStride generalizes outside this domain and can be used as a direct replacement to other decimation meth- ods, such DiffStride and max pooling, in more general applications.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Nasir Ahmed, T Natarajan, and Kamisetty R Rao. Discrete cosine transform. IEEE transactions on Computers, 100(1))90-93, 1974.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical variance trade-off. Proceedings of the National Academy of Sciences, 116(32):1849-15588 2019.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint
2013.
Lu Chi, Borui Jiang, and Yadong Mu. Fast Fourier convolution. Advances in Neural Information Processing Systems, 33:4479 4488, 2020.
Francois Chollet. Deep learning with Python. Simon and Schuster, 2021.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv:1707.08819 2017.
James Cooley and John Tukey, An algorithm for the machine calculation complex fourier series. Mathematics of computation, 19990))277-301, 1965.
CM Da Fonseca. On the cigenvalues of some tridiagonal matrices. Journal of Computational and Applied Mathematics, 400(1):283-25) 226, 2007.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Keinosuke Fukunaga. Statistical pattern recognition. In Handbook of pattern recognition and com- puter vision, pp. 33-60. World Scientific, 1993.
X. Y. Han, Vardan Papyan, and David L. Donoho. Neural collapse under MSE loss: Proximity to and dynamics the central path. In International Conference Learning Reprexentations, 2022. URL httlt:ttp//ooootm //tp:/e:e////eeeeeieeeeee net worum??d==wwddviiiii R3
Clayton A. Harper, Lauren Lyons, Mitchell A. Thortoo,, and Eric C. Larson. Enhanced automatic modulation classification using deep convolutional latent space pooling. In 2020 54th Asilomar Conference on Signals, Systems, and Computers, 2020.
Clayton A. Harper, Avi Sinha, Mitchell A. Thornton, and Eric C. Larson. SNR-boosted automatic modulation classification. 2021 55th Asilomar Conference on Signals, Systems, and Comput- ers, pp. 372-375, IEEE, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Harold Hotelling. Analysis of complex of statistical variables into principal components. Journal of educational psychology, 24(6):417, 1933.
Kari Karhunen. Zur spektraltheorie stochastischer prozesse. Ann. Acad. Sci. Fennicae, AI, 34, 1946.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015. URRlattp://aaxii orq/ abs////s//11 6980
Alex Krizhevsky, Geoffrey Hinton, er al. Learning multiple layers of features from tiny images. 2009.
CD Lai. First order autoregressive markov processes. Stochastic processes and their applications, 7(1):65-72, 1978.
Jinhua Lin, Lin Ma, and Yu Yao. A Fourier domain acceleration framework for convolutional neural networks. Neurocomputing 364:254 266,, 2019.
10

---

Under review as a conference paper at ICLR 2023
Michel Loëve. Sur les fonctions aléatoires stationnaires du second ordre. Revue Scientifique, 83: 297-303, 1945.
Michel Loève. Probability theory. Van Nostrand publishers, 1955.
Ranjan K Mallik. The inverse of tridiagonal matrix. Linear Algebra and its Applications, 325 (1-3):109-139, 2001.
Timothy James O' Shea, Tamoghna Roy, and T. Charles Clancy. Over-the-air deep learning based radio signal classification. IEEE Journal of Selected Topics in Signal Processing, volume 12:1, pp. 168-179. IEEE, 2018.
Junhao Pan and Deming Chen. Accelerate non-unit stride convolutions with Winograd algorithms. In 202/ 26th Asia and South Pacific design automation conference (ASP-DAC), pp. 358-364. IEEE, 2021.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652 24663, 2020.
Judea Pearl. On coding and filtering stationary signals by discrete Fourier transforms (corresp.). IEEE Transactions on Information Theory, 19(2):229-22) 232, 1973.
Harry Pratt, Bryan Williams, Frans Coenen, and Yalin Zheng. FCNN: Fourier convolutional neural networks. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp- 786-798. Springer, 2017.
Ning Qian. On the momentum term in gradient descent learning algorithms. Neural networks, 12 (1):145-151, 1999.
Rachid Riad, Olivier Teboul, David Grangier, and Neil Zeghidour. Learning strides in convolutional neural networks. ICLR, 2022.
Oren Rippel, Jasper Snoek, and Ryan Adams. Spectral representations for convolutional neural networks. Advances in neural information processing systems, 28, 2015.
Daniel L Ruderman. The statistics of natural images. Network: computation in neural systems, (4):517, 1994.
David Saad. On-line learning in neural nefworks. Cambridge University Press, 1999.
Skipper Seabold and Josef Perktold. statsmodels: Econometric and statistical modeling with python. In 9th Python in Science Conference, 2010.
Nandini C Singh and Frédéric E Theunissen. Modulation spectra of natural sounds and ethological theories of auditory processing. The Journal of the Acoustical Society of Americu, 1 114(6):3394- 3411, 2003.
David Snyder, Daniel Garcia-Ronee Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur. X- vectors: Robust DNN embeddings for speaker recognition. In 2018 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pp. 5329-5333 IEEE, 2018.
Stanley Smith Stevens, John Volkmann, and Edwin Broomell Newman. A scale for the measurement of the psychological magnitude pitch. The journal of the acoustical society of america, 8(3):185- 190. 1937.
Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. In Proceedingy of the 57th Annual Meeting of the Association for Compu- tational Linguistics, pp. 331 331--35, Florence, Italy, July 2019, Association for Computational Lin- guistics. doi: 10. 10.18653/v1/P19 1032. URL https/////aaaaaallll.o..ooooo ottp
Marco Tagliasacchi, Beat Gfeller, Félix de Chaumont Quitry, and Dominik Roblek. Self-supervised audio representation learning for mobile devices. arXiv preprint arXiv: /796, 2019.
11

---

Under review as a conference paper at ICLR 2023
Mitchell Aaron Thortoon, Rolf Drechsler, and D Michael Miller. Spectral techniques in VLSI CAD. Springer Science & Business Media, 2012.
Antonio Torralba and Aude Oliva. Statisties of natural image categories. Network: computation in neural systems, 14(3):391, 2003.
Mustafa U Torun and Ali N Akansu. An efficient method derive explicit KLT kernel for first- order autoregressive discrete process. IEEE transactions signal processing, 31154)399443395, 2013.
Michael Unser. On the approximation of the discrete Karhunen-Loove transform for stationary processes. Signal Processing, 7(3):231-249, 1984.
Gregory K Wallace. The JPEG still picture compression standard. Communications of the ACM, 34 (4):30 44, 1991.
Pete Warden. Speech commands: dataset for limited-vocabulary speech recognition. arXiv preprint 11X1v:1804.032000 2018.
Luke Wood and Eric C Larson. Parametric spectral filters for fast converging, scalable convolutional neural networks. In /CASSP 2021-2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 2800-2804. IEEE, 2021.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond em- pirical risk minimization. In International Conference on Learning Representations, 2018. URL
https::/ net/ Forummid-r1ddyp 1twp://vwie.ott/ooooiii------:1
R. F. Ziemer and W. H. Trantor. Principles of communications systems, modulation and noise, 1985.
A APPENDIX
A.I AR(1) 2-DIMENSIONAL DATA
In order to fit 2-D data to AR(I) process, must have method to transform the 2-D data 1- series. However, the method chosen must not introduce edge effects that inhibit the ability for an AR(1) process to fit the data. For example, neighboring pixels in image typically correlated with one another. Unpacking the 2-D data by flattening the rows where the end of the preceding row is adjacent to the start of the following very far from one another and be less likely to be correlated and an AR(1) process. Instead, we unpack 2-D data by traversing the ensuring unpacked data points remain close to points from the original 2-D structure. This procedure can be seen in Figure
could introduce edge effects the pixels could be
Feequnccyhaigt
Figure 5: Overview of unpacking 2-D sample into 1-D series in order to fit an AR(1) process.
12

---

Under review as a conference paper at ICLR 2023
A.2 LINEAR TRANSFORMATION TO MAXIMIZE ENERGY COMPACTION IN INDEPENDENT SPECTRAL COEFFICIENTS: THE KLT TRANSFORM
In our work, we make use of the DCT leverage its energy-compaction properties enable learn- able decimation rates. In this section, we provide the mathematical foundation explaining the DCT's ability represent AR(1) processes with fewer terms than the DFT enabling larger decimation rates without compromising classification performance.
desirable to use transformation that tends compact large fraction of a signal's energy into just a few transform (or "spectral") coefficients. This characteristic is coupled with another desirable feature wherein maximum decorrelation is present among the transform coefficients, that is, the coefficients contain independent portions of the signal's energy components. Let us first define an N-sampled real-valued discrete signal as components of the vector, x. The components of are I&k R where 0 S k S N 1, or succinetly, x E RN. For simplicity while attempting to be as general as possible, we assume that x is of the form
normalized aero-mean random vector thus [ 0 and the zy. variates of a random variable, X, with zero-mean.
The covariance matrix of x with itself is its N x N autocorrelation matrix, Rx = Coofxx] = Exxx"). Note that the total energy of the signal, x, assumed to be |x||² is contained within the coefficients matrix E[xx") = Rx- The component of Rx at row and column j is denoted as R2 (k, j) and encodes the correlation between the signal components Ta and zz comprising the energy term | Moreover, since x E TR,, Rx Rx
Observation 1: Rx is a diagonal matrix.
If components zy and zj are uncorrelated, then R. (k. of the autocorrelation matrix, it takes the form of a diagonal matrix with positive diagonal values equal to |sa/?. Because the signal x of a form wherein the energy of each component, Tk, proportional to |ar|| the total energy of the signal is then the inner product of x with itself, or its squared L2-norm, ||x||?:
for all k * j. Due to the definition
|||| E(x x]
Σ k=
R₂ (k.
We denote the complex conjugate transpose of matrix A as A Likewise, a conjugate transpose of column vector the row vector v*. If the vector is real-valued, as is the case for signal vector x, then V* vT. Let us apply a linear transformation, T, to x as y = Tx where T is an orthogonal transformation matrix. Since T is orthogonal, energy components in the spectral coefficients comprising y are independent. The physical principal of conservation of energy applies such that |2 = ( y ||x|
We also note that the covariance of y with itself is its autocorrelation matrix, Ry, as given in Equa- tion
Ry Cou(yy) Flyy") = F[Tx(TX))) - -TEx TR T
(5)
Equation (5) indicates that if is desirable to compact energy in the fewest M N components of the transformation vector y, then the transformation matrix T should be structured such that the en- ergy of y is contained within the first M coefficients and all y/s = 0 for M < < N 1. We define the M Mddimensionaa vector yM to the the first M coeffients of y Tx. If a suitable transormation matrix T is chosen such that maximum energy compaction results, then from Equation (4), we have Equation (6)-
M-1
||x|| = ||yM|| 12
1ual = Elym'Tyyya = (Tx) = T
(6)
k=0
Because Ym = [yo 81 VM-1 0
0]
x T and Ym = Tx, we see that y = Tx = Ym
yet. the N M row vectors of T are irrelevant since the ym coefficients, yi 0 4 k> M when
13

---

Under review as a conference paper at ICLR 2023
the transformation y Tx is calculated. Thus. replace the transformation matrix T with that of TM where TM has the same row vectors, t. as matrix T for all & M and has zero-valued row vectors, t 0", for all M < & < N 1,, where denotes the null vector. This implies that maximum energy compaction occurs when we rewrite the rightmost term of Equation 6 as given in Equation
ti
E (x"TTTx) = (x =
x² too ti tM
]
M
x
0
0
From Equation 4 know that =|x||== Σ (k, and from Equation (6) that ||x| Ml Txx]: therefore, only the diagonal values of TT result in non-zero coefficients. Alternatively, since T by definition an orthogonal transform, the inner product of any two column vectors, denoted as (tg, [j) 0 for all & # j. Thus, can rewrite Equation (7) as
M
E Bxx'T = E x"
tk
x
(8)
k==
Using matrix algebra identities, we re-arrange the terms of Equation
resulting in Equation (
-
E(x E(x" T
1xxx [x)
E(x"
t Rxts
Equating |x| lym 1| F2 T F
to
Rxtx from Equations and (8)
and ||x| = Exx x} = Ek= R2 (k. k) from Equation
we obtain Equation 10
M-1 E(x"
N-1
Σ 1xx = Σ (k,
(10)
k=0
Because the orthogonal eigenvectors, L, serve as column vectors of the transformation matrix, and Rx is diagonal matrix, the spectral decomposition theorem applies Equation (10) indicat- ing that the expression T RxT satisfies the eigendecomposition where the Ro k) are the 1tth eigenvalue, dg. for the cigenvector t; as given in Equation [II)
(k Autt
(11)
Therefore, the optimal transformation matrix in terms of compacting maximal signal energy compo- nents for a signal, x. in the fewest spectral coefficients such that all components contain independent, or decorrelated, energy values is in fact the covariance (or autocorrelation) matrix of x defined as Rx (xv(x) ]. Therefore,
Rx and is an orthogonal transform.
The particular transformation matrix, T = Rxx that maximizes energy compaction within as few independent or decorrelated spectral coefficients as possible is not a new result and was formulated by Harold Hotelling in 1933 (Hotelling 1933) for the purposes of obtaining Principal Components in a sequence values for statistical analyses and thus sometimes referred to as the "Hotelling transform." or, in more more contemporary times, as "Principal Components Analysis" (PCA). In later publications by Kari Karhunen and Michel Loève, this result was again simultaneously ob- Karhunen 1946) series expansion
tained and published by both researchers (Loève
1945
1955
14

---

Under review as a conference paper at ICLR 2023
method for representing continuous random processes, thus, T Rx is sometimes referred as the "Karhunen-Loose transform" or the "KLT."
In the contemporary of data analytics, the of PCA to obtain T Rx is often applied to data sequences represented as x to estimate the data vector reduced dimensional space as was originally proposed by Hotelling. Modern PCA is typically used to determine reduced set of dominant basis vectors whose linear combinations closely approximate the data vector x thus allowing it, at least close approximation it, be processed in lower dimensional space as described in (Fukunaga 1993).
A.3 PRACTICAL TRANSFORMS TO MAXIMIZE ENERGY COMPACTION WITH INDEPENDENT SPECTRAL COFFFICIENTS
As is apparent from the previous derivations, the property of requiring the spectral coefficients to be independent, our case, of representing decorrelated signal energy components, implies that the desired transformation be orthogonal as is the case of the KLT. Unfortunately, computation of the KLT transformation matrix, the autocorrelation of the signal x, is computationally intense. In general, an N-dimensional vector x, requires N2 scalar multiplications and N2 sealar additions of two operands. Other classes of orthogonal transformations are independent with regard to properties of the signal and comprise fixed transformation matrix.
The use of a fixed transformation matrix is desirable since it may be used repeatedly and independent of the particular structure of the signal. Furthermore, some classes of these transformation matrices have desirable structural properties such as being represented as factors of sparse matrices enabling efficient algorithms to be devised that greatly reduce the number of arithmetic operations required for the computation of the spectral or transformation vector. In particular, the discrete Fourier trans- form (DFT) has both the property of transformation matrix with fixed structure, and of being decomposable into a set of sparse matrix factors. This observation has led to the development of the so-called "Fast Fourier Transform" (FFT) (Cooley & Tukey (965) that is an efficient implementa- tion of the DFT. Since the publication of the FFT algorithm, which some attribute to work as early as that Carl Gauss, many alternative efficient algorithms have been developed. summary of more modern and effect spectral transformation methods and algorithms is described in (Thornton et al. 2012).
A.4 KLT FOR REAL-VALUED PERIODIC PROCESS: THE DISCRETE FOURIER TRANSFORM
In terms of comprising independent energy components, the DFT is often used due to the Wiener- Khinchine theorem that states the power spectral density of a physical signal is equivalent to the Fourier transform of the signal'ss autocorrelation function (Ziemer & Trantor 1985). For these rea- sons, the DFT or FFT is often used to compute energy components of signal.
The DFT optimal whenever the x vector is comprised of periodic signal. That is, Fg. = Fk++ for all values of k. This can be shown to be the case since the autocorrelation matrix, Rx, has circulant structure.
TO Ti
TN-1
FN-1
To
TN-2
Rx E [xxx
(12)
F1 72
To
The eigenvectors of Rx as given in Equation 11 can determined and are found to consist of the Fourier basis vectors, w used to construct the DFT transformation matrix as shown in Equa- tion 13).
R2 (k, k) w -Aw
W
(13)
Thus, the DFT is special case of the KLT for strictly periodic signals. As signal loses its pe- riodic structure, the DFT becomes less optimal in terms of energy compaction, but retains its abil- ity to preserve independence de-correlated energy components in the spectral vector due to the
15

---

Under review as a conference paper at ICLR 2023
Khinch-K theorem. For this reason, the DFT is often the first choice for determining energy components of a signal due to the widespread popularity efficient, or "FFT," algorithms and the fact that many signals in communications systems are in the form of modulated carrier waves and thus have a good degree periodicity.
The spectral coefficients comprising the DFT transform complex, when the transformed signal x is real-valued. In an effort to further reduce computational complexity, there interest in the use of linear transformations that maximize energy compaction yet yield real-valued spectra.
A.5 KLT FOR AR(1) PROCESS: THE DISCRETE COSINE TRANSFORM
If the signal is comprised of components, 2i, that related as Iı p0 + 2k, where 2n a variate (i.e., the kdth outcome) of the Gaussian distributed random variable Z with zero-mean, Pzz 0, and variance o2 and the constant correlation coefficient, P. satisfies |p| In this case, x is a stationary first-order Markov process, or, more generally and autoregressive model of order one, AR(1).
The autocorrelation matrix for x is Rx = Cov (x) = E (xx") and is given Equation 14 as follows from Lai 1978).
0²
0
0
pl
N-2
0
N-3
Rx Oz
0
0
(14)
N-2
N
We wish to compute the eigenvectors of Rx in Equation 14) and make use of the fact that the eigenvectors of Rx and 3²R²² are equivalent wherein the scalar constant 3² is defined as:
p²
8
(15)
+ p2
We also define the scalar constant, or in Equation (16) as:
(16)
+ p2
The form 3²R," is determined as follows from Mallik 2001) and is that of the tridiagonal matrix as shown in Equation (17).
po
C
p
Ct
Ct
p
3²R 1
x
0%
0000
000
0
1 pa
(17)
When the correlation coefficient is close to unity,p ss 1, and the variance is normalized to unity, 22 1, the matrix R² can be simplified to that of Equation (18).
16

---

Under review as a conference paper at ICLR 2023
a
a
0
0
(18)
1
:
000000
1
1
0
The eigenvectors of To are determined per Da Fonseca (2007). leading to the result:
Tates Astek
(19)
Explicitly, these eigenvectors are of the form as provided in Equation (00
+2m=+1 1R
cos ((N (() 1) ] vk = 1,2, 1m;=) ... 1)
V2
008
2N
(20)
The coefficients of t are recognized as the components the "discrete cosine transform" (DCT)
matrix, Te defined in (Ahmed et al. 1974). It is noted that the coefficients of Tr related to the DFT coefficients in Equation 113 as Re [ }. As previously mentioned, the fact that the
DCT yields real-valued spectrum is advantageous from computational complexity viewpoint as compared to the complex-valued spectra arising from the DFT.
17


Reviews:
[{"review_id": "fqoEcvXAxQ", "reviewer": "Reviewer_LwJS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work extends the baseline model DiffStride by replacing the discrete Fourier transform (DFT) with the discrete cosine transform (DCT), which leverages the energy compaction properties of DCT. Experiments are conducted on natural signals non-natural signals. The results in image classification and modulation classification demonstrate the advantageous tradeoff in model complexity and model performance for DCT-DiffStride against DiffStride. In speech classification, at low model complexity, these two methods are comparable\n", "summary_of_the_review": "Overall, the idea of this paper makes sense to me, but I think the experiments results and the novelty of this paper may be marginally below the acceptance.", "strength_and_weaknesses": "1. This paper is well-written and easy to understand. The structure is very clear.\n\n2. The architectures of this work are similar to DiffStride. The main innovation is the use of DCT. Knowledge in signal process is utilized. The energy compaction properties of DCT are well-known and widely used in lossy compression. \n\n3. In Figure 2, DCT-DiffStride achieves the best performing model at 40% model complexity. However, for CIFAR-100, the baseline DiffStride seems achieve better performance at similar model complexity (around 40%) than DCT-DiffStride. Moreover, for ImageNet dataset, is it possible to provide a comparison figure between DCT-DiffStride and DiffStride as Figure 2?\n\n4. For audio dataset, DCT-DiffStride and DiffStride is comparable. In Figure 3 (b), It seems DiffStride even achieves slightly higher accuracy than DCT-DiffStride at the lowest model complexity. Moreover, why the performance of DiffStride considerably decreases at the highest model complexity (the red point between 0.6 and 0.8)?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Y8g8V3DlRyd", "reviewer": "Reviewer_Kc3t", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a DCT-based approach to learning strides to perform spatial pooling. The approach is an improvement over DFT-diffstride and shows benefits over it in a low-complexity regime.", "summary_of_the_review": "The paper is a straightforward improvement over DFT-diffstride and marginal improvements are shown against it.\n\n## Post-rebuttal\nI acknowledge the authors' response. As noted by other reviewers, the contribution is incremental and not sure if it is sufficient for publication at ICLR. Therefore I'm keeping the original score of marginal-accept.", "strength_and_weaknesses": "## Strengths\n1. DCT is shown to be better to represent signals compactly and the paper shows its benefits compared DFT-diffstride.\n2. Overall the paper is clearly written.\n\n## Weaknesses\n1. The novelty is somewhat limited as it is a straightforward modification to DFT-diffstride. Even though, marginal improvements are shown in practice it is not clear if it is sufficient for a publication.\n2. Experiments simply compare against DFT and it is not clear if this could outperform hand-tuned strides in standard architectures. Furthermore, how difficult is it to optimize for the decimation rates, it seems it involves differentiating through DCT. Please comment on this. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AxNRxU7Swog", "reviewer": "Reviewer_NRJF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors consider the problem of reducing the size of hidden layers in convolutional neural networks (CNN). Size reduction is usually achieved by setting an appropriate stride for the convolutional layers. The stride is a hyperparameter and is usually fine-tuned using cross-validation. An alternative approach is DiffStride (DS, [1]). DS is based on the observation that strided convolutions over the activations of a CNN can be viewed as a low-pass filter over the Fourier-transformed activations. The cutoff frequency of a low-pass filter is a continuous parameter that can be learnt using gradient descent. Thus, the original recipe for DS is to compute the activations' discrete Fourier transform (DFT), remove frequencies above the learnable cutoff frequency and inverse DFT the filtered frequencies.\n\nThe authors propose extending DS by using the discrete cosine transform (DCT) instead of the DFT, as they argue that its properties are better suited for the problem at hand. The authors call their method DCT-DiffStride (DCT-DS). Furthermore, the authors propose a regularisation term to ensure that the model does learn a useful cutoff frequency instead of just keeping all the activations.\n\nThe authors benchmark their approach on a couple of standard image and audio classification tasks and a less standard modulation scheme classification task on radio signals.\n\nReferences\n[1] Rachid Riad, Olivier Teboul, David Grangier, and Neil Zeghidour. Learning strides in convolutional neural networks. ICLR, 2022", "summary_of_the_review": "Overall reasonable, though quite incremental contributions mainly lacking some empirical studies/comparisons. I will be happy to raise my score if the authors address my concerns.", "strength_and_weaknesses": "## Strengths\nDCT-DS is a sound, well-motivated idea, and the proposed model complexity measure and regularization term based on it seem reasonable too. The authors perform a reasonable amount of experiments on a good variety of data types. The authors show that in the low model complexity domain, DCT-DS can significantly outperform DS.\n\n## Weaknesses\n - While the idea is sound and reasonably well-executed, it is quite incremental. This is not a problem in itself, but it is unclear how beneficial the method is. This is because, with the exception of the modulation scheme classification task, the authors only show comparisons between DS and DCT-DS, so it is not possible to assess how well these methods perform compared to more traditional methods.\n - As I understand, the only difference between learning the cutoff frequency for a given model complexity for DS or DCT-DS and setting the stride size manually for standard convolutional layers is that during training, the model can allocate the size reduction more flexibly. Therefore, it would also be important to discuss how the  \"distribution\" of size reduction across different layers differs between DS, DCT-DS and the traditional methods.\n - The authors note, \"To ensure stability during training, all methods utilize unit l2-norm global gradient clipping\"  - it is not obvious to me what causes instability in the case of DS and DCT-DS. Is this a weakness of these methods, or is training unstable anyways?\n\n## Questions \n - Why does the squared penalty term in Eq 4 ensure that the model complexities of DiffStride and DCT-DiffStride will be better matched compared to Eq 3? Is it merely the fact that it encourages that the model complexity exactly matches $\\hat{c}$ rather than just making it an upper bound?\n - Could the authors comment on why DiffStride overtakes DCT-DiffStride in Figure 2? Furthermore, why does the accuracy decrease as the model complexity increases? Is this simply showing overfitting behaviour, or is there a different reason?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8Gq3_H7r-MV", "reviewer": "Reviewer_yZ5T", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors improve DiffStride, a pooling layer with learnable strides, by replacing the Fourier transform with a DCT.", "summary_of_the_review": "DCT-DiffStride applies a simple tweak to DiffStride. Simplicity is not a flaw but rather a quality when 1) it serves better performance 2) is clearly justified and explained. Given the inconsistent experimental results, and the lack of analysis of the high performance in the low-computation regime, it's hard to consider this paper worth being published in its current state.", "strength_and_weaknesses": "Strengths:\n* DCT-DiffStride is significantly better than DiffStride in the low computational cost regime.\n\nWeaknesses:\n* The main weakness of the paper is the minimal technical novelty (replacing DFT by DCT in DiffStride), which is hidden behind a lot of verbosity.\n* Section 2.2 is very verbose and could be summarized. For instance, the example given for non-periodic signals is artificial. Also, how are properties of the DCT on AR(1) processes relevant unless authors justify they are a good model for the natural signals considered? What would have needed more details is the last paragraph which vaguely describes the implementation details.\n* Similarly, the appendix provides a generic description of KLT/DCT/DFT that seems unconnected to the narrative of the paper.\n* The weight of the regularization (100) is not discussed, neither how it was cross-validated.\n* The results show that DCT-DiffStride only improves in the very low complexity regime. As such, and given that the technical contribution is minimal, it is hard to consider this paper as a significant contribution on top of DiffStride.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject