End-To-End Input Selection for Deep Neural Networks
Authors: Stefan Oehmcke
Fabian Gieseke
Venue:  (2020)
Keywords: Deep Learning
Input Selection
Gumbel Softmax Trick
Remote Sensing
Feature Selection

Abstract: Data have often to be moved between servers and clients during the inference phase. This is the case, for instance, when large amounts of data are stored on a public storage server without the possibility for the users to directly execute code and, hence, apply machine learning models. Depending on the available bandwidth, this data transfer can become a major bottleneck. We propose a simple yet effective framework that allows to select certain parts of the input data needed for the subsequent application of a given neural network. Both the associated selection masks as well as the neural network are trained simultaneously such that a good model performance is achieved while, at the same time, only a minimal amount of data is selected. During the inference phase, only the parts selected by the masks have to be transferred between the server and the client. Our experiments indicate that it is often possible to significantly reduce the amount of data needed to be transferred without affecting the model performance much.

Under review as a conference paper at ICLR 2020
END-TO-END INPUT SELECTION FOR DEEP NEURAL NETWORKS
Anonymous authors Paper under double-bling review
ABSTRACT
Data have often be moved between servers and clients during the inference phase. This is the case, for instance, when large amounts of data are stored on a public storage server without the possibility for the users to directly execute code and, hence, apply machine learning models. Depending bandwidth, this data transfer can become a major bottleneck. We propose a simple yet effective framework that allows to select certain parts of the input data needed for the subsequent application of a given neural network. Both the associated selection masks well as the neural network are trained simultaneously such that a good model performance is achieved while, at the same time, only a minimal amount of data is selected. During the inference phase, only the parts selected by the masks have to be transferred between the server and the client. Our experiments indicate that it is often possible significantly reduce the amount of data needed to be transferred without affecting the model performance much.
the available
INTRODUCTION
Neural networks have successfully been applied to many domains (Bengio et al. 2013 LeCun et al. 2015). Two trends have sparked the use of neural networks in recent years, Firstly, the data volumes have increased dramatically in many domains yielding large amounts of training data, Secondly, the compute power of today's 's systems has significantly increased as well, particularly those of massively-paralle architectures based on graphics processing units. Those specialized architectures can be used to reduce the practical runtime needed for training and applying neural networks, which has led to the development of more and more complex neural network architectures (Krizhevsky ct al. 2012 He et al. 2016 Huang et al. 2017).
Many machine learning applications require data to be exchanged between servers and clients during the inference phase. This is the case, for example, in remote sensing, where current projects produce petabytes of satellite data every year (Wulder et al., 2012 Li & Roy 2017). The application of a machine learning model in this field to, e.g., monitor changes on a global scale, often requires the transfer of large amounts of data between the server and the client that executes the model, see Figure I Similarly, data have often to be transferred from clients to servers for further processing. For instance, data collected from mobile devices transferred remote servers to be analyzed by virtual assistants such as Alexa from Amazon, Siri from Apple, or the Google Assistant. A similar situation is given when energy-efficicet batteries to collect sensor data from remote locations. Here, the transfer of data is often considered the most expensive operation due to the high power consumption caused by the transmission.
microcontrollers g., the Arduino Uno) are powered by
While the reduction of the training and inference runtimes have received considerable attention (Coates et al., 2013 Han et al.) 2015 Gordon et al. 2018 Nan et al. 2016: Kumar et al. 2017 Xu et al. 2013), relatively little work has been done regarding the transfer of data induced by such server/client based scenarios. However, this data transfer between clients and servers can become a severe bottleneck that can significantly affect the way users leverage available data. In some cases, the necessary data transfer can be reduced based on prior knowledge (c. g., in case one knows that only certain input channels are relevant for the task to be conducted). Also, for some learning tasks, the data transfer can be reduced by extracting a small amount of expressive features from the raw data. In general, such feature based reductions have to be adapted the specific tasks and might also lead to a worse performance compared to purely data-driven approaches
Note that, in case the data resides on public storage server, it is often not possible for the user to execute any code on the server side. This renders manual feature extraction or the application of (parts of) deep neural network impossible the server side.

---

Under review as a conference paper at ICLR 2020
tranaler
transfer
data
masks
selected data
model (client)
(server)
Figure I: Application of a neural network in the context of remote sensing. Here, hundreds of input feature maps might be available (nulti--pecctral image data collected at different times). Transferring data from the server to the client running the model can be extremely time-consiin cons-minnnnnnn... Our framework uses various types of selection masks that can be adapted to the specific transfer capabilities between the server and the client (e.g., if channel- or pixel-wise data transfers possible). Also, a different loss Q; can be assigned to each individual mask to penalize selections made by it. The masks as well as the given network performance and to select only small amounts of the input data. During the inference phase, only the selected parts have to be transferred. Similar bandwidth-restriited scenarios can be found in other Gata-intensive disciplines as well such as astrophysics or in the context of sensor data analytics.
optimized simultaneously in end-to-end fashion to achieve good model
Contribution: We propose a framework that automatically learns to select the relevant parts of the input data for given neural network and its task. In particular, our approach aims select the minimal amount of data needed to achieve a model performance that is comparable with one that can be obtained all the input data. The individual selection criteria can be adapted to the specific needs of the task at hand as well as to the transfer capabilities between the server and the client. As shown in our experiments, our framework can be used sometimes significantly reduce the amount of data needed to be transferred during the inference phase without affecting the model performance much.
2 RELATED WORK
Reducing the training time has gained significant attention in recent years. This includes, for instance. the use of parallel or distributed implementations (Coates et al. 2013 Dean et al. 2012 Li et al.) 2018). Approaches aiming at an efficient inference phase have been proposed as well, including schemes that aim at reducing the weights of networks the amount of floating point operations (Han) ctt al. 2013 Gordon et al.) 2018). Similarly, methods that deploy small tree-based models have been suggested (Kumar et al. 2017 Xu al. 2013). The transfer of data during the inference phase has been addressed as well. For instance, et al. (2016) propose a method that prunes features during the construction of random forests such that only few are needed during the inference phase (thus, avoiding costs for their computation and their transfer). In some cases, data compression can be used reduce the amount of bytes needed be transferred (e. images compressed via JPEG). However, this usually requires to retrain a network to find suitable compression level, which is not known beforehand Deep neural networks have also been used compress image data (Jiang et al. 2018 but the resulting compressed versions are independent of the learning task.
We conduct gradient-driven search to find suitable weight assignments for the selection masks. An alternative to our approach are greedy schemes that, e.g., incrementally select input channels or pixels. However, these schemes might yield suboptimal results since only one channel/pixel selected in each step. Further, these approaches quickly become computationally infeasible in case many channels or input pixels are given. Naturally, an exhaustive search for finding optimal mask assignments is computationally intractable. Our approach can be seen trade-off between these two variants. Finally, our approach is inspired by focused and peripheral vision, where unfocused objects containing less detail still offer useful information (Strasburger al. 2011).
Such compressed versions might also not he available on the server/client side. Our framework can handle these scenarios special case with the optimal compression level automatically being selected during training.
2

---

Under review as a conference paper at ICLR 2020
input
masks
output
input
masks
input
masks
output
input
masks
oulput
Q
Qi
Qa
(a) channel (any) (b) channel (xor)
(e) xe1 (any)
(d) pi (xor)
Different selection masks that can be used to select parts of the input data. For each of the masks, an individual loss Q, can be defined penalize selections made by that mask. While the final masks are discrete, differentiable surrogates are used during training.
Figure 2:
3 LEARNING SELECTION MASKS
We resort masks that can be used to select certain parts of the input data. These masks are adapted during the training process such that (a) the predictive power the network remains satisfying and (b) only minimal amount of the input data is selected. We will focus image data in this work for the sake of exposition, but our approach can also be applied to other types of data.
3.1 SELECTION MASKS
2x
The selection masks allow to select parts of the data such as certain input channels individual pixels of the dif- ferent channels, see Figure 2 For each such mask, an associated cost can be defined, which can be used to adapt the masks to the specific requirements of the task at hand (e. g.. if selecting pixels from one channel causes less data transfer in the inference phase than from another channel). Our optimization approach resorts to the following mask realizations, see Figure
(a) channe" (any) (h) channel (xor)
2*
(c) pixel (any)
pixel (xor)
Figure 3: Implementation of masks
channel (any): To select an arbitrary number of k input channels, a joint mask m {0, 11111xxx22 is used, which contains, for each of the & channels, two weights. For instance, a mask m with m [.1...J = (1, 0) and 2,2 = (0, 1) corresponds to selecting the first
Bxxxxxk the first two axes are broadcasted, which yields a mask m E {0, 1 jwwxxxxx2
but not the second channel. Before applying the mask an image
similar fashion, one can select exactly one of the k input channels by resorting to a joint mask of the form m E {0, 10 1x 1x Here, exactly one of the weights equals one. For instance, mask m with m [[.1..]] = (0, ........... 0, 1) corresponds to only the last channel being selected. As before, the first two axes are broadcasted prior to the application of the mask, yielding mask of the form m E {0, 1|=x
channel (xoz):
,,
pixel (any) To conduct pixel-wise selections, one can directly consider joint masks m" {0, [0,1]xxxxxx* which permit to seleet individual pixels per channel. For instance, a mask D with m 1, = (1,0) and m = (1, 0) for 1,..., corresponds to selecting all pixels on the diagonal for the first two channels.
pixel (xor): : Similarly, one can only allow one channel to be selected per pixel by considering a joint mask of the form m E {0, 1 which contains, for each pixel, exactly one non-zere element corresponding to the selected channel for that pixel.
Note that variants of these four selection schemes can easily be ob- tained. For instance, shapes can be defined that partition the input data into, say. nine rectangular cells by considering masks of the form mn D (0, 13333xx22 where the first two axes are broadcasted to the corresponding cells. Such variants would allow select certain cutouts, see Figure 4 The particular masks can be chosen according to the spe- cific transfer capabilities between server and client. Finally, the different selection masks can also be applied sequentially with individual costs being assigned them, Section
Figure 4: block (any)

---

Under review as a conference paper at ICLR 2020
Algorithm I: Leearnne
Input :oodel f and training set T
m < InitAAl
()
// initialize all selection masks // initialize lambda and tau
7 <- Tn Ini ()
for i <- to Repoch do
for j - Thorkh do
y + GetBatch(Z) b mod 2 =
// get next batch // alternate between axploration/Eixatian
m D .m +- Di Discret zeMaakks(m,T.))
// compute masks // apply masks to input compute prediction // compute adapted loss f, m + Optimize m, m, update weights of masks and model
& < ApplyMasks (x, D)
ÿ f())
< (y(Ú.v) xQ(m²)
10
11
12
A, - AdaptLambdaTau (),
// adapt lambda and tau // extract discretized masks
13
m
D +- Discrett zeMasks false)
14 return f.m
3.2 ALGORITHMIC FRAMEWORK
Let T {(11 (1fii,1),.... y1)...., (In, yn}} C. X x Y be training set consisting of images æ; e R with associated labels y4 R. The goal of the training process is to find suitable weight assignments both for the selection masks well for the neural network f X + Y that is applied to the data.
3.2.1 OPTIMIZATION APPROACH
Our procedure for learning suitable mask and network weights is given see Algorithm
Both the joint selection mask well as the parameters À and T are initialized in Line I and Line respectively. The parameter À determines the trade-off between the task loss C and the mask loss Typically, À initialized with a small positive value (e.g., À 0.1) and is gradually increased during training. Both the selection mask m and the network are trained simultaneously by iterating over pre-defined number Repoch of epochs, each being split into Abatch batches (for the sake of exposition, we assume batch size of 1). For each batch, a discrete mask m is computed via the procedure DiscretizeMaske used to obtain the masked image &. The induced prediction ý is then used to compute the task loss (0. y). addition, the overall mask loss Q(m is computed. Note that the discretized weights m P are used in the forward pass, whereas a mask m with real-valued weights is used in the backward pass in Line II After each epoch, both À and are adapted. As detailed below, the procedure Discret i zeMasks alternates between an "exploration" and a "fixation" phase, specified by the parameter b. The final discrete weights for the joint mask computed in Line 13
by searnSelectiontee
which is
Learning Discrete Masks: Naturally, exhaustive search schemes that find the optimal discrete weights by testing all possible assignments computationally infeasible. Simple greedy approaches such as forward/hackward selection channels become computationally very demanding and are thus ill-suited for pixel-wise selections. Learning such discrete masks is difficult since the induced objective is not differeentablee, which rules out the use of gradient- optimizers commonly applied for training neural networks. One way circumvent this problem is the so-called Gumbel- Max trick, which has been recently proposed in the context of variational auto encoders to learn discrete latent variables (Maddison al. 2014 Gumbel 1954, Jang et al. 2017 The procedure Discreti zeMaska uses this trick discretize the masks m, which contain class probabilities, in the forward pass of Algorithm|] For instance, given a mask m IR 1lx1xxx22 corresponding to
channel (any), the procedure yields a discrete mask m (0. 1] 111x1xX22 via
D m.I.J. one_hot
arg max log 1.1...4 iE(1.2
(1)
'hhe Gumbel-Max trick yielded hetter results compared to other discretization methods such as 1.2- regularization along with truncation of small weights.
4

---

Under review as a conference paper at ICLR 2020
where j {1,...,A, }} corresponds to the j-th channel and where each 9i is either from the Gumbel distribution, depending on which phase is executed (see below). Equation (T) does not provide gradient information because arg max cannot be differentiated. For this reason. the following differentiable surrogate m E RIxIxxx2 is employed for the backward pass in Line 11
sample
logg m p.1...11 + 91 log p.1...) + 92
S 1.1.441 softmax
Thus, the softmax function is used as a surrogate for the discrete arg max operation. The pa- rameter I is called temperature. A large r leads to the resulting weights being close to uniformly distributed, whereas a small value for T renders the values outputted by the softmax surrogate being close to the discrete one-hot encoded vectors. The procedure Discret i zeMasks alternates between "explore" and "fixate", specified by the parameter b. If b is true, then y: is random sample from the Gumbel distribution 9: = - log log (u)) with uniform sample u ~ UO, 1). If is false, I O. In the exploration phase, the optimizer can try out new possible mask assignments, whereas the network weights are adapted to the new data input in the fixation phase. The amount of changes made during the exploration phase is also influenced by the temperature parameter
Initialization and Adaptation: The selection goal influences the initialization of the mask m. In case all input channels for the channel (any) scheme are equally important, the individual masks are set to mp.1j.1 = (I + E, 0 + =) for all j = I,..., , k to initially "select" all of the channels, where E ~ (N, σr) for some small > In case the channels should be treated differently, the initialization can be adapted accordingly. For instance, only the first channel can be selected initially by setting m...1 = (1 + E, + E) for j = and [1.1...]] = (0 + E, 1 + 8) for j #
The procedure Ini teittaamiee LambdaTau initializes both À and The parameter X, which determines the trade-off between the task loss L, and the loss Q associated with all masks, is initialized to a small value (e. À 0.1). The temperature parameter r is initialized to a positive constant Timit (e.g., Tima = 10). The adaptation of both À and т after each epoch are handled by the procedure AdaptLambdaTant In the course of the training process, the influence of À is gradually increased until neeooch epochs have been processed or some other stopping criterion is met (c. g., as soon as the desired reduction w.r.t. Q is achieved). Since the range of values for the model loss L is generally not known beforehand, we resort to a scheduler that increases X in Line 10 of Algorithm T case the overall error L = L + XQ has not decreased for certain amount of epochs. The scheduler behaves similarly to standard learning schedulers, but instead of decreasing the learning rate, the value for A increased by certain factor Afuc (e. g., Aac = 1.1). The temperature influences the outcome of the so f ftmax operation in Equation ( A large value leads to similar weights being mapped to similar ones via the operation, whereas small value for r amplifies small differences such that the outputted weights m are close to zero/one. For each new assignment of X, we resort to some cool-down sequence, where reset to = Tinit and gradually decreased by factor Tdeeny after each epoch g., Tdecay = 0.9). This cool-down sequence let the process explore different weight assignments at the beginning. whereas binary decisions are fostered towards the end.
3.3 EXTENSION AND REDUCTION
Different costs can be assigned to the individual masks, which are jointly taken into account by the overall mask loss ((m"). For instance, given k input channels, one can resort different losses eI,..., 2x to favor the selection of certain channels. This turns out to be useful in case different "versions" for the input channels are available, whose transfer costs vary (c. compressed images or thumbnails of different sizes). Often, pre-trained networks with a fixed input structure are given. The selection of different versions for such networks can be handled via simple operators, see Figure 5] The extend operator can be used to extend given input feature map (c. g., by generating ten compressed versions different quality), whereas the merge operator can combine feature maps in user-defined way (e. g.. by summing up the input channels). For instance, an ext extend operation followed by a channel (xor) selection and merge operation can be used to gradually select certain version of each input channel without significantly changing the input for a given network in each step, thus allowing to learn masks for pre-trained networks without having retrain the network weights from seratch, see Section 4
Figure 5: extend / merge
5

---

Under review as a conference paper at ICLR 2020
1.0
0.9
0.91
20.8
5,900
0.7
0.6
0.89
0.5
1.0
0.9
G60.6
00.4
0.8
0.2
0.7
0.0
30
100
150
200
-0
100
155
30
100
200
(a) remote
(b) supernove
(c) cifarl0
Figure 6: channel (any) mask realization results remote, supernova, and c far10. The black line is the average value of the runs and individual runs are displayed in different colors.
Table 1: Datasets and Models
EXPERIMENTS
Dataset
#train
#test #class
w h c model
We implemented our approach in Python 3.6 using PyTorch (version 1.1).
remote
24 694 24 694
12 35 35 36 AllConvNet
supernova
420
4 4018
3 AllConvNet
cifar10
50000 10 000
10 32 32
3 ResNet101
Except for the trade-off parameter X, default parameters were used for all experiments (Neacch = 128, Timit = 10,
mnist 60 000 10 000
10 28 28
1 LeNet5
svhn
73 257 26 032
10 32 32
3 ResNet101
Tdecay 0.5, and Tiin 0.01). The learning rates 3 for all selection masks were set to
= 0.01. For the networks, the Adam (Kingma & Bal (014) optimizer with AMSGrad (Reddi
al. 2018) and learning rate 0.0001 was used. The initial assignment Xinit well the
factor fac for X can have a significant impact. For this reason, we considered small grid (Ainits Afec) {0.1, 1.0) {1.1, 1.25) of possible assignments. The influence of this parameter is shown in Figure 14: for all other figures, one of the four configurations is presented.
We considered several classification datasets and network architectures, see Table T In addition to the well-knoww ci farl0, mnist, and svhn datasets (Krizhevsky et al.] 2009; LeCun un et al. 2010 Netzer et al. 2011), we considered two datasets from remote sensing and astronomy, respectively. For each instance of remote, one is given an image with 36 channels originating from six multi-spectral bands available for six different dates (Prishchepov et al.) 2012). The learning goal to predict the type of change occurring in the central pixel of each image. The astronomical dataset is related to detecting supernovae (Scalzo et al. 2017). Each instance is represented by an image with three channels and the goal is predict the type of object in the center of the image balanced version of the dataset was used). Both remote e and supernova depict typical datasets in remote sensing and astronomy, respectively, with the target objects being located in the centers of the images. For all experiments, we considered a fixed amount of epochs and monitored the classification accuracy on the hold-out set. Each experiment was conducted Trans 10 times and the lines of the figures represent individual runs (the thicker black line is the aggregated mean over all runs). If not stated otherwise, we considered pre-trained networks before applying our selection approach.
4.1 CHANNEL SELECTION
The first experiment addressed the task of selecting subset of the input chan- nels. We used remote, supernova, and cifarl0 as datasets, for which different outcomes were expected. For each of the channels, we assigned the same mask loss Q; = 1/c. The overall mask loss Q was the sum over all channels, which corresponds to the ratio of the data that need to be transferred. The outcome is shown in Figure As expected, channel-wist selection worked best on remote due to many channels carrying similar information. Only if less than 20% of the channels were selected, the accuracy started to drop. In Figure 7, the selection process is sketched, where each row represents a different epoch (from top to bottom: 0, 50, 100, 150, 200) and where each columns corresponds to one of the channels. For supernova, the removal of single channel did not significantly affect the classification accuracy. For some runs, all channels were removed at once, which indicates that the steps made for À were too large (thus, smaller Xfac should be considered).
Figure 7: Selected channels for remote

---

Under review as a conference paper at ICLR 2020
0.92
0.998
0.95
30000
0.996
00.00
0.85
0.86
0.80
0.84
0.75
1.0
0.5
Ins 2
0.4
0.0
50
100
150
0
100
130
200
100
150
200
(a) remote
(b) supernove
(c) cifar10
Figure 9: pixel (any) mask realization results remote, supernova, and cifar10.
On c far10. only of the three channels could be dropped with minimal degradation of accuracy, Thus, as expected, less channels could be removed for both supernova and ci cifarl0 due to the channels being less redundant.
4.2 PIXEL-WISE SELECTION
Next, pixel-wise selections were addressed (pixel (any)) by conducting a similar experiment using the same datasets. The mask loss Q was obtained by summing over the selected pixels, where weight of w/xx was assigned to each individual pixel. Thus, Q corresponds to the ratio of pixels that need be transferred. The results given in Figure 9 It can be seen that all plots for Q are smoother than for the channel-wist selections, which is due to the fact that the selection decisions be made at each step were much more fine- ine-grained (for cifarlC and supe supernoves væ, only three channels but thousands of subpixels are given). It can be seen that the accuracy drops slightly at the beginning of the training process. This is because the networks were not trained with missing inputs before and, hence, had to learn to compensate the missing input at the beginning. This effect could be lessened by (a) adding dropout layers to the networks or by (b) decreasing both Ainit and Afa to let the approach do less exploration at the beginning. Overall, the achieved reduction w.r.t. the remained accuracy is higher than for the channel-wise selection, although there are notable spikes in supe supernova nove that most likely stem from the removal of subpixels being crucial for the classification task (the removal of some central pixels seem to have had a significant impact). The development of the masks w.r.t. Nepoch is shown in Figure for supernova.
Figure 8: Pixel-wise selections
0.90
4.3 FEATURE MAP SELECTION
accuracy
0.85
In many cases, preprocessed data available the server/client side. The next experiment was dedicated to such scenarios. In particular, sions for the cifaric images of different JPEG qualities q {100, 95, 85, ... 25, 15). The goal was select one of these versions via channel (xor). To capture the varying costs for the transfer of the different versions, we assigned 2g = 7/--100 to each version with quality level g. This mask loss not as directly linked to the transfer costs, the individual JPEG levels can have different impacts on each image, which depends on the JPEG compression algorithm. Also, the masks were initialized in such way that only the version with the highest quality was selected initially.
0.80
0.75
considered ten compressed ver-
1.0
0.8
los
0.6
0.2
50
100
150
200
Teppach
Figure 10: JPEG on cifar10
Figure shows the results. It can be seen that the low- est possible value (0.15) was obtained for Q, for which an accuracy of about 82% remained. Also, an accuracy of about 88% could be maintained while reaching loss of about Q ze 0.5. An illustration of the reduced input over the epochs is given in Figure 11
30
100
200
Repach
Figure 11: Reduced images
7

---

Under review as a conference paper at ICLR 2020
0.90
0.3
2098
0.7
0.80
0.7
U00 0
0.98
drp 2
dog Cogion
0.96
0.2
0.0
los
1.0
0.6
0.6
0.4
0.4
0.4
0.2
20
100
100
150
200
(a) cifar10
(b) svhn
(c) mnist
Figure 13: Results for the combination of selection masks on cifarlc, svhn, and mnist, where
JPEG qualities for each channel were used and, the same time, pixels could be selected.
4.4 COMBINATION OF MASKS
Next, multiple selection masks and mask losses considered. The following op- erations were applied, see Figure 12 First, an extend operation
used to generate different JPEG qualities for each channel. Af- terwards, channel (xor) selection oper- Figure 12: Combination of multiple selection masks. ation was applied, followed by merge operation (sum). Finally, a pixel (any) selection was conducted select subpixels of the merged channels. For this experiment, we used cifarl0. mn mtsst, and s svhn. The joint mask loss Q was set to the product %/-1000 1/exhxx of the two previously defined losses. The results are shown in Figure [3] Note that the models for svhn and mnist were not pre-trained in this case, which is why the accuracies start with lower value. Since mnist is a dataset with many empty border pixels, our approach was able to remove 50% of the pixels in the first few epochs. Also, the lowest possible JPEG quality observed on svhn although seems that is was harder to remove pixels due to more background pixels compared to mn i st. For c fari the results show that the combined masks yielded similar outcomes as for the individual masks, see again Figure 9nd and T0
used. Similar effects be
4.5 INFLUENCE OF À
0.998
The parameter can have big impact on the selection process. Figure [4] shows the influence of the four different configura- tions considered for our experiments given the remote dataset. It can be seen that large Ainät (blue and red line) leads to the mask loss Q quickly decreasing. For such settings, seems that not able to compensate the loss in information, the network which why the accuracy was lower until the network was able to adapt to the new input. A smaller initial value for À leads to the selection process taking less input data away at the beginning, which avoids an initial drop of accuracy. Similarly, large Afec leads to a faster decrease w.r.t. Q, which can be suboptimal in certain cases.
0.99
0.994
0.992
1.0
Auii
0.8
1.100 Ausi 1.0
0.6
1.25, Ausi 0.1
loss
0.4
0.2
0.0
50
100
150
200
Nepach
Figure 14: Influence of À
CONCLUSIONS
The transfer of data between servers and clients can become a major bottleneck during the inference phase of a neural network. We propose framework that allows to automatically select those parts of the data needed by the network to perform well, while, at the same time, minimizing the amount of selected data. Our approach resorts to various types of selection masks that are jointly optimized together with the corresponding network during the training phase. Our experiments show that is often possible achieve good accuracy with significantly less input data needed to be transferred. We expect that such selection masks will play an important role for lata-intensive domains such as remote sensing or astrophysics and for scenarios where the data transfer bandwidth is very limited.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 3598):1798-1828, 2013.
A. Coates, B. Huval, T. Wang, J. J. Wu, B. C. Catanzaro, and A. Y. Ng. Deep learning with COTS HPC systems. In International Conference on Machine Learning (ICML), volume 28 of JMLR Proceedings, pp. 1337-1345. JMLR. org, 2013.
Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Mare Aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, Quoc V. Le, and Andrew Y. Ng. Large scale distributed deep networks. In Advances in Neural Information Processing Systems 25, pp. 1223- 1231. Curran Associates, Inc., 2012.
Ariel Gordon, Elad Eban, Ofir Nachum, Bo Chen, Hao Wu, Tien-Ju Yang, and Edward Choi. Morphnet: Fast & simple resource-constrained structure learning of deep networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1586-1595. IEEE Computer Society, 2018
Emil Julius Gumbel. Statistical theory of extreme values and some practical applications: a series of lectures, volume 33. US Government Printing Office, 1954.
Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. In Neural Information Processing Systems (NeurIPS), pp. 1135-1143, Cambridge, MA. USA, 2015. MIT Press.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Computer Vision and Puttern Recognition (CVPR), pp. 770-778. IEEE, 2016.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261 2669. IEEE, 2017.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameteriization with gumbel-softmax. In International Conference on Learning Representations (ICLR), 2017.
F. Jiang, W. Tao, S. Liu, J. Ren, X. Guo, and D. Zhao. An end-to-end compression framework based on convolutional neural networks. IEEE Transactions on Circuits and Systems for Video Technology, 28(10):300-33018 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Computing Research Repository (CoRR), abs/1412.69800, 2014. URL httlhtttp//axxii httpa/wwwwww..rg////////1//
6980
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). 2009. URL /tt.:////////...............o/t. cit/rriar html
cs www.oto.ee.ooookiii.....
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convo- lutional neural networks. In Neural Information Prucessing Systems (NeurIPS), pp. 1106-1114. Curran Associates, 2012.
Ashish Kumar, Saurabh Goyal, and Manik Varma. Resource-efficient machine learning in 2 KB RAM for the internet of things. In Doina Precup and Yee Whye Teh (eds.), International Conference on Machine Learning (ICML), volume 70 of Proceedings of Machine Learning Research, pp. 1935-1944. PMLR, 06-11 Aug 2017.
Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. AT&T Labs [Online]. Available: http://yann. lecun. com/exdb/nnist, 2:18, 2010.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521: 2015.
Jian Li and David P. Roy. A global analysis of sentinel-2a, sentinel-2b and landsat-8 data revisit intervals and implications for terrestrial monitoring. Remote Sensing, 9(002,, 2017.

---

Under review as a conference paper at ICLR 2020
Youjie Li, Mingchao Yu, Songze Li, Salman Avestimehr, Nam Sung Kim, and Alexander Schwing. Pipe-sgd: A decentralized pipelined sgd framework for distributed deep net training. In Advances in Neural Information Processing Systems 31, pp. 8045-8055 Curran Associates, Inc., 2018.
Chris Maddison, Daniel Tarlow, and Tom Minka. A sampling. In International Conference on Neural Information Processing Systems (NeurIPS), pp. 3086-3094, 2014. URL http://
papers. nips httpps..cc/aaper5449---
Feng Nan, Joseph Wang, and Venkatesh Saligrama. Pruning random forests for prediction on budget. In D. D. Lee, M. Sugiyama, U. V. Luxburg. I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 2334-2342. Curran Associates, Inc., 2016.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop Deep Learning and Unsupervised Feature Learning 2011, 2011. URL http:://fld..... itttafoord.edu. pdf
Alexander V. Prishchepov, Volker Radeloff, Maxim Dubinin, and Camilo Alcantara. The effect of landsat ETM/ETM+ image acquisition dates on the detection of agricultural land abandonment in eastern europe. Remote Sensing of Environment, 126: 209, 2012. ISSN 0034-4257.
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and be- yond. In International Conference Learning Representations (ICLR), 2018. URL https /
R. A. Scalzo, F. Yuan, M. J. Childress, A. Möller, B. P. Schmidt, B. E. Tucker, B. R. Zhang, P. Astier, M. Betoule, and N. Regnault. The skymapper supernova and transient search. Computing Research Repository (CoRR), abs/ 70s200022,558 2017. URL
http:/abs/10000 abbs/1702.0555
Hans Strasburger, Ingo Rentschler, and Martin Jüttner. Peripheral vision and pattern recognition: A review. Journal of vision, 11(5):13-13, 2011.
Michael A. Wulder, Jeffrey G. Masek, Warren B. Cohen, Thomas R. Loveland, and Curtis E. Woodcock. Opening the archive: How free data has enabled the science and monitoring promise of landsat. Remote Sensing of Environment, 122(Supplement C):2 - 10, 2012. Landsat Legacy Special Issue.
Zhixiang Eddie Xu, Matt Kusner, Kilian Q. Weinberger, and Minmin Chen. Cost-sensitive tree of classifiers. In International Conference on Machine Learning (ICML), volume 28 of JMLR Proceedings, pp. 133-141, JML..org, JML. Rrg, 2013.
10

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 RATIO OF FIXATION/EXPLORATIOON
We introduced the fixation phase to allow the network to adapt to the mask changes made during the explo- ration phase. This can be seen as intermediate "post- training" to ensure that the "optimal" result is obtained for a given À (which is increased over time). The al- ternation between the exploration and fixation phase worked well in practice. However, other ratios between these two phases are also possible. This can be achieved by resorting to a corresponding ratio y in Line of Al- gorithm T Here, 7 0.25 corresponds and exploration iteration(s) and = 0.75 to 3 fixation and exploration iteration(s).
sccuaac (val)
10.900
0.25
0.50
0.75
0.850
Q (val)
0.8
0.4
fixation
0.34
E, (triin.)
0.2
0.14
The influence of the ratio y between exploration and fixation is shown in Figure [5] (which depicts an exten- sion of Figure 9c). The ratio does have the expected effect on the results. particular, a larger value (more intermediate "post-training" yields slightly better accu- racies (top) and slightly larger values for the reduction loss Q (middle). Hence, the ratio can be used to influence speed up/slow down the pruning. We decided to omit this from the original algorithm to keep simple since a similar effect can be achieved by adjusting the trade-off parameter and the temperature
200
Repoch
Figure 15: Influence of the ratio between exploration and fixation during training.
A.2 DECREASE IN ACCURACY
One might question if the decrease in accuracy observed in the experiments result from the general- ization gap, i.e., that the networks are simply overfitting. However, steady decrease in accuracy is expected since more and more input data are masked out. Eventually, an accuracy close to the mean estimator will be obtained, since the mask and network weights are adapted according to the joint objective L Xº and since À is increased in the course of the training process. Figure 15 (bottom) shows the training loss L, (cross entropy) for the experiment described above. can seen that the training loss L increases as the validation accuracy decreases, which is strong indicator that overfitting is not responsible for the decrease in accuracy (but the loss of information due to the mask changes). Note that the slope of the initial drop/increase (first 20 epochs) depends on the assignment for Ainis here, smaller values for Ainit lead to less changes at the beginning (see again Figure 14). Finally, it is worth mentioning that our approach very stable w.r.t. the involved parameters (note that all prper-pprameters except for À were fixed).
A.3 STOPPING CRITERIA
Instead using a fixed number Tepocch epochs, other stopping criteria can also used. For example, one could stop training the mask and the model as soon as the loss Q for the masks falls below particular user-defined threshold or as soon as the accuracy has decreased significantly. Once the general training procedure has stopped, one could keep adapting both the mask weights and the network weights for several epochs without increasing À any further, i. i.c without changing C anymore. In addition, could "finalize" the model f by training the model (but not the mask) for several epochs.
11


Reviews:
[{"review_id": "rJgm2PsCFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an approach to discrete input selection for NNs, using the Gumbel-Softmax trick at its core. It motivates this problem in the context of communicating data over a network with limited bandwidth budget. It proposes constructing different kinds of masks that can be applied over channels or pixels in the input, grounding the discussion in the image domain. This can be seen as a special case of Feature Selection, with image specific substructures motivating the choice of mask types. \n\nThere is very little novelty in this work over that presented in Abubakar Abid et. al. [1], where the idea of using Gumbel-Softmax as a differentiable Feature Selection algorithm has already been expounded at depth, both in unsupervised as well as supervised settings. The current work draws directly from the supervised form in [1]. The only incremental contribution in this work is the specific mask types and mask-specific losses.\n\nPros\n•\tInteresting approach to extend the framework in [1] to CNNs, with use of masks and mask-specific loss\n•\tClear motivation for the network bandwidth limited use case\n\nCons\n•\tHardly any technical novelty because the core ideas are already presented as well as applied to the same task in [1]\n•\tIt is very surprising that the authors do not even cite [1] in their paper, despite their work being extremely closely related to it\n•\tMost of the discussion in the Related Work section is unrelated to the specific task they tackled in the paper (i.e., input/feature selection). The second paragraph in this section talks about ‘gradient-driven search’ for discrete selection, which has been recently explored not only in [1] but also related G-S applications like [2], [3], but the authors seem unaware of this line of works\n•\tThe authors do not compare their approach against any existing baselines from literature for this task, again with the most apt being [1] and baselines therein. This makes it hard to understand the true value of their proposals such as mask types, schedule that adjusts both ‘tau’ and ‘lambda’ during training etc.\n\n[1] Abubakar Abid et al., “Concrete Autoencoders for Differentiable Feature Selection and Reconstruction”, ICML 2019, (https://arxiv.org/abs/1901.09346)\n[2] Hanxiao Liu et al., “DARTS: Differentiable Architecture Search”, ICLR 2019\n[3] Bichen Wu et al., “FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search” CVPR 2019\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJgDHU86FS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors argue that data transfer costs for ML inference may be significant in some scenarios, such as for remote sensing. They propose to jointly train a model to maintain good performance while significantly reducing input size by applying a global mask. To allow training with discrete masks, the Gumbel-softmax trick is used. The experiments cover multiple datasets and multiple types of mask. Generally, performance remains reasonably high and degrades gracefully as the input is increasingly masked.\n\nI lean slightly towards rejecting the paper. The problem is interesting and potentially important, but many experiments are  too simplistic and lack strong baselines.\n\nI believe the problem is well motivated, but not very much explored yet. There has been much work on reduce ML system computation costs and memory storage requirements, but mostly by modifying the model instead of the data. The paper proposes a reasonable approach to reduce data transfer costs.\n\nThe authors propose 4 types of masks (channel/any, channel/xor, pixel/any, pixel/xor), which are applicable under different circumstances. Some of them may also be combined. To learn discrete masks, they apply the Gumbel-softmax trick. Results clearly show that learning discrete masks (reducing input size) while maintaining decent performance is feasible .\n\nAs the objective function is modified during training by adjusting \\lambda, the performance/size trade-off is only loosely specified. All presented results are learning curves, but there are no clear final numbers.\n\nThe channel selection task (4.1) is potentially interesting, but lacks a baseline. How does random selection of channels perform?\n\nThe pixel selection task (4.2) is simplistic. Using a cloud of pixels near the center of the images appears sufficient, which could be inferred by looking at a few samples and doesn't necessarily necessitate learning. Could the approach be extended to more complex images, predicting one mask per image instead of a global mask?\n\nFeature map selection (4.3, channel 'xor') could be likely solved with hyper-parameter search, especially if the number of channels is small. Section 4.4 combines the previous two subsections, and it is unclear how much we gain from learning the masks over using simple heuristics.\n\nMore minor points:\n\nIn the related work section, the author could additionally mention distillation.\n\nThe Gumbel noise is used on half of the inputs, while the current argmax is used otherwise. It is unclear whether this is necessary, and there are no related experiments.\n\nAlthough this is not crucial, for the 'any' variants, the last mask dimension appears superfluous (2-class softmax). The binary variant of Gumbel-softmax (Maddison et al. The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables, Appendix B) could be used.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1lFl73pKS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses the problem of high-cost transfer between server and user for machine learning applications. The method proposes to augment inputs with channel/spatial masks that are trained via the Gumbel-softmax trick together with the model's weights trained/finetuned to account for the loss of information. These learned masks are then applied to the image before sending it to a server with where inference takes place to reduce file transfer costs. In the experimental study, the paper shows that on computer vision tasks inputs can be reduced with relatively little drop in accuracy and analyses how hyperparameters of the model affect its performance. The framework is also adapted to the downstream task-guided choice of compression techniques, e.g. which compression quality to choose for JPEG that will be an optimal trade-off in terms of downstream task quality vs data transfer cost.\nOverall, this paper could be a valid algorithmic contribution, however, I have concerns about how this method fares with others, resolving the following issues in the author response will likely increase the score.\n- A discussion on the connection/comparison with representation learning and dimensionality reduction (VAEs, quantization, etc) would help improve the exposition of the paper and help define how and when the suggested method is more appropriate to use, as well as citations to the other lines of work in reducing the model size (knowledge distillation [1], tensor decomposition approaches [2], adaptive computation time techniques [3], etc).\n- I would like to see an experiment that compares other works mentioned as related work. [4] has been mentioned as one of the nontrivial compression methods for image data, how does the proposed method compare to it?\n- Another experiment comparing a method from representation learning, e.g. a VAE trained with the embedding size corresponding to some optimal Q value in this work would be helpful.\n- Please include reference accuracy values for the dataset/NN pairs used in Table 1.\n- In Section 4.5, the paper states that large lambda values correspond to blue and red lines, however in the corresponding figure large lambda values correspond to blue and orange lines which exhibit different behaviors, could you please clarify this?\n\nReferences\n[1] Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. \"Distilling the knowledge in a neural network.\"\n[2] Novikov, Alexander, et al. \"Tensorizing neural networks.\" Advances in neural information processing systems. 2015.\n[3] Figurnov, Michael, et al. \"Spatially adaptive computation time for residual networks.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.\n[4] Jiang, Feng, et al. \"An end-to-end compression framework based on convolutional neural networks.\" IEEE Transactions on Circuits and Systems for Video Technology 28.10 (2017): 3007-3018.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to address the high bandwidth cost when transferring data between server and user for machine learning applications. The input data is augment with channel and spatial mask so that the file transfer cost is reduced. While the reviewers agree that this is a well motivated and interesting problem to study, a number of concerns are raised, including loosely specified performance/size trade-off, how this work is compared to related work, low novelty relative to a few key missing references. The authors respond to Reviewers’ concerns but did not change the rating. The ACs concur the concerns and the paper can not be accepted at its current state.
Decision: Reject