SC2 Benchmark: Supervised Compression for Split Computing
Authors: Yoshitomo Matsubara
Ruihan Yang
Marco Levorato
Stephan Mandt
Venue: Accepted by TMLR ()

Abstract: With the increasing demand for deep learning models on mobile devices, splitting neural network computation between the device and a more powerful edge server has become an attractive solution. However, existing split computing approaches often underperform compared to a naive baseline of remote computation on compressed data. Recent studies propose learning compressed representations that contain more relevant information for supervised downstream tasks, showing improved tradeoffs between compressed data size and supervised performance. However, existing evaluation metrics only provide an incomplete picture of split computing. This study introduces supervised compression for split computing (SC2) and proposes new evaluation criteria: minimizing computation on the mobile device, minimizing transmitted data size, and maximizing model accuracy. We conduct a comprehensive benchmark study using 10 baseline methods, three computer vision tasks, and over 180 trained models, and discuss various aspects of SC2. We also release our code and sc2bench, a Python package for future research on SC2. Our proposed metrics and package will help researchers better understand the tradeoffs of supervised compression in split computing.

Published in Transactions on Machine Learning Research (06/2023)
SC2 Benchmark: Supervised Compression for Split Com- puting
Yoshitomo Matsubara
yoshitom @ uci. edu
Department of Computer Science University of California, Irvine
Ruihan Yang
makgn..ang@uuu uct
Department Computer Science University of California, Irvine
Marco Levorato
levorato uci. edu
Department of Computer Science University of California, Irvine
Stephan Mandt
mandi @ edu
Departments of Computer Science and Statistics University of California, Irvine
Reviewed on OpenReview: http:///gwggo.o.wo..////////t/t/
Abstract
With the increasing demand for deep learning models on mobile devices, splitting neural network computation between the device and more powerful edge server has become an attractive solution. However, existing split computing approaches often underperform compared to naive baseline of remote computation on compressed data. Recent studies propose learning compressed representa- tions that contain more relevant information for supervised downstream tasks, showing improved tradcoffs between compressed data size and supervised performance. However, existing evaluation metrics only provide an incomplete picture of split computing. This study introduces supervised compression for split computing (SC2) and proposes new evaluation criteria: minimizing computa- tion on the mobile device, minimizing transmitted data size, and maximizing model accuracy. We conduct comprehensive benchmark study using 10 baseline methods, three computer vision tasks, and over 180 trained models, and discuss various aspects of SC2. We also release code and sc2 bcc2benchf Python package for future research on SC2. Our proposed metrics and package will help researchers better understand the tradeoffs of supervised compression in split computing.
1 Introduction
Machine learning models increasingly used in intelligent devices such smart devices, wearable devices, au- tonomous drones, and surveillance cameras (Chen & Ran 2019). However, these devices are often computationally weak, which makes it challenging to deploy complex deep learning models on them (Eshratifar et al. 2019a). To address this issue, researchers have proposed lightweight machine learning models that are optimized for low com- putational cost and high supervised performance (Sandler et al. 2018 Tan et al.) 2019; Howard et al., 2019). An alternative approach is to offload heavy computing tasks a more powerful cloud/edge server. In this scenario, weak local devices only send compressed data such as images the cloud/edge server, which carries out heavy computing costs to complex deep learning models. In the context of visual data, neural image compression models have been
"This work was done prior joining Amazon.
benchmark

---

Published in Transactions on Machine Learning Research (06/2023)
Limited computing resource
Limited channel capacity
Powerful computing resource
Battery constraint
Unstable connectivity
Negligible processing time
Minimize Encoder Size
Minimize Data Size
Maximize Model Accuracy
Mobile Device
Communication
Cloud/Edge Server
(()))
O BIJ
Reconstructed Input
Input
Compressor
Decompressor
Full Inference Model
Compressed Input
Prediction
"Zebra"
(a) Input Compression for Full Offloading
Input
Encoder
Decoder
Pruned Inference Model
Compressed Feature
Decoded Feature
Prediction
"Zebra"
(b) Supervised Compression for Split Computing
Figure 1: Input compression (top) vs. SC2: supervised compression for split computing (bottom) for image clas- sification. The input compression approach reconstructs the input image on the could/edge server whereas the SC2 approach produces an compressible feature representation suitable for the supervised task. Note that the training pro- cess is done offline, and then the model will be split for deployment.
attracting much interest (Ballé et al. 2017: Minnen et al. 2018 Yang et al., 2023b). However, compressing the full image using traditional algorithms optimized for input reconstruction is inefficient, as they likely result in embedding information that irrelevant toward the supervised task (Choi & Han 2020).
A potentially better solution is split the neural network model into two sequences (Kang al. 2017). The first sequence executed on the weak mobile (local) device and applies some elementary feature transformations to the input data. Then, the mobile device transmits intermediate, informative features through constrained wireless com- munication channel to the edge server that the second sequence of the model can processes the bulk part of the computation (Eshratifar et al.) 2019b] Matsubara et al. 2019). This approach is called split computing, and it has been advancing with Icarnable data compression pipelines based on feature quantization (Matsubara et al. [2020] Shao
& Zhang 20200] Matsubara & Levorato 2021 Banitalebi-Dehkordd et al. 2021 Assine et al. 2022). or entropy-
coding (Singh et al. 2020 Matsubara et al. 2022c Yuan et al. 2022).
In this paper, we formalize and study the problem of supervised compression for split computing (SC2). We benchmark several established methods, including state-of-the the-art input and feature compression baselines, for various models in multiple supervised tasks from the aspects of the SC2 problem. Our experiments reveal that the choices of the split- ting point, reference models, and the encoder-decoder architectures substantially affect the supervised rate-distortion performance. We argue that the study of these approaches necessitates a thorough characterization of the three-way tradcoff between encoder size, data size, and supervised learning performance.
This benchmark study involves 10 different state of the art methods and more than 180 trained models for three differ- computer vision tasks. It also contains variety of architecture ablations, extended discussions on the limitations of the supervised rate-distortion tradeoff. and the first discussion of three-way tradeoff in supervised compression. Our main contributions are proposing new metrics, an exhaustive benchmarking study, and providing Python package and code repository for future research in the research community.
The rest of this paper is structured as follows. Section we summarize related studies, supervised compression. and the motivations of this benchmark study. Section 3 provides an overview of SC2. In Section 4 we demonstrate the effectiveness of our approach, highlighting the importance our proposed tradeoffs for SC2, and further discuss factors improve SC2 Benchmark results. Finally, conclude this work in Section
2

---

Published in Transactions on Machine Learning Research (06/2023)
2
Background
In this section, we briefly introduce related studies and highlight the main motivation behind this study.
2.1
Related Work
We summarize related ideas from the neural compression and split computing communities.
2.1.1
Neural Image Compression
Neural image compression models are typically trained an unsupervised manner to reconstruct input images while learning compressed representations of the data (Yang et al.] (0236). These models leverage neural networks for nonlinear dimensionality reduction and subsequent entropy coding. Early studies employed LSTM networks to capture
spatial correlations among pixels within image (Toderici et al. 2017 Johnston al. 2018).
One of the pioneering works in image compression using autoencoders was proposed by Theis et al. (017)) The con- nection between image compression and probabilistic generative models was established by variational autoencoders (VAEs)(Kingma & Welling, 2014, Ballé et al., 2017). Building upon this, [Ballé et al. (2018) proposed two-level VAE architectures with a scale hyper-prior for image encoding. These architectures have been further enhanced by
incorporating autoregressive structures(Minnen al. 2018 Minnen & Singh 2020) and optimizing the encoding pro-
(Yang et al.
2020a). This approach has been extended to numerous architectures for images (Cheng et al.,
2020:
Zhu et al. 2021 1 Wang et al. [2022] He et al. [2022 Liu et al. 2022 Yang & Mandt 2022a) and video (Wu et al.
2018
Djelouah et al. 2019 Han et al. 2019 Chen et al.
2019 Lu et al.
2019 Habibian et al. 2019 Agustsson et al.
2020
Yang et al. 2023a).. Active research topics include variable bitrates (Lu et al. (222)). compression without pre- pre-defined
quantization grids (Flamich et al. 2019 Yang et al. 2020b). and exploring compression limits (Alemi et al.
2017:
Yang& Mandi 2022b).
While self-supervised compression architectures for generic image classification have been proposed (Dubois et al. 2021), , one particular approach using Vision Transformer (ViT)-based encoder from the CLIP model (Radford et al. ) by Dubois et a (2021) is characterized by encoder with 87.8 million parameters. However, this high parameter count, which is 627 times larger than the encoder proposed by Matsubara et al.) (202c)), mainly due to the ViT-based encoder, renders it unsuitable for deployment on resource-constraiined edge computing systems.
2.1.2 Split Computing
In many real-time application settings, local (mobile) devices capture sensor data (e.g., images) and often have lim- ited computing resources and battery constraints, thus fully offload computationally heavy tasks to more powerful cloud/edge servers. such full offloading scenarios, the more resourceful edge server receives the sensor data from the mobile device via wireless communication channel and then execute the entire model. To complete the inference in a timely manner, the latter strategy requires high-capacity wireless communication channel between the mobile device and edge server. With low-capacity wireless networks, critical performance metries such as end-to-end la- tency would degrade compared to local computing due to the large communication delay. As an intermediate option between local computing and full offloading (the full computation is on either local device or edge server), split com- puting (Kang et al., 2017) has been attracting considerable attention from the research community to minimize total delay in resiuree-limited networked systems (Eshratifar et al.] 2019b] Matsubara et al.) 2019). For instance, Long Range (LoRa) (Samie et al., (016)
widely used technology for ensource-constraiinn Internet of Things devices and applications, which has a data rate of 37.5 Kbps due duty cycle limitations (Adelantado et al., 2017).
In split computing, a deep learning model is split into two sequences. The first sequence of the model is executed the mobile device. Having received the output of the first section via a wireless communication, the second sequence completes the inference on the edge server. A critical need is to reduce computational load on the mobile device while minimizing communication cost (data size) as processing time on the edge server is often smaller than local processing order to reduce communication cost, recent studies on split computing (Eshratifar et al., [2019b] Matsubara et al. 2019 Shao Zhang] 2020 Matsubara Levorato 202T; Assine al. 2022) introduce bottleneck, whose data size is smaller than input sample to vision models. Those studies
and communication delays (Matsubara & Levorato 2021
3

---

Published in Transactions on Machine Learning Research (06/2023)
Input images
SC IC
Figure 2: Bitrate comparison between a supervised compression (SC) model (Matsubara et al. 2022c) and an input compression (IC) model (Ballé al. 2018). We plot the difference of the bits allocated for each pixel, exemplified three images. Areas where the SC model allocates fewer and more bits for the given image are indicated in blue and red, respectively (best viewed in PDF). It is apparent how supervised compression allocates more bits to information relevant to the supervised object recognition goal.
show for image classification and object detection tasks that a combination of 1) channel reduction (dimensionality reduction) in convolution layers and 2) quantization at bottleneck layer is key to designing such bottlenecks.
2.2 Supervised Compression
Supervised compression refers to the process of learning compressed representations specifically tailored for super- vised downstream tasks, including classification, detection, or segmentation. In this context, a supervised compression model is defined deterministic mapping + z y, where x, y, and z represent the input data, targets, and compressed representations, respectively, for the given supervised downstream task(s).
This formulation bears resemblance to the concept of the deep variational information bottleneck (Alemi et al. 2017)) although should be noted that the latter was originally devised for enhancing adversarial robustness rather than com- pression. For comparison, we also consider deterministic mapping x + + & + y to denote the inference process of an input compression model followed by a supervised model. Here, z" represents the compressed representations of the input data x, while & indicates the reconstructed input data.
It is important highlight that in unsupervised input compression, the input data x can be accurately reconstructed from z" (x = &). However, in the case of supervised compression, the compressed representations z lack the neces- sary information for precise reconstruction of the original input data. This is because supervised compression aims to learn z" in a way that retains relevant information for the specific downstream task(s), allowing for the compression of irrelevant information. For instance, in image classification tasks, not all pixels input images are crucial, and supervised compression can effectively discard such irrelevant information in z while z" requires all information faithfully reconstruct the original input data x.
Since input compression models are trained to reconstruct images, they tend to allocate bits more or less uniformly across the image. In contrast, supervised compression methods use the compressed features for prediction tasks; they should therefore be expected to allocate most of their bits to only relevant regions of the image, i.e., regions that correlate with the prediction task. Figure 2 confirms this intuition, showing the difference in the bitrates between supervised compression and input (image) compression approaches. In more detail, we create spatial maps of the bits consumed for each pixel for an image compression model (Ballé et al.) (017) and a supervised compression model, Entropic Student (Matsubara et al. 2022c) and plot the differences (bottom row). Blue and red areas indicate
4

---

Published in Transactions on Machine Learning Research (06/2023)
pixels for which the neural image compression model allocates more and fewer bits to each pixel than the supervised compression model does, respectively.
In this study, we put our focus on supervised compression for split computing (SC2) and present the details of our SC2 benchmark and experiments in Sections and 4, respectively.
2.3
Motivations
In this section, we discuss the motivation behind the SC2 benchmark study.
2.3.1
Issues in Evaluation Metrics
As described in Section 2.1.2 split computing has been attracting a growing interest from the research community. Contributions from the machine learning community (Singh et al., 2020, Matsubara & Levorato 2021] Matsubara
iet al.
2022c Assine et al. 2022 Datta et al. 2022; Ahuja et al. 2023 specifically aim at improving the tradeoff
between compressed data size and model accuracy. Many of such contributions leverage ideas and techniques widely
used in neural image compression techniques such as reparameterization trick (Kingma & Welling 2014), quantization
with entropy coding Wintz (1722; Netravali & Limb (1800), and rate-distortion autoencoders (Ballé al. 2017).
Such studies from the machine learning community still heavily rely on rate-distortion evaluation metric that is popular in the neural image compression community (Ballé et al. 2017 2018 Minnen et al. 2018 Minnen & Singh 2020 Yang et al., 2020bb a Yang & Mandt 2023). However, the rate-distortion metric does not consider one of the essential aspects to determine the success of these techniques in real- world systems, that is, the cost of encoding given the heavily asymmetric computing power of mobile devices and edge servers. As emphasized in Section [2..2] it is important reduce both the encoding cost (local computing cost on mobile devices) and size of data to be transferred from weak mobile device to powerful cloud/edge server.
For instance, Singh et al. (2020): Datta et al. (2022): : Ahuja et al. (023)) discuss the supervised rate-distortion tradeoff for their approaches in the context of image classification, object detection, and/or semantic segmentation tasks. To achieve better supervised rate-distortion tradeoff, those studies introduce bottlenecks (splitting points) to existing convolution neural network models almost at the end of their layers - c.g., the penultimate layers or last convolution layers/blocks in the original models, which results in approximately 60-170 times larger encoder size than encoder size of Entropic Student (Matsubara al. 2022c) (see Section 4.3 Since those are discriminative models, it should be easier compress data with respect to the input data (e.g., images) when introducing bottlenecks at the later stage of the models. As result, such approaches will put most of the model's inference cost weak mobile devices to compress data, strategy that increases total execution time and imposes high energy consumption.
The neural image compression community (Ballé et al. 2017 2018 Minnen et al., 2018 Minnen & Singh 2020) Yang et al. 2020b Singh et al. (2020); Datta et al. (2022); Ahuja et al. (2023) refers to bits per pixel (BPP) as rate as part of the rate-distortion tradeoff evaluation. Conversely, split computing studies such as (Matsubara et al.) 2019:
Matsubara Levorato| 2021| Assine al. 2022, Haberer & Landsiedel (2022), instead, focus on the actual data size of compressed representations. Reducing the data size directly decreases the data communication time between the mobile device and the edge server, while BPP does not so necessarily (e.g., larger data size can result in small BPP if the image has more pixels). In order to improve the effectiveness of split computing at runtime, the performance evaluation should focus on the actual data size of compressed representations in SC2 problems.
To address those issues, this study we define evaluation criteria for SC2 in Section 3.1 and propose how to incorpo- rate them into existing rate-distortion tradcoff's in Section 3.2
2.3.2 Lack of Comprehensive Benchmark
Besides the issues in evaluation metrics for SC2 (Section 2.3.1 we point out that comprehensive discussion many important aspects of SC2 is lacking in the machine learning community. We report some examples in the following:
Multiple target tasks: Currently, supervised compression studies (Matsubara Levorato 2021 Singh et al. 2020 Matsubara et al. 2022c Yuan et al. (002) discuss the performance of their methods for different tasks, using different evaluation metrics. In order to highlight the differences between the proposed methods, it is
5

---

Published in Transactions on Machine Learning Research (06/2023)
essential to discuss the performance shared set of tasks. e.g., does method consistently outperform other methods on all the tasks?
Bottleneck placement: For efficient inference in split computing, it is critical to reduce both encoding cost and data size of compressed representations. Moreover, determining where to introduce bottlenecks in model is a relevant question as it influences the allocation of computing load in the system. These aspects and tradcoffs should be discussed in state of the art supervised compression methods, also providing comparison with image- codec-based feature compression baselines (Alvar & Bajic 2021
3. Variety of reference models: Image classification models play an important role they are used as backbones for complex computer vision tasks such object detection and semantic segmentation (e.g.. ResNet (He et al. 2016) as backbone for Faster R-CNN with FPN (Ren et al. 2015; Lin et al.) 2017a) and DeepLabv3 (Chen et al.
2017bb respectively). There is a trend in both supervised compression and split computing studies (Eshratifar
et al. 2019b. Matsubara et al. 2019 Matsubara & Levorato 2021; Singh et al., 2020, Matsubara et al. 2022ct Yuan et al. 2022) introduce bottlenecks to some existing models (referred to as reference models in this study) rather than design new models with bottlenecks from scratch. Besides CNN models, there is an increasing interest in Vision Transformer (ViT) (Dosovitskiy al. 2021) in the machine learning and computer vision communities. Thus, investigating the effect of reference model choice for SC2 should be an interest of the communities.
4. More sophisticated encoder-decoder: Several studies from the neural image compression community (Ballé jet al.] 2018 Minnen et al. 2018; Minnen Singh| 2020) show that sophisticated encoder-decoder architec- tures such as a hyperprior significantly outperform simpler encoder-decoder architectures like factorized prior (FP) (Ballé et al. 2018) in terms of rate-distortion tradeoff for image compression tasks (unsupervised com- pression). For supervised compression, Matsubara et al. (2022c) use a FP-based encoder-decoder architecture. Following the study, Yuan et al. (2022) propose a hyperprior- based architecture, but do not compare its per- formance to that of the FP-based encoder-decoder. In addition to the choice of reference models, should be important to discuss the impact of encoder-decoder architectures the context of the SC2 problem.
Addressing the issues in SC2 evaluation metrics (Section 2.3. we tackle each of them through experiments with
strong input and feature compression baselines in Sections 4.1
4.5
3 Evaluation and SC2 Benchmark
In this section, we describe our evaluation criteria in detail. This includes concise definitions of the supervised rate- distortion tradeoff, the proposed tradeoffs between rate, distortion, and computing load, well as selection of baselines. Experimental results will be presented in Section
3.1 Evaluation Criteria
In split computing, the following three components typically considered: low-powered local (mobile) device, capacity-constrained network, and an edge (cloud) server. The overall goal is to distribute neural network M such that the first layers are deployed on the local device, and the remaining layers are deployed on the cloud/edge server. The portion of the model deployed on the mobile device considered an "encoder" because it compresses the data into representation suitable for transmission to the edge server. Given the asymmetric nature of the system, the complexity of the encoder should be minimized, while the remaining part of the model should comprehend most of the computing load, as the edge server is assumed to have larger computing power compared to the mobile device. We remark that the training process is performed offline, and the split computing is executed at runtime only. Training split deep neural network (DNN) models across multiple devices (Gupta & Raskar 2018) (or split learning (Vepakommal et al. 2018)) is a different problem and out of the scope of this study.
As described Section 2.2 we define supervised compression as learning compressed representations for supervised downstream tasks such as classification, detection, or segmentation. The design should aim for three criteria: high supervised performance (low distortion), high compressibility (low rate), and minimal encoder size on the local device (low computing load). We discuss these three aspects in more detail.

---

Published in Transactions on Machine Learning Research (06/2023)
3.1.1 Supervised Distortion
The target metric highly depends on the supervised task; in the simplest case, it could be a form of accuracy, e.g.. in classification. In this paper, we study three applications of supervised compression involving classification, object detection, and semantic segmentation. In these cases, we consider the supervised distortion to be accuracy, mean average precision (mAP), and mean intersection over union (mloU), respectively. Note that compressing intermediate model representations typically requires a discretization step at the bottleneck layer. These supervised distortions are therefore computed after such intermediate discretizations.
3.1.2 Compressed Data Size (Rate)
Rate is defined as the average file size per datum after compression. In the neural compression literature (Ballé et al. 2017] 2018, Minnen al. 2018 Yang al. 2020b, SSinhh et al. 2020), most studies focus on bits per pixel (BPP), defined as the number of bits in the compressed representation divided by the input image size. In this paper, report the rate on the basis of data points since this measures the actual amount of data sent to the edge server. Frameworks should penalize large amount of data transferred from the mobile device to the edge server. Notably, BPP does not penalize absolute amount of data, for instance when feeding higher resolution images to downstream models for achieving higher model accuracy (Touvron et al. 2019). The rate could either be directly related to the size the raw feature representations rounded certain arithmetic precision or result from an additional entropy coding step.
3.1.3 Encoder Size
In addition to minimizing rate and distortion, it is also critical to minimize local processing cost as mobile devices usually have battery constraints and limited computing power (Matsubara et al., 2020 2022c). To estimate the local processing cost, FLOPS (floating point operations per second) and MAC mnultiply-accumule are often used. However, FLOPS is not a static value, and FLOP/MAC not well-defined in practice As a simple proxy the computing cost, we measure the number of model parameters, a static value that widely used discuss model complexity (He al. 2016 Huang et al., 2017; Devlin al., 2019; Matsubara et al. 2022c We define the encoder size Esize the total number of bits needed to represent the parameters of the encoder:
(Zhang et al. 2019)
Esize
Σ EE|
#bits(O,),
(1)
where θ is a set of the encoder's parameters, and #bits(-) indicates the number of bits for its input. For comparison, we also demonstrate the usage of encoder FLOPS in Appendix E
3.2 Supervised R-D Tradeoff and Three-way Tradeoff
In lossy data compression (Yang et al., 2023b), one typically studies the tradeoff between the rate and distortion (R-D) of compression scheme, L..., the quality degradation as function of file size (Ballé al., 2017 2018, Minnen et al. 2018 Singh et al. (020). In analogy to this, split computing approaches typically consider a supervised R-D tradeoffs (Matsubara et al. 2020 Matsubara & Levorato| 2021:(Sigg al. 2020 Matsubara et al. 2022c and replace the reconstruction distortion with the supervised distortion defined above.
The conventional supervised rate-distortion (R-D) tradeoff does not consider the encoder size. This approach is rea- sonable as long as models are compared with similar encoder sizes. However, without restrictions on the encoder size, the supervised compression problem becomes trivial. For example, encoder could simply carry out the classification task on the local device and only compress the label, leading cheap log( log(K) bit representation of the transmitted features. If the bottleneck layer is deployed close to the output layer, similarly small compression costs can be achieved (Singh et al., 2020 Datta et al., 2022] Ahuja al. 2023). However, Matsubara et al. (2022c) emphasize the importance of minimizing the encoder size for achieving efficient split computing.
K-class elassification setup, a powerful
To also take the encoder size into account, we propose to analyze the three-way tradeoff between encoder size, data size, and supervised distortion. Naturally, not all criteria can be simultaneously fulfilled: upon choosing a lighter
FEgg https:/t///// pectps://detectreee readtn2eeeeeddeeeeeeecoo httpstttaddeee.eeeee//aatattaten
htm11 Fycere. nn. FlopCountAnalyst

---

Published in Transactions on Machine Learning Research (06/2023)
Table I: List of methods in this study. IC: Input compression, FC: Feature compression, SC: Supervised compression.
Name (Acronym)
Type
Description
JPEG
Standard lossy image codec
WebP
IC
Classical lossy image codec (Google)
BPG
Statc-of-the-art classical lossy image codec Bellard
FP
IC
Factorized prior (Ballé et al.
2018
SHP
IC
Scale hyperprior (Ballè et al.
(2018)
MSHP
IC
Mean-scale hyperprior (Minnen al.
2018
JAHP
IC
Joint autoregressive hierarchical prior (Minnen et al.
2018
JPEG
FC
JPEG- based intermediate feature compression AAvar & Bajić
2021
WebP
WebP-based intermediate feature compression (Alvar Bajic
2021
CR + BQ
SC
Channel reduction & bottleneck quantization (Matsubara & Levorato
2021
Compressive Feature
SC
End-to-end learning of compresive feature (Singh el al. 2020 Yuan al.
2022)
Entropic Student
SC
Multi-stage fine-tuning with neural compression and knowledge distillation Matsubara et al. 2022c
weight encoder, we naturally have a less compressible bottleneck representation or a drop in accuracy. In Section + we visualize several such tradeoff curves. To simplify plotting and ease model selection, we also propose another tradcoff that takes encoder size and data size multiplicatively into account. We term this quantity ExR-D tradeoff, where we plot the distortion as a function of the product of encoder size and data size (see Section
3.3 Baselines
In this section, we discuss relevant baselines and categorize them as either input compression, feature compression or supervised compression methods. All these baselines and the corresponding acronyms are summarized in Table
3.3.1 Input/ Feature Compression Baselines
In conventional implementations of the edge computing paradigm for computer vision tasks, compressed images are transmitted to the edge server, where all the tasks are then executed. We consider seven baselines in this study referring to the "input compression" scenario that can be categorized into either codec-based or neural input compression.
Classical Image Compression. A first approach relies on using off-the- sheflff-sheff classical image compressors. We evalu- ate each model's performance in terms of the rate-distortion curve by setting different quality values for three codec- based input compression methods: JPEG, WebP (Google), and BPG (Bellard). We use the implementations Pillow and investigate the rate-dia disttriiooiion (R-D) tradeoff for the combination of the codec and pretrained downstream models by tuning the quality parameter in the range of 10 to 100. Since BPG is not available in Pillow, our implementation follows (Bellard), and we tune the quality parameter in the range of 0 50 to observe the R-D curve. We use the x265 encoder with 4:4:4 subsampling mode and 8-bit depth for YCbCr color space, following (Bégaint et al. 2020).
We also introduce codec-based feature compression basclines Alvar & Bajic 2021) (see Section 4.3
Neural Image Compression. As an alternative, we consider state of the art neural image compressors (allé et al. 2018 Minnen et al. al, 2018 Minnen & Singh] 2020) (see (Yang al. 20235) for recent survey). We adopted the neural image compression models whose pretrained weights were available CompressAl (Bégaint al., 2020). These models mainly rely on variational autoencoder architectures and differ in terms of their entropy models (priors) that can have large effect the achievable code lengths. Without going into detail, these models are known under the names of "factorized prior" (Ballé et al.) 2018). "scale hyperprior" (Ballé et al. (018). "mean-scale hyperprior" (Minnen)
et al. 2018), and "joint autoregressive hierarchical prior" (Minnen et al. 2018).
3.3.2 Supervised Compression Baselines
Another group of baseline models originates from frameworks prior to split computing. We broadly divide them into the following three categories.
httpss//pytton-ppt ll...rg/
8

---

Published in Transactions on Machine Learning Research (06/2023)
Channel Reduction Bottleneck Quantization. These split computing baselines (Matsubara et al. 2020 Shao &
Zhang 2020 Matsubara & Levorato 2021 Dong et al. 2022) correspond to reducing the bottleneck data size with
channel reduction and bottleneck quantization: we hence denote them as CR+BQ. These methods quantize 32-bit floating-point to 8-bit integers (Jacob et al. 2018). Matsubara et al. (2020); Matsubara & Levorato (2021) report that post-training bottleneck quantization did not lead to significant accuracy loss. Following (Matsubara & Levorato
2021), we modify these pretrained models and introduce bottlenecks with a different number of output channels in convolution layer to control the bottleneck data size. Using the original pretrained model teacher model, we train the bottleneck-injected model (student) by generalized head network distillation (GHND) and quantize the bottleneck after the training session.
End-to-End Supervised Compression. As an instantiation of information bottleneck framework (Alemi et al. 2017). Singh et al.] (2020) first propose an end and-to-end supervised compression method with an entropy bottleneck for image classification tasks, and Yuan et al. (2022) apply a similar idea to object detection tasks. [Singh et al.) (2020)'s 's approach focuses on a single task and introduces the compressible bottleneck to the penultimate layer. In the considered setting, such design leads to overwhelming workload allocated mobile devices: for example, in terms of model parameters, about 92% of the ResNet-50 (He et al. 2016) parameters would be deployed on the, weaker, mobile device. To make this approach compatible with SC2 setting, we apply their approach to entropic stu- dent models without a teacher model. We find that compared to (Singh et al. 2020), having stochastic bottleneck at an earlier layer (due to limited capacity of mobile devices) leads to a model that is harder to optimize (see Section 4.1
Entropic Student. Our final baseline in this paper is Entropic Student (Matsubara et al. 2022c), a two-stage fine- tuning method that combines the concepts of neural image compression and knowledge distillation (Hinton et al. 2014). At the first stage of the fine-tuning, only the encoder-decoder in the student model is trained to mimic interme- diate representations of its teacher model. The second stage of the method freezes the parameters of its encoder and fine-tune decoder and all the subsequent layers for the target tasks so that the encoder can be reused for other tasks.
For the end-to-end supervised compression and Entropic Student methods, we individually train the same model archi- tectures (including its encoder-decoder) with each of the two training methods. Following (Matsubara et al.) 2022c), we design the encoder with convolution and GDN (Ballé et al. 2016) layers followed by a quantizer described in Appendix B Similarly, the corresponding decoder is designed with convolution and inversed GDN (IGDN) layers to have the output tensor shape match that of the first residual block in ResNet-50 (He et al.] (016). For image classi- fication, the entire architecture of the model consists of the encoder and decoder followed by the last three residual blocks, average pooling, and fully-connected layers in ResNet-50. For object detection and semantic segmentation. we replace ResNet-50 in Faster R-CNN (Ren et al. 2015) and DeepLabv3 (Chen et al. 2017b) with the student model for image classification.
3.4 Choice of Datasets
We use image data with relatively high resolution, including ImageNet (ILSVRC 2012) (Russakovsky et al. 2015). COCO 2017 (Lin et al. 2014), and PASCAL VOC 2012 datasets (Everingham et al., 2012). As pointed out in (Mat- subara et al. 2022b), split computing is mainly beneficial for supervised tasks involving high-resolution images e.g., 224 x 224 pixels or larger. For small data samples, either local processing or full offloading often achieve better operating points in the three way tradeoff compared to split computing 51
3.5 Python Package sc2bench-
To facilitate research on supervised compression for split computing (SC2), we publish an installable Python pack- age named sc2bench (i.., pip install sc2bench) and scripts to reproduce the experimental results reported in this phpsr Python package is built on PyTorch (Paszke et al. 2019) and torchdistill (Matsubara b for repro- ducible SC2 studies, using CompressAl (Bégaint et al. (2000) and Py Torch Image Models (Wightman 2019 for neural compression modules/models and reference models, respectively. Our Python package offers various supervised com- pression models, modules and functions for further studies on SC2, and our repository provides the implementations of our baseline models and training methods, including weights of the models we trained in this study.
Fg.g. the average data sizes of 32 32 pixels images for MNIST LeCun et only 0.966 KB and 1.79 KB, respectively.
1998 (Gray scale) and CIFAR (Krizhevsky 2009 (RGB)

---

Published in Transactions on Machine Learning Research (06/2023)
IPRCI
HPG [Belaoo]]
SR2 2nd
[Miuuass 2018]
Conpecuiie Foalace
-0-- 10: WebP (Coogh) -Ct- IC: FP [Balle ct al 20.5 --p- IC: MSRF [Minion et al 2018
SC:CR BC) Manachere & Levocato 2123
SCh Eutrock: Sturken (NacsEbere ot al. 202
75
75-
70
70
Acmiiaa aomiaaeyy (%%
Anaracy 195)
Ascraey [36)
65-
-05
60
60-
Encnder Suz (N Byrte
1o0gg
45
50
50
1tft
GOODD
Data Size [KBind
18tt
20000
45
100
-0
45
10
10
-0'
10
jor
Data Size [KByte]
Eneoder Sizee [KBstt]] x Data Siee [KBrz]]
Figure 3: SC2 for image classification on ImageNet (ILSVRC 2012). We show the supervised R-D tradeoff (left). the ExR-D tradeoff (middle), and the full three-way tradeoff (right). In all cases, used ResNet-50 our reference model. Grey lines denote projections. Entropic Student performed best in R-D and ExR-D performance.
4 Experiments
We empirically assess state-of-the-art codec-based/neural input compression, feature compression, and supervised compression baselines (Table [ according to the criteria introduced in Section [ We thereby consider the three supervised tasks of image classification, object detection, and semantic segmentation.
4.1 Image Classification
We first discuss the rate-distortion performance of our baselines using large-scale image classification dataset. Specifically, we use ImageNet (ILSVRC 2012) (Russakovsky et al. 2015). that consists of 1.28 million training and 50,000 validation samples. Using ResNet-50 (He et al. 2016) as a reference model, we train the models on the training split and report the top-1 accuracy on the validation split. Appendices and D describe the details of the training configurations, including hyperparameters.
Figure 3 (left) presents supervised rate-distortion curves of ResNet-50 with various compression approaches, where the and y-axes show the expected compressed data size and the supervised performance (accuracy), respectively. For image compression baselines, we considered factorized prior (FP) (Ballé et al. 2018), scale hyperprior (SHP) (Ballé et al. 2018), mean-scale hyperprior (MSHP) (Minnen et al. 2018), and joint autoregressive hierarchical prior models (JAHP) (Mmnen et al. 2018), as well as JPEG, WebP, and BPG codecs. The combination of channel reduction and bottleneck quantization (CR+BQ) (Matsubara Levorato 202T] a popular approach used in split computing studies comparable to JPEG codec in terms R-D tradeoff, but neural image compression models performed better.
Among all the baselines considered, the Entropic Student trained by the two-stage method performed the best. To test the effect of knowledge distillation, we also trained the Entropic Student model without teacher model, which in essence corresponds to (Singh et al. 2020) with an adjusted bottleneck placement. The resulting R-D curve is signif- icantly worse, which we attribute to two possible effects: first, it is widely acknowledged that knowledge distillation generally finds solutions that generalize better. Second, having bottleneck at an earlier layer may make it difficult for the end-to-end training approach without a (pretrained) teacher model to optimize as empirically shown in (Matsubaral
et al. 2020 2022a). We explore the effects of the bottleneck placement in Section
Besides the tradeoff between data size and accuracy, we also investigated tradeoffs incorporating the encoder's model size. Figure 3 (middle) shows the proposed ExR-D tradeoff, and the full diree-dimensionall tradeoff is shown in Fig. 3 (right). Both the figures reveal that when considering encoder size, Entropic Student outperformed the other baselines even more significantly since the encoders of the split DNN models are significantly smaller than those in the neural input compression models. Entropic Student's 's encoder is approximately 40 times smaller than the encoder of the mean-scale hyperprior and can therefore be deployed efficiently on mobile devices. As shown in (Matsubara et al., pipeline (Fig. [ than
2022c), the model also can achieve much shorter latency to complete the the input compression baselines considered for resource-constrained edge computing systems.
10

---

Published in Transactions on Machine Learning Research (06/2023)
IC:JPPRC
ICh BPG (Relaeee]
- 30 SHP [allE d. 2018
----
10: JAHP (Monets al. 2008)
NC1 Comperoiee Poaiane [Sigh 2021
[Cioogle]
I0: Belli 2018) -Pp- 'C" MSHP [Nlmm
2015
CP. PQQ [Mieberltens Tavorado 2001
Entrriic Strdent [Micarieers dl X02
38
36
34
TAA [36]
32
10AP
300
30
Eacoder Size BK to
20 10 10t 16 10 Duta Stee (KByre]
26
24
10P Data Size [KByte]
24
101
10P
10
16
Eneoder Sizee [KBsto] x Data Size [KBrre]]
Figure SC2 for object detection on COCO 2017. We show the supervised R-D tradeoff (left), the ExR-D tradeoff (middle), and the full thy tradeoff (right). In all cases, used Faster R-CNN with ResNet-50 and FPN reference model. Grey lines denote projections. Entropic Student performed best in R-D and ExR-D performance.
IC-11213
IC: [Aelaad] - J0 SHP Tailé
JAHP 200) 19C UR BQ [blatsubbrn de Levornto 2031]
SC, Comporadiv Frature
10: Wild (Gooogh - IC: FP? Balle 1l. 2:18] - 302 MEHP [alnme al. 2018
SCC Entrook: Studenn [Nocsubers or al 222]
80-
80
76
78
ml00U
74
24-
72
Encoder Sten (ICB510) AORINO
000000
CORHO
o0 up of Data Size KKhyto
20003
10
10
10
10
10
Deta Size [[Byte]
Encoder Size [KTytec] x Data Size [KByte]]
Figure SC2 for semantic segmentation on PASCAL VOC 2012. We show the supervised R-D tradeoff (left), the ExR-D tradeoff (middle), and the full three-way way tradeoff (right). In all cases, used DeepLabv3 with ResNet-50 our reference model. Grey lines denote projections. Entropic Student performed best in R-D and ExR-D performance.
4.2 Object Detection and Semantic Segmentation
We further study the rate-distortion performance on two downstream tasks: object detection and semantic segmenta- tion, reusing the models pretrained on the ImageNet dataset. As suggested by et al. (2019), such pre-training speeds up the convergence for other tasks (Russakovsky et al., 2013). Specifically, we train Faster R-CNN with FPN (Ren) al. 2015 Lin et al., 2017a) and DeepLabv3 (Chen et al. 2017b) for object detection and semantic segmentation, re- spectively, using the models pre-trained on ImageNet in Section 4.1 for the supervised compression baselines. Faster R-CNN is a two-stage object detection model; generates region proposals and classities objects in the proposed regions. DeepLabv3 is popular semantic segmentation model (Chen al. 2017a).
For object detection, we use the COCO 2017 dataset (Lin et al., 2014) to fine-tune the models. The training and validation splits in the COCO 2017 dataset have 118,287 and 5,000 annotated images, respectively. For detection performance, we refer to mean average precision (mAP) for bounding box (BBox) outputs with different Intersection- over-Unions (loU) thresholds from 0.5 and 0.95 on the validation split. For semantic segmentation, we use the PAS- CAL VOC 2012 dataset (Everingham al.) 2012) with 1,464 and 1,449 samples for training and validation splits, respectively. We measure the performance by pixel IoU averaged over its 21 classes. It is worth noting that following the PyTorch (Paszke et al., 2019) implementations, the input image scales for Faster R-CNN (Ren et al. 2015) are defined by the shorter image side and set to 800 in this study which is much larger than the input image in the previous image classification task. For DeepLabv3 (Chen et al., (0176)), we use the resized input images such that their shorter size is 513. The training setup and hyperparameters used to fine-tune the models are described in Appendices C and
Figures 4 and show the results for object detection and semantic segmentation, where the left figure shows the su- pervised R-D tradeoff. Compared various split computing and input compression approaches, the Entropic Student approach demonstrates better supervised R-D curves in both the tasks. In object detection, Entropic Student's im-
11

---

Published in Transactions on Machine Learning Research (06/2023)
E3 beer! (Boohes32) F3 1ear2 (RooNet-92)
RB legae (Rodict-di) legerd (Rondrt--0))
-age 2B at arrind (BooNes-20) (Snggee al 2013
Tatmpic Stoderk [Madonieem al. 2002)
80
70
70
Anaraay [X]
Acsarayy [95]
Eacoder Size By Abvemaaae
00 71
60-
60-
50
50
41
00-
40
VILLO
Data Size (KByt
10X0
30
10
10 10
10
-30
10
10
10"
10
10
Data Size [KBBtte]
Eneoder Sizee [KDstto] x Data Size [KBrze]
Figure 6: Entropic Student vs. Re with entropy bottleneck (EB) introduced to different layers. Simply intro- ducing EBs its late layers e.g., layer4 and avgpool improved the conventional R-D tradeoff (left), which results in most of the layers in the model be deployed weak local devices. Our proposed ExR-D and three-way tradeoffs penalize such configurations (middle and right).
provements over BPG and JAHP are smaller than those in the image classification and semantic segmentation tasks, However, as shown in Figs. 4 and [ (middle and right), the proposed ExR-D and three-way tradeoff's show that the combined improvements in encoder size and data size reductions are even more significant.
4.3 Bottleneck Placement
An important design decision in split computing is the choice of the layer where the DNN is split. If the DNN is split at an early layer, the computation on the weak local device will be lightweight, but the learned representation at the splitting point is not easily compressible. When splitting the DNN close to the output, the latent representation contains only relevant information for the supervised task and is therefore compressible, but most of the computation is carried out on the weak local device.
The proposed ExR-D tradeoff considers both the encoder size and the data size of the latent representation. It can guide selecting bottleneck layer that simultaneously leads compressible representation and a lightweight encoder model. Figure 6 shows different R-D tradeoff curves, where we consider five ResNet-50 models that have entropy bottlenecks (EBs) placed at five different layers. Specifically, we introduce the EBs after the 1st, 2nd, 3rd, 4th residual blocks as well as at the penultimate layer; we refer to these models as layerl, layer2, 1ayer layer and avgpool respectively. Note that [Singh et al. (020) introduce an EB at the penultimate layer of ResNet, and Datta let al. ( Ahuja et al. (2023) introduce an EB at layer3 and layer4 of ResNet-50, which results in deploying approximately 60- larger encoder than that of Entropic Student (Matsubara et al. 2022c) on weak mobile devices. We introduce the EB to pretrained ResNet-50 and then fine-tune the model for better performance. For reference, we also add the entropic student model to the plot as it performed the best in Section 4.1
Figure 6 (left) shows that the supervised R-D tradeoff can be misleading for choosing the bottleneck layer, simply favoring models that place their bottlenecks at late layers. With such configurations, we would deploy at least 92% of the entire model on the weak local device. Clearly, in split computing scenarios, we would rather prefer doing most of the computation on the edge server. A complete picture seen on the ExR-D tradeoff (middle) and the three- way tradeoff (right), revealing that layerl would be the best placement for introducing bottlenecks (as done in the entropic student). These findings are not specific to learned entropy models but apply to conventional feature com- pression schemes well. To demonstrate this, we adopt the method of (Alvar & Bajić 2021) that used conventional image codecs to compress feature maps of a pretrained neural network. (Overall, this baseline leads to much worse compression rates, which is why we excluded it in our main experiments.) Thus, we implement a similar baseline by replacing the EB in a pretrained ResNet-50 with JPEG and WebP codecs. To be specific, we treat the network feature maps as a concatenation of 3-channel "sub-images" which can be compressed separately. analogy to Fig. 6 we study the same bottleneck placements. Figure 7 shows the R-D, ExR-D, and three-way tradeoffs of the corresponding method, mirroring the findings of the previous discussion.
12

---

Published in Transactions on Machine Learning Research (06/2023)
PC: IPGG letri
FC: JPEC laser mwe FC: WelP lenzz
FC: IPEG ot lopreS
DC: IPGG at leeel
FC: JPBB rospoel
Ennrople Student
PC: WebP at lowrl
PC; Web2 an layer
- PC; Webl as loxe
-dd PC: WelP anpooil
80
80
TO
Anaracy [X]]
60
60
50-
Acsrany
50-
00
40
Eacoder Size (K to]
10
33-
20
20
Data Stee Byr
40000
xP
10
10
10
10
10
10
10
10
10
209
10
10
Data Size [KTBtte]
Eneoder Size [KBste] x Data Size [KBrae
Figure 7: Entropic Student vs. Re with bosec-based feature compression approaches (instead of EB in Fig,
introduced to different layers. Overall, their data sizes are orders of magnitude larger than those of the supervised compression in Fig. 6
Sputtable Baddd-20 FP based)
Sptitable ResNie. (22-Uaso))
Splizaake ReaNett 6.10P (2P-laned)
Spittaaiee Refuri VIE R.2--5/022 22 (PP.ba Spattabie Heftond VIT 122218/222 228pp (MSSE beso))
Sputmable BoaSec 20 MMHHP boced) ---- Sptiitable Res HI. (MSEP P basedd - Sptiitatie Regfett 6.U3F (Miatt base)) -
Encoder Size BKiBtte Aesuusss (%)
Ameuasy (96)
75-
Acetrwyy [96]
75
50
70-
77
60
65
65-
53
val Data Size [kBrrr los ti Jdig
60
60
65
100
10
55
10
10
Data Size [KByte]]
Erwoder Sixe [KByte]] x Data Size [KTyyee]
Figure 8: Conventional R-D (left), our proposed ExR-D (middle), and three-way tradeoff's (right) for Entropic Student with various reference models and encoder-decoder. Different colors indicate different reference models. Stronger reference models outperform Splittable ResNet-50, the best Entropic Student model in Fig. 3 in terms of R-D, ExR- D, and three-way tradeoffs. The encoder-decoder design well generalizes so that can work with various reference models including Hybrid ViT (Steiner et al.) 2022). transformer-batt model. The more sophisticated (MSHP- based) encoder-decoder design further improved R-D tradeoff at cost of significantly increased encoder size.
ExR-D
4.4 Network Architecture Ablations
To further improve the performance of models for split computing without increasing the encoder size or data size, we put our focus on Entropic Student and investigate the effect of reference models. This section investigates alter- natives to ResNet-50 as our default reference model and in particular considers ResNet-101 (a larger model in the family)
2016). RegNetY-6.440 (Radosavovic et al. 2020), and Hybrid Vision Transformer (ViT) R26+S/32 (224px) (Steiner et al. p022p as reference models. All these models use the same input patch size of 224 224 and the same encoder-decoder architecture.
As shown in Fig. (left), we significantly improved the R-D tradeoff by replacing ResNet-50 with the more accurate reference models. Since we reused the encoder-decoder design in Section 3.3.2 for the configurations i.e., comparable encoder size and data size, we successfully improved the ExR-D and three-way tradeoff's as shown in Fig. g (middle and right). As most of the layers are deployed on the edge server (which we assume is not resource-consterrr constrained), using such more accurate reference models does not significantly affect the architecture's latency. Therefore, we can improve the ExR-D and three-way tradeoffs as long as the decoder can still learn suitable feature representations.
4.5 Supervised Compression with a Hyperprior
In this section, we discuss whether we can improve the supervised R-D tradeoff by adopting a more powerful entropy model for compressing the latent variables losslessly. To this end, we draw the neural compression literature,
13

---

Published in Transactions on Machine Learning Research (06/2023)
where adopt hierarchical design in which the latent representations 2 compressed using entropy model that relies on additional hyperlatents 24-- While the supervised encoder-decoder design in Entropic Student (see Sec- tion 3.3.2) is based on the factorized prior (FP) model of image compression (Ballé et al.) 2018), the hyperprior approach draws on the mean-scale hyperprior (MSHP) model from (Minnen et al., 2018). The hyperprior can be expected to reduce the data size since the distribution of can be better approximated.
In Fig. 8 (left), we confirm that the MSHP-based encoder-decoder consistently improved the R-D tradcoff compared to those with FP-based encoder-decoder. These gains more significant for simpler reference models Le., ResNet-50 and 101. We stress that this extra performance comes the cost of additional encoder complexity, which is reflected in the ExR-D curve that considers data size and encoder size jointly.
5 Conclusion
In this paper, we investigated supervised compression for split computing (SC2), where a machine learning model is split between low-powered device and a much more powerful edge server. We explored optimal ways of splitting the network while aiming for high supervised performance, high compression rates, and low computational costs the edge server. We introduced SC2 Benchmark, a new benchmarking framework of supervised compression for split computing to more rigorously analyze this setting and the various tradeoffs involved. Specifically, we investigated a variety of input/feature compression models, supervised compression models, supervised tasks (such as image clas- sification, object detection, and semantic segmentation), training schemes (such as knowledge distillation), metrics (such as ExR-D and three-way tradeoffs). and architectures (convolutional or Vision Transformers). Altogether, this study involved more than 180 trained models. We showed that Entropic Student (Matsubara et al.) 2022c), super- vised compression model inspired by neural image compression models (with or without hyperpriors), trained in multi-stage knowledge distillation approach, performed best in terms of the supervised R-D, ExR-D, and three-way tradeoffs for the three supervised tasks considered in this study. Hoping that our benchmark will set the stage for a more rigorous evaluation of supervised compression methods and split computing models, we publish s sc2bench, pip-installable Python package to lower the barrier to SC2 studies. We also release our code repository based on sc2bench to offer reproducibility of the experimental results this study.
Acknowledgments
We acknowledge the support by the National Science Foundation under the NSF CAREER award 2047418 and Grants 1928718, 18, 2003237, 2007719, IIS-1724331 and MLWINS-2003237, the Department of Energy, Office of Science under grant DESC002233, the IARPA WRIVA program, as well as Disney, Intel, and Qualcomm.
References
Ferran Adelantado, Xavier Vilajosana, Pere Tuset-Peiro, Borja Martinez, Joan Melia-Segui, and Thomas Watteyne. Understanding the Limits of LoRaWAN, IEEE Communications Magazine, 55(9):34-40, 2017.
Eirikur Agustsson, David Minnen, Nick Johnston, Johannes Balle, Sung Jin Hwang, and George Toderici. Scale-space flow for end-to-end optimized video compression. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8503-8512, 2020.
Nilesh Ahuja, Parual Datta, Bhavya Kanzariya, V Srinivasa Somayazulu, and Omesh Tickoo. Neural Rate Estimator and Unsupervised Learning for Efficient Distributed Image Analyties in Split-DNN Models. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2022-2030, 2023.
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep Variational Information Bottleneck. In International Conference on Learning Representations, 2017.
Saeed Ranjbar Alvar and Ivan V Bajić. Pareto-Optime Bit Allocation for Collaborative Intelligence. IEEE Transac- tions on Image Processing, 30:3348-3361 2021.
14

---

Published in Transactions on Machine Learning Research (06/2023)
Juliano S Assine, JCS Santos Filho, and Eduardo Valle. Collaborative Object Detectors Adaptive to Bandwidth and Computation. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2839-2843. IEEE, 2022.
Johannes Ballé, Valero Laparra, and Eero P Simoncelli. Density Modeling of Images using Generalized Normaliza- tion Transformation. In International Conference on Learning Representations, 2016.
Johannes Ballé, Valero Laparra, and Eero Simoncelli. End-to-end Optimized Image Compression. International Conference on Learning Representations, 2017.
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with scale hyperprior. In International Conference on Learning Representations, 2018.
Amin Monitalebi-Dehkordh Naveen Vedula, Jian Pei, Fei Xia, Lanjun Wang, and Yong Zhang. Auto-Split: A General Framework of Collaborative Edge-Cloud AI. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 2543-2553, 2021.
Jean Bégaint, Fabien Racapé, Simon Feluman, and Akshay Pushparaja. CompressAI: a PyTorch library and evaluation platform for end-to-end compression research. arXiv preprint (xXiv:2011.03029 2020. |tttps://github com/ CenteerDiiitaann/CCCoooo
Fabrice Bellard. BPG Image format. https: http://eelaaadrrreee. [Accessed on February 18, 2023].
Jiasi Chen and Xukan Ran. Deep Learning With Edge Computing: A Review. Proceedings of the IEEE, 107(8): 1635-1674, 2019.
Liang-Chied Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. DeepLab: Seman- tic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. IEEE Transactions on Paternn Analysis and Machine Intelligence, 40(4):834-84) 2017a
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking Atrous Convolution for Semantic Image Segmentation. arXiv preprint arXiv: 2017. 2017b.
Zhibo Chen, Tianyu He, Xin Jin, and Feng Wu. Learning for Video Compression. IEEE Transactions on Circuits and Systems Video Technology, 30(2):566-576, 2019.
Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. Learned Image Compression With Discretized Gaussian Mixture Likelihoods and Attention Modules. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7939-7948, 2020.
Hyomin Choi, Robert A Cohen, and Ivan V Bajić. Back-And-Forth Prediction for Deep Tensor Compression. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4467-4471. IEEE, 2020.
Jinyoung Choi and Bohyung Han. Task-aware quantization network for jpeg image compression. In European Con- ference on Computer Vision, pp. 309-324. Springer, 2020.
Parual Datta, Nilesh Ahuja, V Srinivasa Somayazulu, and Omesh Tickoo. A Low-Complexity Approach Rate- Distortion Optimized Variable Rit-Rate Compression for Split DNN Computing. In 2022 26th International Con- ference on Pattern Recognition (ICPR), pp. 182-188. IEEE, 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, 2019.
A. Djelouah, Campos, Schaub -Myer,, and C. Schroers. Neural inter-frame compression for video coding. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6420-6428, 2019.
15

---

Published in Transactions on Machine Learning Research (06/2023)
Xin Dong, Barbara De Salvo, Meng Li, Chiao Liu, Zhongnan Qu, HT Kung, and Ziyun Li. SplitNets: Designing Neu- Systems. In Proceedings of the IEEE/CVF
ral Architectures for Efficient Distributed Computing on Conference on Computer Vision and Pattern Recognition, pp. 12559-12569, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference Learning Representations, 2021.
Yann Dubois, Benjamin Bloem- Reedd, Karen Ullrich, and Chris Maddison. Lossy Compression for Lossless Pre- diction. In Neural Compression: From Information Theory to applications-Worksshan ICLR 2021, 2021.
Amir Erfan Eshratifar, Mohammad Saeed Abrishami, and Massoud Pedram. JointDNN: an efficient training and inference engine for intelligent mobile cloud computing services. IEEE Transactions on Mobile Computing, 2019a.
Amir Erfan Eshratifar, Amirhossein Esmaili, and Massoud Pedram. BottleNet: A Deep Learning Architecture for Intelligent Mobile Cloud Computing Services. In 2019 IEEE/ACM Int. Symposium on Low Power Electronics and Design (ISLPED), pp. 1 1-6, 2019b.
Mark Everingham, Luc Van Gool, CKI Williams, John Winn, and Andrew Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012). 2012.
Gergely Flamich, Marton Havasi, and José Miguel Hernández-Lobato. Compression without Quantization. In Open- Review, 2019.
Google. Compression Techniques WebP Google Developers, pttpsss//eeeelopaessoooooo..... on (Acessed on February 18, 2023].
webp/ /ttp//dccccccccceee
Otkrist Gupta and Ramesh Raskar. Distributed learning of deep neural network over multiple agents. Journal of Network and Computer Applications, 116:1-8, 2018.
Janek Haberer and Olaf Landsiedel. Activation Sparsity and Dynamic Pruning for Split Computing Edge AI. Proceedings of the 3rd International Workshop on Distributed Machine Learning, pp. 30-36, 2022.
Amirhossein Habibian, Ties van Rozendaal, Jakub M Tomezak, and Taco S Cohen. Video compression with rate- distortion autoencoders. In Proceedings the the IEEE International Conference on Computer Vision, pp. 7033-7042,
2019.
Jun Han, Salvator Lombardo, Christopher Schroers, and Stephan Mandt. Deep generative video compression. Advances in Neural Information Processing Systems, volume 32, 2019.
Dailan He, Ziming Yang. Weikun Peng, Rui Ma, Hongwei Qin, and Yan Wang. ELIC: Efficient Learned Image Com- pression With Unevenly Grouped Space-Channer Channel Contextual Adaptive Coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5718-5727, 2022.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Proceedings of the IEEE Interna- tional Conference on Computer Vision, pp. 2961-2969, 2017.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking ImageNet Pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4918 11117,997721 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in Neural Network. In Deep Learning and Representation Learning Workshop: NIPS 2014, 2014.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chied Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for Mob,leNetV3, In Proceedings of IEEE/CVF International Conference on Computer Vision, pp. 1314-1324, 2019.
16

---

Published in Transactions on Machine Learning Research (06/2023)
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely Connected Convolutional Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700-4708, 2017.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and Training Neural Networks for Efficient Integer-Arithmett--Onni In- ference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2704-2713, 2018.
Nick Johnston, Damien Vincent, David Minnen, Michele Covell, Saurabh Singh, Troy Chinen, Sung Jin Hwang, Joel Shor, and George Toderici. Improved Lossy Image Compression with Priming and Spatially Adaptive Bit Rates for Recurrent Networks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp.
4385-4393, 2018.
Yiping Kang, Johann Hauswald, Cao Gao, Austin Rovinski, Trevor Mudge, Jason Mars, and Lingjia Tang. Neuro- surgeon: Collaborative Intelligence Between the Cloud and Mobile Edge. In Proceedings of the Twenty-Second International Conference Architectural Support for Programming Languages and Operating Systems, pp. 615- 629, 2017. ISBN 978 1188-1-5503-- 778-1-4503-4465- doi: 10. 10.1145/3037697.3333600
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Third International Conference on Learning Representations, 2015.
Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes. In International Conference on Learning Representations, 2014.
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-Base Learning Applied to Document Recognition. Proceedings of the IEEE, (6(11):2278-2
1998.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In European conference on computer vi- sion, pp. 740-755. Springer, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature Pyramid Net- works for Object Detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2117-2125, 2017a.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE International Conference on Computer Vision, pp. 2980-2988, 2017b.
Anji Liu, Stephan Mandt, and Guy Van den Broeck. Lossless Compression with Probabilistic Cireuits. In International Conference on Learning Representationss 2022.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. In International Confer- ence on Learning Representations, 2017. 2017.
Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao. Dve: An end-to-end deep video compression framework. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11006-11015, 2019.
Yadong Lu, Yinhao Zhu, Yang Yang, Amir Said, and Taco S Cohen. Progressive Neural Image Compression with Nested Quantization and Latent Ordering. In 2021 IEEE International Conference on Image Processing (ICIP), pp. 539-543. IEEE, 2021.
Yoshitomo Matsubara. torchdistill: Modular, Configuration-Driveee Driven Framework for Knowledge Distillation.
International Workshop on Reproducible Research in Pattern Recognition, pp. 24 44. Springer, 2021. https: 7/g
17

---

Published in Transactions on Machine Learning Research (06/2023)
Yoshitomo Matsubara and Marco Levorato. Neural Compression and Filtering for Edge-assisted Real-time Object Detection in Challenged Networks. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 2272-2279, 2021.
Yoshitomo Matsubara, Sabur Baidya, Davide Callegaro, Marco Levorato, and Sameer Singh. Distilled Split Deep Neural Networks for Edge-Assisted Time Systems. In Proceedings of the 2019 Workshop on Hot Topics in Video Analytics and Intelligent Edges, pp. 21-26, 2019.
Yoshitomo Matsubara, Davide Callegaro, Sabur Baidya, Marco Levorato, and Sameer Singh. Head Network Distilla- tion: Spliting Distilled Deep Neural Networks for Resource-Constraing Edge Computing Systems. IEEE Access, 8:212177-212199 2020. doi: /Ttp://AACESSS.22........0
Yoshitomo Matsubara, Davide Callegaro, Sameer Singh, Marco Levorato, and Francesco Restuccia. BottleFit: Learn- ing Compressed Representations in Deep Neural Networks for Effective and Efficient Split Computing. In 2022 IEEE 23rd International Symposium on a World of Wireless, Mobile and Multimedia Networks (WoWMoM), pp. 337-346. IEEE, 2022a.
Yoshitomo Matsubara, Marco Levorato, and Francesco Restuccia. Split Computing and Early Exiting for Deep Learn- ing Applications: Survey and Research Challenges. ACM Computing Surveys, 55(5):1-30, 2022b.
Yoshitomo Matsubara, Ruihan Yang, Marco Levorato, and Stephan Mandt. Supervised Compression for Resource- Constrained Edge Computing Systems. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2685-2695, 2022c.
David Minnen and Saurabh Singh. Channel-Wise Autoregressive Entropy Models for Learned Image Compression. In 2020 IEEE International Conference Image Processing (ICIP), pp. 3339-3343, IEEE, 2020.
David Minnen, Johannes Ballé, and George D Toderici. Joint Autoregressive and Hierarchical Priors for Learned Image Compression. In Advances in Neural Information Processing Systems, pp. 10771-10780, 2018.
Arun Netravali and John Limb, Picture coding: A review. Proceedings of the IEEE, 6863))366-406, 1980.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, -erroperformncce deep learning library. In Advances in Neural Information Processing Systems, pp. 8024-8033 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models From Natural Language Supervision. arXiv preprint arXiv:2 103.0000, 2021.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing Network Design Spaces. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10428- 10436, 2020.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In Advances in Neural Information Processing Systems, pp. 91-99, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3):211 21--22,, 2015.
Farzad Samie, Lars Bauer, and Jörg Henkel. loT Technologies for Embedded Computing: A Survey. In 2016 In- ternational Conference on Hardware/Software Codesign and System Synthesis (CODES+ ISSS), pp. 1-10. IEEE, 2016.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chie Chieh Chen. MobileNetV2: Inverted Residuals and Linear Bottlenecks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pp. 4510 4510-4520, 2018.
18

---

Published in Transactions on Machine Learning Research (06/2023)
Jiawei Shao and Jun Zhang. BottleNet++: An end-to-end approach for feature compression in device-edge inference systems. In 2020 IEEE International Conference Communications Workshops (ICC Workshops), pp. 1-6. IEEE, 2020.
Saurabh Singh, Sami Abu-El-Haija, Nick Johnston, Johannes Ballé, Abhinav Shrivastava, and George Toderici. End- to-end Learning of Compressible Features. In 2020 IEEE International Conference on Image Processing (ICIP), pp. 3349-33333 3353. IEEE, 2020.
Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your VIT? Data, Augmentation, and Regularization in Vision Transformers. Transactions Machine Learning Research, 2022. ISSN 2835-8856. URLBttps::/ppeereviiie
net f 1KcP
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V Le. MnasNet: Platform-Awart Neural Architecture Search for Mobile. In Proceedings of the IEEE Conf. Computer Vision and Pattern Recognition, pp. 2820-28222 2828, 2019.
Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár. Lossy Image Compression with Compressive Autoencoders. In International Conference on Learning Representations, 2017.
George Toderici, Damien Vincent, Nick Johnston, Sung Jin Hwang, David Minnen, Joel Shor, and Michele Covell. Full Resolution Image Compression with Recurrent Neural Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5306-53161 -314, 2017.
Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Herve Jegou. Fixing the train-test resolution discrepancy. Ad- vances in Neural Information Processing Systems, 32:8252-8266 2019.
Praneeth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar. Split learning for health: Distributed deep learning without sharing raw patient data, arXiv preprint arXiv:/8/2.000 2018.
Dezhao Wang, Wenhan Yang, Yueyu Hu, and Jiaying Liu. Neural Data-Dependent Transform for Learned Image Compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17389-17388, 2022.
Ross Wightman. Pytorch image models. 2019.
Paul A Wintz. Transform picture coding. Proceedings of the IEEE, (007):809-200, 1972.
Chao-Yuan Wu, Nayan Singhal, and Philipp Krahenbuhl. Video compression through image interpolation. In Pro- ceedings of the European Conference Computer Vision (ECCV), pp. 416-431, 2018.
Ruihan Yang and Stephan Mandt. Lossy Image Compression with Conditional Diffusion Models. arXiv preprint arXiv:2209.0688 2022a.
Ruihan Yang, Yibo Yang, Joseph Marino, and Stephan Mandt. Insights from generative modeling for neural video compression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023a.
Y Yang and S Mandt. Towards empirical sandwich bounds on the rate-distortion function. In International Conference on Learning Representations, 2022b.
Yibo Yang and Stephan Mandt. coanmettrically-poweeee neural image compression with shallow decoders. arXiv preprint arXiv:2304.06244 2023.
Yibo Yang, Robert Bamler, and Stephan Mandt. Improving Inference for Neural Image Compression. In Advances in Neural Information Processing Systems, volume 33, pp. 573-584, 2020a.
Yibo Yang, Robert Bamler, and Stephan Mandt. Variational Bayesian Quantization. In International Conference Machine Learning, pp. 10670-10680. PMLR, 2020b.
Yibo Yang, Stephan Mandt, Lueas Theis, al. An Introduction to Neural Data Compression. Foundations and Trends in Computer Graphics and Vision, 15(2):113-22) 2023b.
19

---

Published in Transactions on Machine Learning Research (06/2023)
Zhongzheng Yuan, Samyak Rawlekar, Siddharth Garg, Elza Erkip, and Yao Wang. Feature Compression for Rate Constrained Object Detection on the Edge. In 2022 IEEE 5th International Conference Multimedia Information Processing and Retrieval (MIPR), 1-6. IEEE, 2022.
Hao Zhang, Dongdong Chen, and Seok-Bum Ko. New Flexible Multiple-Precision Multiply-Accumulate Unit for Deep Neural Network Training and Inference. IEEE Transactions Computers, 69(1):26 -8, 2019.
Yinhao Zhu, Yang Yang, and Taco Cohen. Transformer-based Transform Coding. International Conference on Learning Representations, 2021.
20

---

Published in Transactions on Machine Learning Research (06/2023)
A Image Compression Codecs
As image compression baselines, we use JPEG, WebP (Google), and BPG (Bellard). For JPEG and WebP, we follow the implementations in Pillow and investigate the rate-distortion (RD) tradeoff for the combination of the codec and pretrained downstream models by tuning the quality parameter in range of 10 to 100. Since BPG is not available in Pillow, our implementation follows (Bellard) and we tune the quality parameter in range of 0 to 50 to observe the RD curve. We use the x265 encoder with 4:4:4 subsampling mode and 8-bit depth for YCbCr color space, following (Bé- gaint al. 2020).
B
Quantization
This section briefly introduces the quantization technique used in both proposed methods and neural baselines with entropy coding.
B.1 Encoder and Decoder Optimization
As entropy coding requires discrete symbols, we leverage the method that is firstly proposed (Ballé et al. 2017 7 to lear a discrete latent variable. During the training stage, the quantization simulated with a uniform noise enable gradient-based optimization:
1 1 2
fo(x) + u
At runtime, we round the encoder output to the nearest integer for entropy coding and the input of the decoder:
[fo(x)]
B.2 Prior Optimization
For entropy coding, prior that can precisely fit the distribution of the latent variable reduces the bitrate. However, the prior distributions such as Gaussian and Logistic distributions are continuous, which is not directly compatible with discrete latent variables. Instead, we use the cumulative of a continuous distribution to approximate the probability mass of discrete distribution (Ballé et al. 2017):
x P(z) = f p())dı,
(4)
where is the prior distribution we choose, and P(z) is the corresponding probability mass under the discrete distri- bution P. The integral can easily be computed with the cumulative distribution function of the continuous distribution.
C Channel Reduction and Bottleneck Quantization
A combination of channel reduction and bottleneck quantization (CR + BQ) a popular approach in studies on split
computing (Eshratifar et al. 2019b] Matsubara et al.] 2020 Shao Zhang 2020 Matsubara & Levorato 202T Choi
et
2020
Dong et al. 2022), and we refer to the approach baseline.
C.1 Network Architecture
C.1.1
Image classification
We reuse the architectures of encoder and decoder from Matsubara and Levorato (Matsubara & Levorato 2021
introduced in ResNet (He et al. 2016) and validated the ImageNet (ILSVRC 2012) dataset (Russakovsky et al. 2015). Following the study, we explore the rate-distortion (RD) tradeoff by varying the number of channels in a convolution layer (2, 3, 6, 9, and 12 channels) placed at the end of the encoder and apply a quantization technique (32-bit floating point to 8-bit integer) (Jacob et al. 2018) to the bottleneck after the training session.
21

---

Published in Transactions on Machine Learning Research (06/2023)
C.1.2 Object detection and semantic segmentation
Similarly, we reuse the encoder-decoder architecture used as ResNet-based backbone in Faster R-CNN (Ren et al. 2015) and Mask R-CNN (He et al. 2017) for split computing (Matsubara & Levorato 202I). The same ResNet- based backbone is used for Faster R-CNN (Ren et al. 2015) and DevpLabv3 (Chen al., 2017b). Again, we examine the RD tradeoff by controlling the number of channels bottleneck layer (1, 2, 3, 6, and 9 channels) and apply the same post-training quantization technique (Jacob al. 2018 to the bottleneck.
C.2 Training
Using ResNet-50 (He al. 2016) pretrained on the ImageNet dataset teacher model, we train the encoder-decoder introduced to a copy of the teacher model, that is treated as a student model for image classification. We apply the generalized head network distillation (GHND) (Matsubara & Levorato 2021) to the introduced encoder-decoder in the student model. The model is trained on the ImageNet dataset to mimic the intermediate features from the last three residual blocks in the teacher (ResNet-50) by minimizing the sum of squared error losses. Using the Adam optimizer (Kingma Ba 2015), we train the student model on the ImageNet dataset for 20 epochs with the batch size of 32. The initial learning rate is set 10 and reduced by a factor of 10 the end of the 5th, 10th, and 15th epochs.
Similarly, we use ResNet-50 models in Faster R-CNN with FPN pretrained on COCO 2017 dataset (Lin et al. 2014) and Lab33 pretrained on PASCAL VOC 2012 dataset (Everingham et al. 2012) as teachers, and apply the GHND to the students for the same dataset. The training objective, initial learning rate, and number of training epochs are the same as those for the classification task. We set the training batch size to and 8 for object detection and semantic segmentation tasks, respectively. The learning rate is reduced by a factor 10 at the end of the 5th and 15th epochs.
D Entropic Student
This section presents the details of end-to-end and multi-stage fine-tuning supervised compression baselines for En- tropic Student models. We refer readers to (Matsubara et al. (002c)) for the architectures of Entropic Student models.
D.1 Two-stage Training
Here, we describe the two-stage method proposed to train the Entropic Student models in (Matsubara et al.) 2022c).
D.1.1 Image classification
Using the ImageNet dataset, we put our focus on the introduced encoder and decoder at the first stage of training and then freeze the encoder fine-tune all the subsequent layers the second stage for the target task. At the 1st stage, we train the student model for 10 epochs mimic the behavior of the first residual block in the teacher model (pretrained ResNet-50) in a similar way to (Matsubara & Levorato) [2021) but with the rate term to learn a prior for entropy coding, We use Adam optimizer with batch size of 64 and an initial learning rate of 10-3, The learning rate is decreased by a factor of 10 after the end of the 5th and 8th epochs.
Once we finish the 1st stage, we fix the parameters of the encoder that has learnt compressed features at the 1st stage and fine-tune all the other modules, including the decoder for the target task. By freezing the encoder's parameters, reuse the encoder for different tasks. The rest of the layers can be optimized adopt the compressible features for the target task. Note that once the encoder is frozen, we also no longer optimize both the prior and encoder, which means we can directly use rounding to quantize the latent variable. With the encoder frozen, we apply standard knowledge distillation technique (Hinton et al.) 2014) to achieve hetter model accuracy, and the concrete training objective is formulated as follows:
f= a Los + (1 LxL (o*,o")
(5)
where Lets standard cross entropy. y indicates the model's estimated class probabilities, and y the annotated object category. a and are both hyperparameters, and LxL is the Kullback-Lein divergence. oa and o" represent the softened output distributions from student and teacher models, respectively. Specifically, O$ [oî, ....,,...,00 .c.
22

---

Published in Transactions on Machine Learning Research (06/2023)
where C is set of object categories considered in target task. of indicates the student model's softened output value (scalar) for the i-th object category:
exp %
(6)
Σkeo exp
2k
where is a hyperparameter defined in Eq. | and called temperature. vs denotes a logit value for the i-th object category. The same rules are applied to o for teacher model, which is a target distribution.
For the 2nd stage, we use the stochastic gradient descent (SGD) optimizer with an initial learning rate of 10-3, momentum of 0.9, and weight decay of 5 x 10-4. We reduce the learning rate by a factor of 10 after the end of the 5th epoch, and the training batch size is set to 128. The balancing weight and temperature T for knowledge distillation are set to 0.5 and 1, respectively.
D.1.2 Object detection
We reuse the entropic student model trained on the ImageNet dataset in place of ResNet-50 in Faster R-CNN (Ren et al. (015)) and DeepLabv3 (Chen et al.) 2017b) (teacher models). Note that freeze the parameters of the encoder trained on the ImageNet dataset, following (Matsubara al. 2022c). Reusing the encoder trained on the ImageNet dataset is reasonable approach as 1) the ImageNet dataset contains larger number of training samples (approxi- mately 10 times more) than those in the COCO 2017 dataset (Lin et al. 2014); 2) models using an image classifier as their backbone frequently reuse model weights trained on the ImageNet dataset (Ren et al. (2015) Lin et al.) 2017b).
To adapt the encoder for object detection, we train the decoder for 3 epochs at the 1st stage in the same way we train those for image classification (but with the encoder frozen). The optimizer is Adam (Kingma Bal 2015), and the training batch size is 6. The initial learning rate is set to 10 3 and reduced 10 after the first 2 epochs. At the 2nd stage, we fine-tune the whole model except its encoder for 2 epochs by the SGD optimizer with learning rates of 10-3 and 10 for the 1st and 2nd epochs, respectively. We set the training batch size to 6 and follow the training objective in (Ren et al. 2015), which is combination of bounding box regression, objectness, and object classification losses.
D.1.3 Semantic segmentation
For semantic segmentation, we train DeepLabv3 in a similar way. At the 1st stage, we freeze the encoder and train the decoder for 40 epochs, using Adam optimizer with batch size of 16. The initial learning rate is 10-3 and decreased to 10-1 and 10-5 after the first 30 and 35 epochs, respectively. At the 2nd stage, we train the entire model except for its encoder for 5 epochs. We minimize standard cross entropy loss, using the SGD optimizer. The initial learning rates for the body and the sub-brande brabbbanchhhh (auxiliary module are 2.5 x 10 and 2.5 x 10 10-2, respectively. Following (Chen et al. 20175), we reduce the learning rate after each iteration as follows:
Mare
Ir tro (1
Niter 0.9
Nmmxx
where Iro is the initial learning rate. Niter and Nmax juer indicate the accumulated number of iterations and the total number of iterations, respectively.
D.2 End-to-end Training
In this work, the end-to-end training approach for learning compressive feature (Singh et al. 2020 Yuan et al. 2022 K is treated as a baseline and applied to the entropic student models without teacher models.
D.2.1 Image classification
Following the end-to-end training approach (Singh et al. 2020). we train the entropic student model from scratch. Specifically,
Adam (Kingma & Ba 2015) optimizer and cosine decay learning rate schedule (Loshchilov & Hutter 2017] with an initial learning rate of 10 and weight decay of 10 Based on their training objectives (Eq.
https: htapps://ggttii /ttpthb...ttt//////////htt...h.t www.wc/m//pt..eee http:/pyoorch/yiilee wwwriiiinnnrreeeeeeeeeeee htt///ann//ee///// maaiin/eeereeceea /eieeee//eegnae aegmenttiion
Singh et al. 2020 and Yuan al. (2022 assess their methods only for image classification and object detection tasks, respectively.
23

---

Published in Transactions on Machine Learning Research (06/2023)
E3 beerl (Boshes))) F3 lerr2 (RosNet-22)
RB legred (Rodkct-dt) legrrd (Rodirt-50)
-age 2B at arrgod (RooNes-20) (ingle al 2013
Tetmpie Stoderk [Mldonieem 2002)
70
70
Anaraay [X0
Acsareyy [95]
60-
60
reaedoro Sesuin Eneada GELOPS
50
50
00
40
und-bbolo bio-bbilohchhhhii chochlo Data Size (KByte]
10
10
10
10
30
10
10
10
10
Data Size [KByte]]
Eneoder GFLOPS Data Size [KDy
Figure Entropic Student vs. ResNet-50 with entropy bottleneck (EB) introduced different layers, using encoder FLOPs instead of encoder size (Eq. (T) in Fig. for our proposed ExR-D and three-way tradeoffss (middle and right). Note that the reported FLOPS do not cover all the operations in the encoders Py PyTooch Profiler supports only specific operations such as matrix multiplication and 2D convolution at the time of writing.
we train the model for 60 epochs with batch size of 256 Note that Singh et al. (2020) evaluate the accuracy of their models on a 299 x 299 center crop. Since the pretrained ResNet-50 expects the crop size of 224 x 224 10 we use the
crop size for all the considered classifiers to highlight the effectiveness of the training method.
(o)s(B. -
log Po (f)(x) + c), e~ -nif(-
(8)
distortion
rate
D.2.2 Object detection
Reusing the model trained on the ImageNet dataset with the end-to-end training method, we fine-tune Faster R- CNN (Ren et al. 2015). Since empirically find that simple transfer learning approact to Faster R-CNN with the model trained by the baseline method did not converge, we use the 2nd stage of the fine-tuning method described above. The hyperparameters are the same as above, but the number of epochs for the 2nd stage training is 5.
D.2.3 Semantic segmentation
We fine-tune DeepLabv3 (Chen al. 2017b) with the same student model trained on the ImageNet dataset. Using the SGD optimizer with an initial learning rate of 0.0025, momentum of 0.9, and weight decay of 10-4, minimize the cross entropy loss. The learning rate is adjusted by Eq. (7). and we train the model for 50 epochs with batch size of 8,
E Encoder FLOPS
As mentioned in Section 3.1.3 we defined and introduced encoder size (Eq. (1) as an additional metric for the ExR-D and three-way tradeoffs since FLOPS is not static value, and FLOP/MAC not well supported by existing Py PyTorch frameworks such as PoTorc Profile 12 fvcore, and deepspeed. Figure 9 shows results of the bottleneck placement experiment (Fig. [6) using encoder FLOPS approximated by PyTorch Profiler instead of the encoder size. While its supported operations are limited (e.g. matrix multiplication and 2D convolution at the time of writing), we confirmed
similar trends in Fig. to those in Fig.
For the ImageNet dataset, Singh ct al.
2020 train their models for 300k steps with batch size of 256 for 1.28M training samples, which is
equivalent to 60 qpachs (= 300xx200
2834
24


Reviews:
[{"review_id": "rMvByCawBE", "reviewer": "Reviewer_Y1Q3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strength]\n- This paper is not only well-written but also easy to follow, providing a clear and engaging discussion of the topic.\n- It offers a thorough description of various previous techniques related to split computing, which adds depth to the analysis.\n- The introduction of the benchmark will prove invaluable for researchers seeking to conduct fair and accurate comparisons in the field of split computing.\n\n[Weakness]\n- The paper could benefit from a more concise presentation of previous methods, as some explanations appear to be repetitive across sections.\n- The models and tasks discussed may seem outdated, and incorporating more recent advancements would strengthen the paper's relevance.\n- Given the varying characteristics of split computing, depending on scenarios and advancements in edge devices or servers, it would be helpful if the authors could discuss whether Equation (1) is the most appropriate hardware-agnostic metric. A metric that takes power or hardware cost of edge devices into account might be more challenging to define, but it could be essential for evaluating split computing methodologies.\n- It would be valuable to explore whether the neural image compression model remains the best approach for other tasks, or if the conclusions drawn are specific to certain tasks such as image classification or object detection.\n- The overall contributions of this paper appear to be incremental, and the work presented resembles a technical report rather than a novel research paper. To enhance its impact, the authors could consider incorporating more innovative ideas or findings.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors explore the concept of supervised compression for split computing, a valuable approach when an edge device and an edge server collaborate to execute tasks such as image classification and object detection. They thoroughly investigate various techniques for implementing split computing, taking into account three key metrics: encoder size, data size, and supervised distortion.\n\nTo further support research in this area, the authors introduce a novel benchmark known as the SC2 Benchmark, specifically designed to evaluate the effectiveness of different split computing methodologies. Through their comprehensive analysis, they demonstrate that the neural image compression model outperforms other approaches in terms of supervised rate-distortion performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weakness part mentioned above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "05UITR19VD", "reviewer": "Reviewer_Ley7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengthens:\n(1)\tThe paper is well-motivated. The rate-distortion evaluation metric is not sufficient to evaluate the performance-efficiency trade-off of SC2, given the prevailing of large-scale encoders used in SC2. It is timely to take the size of encoder into account. \n(2)\tThe reason for the urgent requirement of a comprehensive benchmark listed in 2.3.3 is convincing and reasonable.\n(3)\tFigure 2 well explains the difference between SC and IC.\n(4)\tThe empirical results look solid and interesting. Evaluation 10 baselines in one paper requires massive effort and time, which I highly appreciate it.\n\nWeaknesses:\n(1)\tWhile I fully understand evaluating 10 baselines requires massive effort, it is important to report the details of hyperparameters and configurations used to reproduce the results of each baseline to ensure the fairness. \n(2)\tI have a big concern about the usage of parameter count as a proxy for the encoder size. While it is important to measure the parameter count, parameter count itself can not faithfully reflect the real speed of the model/algorithm on hardware. I encourage the authors also report the FLOPs of all the baselines. As long as we guarantee all the numbers are measured using the same hardware, it is a solid contribution and will provide a full picture to the readers.\n(3)\tWhat is the difference between EB at layer1 and Entropic Student in Figure 6? What causes the latter performs much better than the former?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This study provides a comprehensive benchmark for supervised compression for split computing (SC2). Instead of only focusing on the traditional R-D tradeoff, this paper states that the size of encoder should also be considered, and hence proposes another tradeoff that takes encoder size and data size multiplicatively into account. Based on the proposed metric, the authors conduct a comprehensive benchmark study involving 10 different state of the art methods and more than 180 trained models for three computer vision tasks, such as image classification, object detection, and segmentation. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, I believe this paper provides valuable benefits to community and can be accepted if the above weaknesses can be addresses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MAISCWogGY", "reviewer": "Reviewer_MSsY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The paper is well written\n+ Extensive and systematic experiments evalating various relevant design choices\n+ Re-useable benchmark for future work\n\nI did not find significant issues", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Split computing refers to a setup where a low-power edge device (e.g. mobile phone) does part of the computation for neural network inference, and a server does another part. For example, one may wish to classify an image captured on the device, but not run a large neural network there. If part of the computation is done on a server, some data has to be transmitted, and this (compressed) data can also be produced by a neural network (e.g. trained to preserve information useful for classification).\n\nThe present paper performs an extensive benchmark study to evaluate various design choices for split computing and their effect on supervised performance, compression rate, and low computational cost. Specifically, it looks at different (classical and learned) compression methods for inputs or features, supervised compression models, different tasks, architectures, and training schemes. The best performing model was found to be a neural image compression model trained by knowledge distillation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It would be nice to discuss the relation to the well-known information bottleneck, where a supervised loss is combined with a hidden-layer compression loss. The IB literature is a bit more theoretical, but it would be good to make the connection.\n\nTypos:\npage 4: \"growing interest from the research.\"", "broader_impact_concerns": "Split computing seems useful in warfare (e.g. autonomous drones) and mass surveillance, so an impact statement might be appropriate. This being a benchmark study, it is not clear that by itself it will make a difference though.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "t8mbf6QxwD", "reviewer": "Reviewer_X4aF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "---\n\n**Strengths**:\n* The paper is well-written and easy to understand. The authors provide sufficient background on split computing to make it accessible to readers outside the field.\n* This paper addresses an important problem in the emerging field of split computing. By providing a standardized way to measure the performance of split computing systems, this benchmark enables researchers to compare different systems and evaluate their performance objectively. This helps promote the development of split computing and advance the field.\n\n---\n\n**Weaknesses**:\n* The metrics used in this paper may be subject to some questions. Firstly, the encoder cost is measured by the model size, rather than the more commonly used metric of the number of Multiply-Accumulate operations (MACs). While the authors justify their choice, most papers in efficient deep learning use MACs as a proxy for computation cost. Secondly, the authors do not consider the cost of the decoder, even though its computation cost could affect the end-to-end latency. Although the decoder runs on a faster server, its cost should be taken into account. Thirdly, the proposed ExR metric seems arbitrary, and it is unclear why these two metrics should be multiplied. While it is important to use a proxy metric that is not hardware-specific, the authors should justify why their proposed proxy metric is indeed correlated with the end-to-end latency, which is the most important metric in this context. Ultimately, the end-to-end latency includes the inference time of the encoder, transmission time of the compressed data, and the inference time of the decoder, making it the most relevant metric.\n* The benchmarks included in this paper are limited to image workloads. In order to make the benchmark more comprehensive, it is important to include other types of workloads such as speech recognition, machine translation, language modeling, and text summarization. By including a wider variety of workloads, the benchmark can provide a more complete picture of the performance of split computing systems across different tasks. This would be particularly beneficial for researchers and practitioners working in these other areas.\n\n---", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper discusses split computing for deep learning models between mobile devices and edge servers, which is a popular solution for implementing such models on mobile devices. The authors propose a new method called supervised compression for split computing (SC2) and new evaluation criteria to minimize computation on mobile devices, minimize transmitted data size, and maximize model accuracy. They conducted a comprehensive benchmark study and released a Python package to help researchers better understand the tradeoffs of supervised compression in split computing.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weaknesses section for more details.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Please refer to my comments above. 
Decision: Accept (poster)