All-in-One Image Coding for Joint Human-Machine Vision with Multi-Path Aggregation
Authors: Xu Zhang
Peiyao Guo
Ming Lu
Zhan Ma
Venue: NeurIPS 2024 poster (2024)
Keywords: learned image coding
joint human-machine vision
multi-path aggregation
Primary area: machine_vision

Abstract: Image coding for multi-task applications, catering to both human perception and machine vision, has been extensively investigated. Existing methods often rely on multiple task-specific encoder-decoder pairs, leading to high overhead of parameter and bitrate usage, or face challenges in multi-objective optimization under a unified representation, failing to achieve both performance and efficiency. To this end, we propose Multi-Path Aggregation (MPA) integrated into existing coding models for joint human-machine vision, unifying the feature representation with an all-in-one architecture. MPA employs a predictor to allocate latent features among task-specific paths based on feature importance varied across tasks, maximizing the utility of shared features while preserving task-specific features for subsequent refinement. Leveraging feature correlations, we develop a two-stage optimization strategy to alleviate multi-task performance degradation. Upon the reuse of shared features, as low as 1.89\% parameters are further augmented and fine-tuned for a specific task, which completely avoids extensive optimization of the entire model. Experimental results show that MPA achieves performance comparable to state-of-the-art methods in both task-specific and multi-objective optimization across human viewing and machine analysis tasks. Moreover, our all-in-one design supports seamless transitions between human- and machine-oriented reconstruction, enabling task-controllable interpretation without altering the unified model. Code is available at https://github.com/NJUVISION/MPA.

All-in-One Image Coding for Joint Human-Machine Vision with Multi-Path Aggregation
Xu Zhang School of Electronic Science and Engineering
Peiyao Guo
Ming Lu" Zhan Ma
Nanjing University (xu. zhang, peiyao) @ama il, www.u..edeii cn (minglu, mazhan jnju. edu.
Abstract
Image coding for multi-task applications, catering to both human perception and machine vision, has been extensively investigated. Existing methods often rely on multiple task-specific encoder-decoder pairs, leading to high overhead of parameter and bitrate usage, or face challenges multi-objective optimization under unified representation, failing to achieve both performance and efficiency. To this end, we propose Multi-Path Aggregation (MPA) integrated into existing coding models for joint
vision, unifying the feature representation with all- in-one architecture. MPA employs a predictor to allocate latent features among task-specific paths based on feature importance varied across tasks, maximizing the utility shared features while preserving task-specific features for subsequent refinement. Leveraging feature correlations, develop two-stage optimization strategy to alleviate multi-task performance degradation. Upon the reuse of shared features, low 1.89% parameters further augmented and fine-tuned for a specific task, which completely avoids extensive optimization of the entire model. Experimental results show that MPA achieves performance comparable state- of-the-art methods in both task-specific and multi-objective optimization across human viewing and machine analysis tasks. Moreover, our all-in-one design supports seamless transitions between human- and machine-oriente reconstruction. enabling coskccontolllable interpretation without altering the unified model. Code is available at htttps: //tp:::///ttttii github..bb. cwm/NJUVIC WTTNNNVVVVIIOO//E SINMMMPA
Introduction
Coding for multi-task that satisfies both human perception and machine vision has been extensively explored over the past few years 669
80
45 The intuitive and simple solution to achieve
optimal performance for various tasks involves defining distinct eecoderr pairs tailored to
specific instances with multiple bitstreams (separate [49 669 [1 42 7 75) or scalable [5] 80 330 [74]
12 26)). which however incurs significant parameter overhead and inefficient bitrate consumption. attempt to enhance the compression performance across multiple tasks by deriving unified and compact representation of input images. Typically, they rely on a generalized encoder for feature extraction and deploy diverse decoding models to support corresponding vision tasks. However, they still suffer from the parametric inefficiency of multiple dedicated decoders. Recent work has made some progress in designing unified compression model for multi-task, where the encoder generates a unified representation, and the decoder focuses on task-oriented reconstruction. Popular techniques include conditional generation (2) and residual prediction [33 24 41), enabling user-controllable modulation of decoded results through task-driven guidance. While achieving performance comparable to decoder-specins approaches, they solely
As result, alternative efforts [8] [9
22 45
"Coresponning author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

x
x
Enc.
Enc. 2
Enc.
Enc.
Enc.
%
92
8n
Dec.
1
Dec. 2
Dec. m
Dec.
Dec. 2)
Dec.
Dec.
Task
Task .... Task
Task
Task 2
Task
Task
Task
Task
(a) Separate pairs
(b) Unified representation
(c) Unified model
Figure I: Paradigm comparison for multi-task coding.
focus on human-centric requirements, such as perception-oriieteed and fidelity-oriented aspects. More critically, the challenge of multi-objective optimization [62 20), particularly in the context of multi- task learning [66 63). persists in the unified paradigm for joint
machine vision, offering the g. The crux of the problem lies the indiscriminate treatment the unified representation across diverse tasks, which necessitates searching for an accurate Pareto front for transition between tasks. This requires the use of massive and complicated techniques to fit 47 58 (7 75 and multi-objective optimization with variable weights 12, 32 24 41. Its failure can lead significant performance degradation, And as 31
performance largely inferior to that optimized for each task independently [42
7
the number tasks increases, the difficulty of optimal search grows dramatically [15 23). So far, few approaches consider the correlation of features across tasks to ensure efficient multi-task collaboration while also optimizing for the specificity of distinct tasks.
To overcome the challenges above, we propose unified image coding method with Multi-Pate Ag- gregation (MPA) for joint maaanne vision tasks. Specifically, a set of Multi-Layer Perceptron (MLP) branches is inserted to form multiple aggregation paths, replacing the single-path MLP in the
feature transform blocks [53 54 which eurrently dominate compression models [56] 18 55
Each path is customized and tailored to different tasks with varying complexities. Considering diverse importance of features across different tasks, we devise a predictor for importance scores allocate latent features among task-specift paths based on their importance. Leveraging feature correlations across tasks, we develop an efficient two-stage optimization strategy with fine-tuning partial param- eters generalized features to alleviate multi-task performance degradation, avoiding extensive optimization of the entire model. This strategy significantly eases the optimization of multi-task coding while maintaining performance comparable to other fully optimized models. Considering any viewing task for human vision (e.g., low distortion or high realism) or analysis task for machine vision (e.g., high-level low-level vision task), MPA switch flexibly between them with seamless transitions, enabling single representation to be interpreted in different ways within a unified model.
Our contributions are summarized as follows:
Considering different importance of features for different tasks, we propose MPA to -on-uniformly treat features. By developing importance score predictor, MPA can allocate generalized latent features among task-specific paths based on their varying importance. This enables a unified model to support multi-task coding with seamless transitions in an all-in-one manner.
Leveraging feature correlations across tasks, we propose two-stage optimization strategy with fine-tuning partial parameters on generalized features to overcome the challenge of multi-objective optimization in multi-task coding. This strategy allows MPA to be easily extended to support new tasks without independent optimization of separate task-specific models.
Extensive experiments demonstrate that using the unified model, MPA achieves rate-distortion (R- D) and aaeeperceeption (R-P) performance for human vision comparable to other state-of-the-ant (SOTA) models and significantly improves accuracy for analysis tasks close to the fully fine-tuned ones, outperforming other separately optimized methods.
Related work
Separate pairs. Developing multiple task-specific encoder-decoder pairs, as shown in Fig. la casy to optimize and beneficial to high performance but has significant parameter and bitrate

---

Output
Binarize
Sigmaid
Bias
MLP
MPA
Lineart 16"
LN
Mask
LN
Main Path
Side Path
GELU
Task
Linear e
SF
Feature Group
Feature Group 2
Partial
Average
Attention
Attention
GELU
LN
LN
Mask
Linear(C.
Input Features
VIT-Block
MP-Block
Multi-Path Aggregation (MPA)
Predictor
Figure 2: The proposed Multi-Pate Aggregation (MPA). Compared to typical Vision Transformer (ViT) block [73 16 original MLP replaced with MPA. LN and SF are Layer Normalization and Scaling Factors [10]. P. ( and A denote predictor, split and aggregation respectively. C' represents the number of input channels. is the ratio Penc in the encoder or the ratio Pdec in the decoder.
overhead 49
42 7 Song et al. (69 introduced Spatially-adaptive Feature Transform (SFT)
for tuning bit allocation. incurring significant latency for optimizing quality map for each image to achieve best task performance. Chen et al. [II] leveraged visual prompt tuning (VPT) [36] for task-specific optimization, resulting in quadratic complexity due to the self-attention mechanism
with respect to the number of VPT tokens. Scalable coding [51 80
30
74
12
26 improved
bitstream efficiency by embedding multiple bitstreams in scalable manner. However, organizing
representations for various tasks in layered manner without introducing redundancy is challenging.
aimed derive unified and compact repre- sentation of input images but developed separate decoders (Fig. Tb) to ease optimization challenges. Feng et al. 22 compressed intermediate features from vision backhone
Unified representation. Some efforts 8
19
22 45
generic representation, while Duan et al. [9 proposed a plug-and-play adapter to bridge compressed representations with existing vision backbones. both requiring optimization of the backbone for optimal performance. Li and Zhang 45 used semantic enhancement network to improve analysis accuracy without jointly optimizing the vision backbone but still faced parameter overhead due to multiple decoders.
Unified model. The paradigm in Fig. Ic uses a single encoder to generate unified representation and single decoder for task-specific reconstructions 23 2 24
41 32]. Gao et al. 23 optimized the unified model for multiple analysis tasks jointly without transitions, resulting in trade-off's and suboptimal performance. Ghouse al. [24] and Korber et al.
41 reconstructed images with basic model and used additional modules to predict and add residuals for task-specific goals, while Agustsson al. [2] and Iwai al. [32] developed hyper-parameter conditioning modules modulate the transition process. These methods require variable weighted objectives during optimization processes, making challenging extend more tasks.
A unified framework: multi-path aggregation
3.1 Preliminaries
Typically, the LIC model involves an encoder network E(x) that maps the input image æ compact latent representation ÿ. and decoder network G(ğ) that reconstructs & as an approx- imation to æ. For the best compression performance, the optimization objective minimizes the distortion between the original and reconstructed images while reducing the expected bitrate r(ū). using Eg log, Pg (ğ) characterized by entropy model P. To achieve perceptually pleasing reconstruction. a Generative Adversarial Network (GAN) strategy is often incorporated, where a con- ditional discriminator Dccond., input) is used to assist in model optimization. In our work, the MPA implemented in TinyLIC [55] to validate its functionality for joint human-machine vision coding, while the GAN strategy in HiFiC [I] used for guiding perceptual optimization. Furthermore, to support continuously variable-rate coding, we introduce the scaling factor (SF) modulation s, [10]

---

and non-linear interpolation [3] in each Multi-Path Block (MP- in Fig.2). The compression quality q [1. max] matching the encoder and decoder determines the bitrate consumption, while the task orientation a E [0, 1] in the decoder controls the continuously transitions between tasks.
3.2 Multi-path aggregation
To support various task optimizations flexibly with the unified model, we develop an efficient architecture named MPA. As illustrated in Fig. 2 the proposed method contains two key modules, multiple paths with MLP layers of varying complexity, and predictor for binarized mask M. Different from recent advances in ofatuure-ofEExerrr (MoE) [7 21], the paths are divided into two groups: the main and the side paths. The main path captures generalized features for various tasks, while the side paths are fine-tuned based on the generalized features. The input features are allocated into different paths the basis of M. The features corresponding "1" in M will enter the main path, while the remaining features will enter the side paths. Unlike existing methods, which apply all features, MPA only exploits a few parameters in the side path to refine the generalized features for task transitions.
consistent transformations
(24] 41 2 32
On the encoder side, set the original MLP in VIT block as the main (high-quality) path Φng adapted to high bitrate features and add bottleneck MLP as the side (low-quality) path σiq specific to low bitrate coding. For controlalllle LIC, the features op- timized for perceptual loss are the most generalized ones [1]. Therefore, on the decoder side, the main path is the Perc. (Perceptual) Path Peee (-), while the side paths contain the MSE path (MMS((). the Cls. (Classification) Path dets and the Seg. (Segmen- tation) Path (-) standing for four representative tasks to showcase the versatility of MPA, i.e., high- realism, low-distortion, high-level visual, and low- level visual image reconstructions. Setting the task index lusk by the user, MPA can realize the aggregation of the main path and one of the side path. To be efficient, we configure each path for different complexity depending on the characteristics of tasks. Specifically, the paths for human vision (i.e., Ppere() (.) and dMMs( are designed as an inverted (inv.) bottleneck MLP (54) to achieve higher realism and lower distortion, while the paths for machine vision (i.e., dets(-) and &seg (-)) was designed as a bottleneck MLP to reduce inference latency. Having an input feature x E HxxXXC fask and mask M E {0, 1} #x MPA processes as Alg. In our implementation, decoder, without the 4th stage since its spatial size is too small to produce a smooth transition.
Algorithm 1 Multi-Path Aggregation
Require: feature æ. mask M, task index Mask I: if encoding then
2: Omain f -bgy Pside Qig
3: else if decoding then
Ómain + Openc
5: Øside + {ØMSE, Oet, &seg [[mck]
end if
7: { + Split(zx, M)
1 + Omain (01),22 x2 x Osido (siic( 9: x + Aggregate( (atregate((1,,1 xt2)
return x
integrate MPA into the 1st, 2nd and 3rd stages in both the encoder and
3.3 Importance score predictor
For the portability of MPA, we introduce a lightweight importance score predictor as illustrated in Fig. 2 (denoted as (P) to generate mask and allocate features to each path. In addition only three parametric linear layers for fast point-wise computation, we use non-parametric partial average layer A(-) to capture multi-scale information. Given intermediate feature u E R xW xC' A(-) performs global average pooling on the latter C²/2 channels of intermediate feature followed by expansion to the spatial size of feature, which aggregates information from all features as global information. The other /²/2 channels are left unchanged as local information. Formally, the partial average operation can be formulated as
HxW S DW m((...c))
if C²/2,
A (u(h, w. w.)))
(I)
(h, w.c),
otherwise.
A hyper parameter (q a) is leveraged to control the feature allocation between the main and side paths, which indicates the performance transition between tasks. In the encoder, the aggregation of dhg and dhg is associated with bitrate, thus we approximate linear relationship between bitrate and aggregation ratio Pene following 43) by an inverse log transformation
3 1) 1)
Penc F(g)=
E [0, 1], E [1, Qmax),
(2)
3

---

where 3 is the base of the log transformation and Qmax represents the maximum value of the range of q. Thus, as q increases, more features will enter (tg(+) for supporting high-quality reconstruction. The aggregation in the decoder is irrelevant to bitrate, and Pdec 1 - ar so that as increases, more features will enter the side paths. For efficiency, the mask is only generated once the beginning of stage and shared by all MP-blocks throughout the stage.
During training, since the direct sampling of M based on p from the importance score is non- differentiable, we first bias the output u' of the last Linear(-) in the predictor using set of learnable parameters b to get shifted logits corresponding to each discrete q or a, and then use Gumbel- Sigmoid [35] 60) with threshold r = 0.5 to soften the sampling process as Eq. (3. After training, the predictor can binarize the score map according to p. and the bias layer in Fig. will be discarded.
M = Gumbel-Sigmoid(e + b, T).
(3)
4
Optimization strategy
4.1 Stage 1: training generalized basic model
Since the perceptually optimized LIC model is well aligned with both human and machine perception, we first train variable-rate model based on Tin [55] and GAN method from HiFiC [6]] for subsequent extension. We add SF in all ViT blocks and implement MPA in the encoder. Following the common practice literatures [ 2 32,, the optimized losses are Eqs. O and (7) for the joint optimization of the encoder E, decoder G, entropy model P and discriminator D:
HO W
1 S
Cratio
Pene
W(()
Σ Σ
M (h, (h,
(4)
1=1
LG = Egrpy | -oogg Dgg, G(g)))),
(5)
LD = Eg Egrpy log(1 D(g. G(ū)) + Exwpal |- log D ((E(2)), x)),
(6)
LEGP E Exppa (()) + d(x } AcLa + Apece + Avatio Cration
where x, &. y and á are the input image, reconstructed image, compressed latents before and after quantization, respectively. Lrtii is introduced to optimize the predictors in MP-Blocks which constrains the ratio "1"s in M at all encoder stages to the encoder ratio target Pene is the number of stages applying MPA, while H/(x and W(() represent the spatial size at Stage s. r(-) and d(.,-) represent the bitrate estimated by the entropy model P and 0.01 0.01MSE(:;) (pixel values are scaled to [0, 255).. x in Eq. 7 varies according to the sampling of quality level during training. Learned Perceptual Image Patch Similarity (LPIPS) [8]] is chosen for Eyere.
4.2 Stage 2: extending and optimizing side paths for multi-task
Obtaining a generalized model, we can add side paths to the decoder achieve cask-controllable image coding. In this training stage, there is no need to train the model entirely, but only to optimize the added parameters, i.c., the added side path @side and the corresponding predictor éped By sampling a during training, the main path and the added side path are randomly aggregated to fit the transition between different tasks. The training objective is simplified to
(dlide (peet)
arg min Egropp (A(9) r()) + Crakk + A Cratio
(8)
Phide Oppeel
where didee and gied are the optimal parameters of the added side path and predictor, Lusk represents the task loss, and the target ratio Craio is changed to decoder ratio Paec The options for task loss are varied. When optimizing for MSE, Liss is just a simple MSE loss measured between æ and a. When optimizing for a visual analysis task, Lusk is a compound loss containing d(æ, .). Epere and the full loss function of the task. Normalization should be applied if needed for augmentation. The task model frozen during optimization. For instance, using the classification model ClsModell.) with Norm(-), cross-entropy is used to compute loss between the classification result of å and the ground truth GT:
Lusk CrossEntropy (CaaeetrrooyyCCMMoeeeeNooooo (x))) GT) + Enrpa [d(x, + Apere
(9)
5

---

Kocalc (1sstortion)
(Distortion)
ImageNet (Cts, ConvNeXe
38
36
MIM (Fuil Perc.)
MSD
38
MPA (Pull Penc))
MPAiFull
80-
MBA (Full Perc.)
Clt.)
CRDR (8-0.0)
CRDB (B-0.0)
Top3 (1))
MPA (Pull MSE)
PSNR 1000
CRDR (B-3.84)
PSNR 1600
(F-2.84)
TransTIC
DIRAC-1
TTrenTTC (Baon)
DIRAC-100
24-
DIRAC-100
SFT (GrvOCAM)
MRIIC (0=00.0)
MRIC (f=0)
SIrl (Untform))
MRIC (B-2.56)
(B-2.56)
HiFIC
HIFAC
70
Full Fize-kuning
201
TinyL:I:
VIN17. Unoompreosed
VTM17.1
30
VTM117.1
26
0.2
0.3
0.6
0.9
0.1
0.2
0.3
0.6 0.7 0.8 0.9
0.4
1.0
1.2 (apo)
1.4
1.6
1.0
Dapp)
Bii-ute
Kodak (Reclicm)
CLJC (Realism)
ADE22K (Scg.. PSPNet50)
0.200
20.0
42
MBA (Phll MSE)
NBA (Full MSEI
0.175
MPA (Pal
17.5
IPuIl Feec.1
40
0.150
CRDB (3-3.84)
15.0
CRDR (2-2.84)
Q1:0.00
30
MIA (Pull Seg
DIRAC
12.5
DIRAC 3
MPA (Pull Pert
0.075
0.100
TInyLIC
DIRAC-100
10.0
MROC (3-2.50)
MRIC
DIRAC-100
36
24
MRA (Pull
TransTIC (VPT)
TransTIC: (Heos)
0.050
VTM17711
5.0
TinyLIC
HiFIC
32
TIryLIC
0.025
VTM17.1
30
VTM17.1
Uncunngeeeeee
0.000
0.0
20
0.5
0.5 0.6
0.0
0.9 1.0 L1
Bit-cate (bpp)
Bus-caca (bpp))
Bicarate (bos)
Figure 3: Multi-task performance. The curves of variable-rate models are plotted as solid lines, while dashed lines are for single-rate models. Colored areas represent the adjustable range of MPA.
5 Experiments
5.1 Experimental settings
Dataset. When training for perceptual quality and MSE, we use a combined dataset including Flicker2W [48. DIV2K
33. and CLIC [71 training sets, about 23K images in total. ImageNet-
IK [14] and ADE20K 83 are used to train dds and éxeg. respectively. We evaluate the model
on Kodak 339 and CLIC test set (71) for image compression, ImageNet validation set [14] for classification, and ADE20K validation set 883 for semantic segmentation.
Training. We set 8= 5 in Eq. 2 Qmn 8, x
{18, 9.32, 4.83, 2.5, 1.3, 0.67, 0.35, 0.18),
ÅG = 2.56, Aper: = 4.26. Ausk
I, and Aratio 10 for all training stages. 9 and are uniformly
sampled from {1. 2, 3, 4, 5, 6, 7, 8} and {0, 2, 3, 4, 5, 5,6,7 7 respectively. The training steps of Stage in Sec. 4.1 are set to 3M, the first half without LG and the last half with Lg- The training steps of Stage 2 are set to 500K. In each training stage, the initial learning rate is set to 10- decayed to 10 for the last 25% steps. When training Spere and Ó'NSE. images are randomly cropped to 256x 256x3, and random horizontal/vertical flip is applied. ConvNeXt-Tiny (54) is used for training dds, with images randomly resized and cropped to 256x256x3 followed by a random horizontal flip. PSPNet50 [82) is used for training daeg. with images randomly resized, flipped and cropped to 256x256x3. The batch size is set to 8 for all tasks. Adam 38 is used for optimization.
Evaluation. For human vision evaluation, we use Peak Signal-to-Nor -oiise Ratio (PSNR) to measure distortion, and use Fréchet Inception Distance (FID) 28 and LPIPS [T to measure realism. We use the same protocol in [61 2 24 32 to calculate FID, i.e., cropping images to overlapped patches of size 256x256x3 2. Note that FID is not calculated on Kodak because it yields only 192 patches, which is not sufficient for measuring FID. For the classification task, we use top-1 accuracy (acc.) to present the performance, with images first resized to a short edge of 292 and then center-cropped to 256x x25xx3. 3. For the segmentation task, performance, with images first resized to short edge of 512 and then center-cropped to 512x512x3.
mean Intersection over Union (mloU) to present the
5.2 Results of multi-task performance
For human vision, we compare the proposed MPA to the SOTA baselines of the unified model to evaluate R-D and R-P performance, i.e., MRIC (2), DIRAC 24). CRDR (32. In addition, we

---

Reconstruction
FullGrad 70
Map (3-rd Stage)
Score Map (2-nd Stage)
Score Map (I-st Stage)
Cls. Path: 29.41dB
Confidence: 81.26%
The score maps are inverted for better visualization (higher for the side path).
C.88
0.8
00
-04
04
0.
-02
0.2
62
G.
0.0
6.0
MSE Path: 29.50IB
Confidence: 37.55%
1.0
+.8
002
0.8
0666
066
0.0
0.4
0.4
0.4
0.2
022
62
Perc. Path: 28.65dB
Confidence: 77.46%
Bit Allocation
Original: va va100035565
100
50
Figure 4: Visualization of the reconstructed images, FullGrad 770 and score maps. The image is from ImageNet [14] and resized to 256x256xx The regions with warmer colors in FullGrad have larger gradients, indicating a stronger impact on the classification decision. The bitrate is 0.0888bpp.
add HiFiC [6 and VTM17.1 [ for comparison as the anchors of fully perceptual optimization and traditional coding method, respectively. For machine vision, the classification task is the understanding of the global semantic information of an image and represents high-level vision task. In contrast, the segmentation task challenges the model's ability to understand pixel-level semantics and represents low-level vision task. Thus, we test the reconstructed images on classification and semantic segmentation tasks to comprehensively evaluate the performance of MPA for machine vision. compare MPA to SOTA baselines with task-specific optimization, SFT 669 and TransTIC [[], along with VTM17.1 (1.
As shown in Fig. MPA can achieve performance comparable existing SOTA models. For human vision, the FID of MPA is lower than that of the best best-performing model, CRDR, at a low bitrate. Since perceptual performance is more reflective of the human visual system low bitrate, this part of the gain is significantly beneficial. Considering the distortion, even with only partial parameters optimized for MSE, MPA still has a wide adjustable range comparable other fully fine-tuned models, especially at low bitrate. Note that we achieve such performance with smaller decoder (9.295M for MPA vs. 13.38M for CRDR), fewer training steps (3.5M steps for MPA vs. 5M steps for CRDR), and partial fine-tuning (7.27% for MPA and 100% for CRDR). In terms of machine vision, pperes det and @veg perform significantly better than models optimized for MSE, outperforming SFT and TransTIC which are specially optimized for vision tasks. With only 1.89% parameters fine-tuned for machine vision. MPA can even achieve accuracy comparable to the fully fine-tuned models, showcasing its powerful task transition capability. Note that any point in the colored areas in Fig. 3 can achieved by adjusting & and a, as MPA supports continuously variable-rate coding and seamless transitions between all tasks using unified model.
We choose the perceptually optimized path serve the main path due to its ability to preserve high-level semantic features that are useful across various tasks. This generalization achieved by using pre-trained classification models for measurement (VGG (68] for training and AlexNet [40] for testing). This method aligns well with both human and machine perceptions in the latent space [8I] The pre-trained VGG implicitly incorporates label information, making the distribution of the reconstructed image semantically closer the original one, thus reducing
Discussion.

---

Reconstruction
Segmentation Map
Map (3-rd Stage)
Score Map (2-nd Stage)
Score Map (I-st Stage)
Seg. Path: 31.62BB
The score maps are inverted for better visualization (higher for the side path).
C..8
C.6
0.
-04
0.4
0.4
-22
0.2
602
G.M
0.0
6.0
MSE Path: 31.871B
1.0
+08
02
0.8
006
06
0.0
-44
04
0.4
0.2
02
62
Perc, Path: 31,30dB
Bit Allocation
Original: va. 00000269
Ground Truth
20
Figure 5: Visualization of the reconstructed images, segmentation maps and score maps. The image from ADE20K [8] and resized to 512x512x3. The bitrate is 0.071 0.0718bpp.
perceptual distance and enhancing accuracy. Additionally, det. and Oseg use task-specific pre-trained models to compute cross-entropy loss (image-wise for classification and pixel-wise for segmentation), directly targeting task-specific accuracy. These constraints, along with semantic information from the first training stage, allow Oets and @seg to achieve higher accuracy than pper with lower computational complexity. Moreover, perceptually optimized features can be easily adapted for MSE optimization, achieving higher PSNR. Perceptual loss preserves high-level semantic features, which supports the minimization of pixel- exee-wise distortion when fine-tuned with MSE loss especially Thus, perceptual optimization not only unify human and machine vision tasks but also enhance performance in traditional metrics like PSNR, making it suitable method for the generalized model.
low bitrate (18).
5.3 Visualization
To investigate the effects of different paths, visualize the qualitative results in Figs. 4nd MSE optimization prioritizes image textures, with fine-grained textures scoring higher, while analysis tasks focus semantic regions. Score prediction differences each stage highlight distinctions between MSE and machine vision perception. MSF-optimized features show significant score differences lower-level stages (Ist and 2nd), enhancing PSNR by improving textures. Conversely, machine vision-optimizee features show significant score differences at all stages, indicating that both high- and low-level semantic features crucial for analysis tasks. Consequently, images optimized for MSE without semantic supervision are visually smoother at low bitrate but may lose important semantic features, reducing accuracy. Paths optimized under LPIPS supervision retain more semantic information, ensuring the reconstructed image is semantically closer to the original one. By introducing task loss in Eq. (8). images decoded by éets and dveg have more task specifcc features for classification and segmentation respectively, leading to higher accuracy than Opere.
5.4 Diving into MPA
To further explore the configuration of the MPA, we perform series of evaluations. Kodak [39] ImageNet-1K [14] and ADE20K [83] are used for evaluate BD-rate (6), top-1 accuracy and mloU respectively. BD-rate is computed over the whole R-D curve. Top-1 accuracy and mloU are computed

---

Table Effects of path complexity
Table 2: Cross-validations on path choices
MLP Type
UMSE (BD-Rate J)
Oets (Acc. 1)
tses
Task (Metric)
Operc
ÓMSE
Ocls
Oveg
(mloU 1)
MSE (BD-Rate U)
49.61 %
16.04% %
32.81%
34.19%
Bottlenock
19.51% 16.04%
76.72%
37.41%
Cls. (Acc. 1)
76.66%
60.59%
76.77 %
73.57%
Inv. Bottleneck
77.16%
37.76%
Seg. (mloU T)
36.17%
28.34%
35.34%
37.41%
Table 3: Ablations on encoder
Table 4: Comparison complexity
BD-Rate $
Models
#Param.
KFLOPs per pixel
Latency (ms)
Components
against VTM
MRIC
69.14M
1118.17
11.89
Tic 55
28.46M
439.29
12.68
Full MPA 16.04%
MLPs
+0.51M~~42.04
-56.68-40
-0.33~-40
w/o Predictors
16.25%
Predictors
+0.03M
+2.23
+0.09
w/o ónq 17.05%
S & A
+0.28
w/o Oig
17.18%
MPA 29. 29.00M-30.53M 484.54
12.72~11.05
at the lowest bitrate, i.e., 0.1521bpp on ImageNet-IK and 0.0948bpp on ADE20K. For the effects of complexity, revealss that human iiiinterrenttdd tasks are more sensitive to the complexity of the paths, which suggests the use of inverted bottleneck MLPs, while machine dision-orientee paths use bottleneck MLPs because the complexity has relatively less impact the accuracy.
As for the choice of paths, we conduct cross-validations as shown in Table which demonstrates that the paths do learn the task-specific features to make their corresponding tasks perform optimally. To evaluate the effect of MPA in the encoder, we conduct ablations on each MPA component. We replace the predicted mask with random mask and disable onq and dig during encoding. The results in Table 3 demonstrate that the predictors capture different features critical to high and low bitrate compression, and dng and Dig are specialized to corresponding features respectively.
To evaluate the generalization of the learned features for the same task, test dets Swin Transformer V2 [52]. Fig. f|shows that dets has a similar performance gain com- parable to that on Xo [54] which is originally used for training. Note that dets is only trained once without
ImageNet-1K SwinV2-T)
MPA (Full Cls.)
MPA (Full Perc.)
Top-1 Acc. (9))
MPA (Full MSE)
75-
TransTIC
fine-tuning Swin V2, reflecting the generalization of the learned features in MPA.
TransTIC (Base)
SET (GraCCMM
70
(Uniform)
The complexity of MPA is shown in Table Here, the input size is 256x256x3. Latency is averaged over a batch of 16 images running 5000 times RTX3090 GPU through an encoder, a hyperprior entropy model, and decoder. With the negligible computational overhead, MPA achieves comparable performance to the full fine- tuning model and has seamless transitions between tasks. Figure 6: Validation on Swin V2-T 52
TInyLIC
65
Pull Fine tuning VTM17.1
Uncompressed
60
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
1.8
(bpp)
The main limitations of MPA are its increased latency compared to the baseline and the need for separate training for each task. The former can be mitigated by developing specialized operator to eliminate the frequent flattening, selection, and reassembly operations, while the latter can be addressed by leveraging multi-task learning techniques [72], which will be explored future work.
6 Conclusion
In this paper, we propose Multi-Path Aggregation (MPA) to tackle the problem of coding for multi- task applications an all-in-one manner. By aggregating task task- specific paths, MPA can support variety of tasks with seamless transitions between them for joint human-macee machine vision using a unified model and unified representation. Merely fine-tuning as low as 1.89% additional parameters, MPA can achieve comparable performance to that of separable models optimized using dedicated criteria, showcasing its strong versatility and scalability. Future work will explore joint optimization of multiple paths to further unify training process and improve multi-task performance.

---

Acknowledgments and Disclosure of Funding
This work was supported in part by Natural Science Foundation of Jiangsu Province under Grant BK2024 1226 and BK20243038, and Jiangsu Provincial Key Research and Development Program under Grant BE2022155. The authors would like to express their sincere gratitude to the Interdis- ciplinary Research Center for Future Intelligent Chips (Chip-X) and Yachen Foundation for their invaluable support.
References
[1] Versatile video coding reference software version 17.1 (VTM-17.1). https://vogit: hhi /ecuutoofer.d///eee/e//oovvoaaea
17.1 July 2022.
[2] Firikur Agustsson, David Minnen, George Toderici, and Fahian Mentzer. Multi-realism image compression
with a conditional generator. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition ( (CVPR), pages 22324-22333, June 2023.
[3] Eirikur Agustsson and Radu Timofte. Ntre 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Workshops, July 2017.
[4] Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified vision-language pre-training with mixture-of-modalii experts. In Advances in Neural Information Processing Systems, volume 35, pages 32897-32912. Curran Associates, Inc., 2022.
[5] Jean Bégaint, Fabien Racapé, Simon Feltman, and Akshay Pushparaja. Compressai: a pytorch library and evaluation platform for end-to-end compression research. arXiv preprint arXiv:201 2020.
[6] Gisle Bjontegaard. Calculation of average psnr differences between rd-curves. ITU SG16 Doc. VCEG-M33, 2001.
[7] Lahiru D. Chamain, Fabien Racapé, Jean Bégaint, Akshay Pushparaja, and Simon Feltman. End-to-end optimized image compression for machines, study. In 2021 Data Compression Conference (DCC), pages 163-172, 2021.
[8] Lahiru D. Chamain, Fabien Racapé, Jean Bégaint, Akshay Pushparaja, and Simon Feltman. End-to-end optimized image compression for multiple machine tasks, 2021.
[9] Tianlong Chen, Xuxi Chen, Xianzhi Du Abdullah Rashwan, Fan Yang, Huizhong Chen, Zhangyang Wang, and Yeqing Li. Adamy-moe: Adaptive multi-task vision mixture-of-expee In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 1 17346-1753 October 2023.
[10] Tong Chen and Zhan Ma. Variable bitrate image compression with quality scaling factors. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2163-2167, 2020.
[11] Yi-Hsin Chen,
Weng, Chia-Hao Kao, Cheng Chien. Wei-Chen Chiu, and Wen-Hsiao Peng. Transtic: Transferring transformer- based image compression from human perception to machine perception. In Proceedingy of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 23297- 23307, October 2023.
[12] Hyomin Choi and Ivan V. Bajić. Scalable image coding for humans and machines. IEEE Transactions on Image Processing, 31:2739-2754, 2022.
[13] Ze Cui, Jing Wang, Shangyin Gao, Tiansheng Guo, Yihui Feng, and Bo Bai. Asymmetric gained deep image compression with continuous rate adaptation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 10532-10541, June 2021.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
[15] Nikolaos Dimitriadis, Pascal Frossard, and François Fleuret. Pareto manifold Icarning: Tackling multiple tasks via ensembles of single-task models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 8015-8052. PMLR, 23-29 Jul 2023.
[16] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
10

---

[17] Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P Bosma, Zongwei Zhou, Tao Wang, Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hlellstem, Toju Duke, Lucas Dixon, Kun Zhang, Quoe Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. GLaM: Efficient scaling of language models with -o-uuree---xpeetts... In Proceedings of the 39th International Conference Machine Learning, volume 162, pages 5547 5569. PMLR, 17-23 Jul 2022.
[18] Zhihao Duan, Ming Lu, Zhan Ma, and Fengqing Zhu. Lossy image compression with quantized hierarchical vaes. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 198-207, January 2023.
[19] Zhihao Duan, Zhan Ma, and Fengging Zhu. Unified architecture adaptation for compressed domain semantic inference. IEEE Transactions on Circuits and Systems for Video Technology, (3(8))41084412) 2023.
[20] Matthias Ehrgott. Multicriteria optimization, volume 491. Springer Science & Business Media, 2005,
[21]
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(00)111-9, 2022.
[22] Ruoyu Feng, Xin Jin, Zongyu Guo, Runsen Feng. Yixin Gao, Tianyu He, Zhizheng Zhang, Simeng Sun, and Zhibo Chen. Image coding for machines with omnipotent feature lcarning. Computer Vision ECCV 2022, pages 510-528, Cham, 2022. Springer Nature Switzerland.
[23] Changsheng Gao, Dong Liu, Li Li, and Feng Wu. Towards task -enerric image compression: A study of semantics-oriented metrics. IEEE Transactions on Multimedia, 25:721-735, 2021.
[24] Noor Fathima Ghouse. Jens Petersen, Auke Wiggers, Tianlin Xu, and Guillaume Sautière. A residual diffusion model for high perceptual quality endec augmentation, 2023.
[25] Jacob Gildenblat and contributors. Pytorch library for cam methods. 2021.
com/
Decobg1//pytorch-g0
[26] Alon Harell, Anderson De Andrade, and Ivan V. Bajić. Rate-distortion in image coding for machines. In 2022 Picture Coding Symposium (PCS). pages 199 203, 2022.
[27] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, June 2016.
[28] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[29] Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Reprexentations, 2022.
[30] Yucyu Hu, Shuai Yang, Wenhan Yang, Ling- Yu Duan, and Jiaying Liu. Towards coding for human and machine vision: A scalable image coding approach. In 2020 IEEE International Conference on Multimedia and Expo (ICME), pages 1-6, 2020.
[31] Yuzheng Hu, Ruicheng Xian, Qilong Wu, Qiuling Fan, Lang Yin, and Han Zhao. Revisiting scalarization in multi-task learning: A theoretical perspective. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 48510 48533. Curran Associates. Inc., 2023.
[32] Shoma Iwai, Tomo Miyazaki, and Shinichiro Omachi. Controlling rate, distortion, and realism: Towards single comprehensive neural image compression model. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2900-2909, Ja00-ay 2024.
[33] Shoma Iwai, Tomo Miyazaki, Yoshihiro Sugaya, and Shinichiro Omachi. controllco extreme image compression with generative adversarial networks. In 2020 25th International Conference on Puttern Recognition (ICPR), pages 8235-8242, 2021.
[34] Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79-87, 1991.
[35] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations, 2017.
[36] Menglin Jia, Luming Tang, Bor-Chun Chen. Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision Nature Switzerland.
ECCV 2022, pages
Cham, 2022. Springer

---

[37]
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
[38]
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In the 3rd Int. Conf. on Learning Representations, 2015.
[39] Eastman Kodak. Kodak lossless true color image suite (photocd pcd0992), 1993.
[40] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25, Curran Associates, Inc., 2012.
[41] Nikolai Körber, Eduard Kromer, Andreas Siebert, Sascha Hauke, Daniel Mueller- Crischer-r and Bjöm Schuller. Egic: Enhanced low-bit-rate generative image compression guided by semantic segmentation. 2023.
[42] Nam Le, Honglei Zhang, Francesco Cricri, Ramin Ghaznavi-Youn Youvalari, and Esa Rahtu, Image coding for machines: an end-to-end learned approach. In ICASSP 202/ 2021 IEEE International Conference on Acousticx, Speech and Signal Processing (ICASSP), pages 1590-1594, 2021.
[43] Jooyoung Lee, Seyoon Jeong, and Munchurl Kim. Selective compression learning of latent representations for variable-rate image compression. In Advances in Neural Information Processing Systems, volume 35, pages 13146-13157. Curran Associates, Inc., 2022.
[44] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. (GS fhard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.
[45] Huanyang Li and Xinfeng Zhang. Human
Human-machine machine collaborative image compression method based on implicit neural representations. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, pages 1-1, 2024.
[46] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays. Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects context. David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision - ECCV 2014, pages 740-755, Cham, 2014. Springer International Publishing.
[47] Xi Lin, Hui-Ling Zhen, Zhenhua Li. Qing-Fu Zhang, and Sam Kwong. Pareto multi-task learning. In H. Wallach, H. Larochelle, A. Beygelzimer, d' d'Alché-Bue, Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[48] Jiaheng I Liu, Guo Lu, Zhihao Hu, and Dong Xu. A unified end-to-end framework for efficient deep image compression. arXiv preprint arXiv:2002.0333 037.000..33 2020.
[49] Jinming Liu, Heming Sun, and Jiro Katto. Improving multiple machine vision tasks in the compressed domain. In 2022 26th International Conference on Pottern Recognition (ICPR), pages 331-337, 2022.
[50] Jinming Liu, Heming Sun, and Jiro Katto. Learned image compression with mixed transformer-cnn architectures. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14388-14338 June 2023.
[51] Kang Liu, Dong Liu, Li Li, Ning Yan, and Houqiang Li. Semantics-to-signal scalable image compression with learned revertible representations. International Journal of Computer Vision, 129(9):2605 2621, 2021.
[52] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong. Furu Wei, and Baining Guo. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 22009-12019, June 2022.
[53] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF
International Conference Computer Vision (ICCV), pages 10012-10022 October 2021.
[54] Zhuang Liu, Hanzi Mao, Chao-Yuaa Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xic. convnet for the 2020s. In Proceedings of the IEÉE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11976-11986, June 2022.
[55] Ming Lu, Fangdong Chen, Shiliang Pu, and Zhan Ma. High-efficiency lossy image coding through adaptive neighborhood information aggregation. arXiv preprint arXiv:2204. 1448, 2022.
[56] Ming Lu, Peiyao Guo, Huiqing Shi, Chuntong Cao, and Zhan Ma. Transformer-bast based image compression. In 2022 Data Compression Conference (DCC), pages 469-469. IEEE, 2022.
[57] Pingchuan Ma. Tao Du and Wojeiech Matusik. Efficient continuous pareto exploration multi-task learning. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 6522-6531, PMLR, 13-18 Jul 2020.
12

---

[58] Debabrata Mahapatra and Vaibhav Rajan. Multi-task learning with user preferences: Gradient descent with controlled ascent in pareto optimization. In Hal Daumé III and Aarti Singh, editors. Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 6597-6607, PMLR, 13-18 Jul 2020.
[59]
Vision maintainers and contributors. Torchvision: Pytorch's computer vision library. attps
//github. www.//ytttc///.tt.ccoooo s onl 2016.
[60] Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser-Nam Lim. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patiern Recognition (CVPR), pages 12309-12318, June 2022.
[61] Fabian Mentzer, George D Toderici, Michael Tschannen, and Eirikur Agustsson. High-fidelity generative image compression. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 11913- 11924. Curran Associates, Inc., 2020.
[62] Kaisa Miettinen. Nonlinear multiobjectiive optimization, volume 12. Springer Science & Business Media, 1999.
[63] Michinari Momma, Chaosheng Dong, and Jia Liu. A multi-objective / multi-tast learning framework induced by pareto stationarity. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 15895-15907. PMLR, 17-23 Jul 2022.
[64] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan. Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang. Zachary VeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, tegh-perforance deep learning library. In Advances in Neural Information Processing Systems, volume 32, pages 8024-8033 8035. Curran Associates, Inc., 2019.
[65] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6):1137-11499 2017.
[66] Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In S. Bengio. H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett. editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[67] Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The paarellyggtted ofxture-of-expper layer. In International Conference on Learning Representations, 2017.
[68] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In International Conference on Learning Representations (ICLR), 2015.
[69] Myungseo Song, Jinyoung Choi, and Bohyung Han. Variable-rate deep image compression through apatially-adaptive feature transform. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2380-2389 October 2021.
[70] Suraj Srinivas and François Fleuret. Full-gradient representation for neural network visualization. In Advances in Neural Information Processing Systems, volume 32, pages 4124-4133. Curran Associates, Inc., 2019.
[71] George Toderici, Wenzhe Shi, Radu Timofte, Lucas Theis, Johannes Balle, Eirikur Agustsson, Nick Johnston, and Fabian Mentzer. Workshop and challenge on learned image compression (clic2020). 2020.
[72] Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansheke, Mare Proesmans, Dengxin Dai, and Lue Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 44(7)33114-36)
2022.
[73] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[74] Shurun Wang, Shiqi Wang, Wenhan Yang, Xinfeng Zhang, Shanshe Wang, Siwei Ma, and Wen Gao. Towards malysis-friendly face representation with scalable feature and texture compression. IEFF Transactions on Multimedia, 24:3169-3181, 2022.
[75] Shurun Wang, Zhao Wang, Shiqi Wang, and Yan Ye. Deep image compression toward machine vision:
A unified optimization framework. IEEE Transactions Circuits and Systems for Video Technology. 33(6):297-22999, 2023.
[76] Wenhui Wang. Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as a foreign language: Beit pretraining for all vision and vision- language tasks. arXiv preprint 2200v:2208. 10442, 2022.
13

---

[77]
Ross
Wightman.
Pytorch image models.
https://github. com/ rwightman/
bytorch-imaçe image- 2019.
[78]
Yuxin Wu, Alexander Kinillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Delectron2. ht httpa
V/github. com/ httpp//sw......ccoooooeeeeeaaeee
2019.
[79] Fuzhao Xue, Ziji Shi. Futao Wei. Yuxuan Lou, Yong Liu, and Yang You. wider instead deeper. Proceedings of the AAAI Conference on Artificial Intelligence, 36(8):8779-88)) -77777777777 Jun. 2022.
[80] Ning Yan, Changsheng Gao, Dong Liu, Houqiang Li, Li Li, and Feng Wu. Sssie: Semantics-to-sigee scalable image coding with learned structural representations. IEEE Transactions on Image Processing, 30:8939- 8954, 2021.
[81] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features a perceptual metrie. In Proceedings of the IEFF Conference on Computer Vision and Pattern Recognition (CVPR), pages 586-595, June 2018.
[82] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid seene parsing network. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition (CVPR), pages 2881-2890, July 2017.
[83] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302-324 2019.
[84] Yinhao Zhu, Yang Yang, and Taco Cohen. Transformer-bass transform coding. In International Conference on Learning Representations, 2022.
14

---

Appendix
A Broader impacts
MPA provides a flexible and task task-oriented optimization approach for a unified coding framework. By leveraging the different importance of features for different tasks and feature correlations across tasks, MPA can empower existing learned image compression models with the ability reconstruct images with various orientations based on the needs of users or machines (such as fidelity and accuracy), significantly enhancing their versatility. This research not only proposes and sets a new baseline for multi-task coding but also offers a new avenue for the exploration of joint human-machine vision. By exploiting generalized features and few-step fine-tuning for a tiny proportion of parameters, MPA enables joint human-maching vision for multiple tasks in an all-in-one manner, without massive handcrafted techniques. This advancement represents a significant step toward creating a versatile coding framework that can adapt to wide range of applications and requirements. This can have positive societal impact by saving a massive amount of storage space, as only a unified model and unified representation need to be stored rather than multiple coding models and representations. By reducing storage requirements, it can lower the cost of data storage and transmission, making it more accessible for a wide range of applications. Additionally, this reduction in storage can lead to more efficient use of resources and energy, contributing to environmental sustainability. Lower storage costs and improved resource efficiency also facilitate broader adoption of advanced coding technologies in various industries, enhancing scalability and operational efficiency. Although there are limitations in scenarios requiring extremely high fidelity or accuracy, which may lead to potential negative societal impacts such as misjudgments on reconstructed images, these issues can be addressed by future work aimed performance improvements the development of solutions tailored to specific scenarios.
B Implementation details
B.1
Architecture
Our implementation is based on TIC 55 The multi-path mechanism is fulfilled by replacing the default Single-Path Residual Neighborhood Attention Block (SP-RNAB) with Multi-Path RNAB (MP-RNAB, i.c.. MP-Block in Fig. 2 at the first three stages of the main encoder and decoder (see Fig. 7 To build a continuously variable-rate baseline, we introduce the scaling factor (SF) modulation sy [00 and non-linear interpolation [33 43 as Eq. TOpp. The SF layer and its inverse operation (ISF) are embedded for the latent representation and each Transformer Block. Other settings are consistent with TInyLIC.
8g:
if E (1,2, 0max), otherwise.
8g
(g- Ial)
52 2-19) [
(10)
In terms of path configuration, ÓMSE) and bottleneck MLPs for machine vision (i.e., det and @seg). as shown in Fig g The bottleneck MLP is used to address low complexity demands, while the inverted bottleneck MLP is used to address high complexity demands. Specifically, the bottleneck MLP first down-samples input channels to C" followed by GELU activation, then up-samples to C" with activation and then down-samples it to C. In our implementation, we set C" 2C as TinyLIC (55] does, and set C" C/2 to align the ppddown-samplin ratio.
inverted bottleneck MLPs for human vision (i.e., Opere and
C, while the inverted one first up-samples C
B.2 MPA in encoder
As we mentioned in Sec. 3.2 we also implement MPA in the encoder, i.c., dny and dig. Setting 3 in Eq the relationship between and the aggregation ratio Penc is visualized in Fig. which fits an approximate linear relationship between bitrate and Pene on both Kodak (39 and CLIC 7
B.3 Reproducibility
All experiments are conducted on server with two Intel Xeon Silver 4210 CPUs and a single NVIDIA RTX3090 GPU. Our implementation relies on the open-source repository of TinyLIC (55).
15

---

Encoder
Hyper Encoder
Impur Image
1%
Convis
MP -NAR
Conv
MP RNAB
Convi 2
MP -NAA
Conv(3.2)
RN-A
Conv 21
SP RN-A
Convi 22
SP 0P-RNAB
X4
xd
Xd
xd.
Factorized Entropy Model
Context Model
AD
Rec Image
TConv/5,2 52
MP RN-B
TConw,3,22 2)
MP -NAB
TConw(3 2)
MP RNAB
TConv(3.2)
SP-RNAB
f
TConv(3,2)
SP. RN-
TConv/3,2)
SP-RNAB
ISF
ISF
Decoder (C)
Hyper Decoder
Entropy Model PC)
Figure MPA in TInyLIC [55]. d, is the number of RNABs used i-th stage. Convolution Conv(k.s)
and its transposed version TConv(k,s) apply the kernel at size of x k and a stride of s. Uniform
Quantization is used in Q: AE and AD stand for respective Arithmetic Encoding and Decoding.
Linear(C,c')
Linear(C,c")
GELU
GELU
Linear(C', ,C)
Linear(C" ,C)
Bottleneck MLP
(b) Inverted Bottleneck
Figure 8: Two types of MLP. Linear(C, C²) denotes fully-connected layer with C input channels and output channels. set C' C/2 and C" 2C in our implementation.
(B=5)
1.0
1.0
0.8
Rotio
Rado
0.4
0.2
0.2
Kodak
CLIC
0.0
0.0
5
0.0
0.2
0.4
0.6
0.B
1.0
1.4
1.6
Biturate (bpp)
Figure 9: The relationship between q and the aggregation ratio Paac-
16

---

which is based on PyTorch [64] and CompressAl [5]. The open-source baselines, i.e., SFT [69 and TransTIC [TT] are re-tested using their public repository. For the comparison baselines without public code and weights, we align our experimental setup with theirs. ConvNeXt Tiny (54 and Swin Transformer V2 Tiny [52] are created using timm [77] with officially published pre-trained weights. PSPNet30 (54] and its pre-trained weights are obtained from the authors' official repository.
FullGrad 70 is visualized using an open-source PyTorch library [25].
C
Comparison to other related work
Ketture-of-Experre (MoE) 34 67
enables Transformer models 73 16 to have non-uniform
computing patterns, improving downstream task performance [44 17
gaining multi-modality modelling capabilities 4 76 or enabling multi-task learning [9] They replace the Ferw-Forward
21
79
Network (FFN) with multiple experts and routing tokens via gating networks to the appropriate experts for inference. Unfortunately, the strengths of MoE are most predominantly represented in large language models due the scaling law [37], which is not suitable for storage- and complexity- sensitive scenarios. In comparison, MPA is quite different from the common practice of MoE, as listed in Table 5
Table 5: Comparison between MPA and MoE
Aspect
MPA
MoE
Architecture
Unified all-in-one model with multiple task-specific paths
Multiple expert models selected by gating network
Optimization
Two-stage optimization with partial pa- rameter fine-tuning
Joint optimization with expert selection via gating network
Routing
Leverages feature importance and corre- lations across tasks
Utilizes the expertise of different experts for specific inputs
Efficiency
Designed specifically for storage- and computation-ssenstivee multi-task scenar- ios
Specially used in large models to sig- nificantly increase parameter count for performance boost
Usage
User-defined selection of task paths
Expert allocation entirely decided by the gating network
Low-Rank Adaptation (LoRA) 29 is technique that reduces the number of trainable parameters in large pre-trained models by factorizing weight matrices into low-rank forms, enabling efficient fine-tuning with minimal computational cost while maintaining performance across various tasks. Our experiments recognize the effectiveness of LoRA in Table 6 Although MPA involves more fine-tuning parameters, it also achieves better performance. Moreover, our work has distinct features. We utilize predictors support coding for multi-task applications and smooth transitions between tasks within an all-in-one framework. The low-rank structure design of LoRA inspires the future work to consider improvements for the side path.
Table 6: R-D performance comparison between MPA and LoRA. The ranks of LoRA are set to 64, 16, and respectively, and the lora lora_alpha (NOT the MPA) is fixed 1. We optimize MSE paths for low distortion. Other experimental settings are consistent with those in Sec.
BD-Rate against VTM
Dataset
MPA (7.27% ft.)
LoRA (3.04% ft.)
LoRA (0.83% ft.) LoRA (0.28% ft.)
Kodak 39
16.04%
16.26%
16.39%
16.83%
CLIC 7
21.43%
21.80%
21.95%
22.59%
17

---

D Loss functions
Our loss functions follow the same forms and hyperparameters those in 21 which have been thoroughly evaluated. For the task loss Lusk we propose, we provide detailed ablation studies in Table ( to demonstrate that our current combination achieves a competitive trade-off. Regarding the role of each term, LPIPS enriches the semantic information of the reconstructed images, improving generalization; MSE loss enforces pixel value consistency between the reconstructed and original images; and the cross-entropy loss directly optimizes classification accuracy.
Table 7: Ablations loss terms in Lusk- We the same experimental settings in Sec. 4 and re-train the classification path for each case. Metrics are evaluated at 0. 0.p521bpp on ImageNet- IK [14]. The top-3 results underlined. LMSE, fpere and Loe denote MSE loss, perceptual loss and cross-entropy loss respectively.
Top-1 Acc. 1 on ConvNeXI-T
Top-1 Acc. t
No.
MSSE
Epere
Lee
PSNR
LPIPS +
on SwinV2-1 52
1
26.7dB
0.330
62.22%
59.27%
2
V
25.7dB
0.229
74.21%
72.25%
3
V
24.2dB
0.305
76.79%
75.16%
4
V
V
25.7dB
0.244
77.01%
75.96%
5
V
V
26.5dB
0.275
76.17%
74.90%
6
V
26.6dB
0.238
74.28%
73.01
7
V
V
v
26.5dB
0.244
76.77%
75.61% 61%
E
Licenses for released assets
We list the licenses for the released assets we in Table g
Table 8: Licenses for released assets
Asset
License
TInyLIC
55
Apache-2.0 license
TransTIC
Apache-2.0 license
SFT 69
Not specified
ConvNeXt 54
MIT license
PSPNet 82
MIT license
Swin Transformer V2
MIT license
ResNet [7] (in Torch Vision
55
BSD-3-Clause license
Faster R-CNN 65
(in Detectron2
78
Apache-2.0 license
VTM17.1 I
BSD license
Flicker2W 48
Custom (rescarch-only)
DIV2K
Custom (research-only)
CLIC
Not specified
Kodak 339
Not specified
ImageNet- 14
Custom (research-only, non-commercial)
ADE20K 83
Custom (research-only, non-commercial)
MS-COCO 2017 46
Custom
18

---

F
More results
E1
Quantitative results
Kodak (Distartion)
CLIC (Distortion)
36
30
PSNR (dB)
34
PS5N (dB)
36
MPA (Pull MSE) (Perc. Main)
MPA (Full MSE) (Perc. Main)
MPA (Pull Perc.) (Ponc. Main)
MPA (Pull Perc.) (Perc. Main)
32-
MPA (Full MSE) (MSE Main)
MPA (Full MSE) (MSE Main)
MPA (Full Perc.) (MSE Main)
MPA (Full Perc.) (MSE Main)
VTM17.1
30
VTM17.1
0.2 0.3 0.4 0.5 0.7 0.8 0.9
1,2
0.1
0.4 0.5 0.6 0.7 0.8 0.9
Bit-rate (bpp)
Bit-rate (bpp)
Kadak (Realism)
CLIC (Realism)
MPA (Full MSE) (Perc. Main)
MPA (Full MSE) (Pere. Main)
0.175
MPA (Full Perc.) (Perc. Main)
35
MPA (Full Perc.) (Pert. Main)
0.150
MPA (Full MSE) (MSE Main)
MPA (Full MSE) (MSE Main)
MPA (Full Perc.) (MSE
MPA (Full Perc.) (MSE Main)
0.125
VIM17.1
25
VTM17.1
LPIPS
0.100
0.075
15
0.050
10
0.025
5
0.000
0.1 0.2 0.3
0.4
0.5
0.6 0.7 0.8
0.9
1.0
111
1.2
1.3
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Bit-rate (bpp)
Bit-rate (bpp)
Figure 10: Multi-task performance with optimizing MSE path the main path. The results reveal that perceptual optimization is more generalized than MSE optimization for multi-task applications. Although MSE optimization achieves lower distortion, its perceptual quality is more severely degraded compared to that of perceptual optimization, with significantly narrower adjustable range.
20.0
0.200
MPA (0-1.0) 1.0)
XEP (a- 00
38
17.5
MPA (m-0 (m-0.8571)
MTM (rm0 7182)
0.175
MPA (au0 8571)
MPA (sm0 7142)
(x-0
(s-0.5714)
8571)
MDA (c-0
30
MPA (o-0.7142)
MPA 2857))
XEA (a-0.2857)
MFA (0514)714
MPA (x-0 1420)
XER (8-0.1428)
4285)
MPA 6m
1142)
MBL (c000087)
(0-0.0657)
PSN8R:
MPA 1428)
10.0
MPA 0.071)
0.100
XPA 0.001)
MEA 1142)
MPA (m-0 0285)
MPA (2u0 0285)
MPA Ja-0 0857)
7.5
MTA 6rm0 01)
0.075
MPA (om0 01)
(.-0.001)
0013
22-
0265)
5.0
MEA (c-0
0.050
MPA to-0.011
MEA (0-0.00 0.0)
XEA (2-0.0)
MES to-0.001]
0001)
0.1
0.2 0.3 044 0.5
0.6
0.7
0.8 09 1.0
0.1 0.2
0.3
0.4
0.5
0.6 0.7
0.0
0.9
1.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8 0.9 1.0
Ribente (hpp)
Bilendt lkpp!
Figure 11: R-D and R-P performance of MPA with variable a on CLIC dataset 71
19

---

Kodak (Distortion)
Kodak (Realism)
38
0.200
(a-1.0)
0.175
36
MPA (a=0.7142)
MPA (a-1.0)
MPA (o-0.5714)
MPA (Q-0.8571)
MPA (a=0.4285)
MPA (o=0.7142)
MPA (a=0.2857)
34
(m-0.5714)
0.125
MPA (am0.1428)
PSNR (dB)
(a=0.1142)
MPA (m=0.2857)
LPIPS
32
0.100
MPA (a=0.1428)
MPA (a=0.0571)
MPA (a=0.1142)
MPA (a-0.0285)
MPA (a-0.0857)
0.075
MPA (a-0.01)
30
MPA (a-0.0571)
MPA (a=0.001)
(a=0.0285)
0.050
(a=0.0001)
(a-0.01)
(a=0.0)
28
MPA (a=0.001)
MPA (a=0.0001)
0.025
MPA (u=0.0)
26
0.000
0.1 0.2 0.3
0.5 0.6 0.9 1.0 1.3
0.1 0.2 0.3 0.4
0.5 0.6 0.7 0.8 0.9
1.0
1.2 1.3
Bit-rate (bpp)
Bit-rate (bpp)
Figure 12: R-1) and R-P performance of MPA with variable a on Kodak dataset 39
ImageNet-IK (Cls., ResNet50)
COC02017 (Det., Faster R-CNN R50 FPN 3x)
40
75
39
MPA (Pull Cls.)
Top-1 Acc. (86
MPA (Full Perc.)
38
(Full
70
MPA (Pall MSR)
37
MPA (Puil Perr.)
TransTTC (VPT)
MPA (Pull MSE)
TransTIC (Base)
mAP
36
TransTIC (VPT)
SPT (GradCAM)
35
TransTIC (Base)
SFT (Unilorm)
TiayLIC Full taning
34
60
tuning VTM17.1
33
VTM17.1 Uncompressed
Uncompressed
32
55
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
1.8
0.05
0.10
0.15
0.20 0.25 0.30 0.35 0.40 0.45 (bpp)
(bpp)
Figure 13: Classification and objection detection performance MPA on ImageNet-IK [4] and MS-COCO 2017 [46]. ResNet50 [27] and Faster R-CNN R50 665 are used for inference. Top-1 accuracy and mAP (mean Average Precision) are used metrics for comparison.
ImageNNt-I1 (Distortion)
ADE20K (Distortion)
42
40
38
PSNR (dB)
Cls.)
36
(Full Seg.)
PSNR
MPA (Full Perc.)
MPA (Full Perc.)
MPA (Full MSE)
34
MPA (Full MSE)
TransTIC (VPT)
TransTIC (VPT)
32
TransTIC (Base)
TransTIC (Base)
Full Fine tuning
Full Fine-tuning
VTM17.1
VIM17.1
1.2
1.8
0.7 0.8 0.9 1.0
Bit-rate (bpp)
Bit-rate (bpp)
Figure 14: R-D performance of MPA on ImageNet-IK
14 and ADE20K 83
20

---

F.2
Qualitative results
Original
MRIC
MPA
MPA
MPA
MPA
MPA
MPA
MRIC H
0.0001)
0.01)
0.0540bpp
0,0505/ppp
0,0515hpp
0,0599bbp
0,0505/bpp
0.05050pp
0.0509hpp
0,0544hppp
3E273
31.16dB
30.0038
30.07aB
30.50aB3
31.08418
31.32dR
404B
32.17dR
0.0175bpp
0.0459bpp
0.0459bpp
0.0159bpp
0.0459bpp
0.0r596pp
0.0159bpp
0.0475bpp
32,29d88
31,35J8
31,4208
31,8248
32.52J8
32.8208
32.94dB
33,73J8
0.0755bpp
0.0636bpp
0.0636hpp
0.0633bbpp
0.0636/pp
0.06666ppp
0.0636bpp
0.0755bpp
29.36dB
28.443B
33.48:3B
30.01
30.09dB
0.0750bpp
0.066kbpp 29.1 1dH
0.0660bpp 29. 143H
0.0660lbpp 29, 40.1
0.0660hpp 1518
0.06606pp 30.8dRR
0.066/bpp 30.61d8
0.0750hpp
E5003
34dB
31.71d8
We visualize the reconstructed images MPA with variable a compared to MRIC 2 The image is from CLIC test set [71] and is cropped 256x256xx The selected paths éper and ØMSE Larger 3 for MRIC and smaller a for MPA stand for higher realism. Note that both the bitrate and the PSNR calculated uncropped images.
Figure 15:
21

---

Original: va130035565
Score Map (I-st Stage)
Score Map (2-nd Stage)
Score Map (3-rd Stage)
0.8
0.8
0.8
0.8
0.4
0.4
0.0
0.0
0.0
Allocation
1 (0.0888bpp)
(0. 166bbp)
(0.1630bpp)
(0.2294bpp)
200
300
150
150
200
100
50
50
5 (0.3 168bpp)
6 (0.4560bpp)
(0.6562hpp)
8 (0.9238bpp)
400
400
200
200
100
200
Figure 16: We visualize the bit allocation and score maps in the encoder. The image is from ImageNet validation set [14] and resized to 256x256x3. The 1st, 2nd, and 3rd stages are the first three stages of the encoder, consecutively.
22

---

Reconstruction
FullGrad
Map Stage)
Map (2-nd Stage)
Map (1-st Stage)
Cls. Path: 29.41dB
Confidence: 81.26%
The score maps are inverted for better visualization (higher for the side path).
0.0
0.6
06
0.4
04
0.4
-03
0.2
03
0.0
0.0
Seg. Path: 29.41dB
Confidence: 68.26%
-.8
0.8
0.8
-0.6
006
-06
0.4
04
04
-02
0.2
0.2
Path: 29.50dIB
Confidence: 37.55%
1.0
1.0
1.0
G.8
0.5
6.8
0.8
0.6
0.8
84
Perc. Path: 28.65dB
Confidence: 77.46%
Original: va.10033566
Figure 17: We visualize the reconstructed images, FullGrad 700 and score maps predicted by importance predictors in each path. The image is from ImageNet validation set [14] and resized to 256x256x3. The 3rd, 2nd, and 1st stages the last three stages of the decoder, consecutively. The regions with warmer colors in FullGrad represent areas with larger gradients, indicating a stronger impact on the classification decision. q set 1 for more distinct comparison. Note that the score maps are inverted for better visualization, i.c., larger scores indicate prioritized for entering the selected side path. The bitrate is 0.0888bpp.
23

---

Original: va130017577
Score Map (I-st Stage)
Score Map (2-nd Stage)
Score Map (3-rd Stage)
0.8
0.8
0.8
0.8
0.4
0.4
0.0
0.0
0.0
Allocation
1 (0.1152bpp)
(0. (6.6pp)bp
(0.2163bpp)
(0.2998bpp)
100
150
200
250
150
100
40
100
20
50
50
5 (0.4150bpp)
6 (0.5874bpp)
(0.8256hpp)
8 (1.1 (1.66pppp))
400
000
600
200
400
200
200
200
100
Figure 18: We visualize the bit allocation and score maps in the encoder. The image is from ImageNet validation set [14] and resized to 256x256x3. The 1st, 2nd, and 3rd stages are the first three stages of the encoder, consecutively.
24

---

Reconstruction
FullGrad
Map Stage)
Map (2-nd Stage)
Map (1-st Stage)
Cls. Path: 27.46dB
Confidence: 76.22%
The score maps are inverted for better visualization (higher for the side patb).
0.0
644
04
0.4
-02
0.2
03
0.0
0.0
Seg. Path: 27.18dB
Confidence: 62.71%
1.0
0.8
088
0.8
-0.6
00
-0.6
6.4
04
0.4
-02
0.2
0.2
27.85dB
Confidence: 44.31%
1.0
1.0
1.0
0.8
08
6.8
0.6
0.6
0.8
Pere. Path: 26.91dB
Confidence: 75.23%
Original: va100017577
Figure 19: We visualize the reconstructed images, FullGrad 770 and score maps predicted by importance predictors in each path. The image is from ImageNet validation set [14] and resized to 256x256x3. The 3rd, 2nd, and 1st stages the last three stages of the decoder, consecutively. The regions with warmer colors in FullGrad represent areas with larger gradients, indicating a stronger impact on the classification decision. q set 1 for more distinct comparison. Note that the score maps are inverted for better visualization, i.c., larger scores indicate prioritized for entering the selected side path. The bitrate is 0.1 1.1ppp.
25

---

Original: va100022926
Score Map (I-st Stage)
Score (2-nd Stage)
Score Map (3-rd Stage)
0.8
0.8
0.8
0.8
0.4
0.0
0.0
0.0
Allocation
1 (0.1279bpp)
(0.2343bpp)
(0.3984bpp)
(0.6533bpp)
125
300
100
150
200
50
50
25
50
5 (0.9980bpp)
6 (1.4140bpp)
(1.9096hpp)
8
(2.4282bpp)
600
600
400
200
400
100
100
Figure 20: We visualize the bit allocation and score maps in the encoder. The image is from ImageNet validation set [14] and resized to 256x256x3. The 1st, 2nd, and 3rd stages are the first three stages of the encoder, consecutively.
26

---

Reconstruction
FullGrad
Map Stage)
Map (2-nd Stage)
Map (1-st Stage)
Cls. Path: 24.80JB
Confidence: 83.21%
The score maps are inverted for better visualization (higher for the side patb).
-0.8
06
6.4
04
04
-03
0.7
03
0.0
0.0
Seg. Path: 24.80dB
Confidence: 64.81%
1.0
-..8
0.8
0.8
-0.6
00
-06
6.4
04
044
-0.2
0.2
0.2
25.
11.06%
1.0
1.0
0.8
0.5
6.8
0.8
0.6
6.8
84
Perc. Path: 23.29dB
Confidence: 73.79%
Original: va110022926
Figure 21: We visualize the reconstructed images, FullGrad 70 and score maps predicted by importance predictors in each path. The image is from ImageNet validation set [14] and resized to 256x256x3. The 3rd, 2nd, and 1st stages the last three stages of the decoder, consecutively. The regions with warmer colors in FullGrad represent areas with larger gradients, indicating a stronger impact on the classification decision. q set for more distinct comparison. Note that the score maps are inverted for better visualization, i.c., larger scores indicate prioritized for entering the selected side path. The bitrate is 0. 1.79bpp.
27

---

Original: val 10100000269
Score Map (I-st Stage)
Score Map (2-nd Stage)
Score Map (3-rd Stage)
1.0
1.0
0.6
0.6
0.6
0.2
0.2
0.2
Bit Allocation
g (0.071 (8bpp)
q (0.0972bpp)
9
(0.1383bpp)
q
(0.1998bpp)
800
125
60
100
40
75
100
50
20
20
50
25
q 5 (0.2739bpp)
(0.3784bpp)
9 (0.510Sbpp)
q
(0.6706bpp)
200
250
300
200
300
150
200
200
100
Figure 22: We visualize the bit allocation and score maps in the encoder. The image is from ADE20K validation set [83] and resized to 512x512x3. The 1st, 2nd, and 3rd stages are the first three stages of the encoder, consecutively.
28

---

Reconstruction
Segmentation Map
Score Map (3-rd Stage)
Score Map (2-nd Stage)
Score Map (1-st Stage)
Cls. Path: 31.53dB
The maps inverted for better visualization (higher for side path).
1.0
Seg. Path: 31.62BB
MSE Path: 31.87d1
0.6
-044
04
044
122
02
0.0
0.0
0.0
Pere. Path: 31.30dB
Original: val 00000269
Ground Truch
We visualize the reconstructed images, segmentation map and score maps predicted by importance predictors in each path. The image from ADE20K validation [83 and resized to 512x512x3. The 3rd, 2nd, and 1st stages are the last three stages of the decoder, consecutively. is set to 1 for more distinct comparison. Note that the score maps are inverted for better visualization, i... larger scores indicate prioritized for entering the selected side path. The bitrate is 0.07 18bp..
Figure 23:
29

---

Original: va130301200
Score Map (I-st Stage)
Score Map (2-nd Stage)
Score Map (3-rd Stage)
0.8
0.8
0.8
0.8
0.4
0.0
0.0
0.0
Allocation
1 (0.180 1806bpp)
(0.2667bpp)
(0.3870bpp)
(0.5521bpp)
150
150
200
50
100
50
5 (0.7644bpp)
6 (1.0471bpp)
(1. (1.4075hpp)
8 (1.822 1bpp)
500
800
000
400
600
800
200
200
200
Figure 24: We visualize the bit allocation and score maps the encoder. The image is from ADE20K validation set [8] and resized to 512x512x3. The 1st, 2nd, and 3rd stages are the first three stages of the encoder, consecutively.
30

---

Reconstruction
Segmentation Map
Score Map (3-rd Stage)
Score Map (2-nd Stage)
Score Map (1-st Stage)
Cls. Path: 25.I3dB
The maps inverted for better visualization (higher for side path).
1.0
0.8
Seg. Path: 25.178B
MSE Path: 25.45dB
C.6
-04
04
044
102
02
0.0
0.0
Pere. Path: 24.28dB
Original: val va100001200
Ground Truch
Figure 25: We visualize the reconstructed images, segmentation map and score maps predicted by importance predictors in each path. The image from ADE20K validation [83 and resized to 512x512x3. The 3rd, 2nd, and 1st stages are the last three stages of the decoder, consecutively. is set to for more distinct comparison. Note that the score maps are inverted for better visualization, i... larger scores indicate prioritized for entering the selected side path. The bitrate is 0. 0.180bbpp.
31

---

Original: val 00001969
Score Map (I-st Stage)
Score Map (2-nd Stage)
Score Map (3-rd Stage)
0.8
0.8
0.8
0.8
0.4
0.4
0.0
0.0
0.0
Allocation
1 (0.1 164bpp)
(0.11444) 1744bpp)
(0.2586bpp)
(0.3856bpp)
200
300
100
200
100
25
5 (0.5589bpp)
6 (0.8149b)) (0.8149bpp)
(1. (1.1666ppp)
8 (1.566 Ibpp)
600
800
400
600
400
600
200
400
200
200
200
Figure 26: We visualize the bit allocation and score maps the encoder. The image is from ADE20K validation set [33 and resized to 512x512x3. The 1st, 2nd, and 3rd stages are the first three stages of the encoder, consecutively.
32

---

Reconstruction
Segmentation Map
Score Map (3-rd Stage)
Score Map (2-nd Stage)
Score Map (1-st Stage)
Cls. Path: 27.03dB
The maps inverted for better visualization (higher for side path).
1.0
1.0
0.8
Seg. Path: 27.078B
MSF Path: 27.50dIB
C.6
-044
04
044
122
02
0.0
0.0
0.0
Pere. Path: 26. 10dB
Original: vai 00001969
Ground Truch
Figure 27: We visualize the reconstructed images, segmentation map and score maps predicted by importance predictors in each path. The image from ADE20K validation [83 and resized to 512x512x3. The 3rd, 2nd, and 1st stages are the last three stages of the decoder, consecutively. is set to 1 for more distinct comparison. Note that the score maps are inverted for better visualization, i... larger scores indicate prioritized for entering the selected side path. The bitrate is 0. 0.11644111 1164bbp.
33

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We claim the contributions and scope in the abstract and Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in Sec. 5.4 and Appx.
A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
34

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All experimental details are provided in Secs. and 5.1 and in Appx.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
35

---

Answer:
[Yes]
Justification:
We provide the instructions in Sec. [ Code will be available upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips. cc/
publ i des Codeeubmmes ss cisionnPlico icy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips. cc/ wwwblic/guideen / //odiddee/Cdeeeeeeee Cooeeeooeeeeeeeeee SudeSuumiiee Swwmisssiiii ssssnoooiccyyyyoco for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the training and test details in Sec. 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
The experiments do not involve statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
36

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the information on the computer resources in Appx. B.3
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the societal impacts in Appx.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
37

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The original papers are cited. The licenses are listed in Appx.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paper thcode com dataaetteett has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
38

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Code is available alltppss//gehhab...cc///uuvv////
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
39


Reviews:
[{"review_id": "30kWjUsfAr", "reviewer": "Reviewer_sciG", "summary": "The paper introduces a Multi-Path Aggregation (MPA) architecture designed to unify image coding for both human perception and machine vision tasks. By integrating the side path, the authors aim to optimize performance across various tasks while maintaining efficiency in terms of parameter and bitrate usage. This approach promises seamless transitions between tasks and improved performance.", "questions": "1. In the paper, MPA is used for classification and segmentation tasks, achieving good results. Can MPA be applied to other fundamental vision tasks, such as object detection? Alternatively, can MPA be used for knowledge transfer in cross-domain tasks, such as shifting from learning \"task-specific\" features to learning \"domain-specific\" features?\n\n2. If possible, I would like to know whether this method is sensitive to changes in hyperparameters (such as those mentioned in Section 4) and how these hyperparameters affect the experimental results.\n\n3. In terms of LPIPS and FID (from Bitrate 0.3 to 0.7), it seems that CRDR performs better. I completely accept that such a situation can occur, but I am curious about why this happens.\n\n4. Can MPA be compared with parameter-efficient fine-tuning methods such as LoRA? Is it possible to conduct a simple performance comparison between them?", "limitations": "Please refer to the Weakness and Question sections. During the rebuttal process, I am willing to actively and promptly discuss with the authors and other reviewers. If the authors can adequately address my main concerns, I am willing to increase my score.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is presented very well, including but not limited to its readability and the rational presentation of experiments.\n\n2. Each part is highly motivated. For example, in the MPA module, a learnable mask is first trained to decouple the features into task-shared features and task-specific features. By optimizing the side path, it adapts to various downstream tasks. Techniques such as Gumbel-Sigmoid are used to handle non-differentiable problems. By introducing the MPA, this paper unifies the encoder and decoder models. These techniques can be referenced and adopted by the corresponding fields. Additionally, the experiments are conducted very thoroughly.", "weaknesses": "1. If we do not consider the issue of feature decoupling, introducing and fine-tuning the side path to adapt to downstream tasks is not a novel idea. For instance, in [1], a similar bypass fine-tuning method was introduced.\n\n2. The proposed two-stage training strategy is somewhat complex, and it introduces a large number of training losses and hyperparameters, which may pose some inconvenience for its generalization and use.\n\n[1] Y. Guo, H. Shi, A. Kumar, K. Grauman, T. Rosing, R. Feris. SpotTune: Transfer Learning through Adaptive Fine-tuning. In CVPR 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, we really appreciate the reviewer's careful comments. We offer the following response to the reviewer's concerns:\n\n# Response to the Weaknesses\n\n1. **Comparison to SpotTune.**\n\n**[Reply]**\nWe appreciate the reviewer's attention to the differences between MPA and SpotTune. It is essential to emphasize that feature decoupling is central to MPA's effectiveness, an aspect that should not be overlooked, especially crucial for coding for multi-task scenarios. By considering feature decoupling, MPA supports smooth transitions between tasks, allowing for controllable and flexible reconstruction orientations, catering to diverse coding applications. In the encoding phase, feature decoupling enhances compression efficiency. In the decoding phase, it enables controllable reconstruction, which is vital for task-controllable image coding. In contrast, SpotTune only considers the impact of fine-tuning across different layers, applying indiscriminate routing to all features, which restricts its utility in task-controllable image coding.\n\n2. **Complexity of training strategy.**\n\n**[Reply]**\nWe thank the reviewer for the concerns regarding the generalization and usability of MPA. In fact, our proposed strategy originates from the widely-used multi-stage training strategy for generative image compression [2], which has been broadly validated for its stability and generalization, while not being overly complex. Here, we provide a further explanation. First, the initial stage involves training a general basic model. In this stage, the losses and hyperparameters we use strictly follow MRIC, with the only difference being that we fix MRIC's β at 2.56 (cf. Eq. (4) in [3]) and add a ratio loss to optimize the predictor in the encoder. Second, the subsequent stage of training only optimizes the side path and predictor in the decoder. When optimizing for MSE, the loss we use is the MRIC loss with β fixed at 0 (cf. Eq. (4) in [3]), plus the ratio loss. For vision task optimization, the loss we use removes the GAN loss from the first stage and adds a vision task-specific loss. Therefore, our proposed method ensures the generalizability and usability.\n\n# Response to the Questions\n\n1. **Applications for other fundamental tasks and cross-domain tasks.**\n\n**[Reply]**\nWe thank the reviewer for the insightful question regarding the application of MPA to other tasks. As we mentioned in our response to Weakness #2, our method demonstrates the generalizability and can adapt to various application scenarios. Here, we conducted additional tests on MPA's performance in object detection shown in Figure 1 of the global response attachment, using the same object detection model and task loss as TransTIC [4]. Regarding knowledge transfer in cross-domain tasks, our exploration mainly focuses on task-specific optimization and has not yet extended to domain-specific features. This is a promising direction for future research, and we will continue to follow up on this.\n\n2. **Hyperparameters and loss terms.**\n\n**[Reply]**\nWe thank the reviewer for the insightful question. Regarding the hyperparameters, we use the same settings as in [3] which have been thoroughly evaluated by the authors (cf. Section 5.2 in [3]). It's hard to reproduce the ablations on hyperparameters in such a limited period of rebuttal phase, so we respectfully suggest you refer to [3] for details. As for the loss terms, we provide detailed ablation studies in Table 1 of the global response attachment to demonstrate that our current combination can achieve a competitive trade-off. Regarding the role of each term, LPIPS enriches the semantic information of the reconstructed images, improving generalization, MSE constrains the pixel value consistency between the reconstructed and original images, and the task loss directly optimizes accuracy. The rate is determined by the encoder and entropy model, which are frozen and do not affect the second-stage optimization. We hope our ablations on the loss terms should help you more directly understand the role of each loss and the potential results of adjusting hyperparameters.\n\n3. **Performance gap.**\n\n**[Reply]**\nThere are two main reasons for the observed performance differences. First, the model size limits the performance especially at relatively higher bitrates. As shown in Table 3 of the global response attachment, TinyLIC, the base model we use to implement MPA, is much smaller than MRIC which is the base model of CRDR. Second, as a common sense in the community of image coding, optimizing with a relatively narrower bitrate coverage range can facilitate the coding performance for variable-rate models. As shown in Figure 3 of our submission, the bitrate coverage range of CRDR (0.08\\~0.72bpp) is much smaller than that of our implementation (0.07\\~1.20bpp). Thus the performance gap is reasonable and acceptable.\n\n4. **Comparison to LoRA.**\n\n**[Reply]**\nWe appreciate the reviewer's suggestion regarding LoRA. Our experiments recognize the effectiveness of LoRA in Table 2 of the global response attachment. Although MPA involves more fine-tuning parameters, it also achieves better performance. Moreover, our work has distinct features. We utilize predictors to support coding for multitask and smooth transitions between tasks within an all-in-one framework. The low-rank structure design of LoRA inspires us to consider improvements for the side path in our future work.\n\n# References\n\n[1] Y. Guo, et al. SpotTune: Transfer Learning through Adaptive Fine-tuning. In CVPR 2019.\n\n[2] F. Mentzer, et al. High-Fidelity Generative Image Compression. In NeurIPS 2020.\n\n[3] E. Agustsson, et al. Multi-Realism Image Compression with a Conditional Generator. In CVPR 2023.\n\n[4] Y. Chen, et al. TransTIC: Transferring Transformer-based Image Compression from Human Perception to Machine Perception. In ICCV 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NODsfIwmbz", "reviewer": "Reviewer_N4Rb", "summary": "This paper explores image coding for multi-task applications and introduces Multi-Path Aggregation (MPA), integrated into existing models to facilitate joint human-machine vision through a unified architecture. The MPA employs a predictor to distribute latent features among task-specific paths according to their importance, thus maximizing the utility of shared features and preserving task-specific features for further refinement. Additionally, a two-stage optimization strategy is proposed to mitigate multi-task performance degradation. Experimental results show that MPA achieves performance comparable to state-of-the-art methods in both task-specific and multi-objective optimization across human viewing and machine analysis tasks.", "questions": "Please check the weakness.", "limitations": "The authors have discussed the limitations of the work in Sec. 5.4 and Appx. A.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1) This paper explores image coding for multi-task applications and introduces Multi-Path Aggregation, integrated into existing models to facilitate joint human-machine vision through a unified architecture. \n\n2) Experimental results show that MPA achieves performance comparable to state-of-the-art methods in both task-specific and multi-objective optimization across human viewing and machine analysis tasks. \n\n3) The paper is exceptionally well-written and has clearly articulated the problem, proposed method, and the significance of their contributions.", "weaknesses": "I'm sorry that this paper is quite different from my research field, and I cannot make an accurate judgment on this paper. However, I think that this paper is exceptionally well written and has clearly articulated the problem, proposed method, and significance of their contributions. Therefore, I have no issues with it and am willing to defer to the opinions of other reviewers before making a final decision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere gratitude to you for the considerate feedback. We appreciate your recognition of the clarity and quality of our paper, particularly in articulating the problem, proposed method, and significance of our contributions.\n\nWe want to affirm that your understanding of the advantages of our proposed method aligns well with the insights provided by other reviewers. Your perspective is valuable and contributes to a well-rounded evaluation of our work.\n\nTo further assist you in understanding the contributions and significance of our work, we encourage you to review our responses to the other reviewers' comments. These responses provide additional context and detailed explanations that might help in appreciating the strengths and innovations of our approach.\n\nThank you again for your supportive comments and for taking the time to review our work. We hope this information enhances your confidence in your initial assessment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kxCJ9GTCBu", "reviewer": "Reviewer_TGUq", "summary": "The paper proposed a unified image compression method with multi-path aggregation for joint human-machine vision tasks. The authors utilized a lightweight predictor to generate masks to allocate features into main and side paths. By leveraging the pre-trained main path module, shared features can be reused to support varied tasks by finetuning a relatively low amount of parameters.", "questions": "Can the author provide more sufficient experiments to verify the effectiveness of the predictor module？\n\nAs for the base model, is it possible to apply the proposed MPA to other more recently published learned image models?", "limitations": "From my point of view, the proposed unified model needs to fine-tune as many sub-modules as the task amounts, and there are multiple training stages to adjust according to the specific tasks. The proposed framework still needs to accommodate different models adapted to the specific tasks, rather than an all-in-one image compression method.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The authors provide an innovative unified image compression method with multiple path aggregation that can support multiple human-machine vision tasks with shared features. \n\n2. Experimental results show that the proposed method achieves the SOTA performance.\n\n3. The paper is easy to understand and the experimental results are well presented.", "weaknesses": "1. For experimental results, the author compared several SOTA methods without the basic TinyLIC model. Besides, I think it is necessary to compare with SOTA baselines of unified models in terms of parameter amounts, time complexity, and computational complexity.\n\n2. The effectiveness of the predictor remained to be verified. According to Table 3, the proposed Predictor provided a tiny improvement in performance with only 0.19% bitrate reduction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing and acknowledging the strengths of the proposed approach. We will address the raised concerns below:\n\n# Response to the Weaknesses\n\n1. **Comparisons to TinyLIC and other SOTA baselines.**\n\n**[Reply]**\nWe thank the reviewer for the valuable suggestion. It is indeed necessary to provide a more comprehensive evaluation by comparing TinyLIC as a baseline and comparing the complexity with other SOTA methods. To address this, we have added the relevant comparisons, as shown in Figure 1 and Table 3 of the global response attachment. TinyLIC baseline is a variable-rate version optimized for MSE, while the perception-optimized version is equivalent to MPA (α=0). Since CRDR is also a modification based on MRIC, we include MRIC as a baseline for complexity comparison. As shown, our implementation has lower parameter and computational complexity compared to MRIC, while maintaining similar latency. This similarity in latency is because MRIC, being composed solely of convolutions, is more GPU-friendly than TinyLIC, which consists of transformer structures that are not yet well optimized for GPU. We promise that these modifications will be presented in the final version.\n\n2. **Effectiveness of the predictor.**\n\n**[Reply]**\nWe understand the reviewer's concerns regarding the effectiveness of the predictor. In fact, Table 3 in our submission only evaluates the encoder, primarily to verify the enhancement of coding efficiency by MPA. However, this table shows only a small part of the effectiveness of the predictor, and the predictor's role in the decoder is even more important which cannot be overlooked, as it enables the base model to achieve coding for multi-task. Our experiments in the submission have already demonstrated its effectiveness. As shown in Figure 3 of our submission, our approach achieves comparable performance to methods like MRIC and CRDR, which are optimized solely for distortion and realism, and matches the vision task performance of fully fine-tuned models, all within a unified model. The key to achieving this performance is that the predictor decouples features across different tasks by predicting their importance for each task, thereby making side paths always focus on the features that are most important for task optimization at any ratio, and easing the complexity of optimizing coding for multi-task. Figures 4, 5, and Appx. E.2 in the submission show that the predictors assign different importance scores for different tasks, demonstrating that the predictor enables the additional side paths to capture the most critical features for each task. Related explanations have been given in Section 5.3 of the submission.\n\n# Response to the Questions\n\n1. **Effectiveness of the predictor.**\n\n**[Reply]**\nYes. To address the concerns and the question, we have explained the effectiveness of the predictor module in our response to Weakness #2.\n\n2. **Applying to other base models.**\n\n**[Reply]**\nYes. In recent years, similar MetaFormer structures [1] (including Swin Transformer [2] and ConvNeXt [3] which have channel MLPs) have been widely applied as general backbones in the latest learned image coding research [4-8], demonstrating their effectiveness and generalization. Our MPA is an innovation based on the channel MLPs within such MetaFormer structures, enabling learned image coding models to support coding for multi-task in a natural way. Therefore, the proposed MPA can be applied to any model using a MetaFormer backbone.\n\n# Response to the Limitations\n\n1. **Not all-in-one.**\n\n**[Reply]**\nWe sincerely thank the reviewer for the insightful feedback. We respectfully acknowledge that our method needs to fine-tune some of the parameters as the reviewer mentioned which is indeed a common practice to optimize coding for multi-task. We would like to clarify that our understanding of an all-in-one model is in the context of practical applications. As the reviewer mentioned, even a unified model still requires fine-tuning for specific tasks. But during the actual deployment phase, it is sufficient for the optimized unified model to handle all required tasks using a single encoder and decoder pair. Our MPA can achieve this by performing one encoding and user-controllable decoding, allowing a unified model to reconstruct an image for the target we want. This is what we mean by \"all-in-one\" coding. We appreciate the reviewer's understanding, and if the reviewer still finds this term unsuitable, we are open to adjusting the wording in the final version.\n\n# References\n\n[1] W. Yu, et al. MetaFormer Is Actually What You Need for Vision. In CVPR 2022.\n\n[2] Z. Liu, et al. Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows. In ICCV 2021.\n\n[3] Z. Liu, et al. A ConvNet for the 2020s. In CVPR 2022.\n\n[4] Y. Zhu, et al. Transformer-based Transform Coding. In ICLR 2022.\n\n[5] R. Zou, et al. The Devil Is in the Details: Window-based Attention for Image Compression. In CVPR 2022.\n\n[6] J, Liu, et al. Learned Image Compression With Mixed Transformer-CNN Architectures. In CVPR 2023.\n\n[7] Z. Duan, et al. Lossy Image Compression With Quantized Hierarchical VAEs. In WACV 2023.\n\n[8] H. Li, et al. Frequency-Aware Transformer for Learned Image Compression. In ICLR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate all reviewers for their recognition of the strengths of MPA, as well as their insightful feedback and constructive suggestions. We have identified several common themes in the comments and would like to address them comprehensively in this global response.

# Advantages of MPA and Comparisons to Other Methods

The reviewers have shown interest in the advantages of MPA. MPA's core lies in its feature decoupling capability, enabling smooth task transitions. A significant advantage of MPA is its all-in-one coding approach, where all tasks can be achieved through extended paths within a single model, eliminating the need to train multiple models for different tasks. Compared to other methods catering to fine-tuning like SpotTune [3] and LoRA [4], MPA leverages different importance of features for different tasks to enable coding for multi-task and task transitions. Compared to other unified models like MRIC [1] and CRDR [2], MPA can achieve comparable performance and easily support more tasks with lower complexity. To address the reviewers' concerns, we have added the comparisons to the TinyLIC's performance and MRIC's complexity as Figure 1 and Table 3 in the attachment, and will be presented in the final version.

# Effectiveness of the Predictor

The reviewers have raised concerns regarding the predictor's effectiveness. As shown in Figure 3 of our submission, our approach achieves comparable performance to methods like MRIC and CRDR optimized solely for distortion and realism, and matches the vision task performance of fully fine-tuned models, all within a unified model. The key to achieving this performance is that the predictor decouples features across different tasks by predicting their importance for each task, thereby making side paths always focus on the features that are most important for task optimization at any ratio, and easing the complexity of optimizing coding for multi-task. Figures 4, 5, and Appx. E.2 in the submission show that the predictors assign different importance scores for different tasks, demonstrating that the predictor enables the additional side paths to capture the most critical features for each task. Related explanations have been given in Section 5.3 of the submission.

# Training Strategy

Regarding the training strategy, the reviewers have highlighted the importance of discussing complexity, loss terms and hyperparameters. Here we give a further explanation:

1. **First Stage:** Training a generalized basic model. During this stage, the loss terms and hyperparameters strictly follow those used in MRIC, with the addition of a ratio loss to optimize the predictor in the encoder. The primary difference is that we fix MRIC's β at 2.56 (cf. Eq. (4) in [2]) since we do not consider task transitions in this stage.

2. **Second Stage:** Optimizing the side path and predictor in the decoder. The transitions between tasks are only controlled by the predictor, which is optimized by minimizing a separate ratio loss. Thus for MSE optimization, we can use the MRIC loss with β fixed at 0 (cf. Eq. (4) in [2]), along with the ratio loss. For vision task optimization, we remove the GAN loss since GAN is not used now, and add a vision task-specific loss.

The strategy ensure that our method remains aligned with established practices to achieve stable training. We used the same hyperparameters as in the MRIC paper, which have been thoroughly evaluated for their influences. To address the reviewers' concerns, we have conducted a detailed ablation study (cf. Table 1 in the attachment) to showcase the necessity of each loss term. The results will be included in the final version. Note that the rate term do not affect the task performance theoretically since it is determined by the encoder and the entropy model which are frozen in the second stage.

# Generalizability and Applicability

The reviewers have raised concerns regarding the generalizability and applicability of MPA to various tasks and base models. Our additional experiments and the detailed results in the attachment's Figure 1, the main paper's Figures 4, 5 and Appx. E.2 demonstrate that MPA adapts well to different application scenarios including object detection. The generalizability is guaranteed by our proposed training strategy, which is inherited and developed from the widely used optimization methods, as we discussed in the previous section. Furthermore, MPA can be integrated into any model using MetaFormer [5] backbones with channel MLPs (like Swin Transformer [6] and ConvNeXt [7]) which has been widely used by the learned image coding community [8-12], ensuring the versatility and practical applicability of the proposed MPA. For the application of MPA in cross-domain tasks, we think this is a promising direction for exploration and will continue to follow up.

# References

[1] E. Agustsson, et al. Multi-Realism Image Compression with a Conditional Generator. In CVPR 2023.

[2] S. Iwai, et al. Controlling Rate, Distortion, and Realism: Towards a Single Comprehensive Neural Image Compression Model. In WACV 2024.

[3] Y. Guo, et al. SpotTune: Transfer Learning through Adaptive Fine-tuning. In CVPR 2019.

[4] E. Hu, et al. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR 2022.

[5] W. Yu, et al. MetaFormer Is Actually What You Need for Vision. In CVPR 2022.

[6] Z. Liu, et al. Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows. In ICCV 2021.

[7] Z. Liu, et al. A ConvNet for the 2020s. In CVPR 2022.

[8] Y. Zhu, et al. Transformer-based Transform Coding. In ICLR 2022.

[9] R. Zou, et al. The Devil Is in the Details: Window-based Attention for Image Compression. In CVPR 2022.

[10] J, Liu, et al. Learned Image Compression With Mixed Transformer-CNN Architectures. In CVPR 2023.

[11] Z. Duan, et al. Lossy Image Compression With Quantized Hierarchical VAEs. In WACV 2023.

[12] H. Li, et al. Frequency-Aware Transformer for Learned Image Compression. In ICLR 2024.

Meta-review: This paper proposes Multi-Path Aggregation (MPA) as an image coding method that can be commonly used in multitask scenarios to satisfy both human perception and machine vision. The problem addressed in this paper is applicable to various computer vision tasks, and its effectiveness has been thoroughly evaluated through experiments across multiple tasks in experiments. One of the significant contributions of the proposed method is its simplicity in training, as it can handle various tasks with a single model. The readability and strong motivation of the paper have been recognized by most reviewers, and all reviewers have given evaluations leaning towards acceptance. The authors' rebuttal effectively addressed the reviewers' concerns, resulting in the maintenance of the reviewers' scores. Some reviewers' suggestions, such as comparisons with LoRA, remain as future work. Additionally, one reviewer noted that the proposed method is not entirely an all-in-one approach, citing this as a limitation. It is expected that responses to this concern will be reflected in the final manuscript.
Decision: Accept (poster)