Which Tasks Should Be Compressed Together? A Causal Discovery Approach for Efficient Multi-Task Representation Compression
Authors: Sha Guo
Jing Chen
Zixuan Hu
Zhuo Chen
Wenhan Yang
Yu Lin
Xing Jiang
LINGYU DUAN
Venue: ICLR 2025 Poster (2025)
Keywords: Video Coding for Machine
Image Compression
Multi-task Learning
Causal Discovery
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Taskonomy-Aware Multi-Task Compression

Abstract: Conventional image compression methods are inadequate for intelligent analysis, as they overemphasize pixel-level precision while neglecting semantic significance and the interaction among multiple tasks.  This paper introduces a Taskonomy-Aware Multi-Task Compression framework comprising (1) inter-coherent task grouping, which organizes synergistic tasks into shared representations to improve multi-task accuracy and reduce encoding volume, and (2) a conditional entropy-based directed acyclic graph (DAG) that captures causal dependencies among grouped representations. By leveraging parent representations as contextual priors for child representations, the framework effectively utilizes cross-task information to improve entropy model accuracy. Experiments on diverse vision tasks, including Keypoint 2D, Depth Z-buffer, Semantic Segmentation, Surface Normal, Edge Texture, and Autoencoder, demonstrate significant bitrate-performance gains, validating the method’s capability to reduce system entropy uncertainty. These findings underscore the potential of leveraging representation disentanglement, synergy, and causal modeling to learn compact representations, which enable efficient multi-task compression in intelligent systems.

Published as a conference paper at ICLR 2025
WHICH TASKS SHOULD BE COMPRESSED TOGETHER? A CAUSAL DISCOVERY APPROACH FOR EFFICIENT MULTI-TASK REPRESENTATION COMPRESSION
Sha Guo, Jing Chen, Zixuan Hu,
Zhuo Chen,
Wenhan Yang,
Yu Lin, Xing Jiang, Ling-Yu Duan 1.4
School of Computer Science, Peking University 2 Peng Cheng Laboratory, Shenzhen, China
3 Fuzhou Chengtou New Infrastructrre Group, Fuzhou, China
sandykwekcse pku. cdu. en, chenjing15@pke en, hzxuanêpku. edu. edu. en, {chenzh08, yangwh) @pcl.ac.com (linyu2679, jiangxing315 @email.... com, Lingyu@pku.edu.c
ABSTRACT
Conventional image compression methods are inadequate for intelligent analysis, they overemphasize pixel-level precision while neglecting semantic significance and the interaction among multiple tasks. This paper introduces a Taskonomy- Aware Multi-Task Compression framework comprising (1) inter-coherent task grouping. which organizes synergistic tasks into shared representations to improve multi-task accuracy and reduce encoding volume, and (2) conditional entropy- based directed acyclic graph (DAG) that captures causal dependencies among grouped representations. By leveraging parent representations as contextual priors for child representations, the framework effectively utilizes cross-task information improve entropy model accuracy. Experiments on diverse vision tasks, including Keypoint 2D, Depth Z-buffer, Semantic Segmentation, Surface Normal, Edge Texture, and Autoencoder, demonstrate significant surrae-perffrmance gains, vali- dating the method's s capability to reduce system entropy uncertainty. These findings underscore the potential of leveraging representation disentanglement, synergy, and causal modeling learn compact representations, which enable efficient multi-task compression in intelligent systems.
INTRODUCTION
Multimodal models like CLIP (Radford et al. (021), GPT-4 (Achiam et al. 2023), and Sora LLiu
al. 2024a) exhibit human-level comprehension and reasoning (Achiam al. 2023 Chang et al. 2024 Zheng et al.) 2023: Laskar et al. 2023). making them potential consumers of visual and multimedia content. This highlights the need for semantic representation compression support efficient multi-task processing. However, current compression techniques, including both handcrafted video codecs (Bross et al.) 202T Pennebaker & Mitchell 1992] Si Shen, 2016) and end-to-
end learning-based approaches (Jiang et al.) 2023; He et al., 2022 Zou et al.) 2022 Chen et al. 2021 Bailé et al., (2017), primarily focus on rate-distortion optimization (Shannon et al. 1959). Handcrafted methods typically rely traditional techniques to minimize redundancy and maintain visual quality, whereas end-to-end learning-base methods often involve constraining the entropy model to accurately estimate the probability distribution of latent space symbols while simultancously maximizing the pixel-level likelihood between the reconstructed and original images. Nevertheless, conventional compression methods lack semantic representation constraints, limiting their ability to preserve task-relevant information while reducing redundancy. As illustrated in Fig. Ta assessment on MS-COCO (Lin et al. (2014) tasks with images compressed 0.155 bpp shows learning-based methods MLIC+ (Jiang et al. 2023) and ELIC (He et al.) 2022) exhibit notable superiority compared to the handcrafted WebP (Si & Shen 2016) and VTM- 17.0 Bross al. (021), particularly in multimodal tasks like Video Question Answering (Zhang et al. 2023). This advantage can be further enhanced by task-specific auxiliary loss functions, highlighting the necessity to capture richer semantics for intelligent multimedia analysis.
"Corresponding Author.

---

Published as a conference paper at ICLR 2025
Seg
Enjject
Aentlhe
Relative Performance Inferenced
ObjectDet.Insten InstanceSeg PannSSeg (PnnoSoe Keypoints |
[lassifier/SemSeg
MLCC
V136 15a
Trained With
Dbj
5.19%
5.13%
3.34%
5.57%
-0.39%
1.62%
InstanceSeg.
4.90%
5.50%
5.20%
3.00%
-1.00%
2.10%
Setsortio
Bin Rade
PunoSeg.
4.70%
5.10%
5.70%
3.10%
-1.20%
4.50%
Keypoinis
4.00%
3.80%
3.90%
5.80%
1.50%
3.10%
Classifier
4.50%
3.20%
2.80%
3.60%
5.50%
2.00%
SemSeg.
3.70%
4.20%
4.60%
2.90%
2.10%
5.70%
Panoptic Sea. Video Quatien Auswerica
(b)
Keypoint
Rutoencuded
Visualiztiio of latent representation of multi-tasks
Heatmap of local 3x3 context causal correlation" Semantic Seg. and Surface Normal
Hentmap of channel-wise global cuusul correlation of Semantic Seg. and Surface Normal
Figure I: Motivation of Taskonomy-Awarg Multi-Task Compression. (a) Multi-task Compression Performance: Normalized performance (0=worst, 1=best for clarity) of compressors in diverse tasks. (b) Necessity of Task Grouping: Relative performance (%) when compressors are trained on one task and tested on another. Positive values indicate collaboration, and negative values highlight conflicts, underscoring the need for task grouping. (c) Necessity Taskonomy-Aware Causal Modeling: Semantic Seg. and Surface Normal tasks' local context correlation (middle) and global channel-wise correlation (right) visualized, showing the potential for capturing redundancy and achieving bit savings through task-aware causal modeling. The indicates the current decoding point, while the indicates the most similar reference positions.
In response to these limitations, the paradigm of Video Coding for Machines (VCM) (Yang et al. 2024; Choi & Bajic 2022 Duan al. 2020 has emerged promising solution. VCM integrates
image compression with feature representation to achieve both compactness and efficiency, aiming to
meet the dual objectives of high-fidelin human vision and high-precision machine vision (Ge et
2024
Li et al.
2024 Liu et al. 2023 Bai et al. 2022).
Despite advances VCM, most methods still treat tasks in isolation (Liu al. 2021 Bai al.
2022) or focus only predefined tasks (Liu et al. 2023 Li et al. 2024), overlooking the benefits of grouping supportive tasks or complex relationships between feature spaces of tasks. Preliminary studies (Shi et al. 2023 Fifty et al. 2021: Standley et al. 2020 Zamir et al. 2018) and our findings in Fig. reveal statistically significant correlations among tasks, both positive and negative. These correlations highlight the synergies and conflicts in multi-tate task compression. Therefore, identifying task groups that leverage synergies and reduce conflicts is essential for improved multi-task learning and optimized compression performance. Furthermore, Fig. demoonstttts the existence of local and global causal contextual relationships across tasks with different semantic granularities. Exploiting conditional relationships in representations enables more accurate prediction of symbol distributions through conditional entropy, thereby improving compression efficiency. Conventional methods, however, process tasks independently. ignoring inter-task dependencies, which leads to redundancy.
Thus, an essential question arises: How can we discern and exploit the interdependencies among tasks to achieve efficient multi-task representation compression? Addressing this question requires models that can (1) discern mutually beneficial and conflicting tasks to group them effectively, and (2) dynamically decouple and model the dependencies between sub-tasks, enabling independent optimization of each sub-task while preserving the hierarchical task relationships.
To address this challenge, we propose a paradigm shift with Taskonomy-Awar Multi-Tane Com- pression (TAMC), which integrates task grouping and causal discovery for compact multi-task representation compression. By leveraging causal relationships between tasks, TAMC enhances task performance and improves overall compression efficiency. Our approach consists of two key components. First, we cluster inter-coherent tasks into groups that share universal representation. leveraging synergies among mutually beneficial tasks to improve accuracy and reduce encoding
2

---

Published as a conference paper at ICLR 2025
volume. This is the first work to systematically group tasks for compression, mitigating conflicts and preventing performance degradation. Second, we construct a directed acyclic graph (DAG) based conditional entropy to capture causal relationships among task representations. This approach identifies task dependencies across different abstraction levels, uncovering inter-task relationships and mapping information flow through directed graphs. By traversing causal paths, parent task representations provide informative cross-task contexts for child tasks, thus reducing uncertainty and enhancing compression efficiency and scalability.
Our contributions: 1. We demonstrate that leveraging intricate inter-task relationships significantly improves -etfooreformnnce efficiency by clustering tasks for collective compression and establishing causal links between clusters. 2. We introduce a DAG-based causal discovery framework via conditional entropy, which captures semantic dependencies across abstraction levels to enhance system certainty and reduce information entropy, thereby improving compression compactness. 3. The proposed TAMC achieves superior performance across diverse downstream task benchmarks while remaining competitive in universal image reconstruction. Extensive experiments on key computer vision tasks using the Taskonomy dataset validate the effectiveness of our approach.
RELATED WORK
Multi-Task Learning. Multi-task learning (MTL) improves performance by introducing inductive
biases and emphasizing relevant features (Zhng & Yang. 2021). However, task competition for
model capacity and ineffective shared representations often hinder MTL. Grouping compatible tasks is crucial for reducing conflicts and boosting performance (Lu et al. 2020 Yu et al.) 2020 Chen
et al. 2020: Kendall et al.] 2018), J, yet current approaches often rely on human intuition (Zhang &
Yang 2021). Recent studies (Fifty et al. 2021 Wu al., 2020 Standley et al. 2020) highlight the
need for systematic task grouping to advance the field.
End-to-End Image Compression for Human and Machine Tasks. In E2E-learned image compres- sion (Ballé et al. 2017), an image æ is first encoded into latent representations using an analysis
transform Ga ( 0e). then quantized discrete values ÿ. With learned probability model (b(") ()) .))9 y can be losslessly coded using arithmetic coding. On the decoder side, a synthesis transform ge (i: 0a) reconstructs the image ø from ÿj:
y Ia (x; 0c) -())))))))))))) a ( 0a).
To improve compression efficiency by decorrelating the latent space and estimating symbol proba- BBlittes,BBallé et al. (2018) introduces hyperprior model that reduces spatial redundancies among
bilities, [Ballé latent variables, adding a few extra bits to convey spatial structure. This hyperprior model enables a more accurate entropy model and better estimation of Py (g). .. It can be divided into hyper analysis transform ha (y: )he) and a synthesis transform (2; Ond):
ha 0ne), & = Q(=), Paja (ūl=) = (=,(2; 0xa).
Minnen et al. (018)) proposed more accurate entropy model which jointly utilizes an autoregressive context model 9em- The predicted Gaussian parameters [(u, o) of the distribution Pe (Ú) are functions of the learned parameters of the hyper-decoder, context model, and entropy parameter networks (Ond, Ocm, and Dep. respectively):
PO(0 2, Ord, Oem Pep) = II (N(H). =1 (-2
1 (0.).
(3)
with Hz, (v) di: 0ep), g2 0nd), and d; 9em Cim Pcm) u (- (-2. 2) is uniform noise to approximate quantization during training. The overall loss function is:
21
C = R(g) + R D(a, x) E log, (po)s (+)=)) + + 1- (2)) +X D(x, a).
rato(latents)
ratethyper lateens
distootion
where À controls the rate-distortion tradeoff. The first term is the rate that corresponds to the cross entropy between the natural (marginal) distribution and the learned entropy model. The second term is the rate to transmit hyperprior. The third term measures the reconstruction quality according to the given distortion metric d (e.g., PSNR MS- MS-SSIM). Recent advancements with improved entropy
3

---

Published as a conference paper at ICLR 2025
Tasks Shared Representation
Construct Conditional
Graph
Scalable Compresion
Facloriadd Entropy Model
AEAAD
P
N
Whai Eytee Maaie
Gringed
Ingetberr Comeganciin
blges that define groph's tepolegy
Conditional Conted
Painwdta Conditional Entropint
AEVAD
Deegl Palruate Grodlers
Stapp Lnbk AfTiniios kingli Groupp
Vyy
Vy
WY,
Dr
W
A
AEAAD
Aleutal
T.
(b)
D
Croep! Phredt lnter
Greep
Backer
N. V.Saebeltomeel Siidge buner
Katemaala Sharel Leiend
Deing Lamez
Aetorattc Codig
Traupprnnchlae Daring Thisky During bilenme
Figure 2: Overview of TAMC. Given a series of tasks, how can we effectively cluster them in the latent semantic space and construct causal graph to optimize pitrate-performance? To find a feasible solution, we follow these steps: (I): Group tasks based on inter-task coherence under bitrate constraints. (II:: Construct DAG via conditional entropy capture causal relationships. (II):: Compress grouped representations according to DAG in a scalable manner.
models (e.g., hierarchical (Ballé et al. 2018). auto regressvee (Minnen al., 2018 He al. 2021
Xiang et al. 2022)), multireference entropy (Qian et al.
2020), and innovations such as Shannel-wiss He et al. 2021 Jiang et al. 2023 He et al.
and spatial-wise acceleration (Minnen Singh| 2020
2022), codebooks and vector quantization (Zhu et al.
2022), and hierarchical VAEs (Duan et al. 2023), have significaatyyy improved compression performance. Many existing techniques overlook
their impact on downstream tasks like classification, detection, and segmentation, which require task- specific feature retention. Task-aware paradigms such as Vidco Coding for Machines (VCM) (Yang
et al.
2024 Choi & Bajie 2022) focus machine vision-targeted compression(Li et al. 2024
et a..
2023). including feature- assisted coding (Liu et al.) 2024b), scalable coding et al. 2022 I,
[023 Chen et al. 2021]. While these methods effective for individual tasks(Liu et al. [202]] Li et al. 2024 Liu et al. (023). they often fall short in capturing the full complexity of multi-task relationships, highlighting the need for more effective multi-task compression strategies.
and intermediate feature compression Kim
3
APPROACH
3.1
ARCHITECTURE
An overview of TAMC is provided Fig. 2 It comprises three key components: (1) a inter- coherent task cluster that groups mutually coherent tasks into shared representation space; (2)
conditional entropy graph, constructed using causal discovery to reveal dependencies; and (3) a scalable compressor, which compresses multiple feature layers along graph paths.
Components (1) and (2) draw inspiration from the lookahead stage in traditional video encoding (Li 2003 He & Mitra 2002 Wang & Kwong 2008; Ma al. 2005). which performs preliminary analysis to optimize the encoding performance of Component (3) under bitrate constraints. In Component (1), 2x downsampled low-resolution images are used inputs, combined with low- complexity feature extraction and decoding backbone (Standley et al. 2020)) to efficiently pre-analyze task grouping performance under the constraint of bitrate consumption. The task grouping strategy leverages gradient coherence to cluster tasks, maximizing shared information while minimizing redundancy. In Component (2), a conditional entropy graph organizes the grouped tasks hierarchically, facilitating information transfer from parent tasks to child tasks. This hierarchical structure improves representation certainty and encoding efficiency by providing a more precise cross-task entropy estimation model.
3.2 GROUP COTER-COEERENT TASKS
The objective of task grouping is to cluster tasks that can share representations, thereby optimizing the performance of multiple tasks under given bitrate budget b. As task grouping is conducted in
4

---

Published as a conference paper at ICLR 2025
the pre-analysis module, the bitrate budget is loosely approximated the number of task groups Formally, a set of n tasks T where each task Ti E T belongs to at least subgroup, i.e., VT E T, T.3c, E C T c k S - 'T shares a feature extractor (encoder) 9a (æ;; 2e), as defined in Eq. To
which is partitioned into & subgroups = {Ci..................... Ck
and each subgroup distinguish between the lookahead and formal encoding stages, the set of shared feature extractors
is denoted as & = {la ..... shared representation y/ = lag (a:; )..). Each task T; associated with unique decoder l. (3: Os:) corresponding to the synthesis transform Is (i): 0a) in Eq. 1 The decoders D = (?,, lez. la.
.......... J. For a raw image æ, the encoder la in subgroup Cg generates
map the latent feature yn(i) into task space, where g(i) denotes the subgroup index for TN. The predicted output ab L. (t)) 0d,) is then compared with the ground truth for evaluation.
Task performance of Tæ is measured using task-specific loss functions Li, e.g., PSNR/MS-SST for pixel reconstruction and cross-entropy loss for segmentation. For a given grouping strategy C = {CI,..., ,Ck), the overall performance of the task grouping is aggregated as " ming (L(T; E cg) (.) .Te bitrate R. required to transmit the latent feature Vy for each group cg is approximated by by By where B is x the amount of data transmitted, with each shared representation consuming a unit cost of Our goal to minimize the total loss subject to the total bitrate constraint b:
12
L Σ A min (I(T, E (y) + Σ B(U)), s.t. B(y, S b.
(5)
f=1
To achieve this, we first analyze the influence of task gradients on each other to determine the subgroup formation. Task grouping is then guided by gradient consistency, ensuring effective collaboration among tasks. By sharing gradient updates, encoding parameters, and feature representations, tasks within the same group enhance mutual learning. The specific methodology is as follows:
Consider a multi-task model that trains all tasks together with parameters O2 U {O,|2 E 7}, where 9g { denotes shared parameters for all tasks, and θ {0a, } represents task-specific parameters for тj. Given input images X, we optimize the overall task performance by following the total loss Leotat:
Liotal = O. O2)
(6)
iET
At the time step 1, for the input batch s", the gradient descent update for the shared parameters θ with respect to Tu at step t + 1 is denoted as 66-1 and is computed as follows:
@t + O' Ve La': 0 O.))
Using these updated shared parameters 1 we calculate the forecast loss for other tasks Ty while keeping task-specific parameters O, and inputs x' unchanged. Gradient coherence between tasks Tu and Tu is then measured as:
Lo(x;; 0++1 00) glu
Chiso =
(8)
(x O O
A positive Chase indicates that the update from Tu reduces the loss for Te, while a negative value Ch-se value indicates conflicting parameter update directions. Tasks with high gradient coherence are grouped to share encoders and representations. After T timesteps, the cumulative coherent measure 7 is calculated as: Ču-. $ Σi= Ct Ct After obtaining the gradient coherence measures between all pairs of tasks, the number of possible grouping for n tasks is 2 1, which grows exponentially with n. To address this computational complexity, we apply relaxed estimates for higher-order cluster coherent measures. Specifically, for a triplet of tasks {Ta, Tw, Tw the triplet cost is estimated as the average of pairwise coherence measures Ću-re and Cu-te- Assuming each cluster has a unit bitrate cost, we adopt a branch-and-boune bound method, as in prior works (Fifty al., [021]] Standley et al.) 2020) Zamir et al. 2018), to search for optimal grouping strategies under the given bitrate budget b.
Note: In the E2E compression training phase, bitrate consumption is detined as Bg [- log, Pe (9)) During deployment, the cumulative distribution function (CDF) of (ÿ) used for arithmetic coding to determine the final bitrate.
2 As shown in Fig. tasks may belong to multiple groups Transparent tasks are used during training to boost group performance but discarded during inference.
5

---

Published as a conference paper at ICLR 2025
3.3 CAUSAL GRAPH-BASED BASED COMPRESSION METHOD
Building upon the task grouping strategies and the group-shared variables V/ we now fo- cus on how to construct the directed acyclic graph (DAG) that represents the causal rela- tionships among these variables. Let y
Algorithm 1 DAG Construction via Conditional Entropy
Require: Latents set y {Y,,... Yx). conditional
entropy oracle H
Ensure:
Returns the causal DAG G = (V, F)) 1: Initialize V 3 and Æ =
(Y, ...... Y be a set of group-shared ran- dom variables, where each Y; corresponds to the shared representation y; for task group G,.
2: y = Soot()), key (Y),,, reverse True)
3: for each latent Y. y
We say that Y, causes Y, if there exists fune-
4: Initialize minH (Y.l) < 20, Y, t None
tion fy such that Y, Cy), where Cj represents the contextual information influenc- ing Yj. including both observable and latent factors. This causal relationship is represented
5:
for each latent Y, E \ [Y.].] > d
6:
Look up H(Y)Y))) from H
7:
H(Y(())) < (Y.nH(Y.))) then Update minH(Y..)) + H(Y)))
8:
by the edge Y; + Yj in directed acyclic graph (DAG) G = (V, E), where V denotes the ver- tex (i.e., y in our context) and E denotes the
9:
Update y, < Yj
end if 11: end for
10:
if Y *
None then Add edge Y, + Ya to E
set of edges.
13:
Identifying the true causal graph without ex-
14: end if
periments or strong assumptions is generally infeasible. To efficiently compress the repre-
15:
end for
16:
return (V,
sentations V. we approximate causal relationships by minimizing the conditional entropy, which quantifies the uncertainty of one variable given another. Specifically, given Y; and Y; with domains ) and V, respectively, the pair-wise conditional entropy H(Y,(Y,) (also denoted as the conditional entropy oracle V in Algorithm I is obtained using the Kernel Density Estimator (KDE) from CCoii & Bajic 2022
Saxe et al. 2019), and defined as:
H(Y)Y))) = (X.) - ((),)) = Elv) logp())) - >''heee ply: yı) log Plye Ys
p(y: ((()))
y..83
We define the parent Y, of the children variable Y. as the one that minimizes the conditional entropy:
Yp arg
Y, (,),, H(Y)Y)))
min
(10)
By applying this criterion iteratively to all representations in V. we construct the edges of the causal graph G. The causal discovery process is detailed in Algorithm [
Scalable Compression Using the Causal DAG. After constructing the directed acyclic graph (DAG) G, inspired by the causal context entropy model Guo et al. (2021) and MLIC+ (Jiang et al. 2023). we perform compression by traversing the graph in topological order. The parent representation additional cross-task context for the child representation ye, thereby improving
Yp serves compression efficiency. Specifically, when the parent latent representation Yp is assumed to cause the child representation ye. the distribution of yp is modeled using the prior framework in Eq. 3
The estimation of Pú, (9p) is performed using a masked convolution Φ,.i = 9cm (Ūp < i; 0 pm which extracts the local context pp... This context is then integrated with the hyperpriors 2p to estimate the Gaussian distribution parameters for úp- When it comes to ecoding Gc. we aggregate both the decoded latent Üp and the first half latent in the current spatial location ye. generating more informative contexts Φe.i. As shown in Fig. Bal the whole process can be extended from Eq and formulated as:
Pg. (ie 2e, Ond, 2 Pem+ Sep) = II (N(He +u
1 2 2'2
(He.),,
(11)
with Mc,i, gep (bbe, &e.i: 0ep) ), 2he gn (2:; Ond) and Φe,i = 9cm+ (Úe i, Mpi Omm+) The differences from Eq. highlighted in blue.
The above causal context model primarily captures local conrelations while ignoring long-range dependencies. To address this limitation, we utilized a causal global prediction model, which leverages long-range correlations between the parent Dp and child ý.- The overall process shown in

---

Published as a conference paper at ICLR 2025
Saltua
DWConvax
HW
Cassal
HW
Cone
W
H
HW
Soilmax
Tramspoo
HW
WW
Aue
89
Seff-mmemiion
Com
ComtEx
latent 9,
Latent y:
SX5 causal mask
(a)
HW
HW
W
W
Latent
Reshape
Reshape
(b)
Figure 3: The structure of proposed cross-task causal context compression: (a) Cross-task local
causal context mask convolution. (b) The parent Ún provides a global causal context for the child y.-
Fig 3b and is formulated as:
Ve.4 softmax2
softmax, (0) <
Ye.t
conv K xK ( &ges DepthRB(g...
non-negative
(12)
where 86 <, yp < i = Embedding(üj, < i). 36 < Embedding( < <). The Embedding
layer consists of a X convolutional layer and 3 3 depth-wise convolutional layer. The 3 x depth-wise convolutional layer is employed for learnable position embedding. DepthRB is proposed by (Jiang al. 2023) to enhance the non-linearity. Denoting the trainable parameters in the causal global prediction model as I gc: Eq extended as:
Púj. (gel Ec, Ond, Oem, O Pep)
II (N(Hes, x
1
(ūr,)),
(13)
2'
with Me, Oc,i geplebe: De.ii ge Dep), w (2:; Gnd). and de. 9em (ue
i, Opi fem), Pge,s = (e < i, Bpi 2 gge). The differences from Eq. are highlighted in green.
4 EXPERIMENTS
4.1 TASKS AND DATASETS
To quantify the performance across diverse downstream tasks, evaluate the Taskonomy dataset (Zamir et al.) 2018) across 6 tasks. Taskonomy is a large-scale computer vision dataset that includes over 4.5 million images from more than 500 buildings. Each image has 18 annotations covering 2D, 3D, and semantic tasks. The total size of the dataset is 11.16 TB. Due to limited computational and storage resources, we used the Tiny split for our experiments, which consists of 872,517 images in the training set and 16,000 images in the validation set and test set, respectively. conducted experiments on 6 tasks selected from the annotated tasks, i.e., Semantic Segment, Keypoint 2D, Edge Texture, Surface Normal, Depth 7-buffer and Autoencoder. More details of tasks and loss measurements are provided in A.2
4.2 BASELINES
Our method is compared against several baselines, including traditional codecs such JPEG (Pen nebaker & Mitchell, 1992), WebP (Si Shen 2016), and VTM-17.0 (Bross et al. 2021), as well as learning-base compression methods, i.e., ELIC (He et al., 2022) and MLIC++ (Jiang et al. 2023
To evaluate the performance of different compression methods across a variety of tasks, we utilized the official open-source implementations of these methods compress input images. The reconstructed images were then evaluated on multiple downstream tasks by analyzing the features extracted by pre-trained task models provided by (Standle al. 2020
httpss://riii googl le http:///ww...oeeeooo/ooooooeeee/ uap=sharing
VMS_w5R52 w5R5?

---

Published as a conference paper at ICLR 2025
the encoder backbone, and the task-specific decoder consists of four transposed convolutional layers and four convolutional layers. Initially, we trained the group-task shared encoder-decoder without compression loss for 60 epochs using the SGD optimizer (Ruder) (016). with the learning rate decaying from 0.1 to le-4. Then, based on the conditional entropy of the shared representation, constructed directed acyelic graph (DAG) among the child nodes of the shared representation. Finally, we learned entropy models for different compression rates by following paths from parent nodes to child nodes. We the task learning rate to le-4, and the learning rate of the hyperprior entropy model to le-4. We continued training for 50 epochs, adjusting the parameter in the distortion-rate trade-off values from [0.04, 0.072. 0.14, 1, 1.932] to learn encoder-decoder models parameters and entropy model parameters for different bitrates.
For method, first employed Xception (Chollet 2017)
+ R with
4.3 RESULTS
4.3.1
PERFORMANCE OF COMPRESSION FOR MULTIPLE TASKS
Samantic Segmentation Parformance [4]
Surface Performance [4]
Edga Texture Performance [4]
Archor
Archor
EUC
EJC
EUC
JRG
JTC
Vite
Ourt
dus
WEBP
VTV
MuC+-
Wese
VIM
Cuo
58- Dop!
top))
Depth Zhuffer Nerformance [1]
Keypoint 2D Performance [4]
Autoencader Performa ncw [-]
EUi
Euc
MuC
FEG
ML
VTM
VLI
Oura
WESP
Durs
BY. Br-Robo Dogt
26. 26-Att 11pe0
(d)
(c)
Figure 4: Comparison of performance-rate curves for 6 tasksusing baseline compression methods and our proposed TAMC. "Anchor" refers to the optimal performance of supervision task obtained using uncompressed images as input. The area of the shade color visualizes bitrate-pent perratepprffrmance gains.
Our results, summarized in Fig. demonstrate that TAMC achieves a new state-of-the-at in compression for multitask scenarios, significantly outperforming other baselines in 5 out of 6 tasks. In multitask learning and compression, different tasks impose distinct requirements on image information. For example, Depth Z-buffer in Fig. [4 on geometric details. Therefore, effective compression for Depth Z-buffer must retain global structural cues to preserve depth continuity. In contrast, Surface Normal estimation in Fig. 44b and Edge Texture detection in Fig. 4c] primarily depend on local sharp edges and fine-grained texture. Consequently, compression strategies for these tasks should prioritize preserving compression methods (i.e., JPEG, WebP, VTM) and end-to-end deep learning approaches (i.e., ELIC, MLIC++) typically use a unified compression representation for all tasks, overlooking the uniqueness, conflicts, and collaboration among multiple tasks. This often results in suboptimal performance for certain tasks. A more detailed analysis is provided in A.3. TAMC addresses this limitation by first partitioning tasks into complementary groups, ensuring that collaborative tasks within the same group share representations. We then employ causal discovery through conditional entropy to identify dependencies among groups These shared representations are scalably compressed following parent-child relationships, effectively leveraging the context priors from parent nodes to reduce the uncertainty of child nodes. A more detailed analysis is provided in A.3.2
details rather than global structure. Traditional
8

---

Published as a conference paper at ICLR 2025
4.3.2
PERFORMANCE OF IMAGE COMPRESSION
ELI
BBC
JRG
TEC
VTM
VTW
XTV WE2P
MEEP
0ute
445
0b-ais Bop)
01-1a 18pp)
(a)
(b)
(c)
Figure 5: Comparison of performance-rate curves for image compression the Taskonomy dataset
using baseline methods and our proposed TAMC.
As shown in Fig. 5 TAMC demonstrates competitive performance compared to the VTM baseline. For structure- and cecturre-sensitie metries, such as MS-SSIM (Wang et al. )003). TAMC is slightly inferior to VTM. For semantic-sensitive metrics, including KID (Binkowski al. 2018), FID (Heusel et al., 2017) in Fig. [ and LPIPS (Zhang et al. (2018) in Fig. [3] TAMC is slightly superior to VTM, reflecting its strong capacity for semantic understanding and perceptual quality retention. However, the Peak Signal-to-Noise Ratio (PSNR) results in Fig. [ comparatively lower, indicating room for improvement in pixel-level fidelity. This could be attributed our choice of Xception (Chollet 2017) as the encoder backbone and a task-specific decoder consisting of four transposed convolutional layers and four convolutional layers, which aligns with the architectures pre-trained for machine tasks (Fifty et al. 2021) but does not yet integrate the full advantages of advanced backbone modules for image compression, e.g., GDN (Ballé et al.) 2018), residual networks (Cheng al. (2020). and transformers (Zou et al. 2022 Lu et al. 2022). suggesting potential for further exploration.
ANALYSIS
5.1
ABLATION OF GROUP
TASKS
Task Groupingl
Task Grouping2
Task Grouping3
Method
Bitrate
Segment.
Depth
Normal
Normal
Keypoint
Segment.
Normal
Texture
JPEG
0.283
44.32%
29.86%
-7.05%
-14110%
-44.58%
28.40%
-17.56%
-0.06%
WebP
0.281
3.13%
20.08%
-3.69%
2.79%
-49.00%
-0.91%
-5.33%
-0.59%
VTM-17.0
0.229
-0.24%
-5.28%
8.24%
11.16%
-46.65%
-1.82%
2.84%
-0.37%
ELIC
0.302
-1.50%
-0.03%
6.48%
9.69%
-39.11%
-2.47%
2.27%
-0.90%
MLIC++
0.289
-1.18%
-1.05%
6.65%
9.79%
-40.40%
-2.26%
2.47%
-0.93%
TAMC
0.224
-10.08%
3.33%
0.21%
12.43%
-45.02%
-4.84%
5.38%
-1.05%
Table 1: Performance loss reduction of grouped inter-coherent tasks relative to single-task training.
We also conducted ablation evaluate the effectiveness and generalization of task grouping in the context of multi-task compression. Given 5 downstream tasks: (Semantic Segmentation. Depth Z-buffer, Edge Texture, Surface Normal, Keypoints 2D), our task grouping results based on gradient coherence between task pairs are as follows: Group 1 (Semantic Segmentation, Depth Z-buffer, Surface Normal), ), Group 2 (Surface Normal, Keypoints 2D) and Group 3 [Semantic Segmentation, Surface Normal, Edge Texture. During inference, only the tasks marked in green within each group are decoded and referenced for downstream tasks, while the remaining tasks are solely used to enhance performance during training and discarded during inference.
For Saapress-thenn-Aallye (CTA) methods, we assess the relative performance improvement of downstream tasks with without task grouping when compression is performed before feature extraction. For our Analyze-then-Ce thhee--oppresss (ATC) paradigm, we evaluate the impact of task grouping in latent space, comparing performance with without task grouping when feature extraction is conducted prior to compression. To isolate the effect of task grouping, the causal discovery module is not employed in this ablation study. The experimental results in Tab. reveal that, for most baseline

---

Published as a conference paper at ICLR 2025
methods, the performance of tasks such Semantic Segmentation in Group 1, Keypoints 2D in Group 2, and Edge Texture in Group exhibits loss reductions compared to single-task training, benefiting from information shared with other tasks. Depth Z-buffer and Surface Normal in Group 1, while not demonstrating significant improvements and even experiencing slight performance declines, contribute positively the performance enhancement of other tasks within their respective groups. JPEG and WebP suffer the most in Semantic Segmentation and Depth Z- Z-buffer, likely due to pixel-Ievel distortions that introduce deviations feature distributions, ultimatelyy leading to degraded performance in task grouping. TAMC does not significantly degrade performance in any task, making the most stable method. Furthermore, TAMC outperforms all other methods, demonstrating strong generalization across different task groups.
5.2
ABLATION OF CAUSAL DISCOVERY TOPOLOGY
To better understand how causal discovery graphs constructed via conditional entropy optimize both collaborative bitrate and task performance, analyzed their effectiveness in determining the optimal trade-off between these factors. As visualized in Fig. [ different graph construction methods compared across various settings. In Setting A, where the graph follows principles of causal discovery via conditional entropy, the resulting tasks exhibit better coordination, balancing both bitrate efficiency and task performance. In contrast, Setting B and Setting C represent random graph constructions that do not adhere to causal discovery principles. The visualized results show that Setting B requires a higher bitrate achieve similar performance while Setting C not only consumes more bitrate but also results in performance degradation. This comparison highlights the importance of proper graph construction in multi-task compression systems.
A Bupass
Ediop
Dapa Dharts
Bunass
Edroo
a
Bugass
ddnup
Berate
Figure 6: Impact of different DAG topologies on bitrate and multi-task performance (lower left indicates better solutions, where both bitrate and performance loss minimized). Left: Example of graph construction methods. Middle: Task performance of graph node and bitrate consumption of the entire causal graph. Note: Different colors denote different graph topologies, and different shapes represent different nodes. Right: A more detailed breakdown of the middle, showing bitrate consumption and performance for different nodes in each causal graph. Note: Different colors represent different topologies, with varying transparency of the same color indicating different nodes within the same graph. Different shapes represent different tasks, Red-bordered shapes denote the best-pent rest-performing task in each topology (prioritizing task performance over bitrate).
CONCLUSION
In this work, we introduced a novel multi-task representation compression framework that leverages causal discovery via conditional entropy to optimize the trade-off between bitrate efficiency and task performance. By grouping mutually beneficial tasks and constructing a DAG to characterize their interdependencies, Through extensive experiments on key computer vision tasks, we demonstrated the effectiveness of our approach in both bitrate reduction and task accuracy. Our findings highlight the importance of properly structured task groupings and causal relationships in multi-task compression, offering a promising direction for future work in video coding for machine learning and multi-task optimization.
method enables efficient compression of disentangled representations.
ACKNOWLEDGMENTS
This work supported by the Program of Beijing Municipal Science and Technology Commission Foundation 1241100003524010 in part by the National Natural Science Foundation of China under Grant 62088 102, in part by AI Joint Lab of Future Urban Infrastructure sponsored by Fuzhou
10

---

Published as a conference paper at ICLR 2025
Chengtou New Infrastructure Group and Boyun Vision Co. Ltd, and in part by the PKU-NTU Joint Research Institute (JRI) sponsored by a donation from the Ng Teng Fong Charitable Foundation.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
Yuanchao Bai, Xu Yang. Xianming Liu, Junjun Jiang, Yaowei Wang, Xiangyang Ji, and Wen Gao. Towards end-to-end image compression and analysis with transformers. AAAI, pp. 104-112, 2022.
Johannes Ballé, Valero Laparra, and Eero P Simoncelli. End-to-end optimized image compression. In ICLR, 2017.
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. image compression with scale hyperprior. In ICLR, 2018
Variational
Mikotaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd 2018.
gans. arXiv preprint
Benjamin Bross, Ye-Kui Wang, Yan Ye, Shan Liu, Jianle Chen, Gary Sullivan, and Jens-Rainer Ohm. Overview of the versatile video coding (vve) standard and its applications. IEEE TCSVT, 31 (10):3736-3764, 2021.
Yupeng Chang, Xu Wang, Jindong Wang. Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1-45, 2024.
Tong Chen, Haojie Liu, Zhan Ma, Qiu Shen, Xun Cao, and Yao Wang. End-to-End learnt image compression via non-local attention optimization and improved context modeling. IEEE TIP, 30: 3179-3191,202
Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang I Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. Advances in Neural Information Processing Systems, 33:2039-2050, 2020.
Zhengxue Cheng. Heming Sun, Masaru Takeuchi, and Jiro Katto. Learned image compression with discretized gaussian mixture likelihoods and attention modules. In CVPR, pp. 7939-7948, 2020.
Hyomin Choi and Ivan V Bajić. Scalable image coding for humans and machines. IEEE Transactions on Image Processing, 31:2739-2754, 2022.
François Chollet. Xception: Deep learning with depthwise separable convolutions. Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1251-1258, 2017.
Lingyu Duan, Jiaying Liu, Wenhan Yang, Tiejun Huang, and Wen Gao. Video coding for machines: A paradigm of collaborative compression and intelligent analyties. IEEE TIP, 29:8680-8695. 2020.
Zhihao Duan, Ming Lu, Zhan Ma, and Fengqing Zhu. Lossy image compression with quantized hierarchical vaes. In WACV, pp. 198-207, 2023.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pultern recognition, pp. 12873-12883, 2021.
Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identifying task groupings for multi-task learning. Advances in Neural Information Processing Systems, 34: 27503-27516, 2021.
11

---

Published as a conference paper at ICLR 2025
Xingtong Ge, Jixiang Luo, Xinjie Zhang, Tongda Xu, Guo Lu, Dailan He, Jing Geng, Yan Wang. Jun Zhang, and Hongwei Qin. awske encoder control for deep video compression. In CVPR, pp. (6036-26045, 2024.
Zongyu Guo, Zhizheng Zhang, Runsen Feng, and Zhibo Chen. Causal contextual prediction for learned image compression. JEEE TCSVT, 32(4):2329-233) 2021.
Dailan He, Yaoyan Zheng, Baocheng Sun, Yan Wang, and Hongwei Qin. Checkerboard context model for efficient learned image compression. In CVPR, pp. 14771-14780, 2021.
Dailan He, Ziming Yang, Weikun Peng, Rui Ma, Hongwei Qin, and Yan Wang. ELIC: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding. In CVPR. pp. 5718-5727, 2022.
Zhihai He and Sanjit K Mitra. Optimum bit allocation and accurate rate control for video coding via/spl rhol-domain source modeling. IEEE transactions Circuits and Systems for Video Technology, 12(10):840 04184400-999-000 2002.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Forrest N landola. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv: 2016.
Wei Jiang, Jiayu Yang, Yongqi Zhai, Peirong Ning, Feng Gao, and Ronggang Wang. MLIC: Multi- reference entropy model for learned image compression. In ACM MM, pp. 7618-7627, 2023.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7482-7491, 2018.
Ycongwoong Kim, Hyewon Jeong, Janghyun Yu, Younhee Kim, Jooyoung Lee, Se Yoon Jeong, and Hui Yong Kim. End-to-end learnable multi-scale feature compression for vem. IEEE TCSVT, 2023.
Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Xiangji Huang. systematic study and comprehensive evaluation of chatgpt benchmark datasets. arXiv preprint arXiv:2305. 18446 2023.
Han Li, Shaohui Li, Shuangrui Ding, Wenrui Dai, Maida Cao, Chenglin Li. Junni Zou, and Hongkai Xiong. Image compression for machine and human vision with peatial-frequenge adaptation. arXiv preprint MrXiv:2407.09853 2024.
Zhengguo Li. Adaptive basic unit layer rate control for jvt. In JVT 7th Meeting, Pattaya, Mar2003, 2003.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, pp. 740-755. Springer, 2014.
Kang Liu, Dong Liu, Li Li, Ning Yan, and Houqiang Li. Semantics-to-signall scalable image compression with learned revertible representations. IJCV, 129(9)2605-2266 2021.
Lei Liu, Zhihao Hu, Zhenghao Chen, and Dong Xu. Iemh-net: Neural image compression towards both machine vision and human vision. In ACM MM, pp. 8047-8056, 2023.
Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, et al. Sora: A review on background, technology. limitations, and opportunities of large vision models. arXiv preprint a7X1v:2402.17177, 2024a.
Yuxi Liu, Wenhan Yang, Huihui Bai, Yunchao Wei, and Yao Zhao. Region-adaptive segmentation prior for image compression. arXiv preprint arXiv:2403.00628, 2024b.
adaptive transform with
12

---

Published as a conference paper at ICLR 2025
Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi-task vision and language representation learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 10437-10446, 2020.
Ming Lu, Peiyao Guo, Huiqing Shi, Chuntong Cao, and Zhan Ma. Transformer-based image compression. In DCC, pp. 469-469, IEEE, 2022.
Siwei Ma, Wen Gao, and Yan Lu. Rate-distortion analysis for h. 264/avc video coding and its application to rate control. IEEE transactions on circuits and systems for video technology. 15(12): 1533 1544, 2005.
Qi Mao, Tinghan Yang, Yinuo Zhang, Zijian Wang, Meng Wang, Shiqi Wang, Libiao Jin, and Siwei Ma. Extreme image compression using fine-tuned vqgans. In 2024 Data Compression Conference (DCC), pp. 203-212. IEEE, 2024.
David Minnen and Saurabh Singh. Channel-wise autoregressive entropy models for learned image compression. In ICIP, pp. 3339-3343. IEEE, 2020.
David Minnen, Johannes Ballé, and George D Toderici. Joint autoregressive and hierarchical priors for learned image compression. NeurIPS, 31, 2018.
William Pennebaker and Joan L Mitchell. JPEG: Still image data compression standard. Springer Science & Business Media, 1992.
Yichen Qian, Zhiyu Tan, Xiuyu Sun, Ming Lin, Dongyang Li, Zhenhong Sun, Hao Li, and Rong Jin. Learning accurate entropy model with global reference for image compression. arXiv preprint arXiv:2010.08321, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04447, 2016.
Andrew M Saxe, Yamini Bansal, Joel Dupello, Madhu Advani, Artemy Kolchinsky, Brendan D Tracey, and David D Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019(12): 2019.
Claude E Shannon et al. Coding theorems for a discrete source with a fidelity criterion. IRE Nat. Conv. Rec, 4(142-163):1, 1959.
Shi, Qimai Li, Wenlong Zhang, Jiaxin Chen, and Xiao-Ming Wu. Recon: Reducing confticting gradients from the root for multi-task learning. In The International Conference on Learning Representations, 2023.
Zhanjun Si and Ke Shen. Research on the webp image format. Advanced graphic communications, packaging technology and materials, pp. 271-277. Springer, 2016.
Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In International conference on machine learning, pp. 9120-9132. PMLR, 2020.
Hanli Wang and Sam Kwong. Rate-distortion optimization of rate control for h. 264 with adaptive initial quantization parameter determination. IEEE Transactions on Circuits and Systems for Video Technology, 18(1):140-144 2008
Zhou Wang, Eero Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In The Seventh Asilomar Conference on Signals, Systems & Computers, 2003, volume 2, pp. 1498-1402. Icee, 2003.
Sen Wu, Hongyang R Zhang, and Christopher Ré. Understanding and improving information transfer in multi-task learning. arXiv preprint arXiv:2005.00900 2020. 2020.
13

---

Published as a conference paper at ICLR 2025
Jinxi Xiang, Kuan Tian, and Jun Zhang. MIMT: Masked image modeling transformer for video compression. In ICLR, 2022.
Wenhan Yang, Haofeng Huang, Yueyu Hu, Ling-Yu Yu Duan, and Jiaying Liu. Video coding for machines: compact visual representation compression for intelligent collaborative analytics. IEEE TPAMI, 2024.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances Neural Information Processing Systems, 33: 5824-5836, 2020.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In CVPR, pp. 3712-3722, 2018.
Ce Zhang. Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A simple I1m framework for long-range video question-ane centiionannseerig. arXiv preprint arXiv:2312.17235, 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE transactions knowledge and data engineering, 34( 2):5566-5609, 2021.
Zibin Zheng, Kaiwen Ning, Yanlin Wang, Jingwen Zhang, Dewu Zheng, Mingxi Ye, and Jiachi Chen. survey large language models for code: Evolution, benchmarking. and future trends. arXiv preprint arXiv:2311. 2023.
Xiaosu Zhu, Jingkuan Song. Lianli Gao, Feng Zheng, and Heng Tao Shen. Unified multivariate gaussian mixture for efficient neural image compression. In CVPR, pp. 17612-17621, 2022.
Renjie Zou, Chunfeng Song, and Zhaoxiang Zhang. The devil is in the details: Window-based attention for image compression. In CVPR, pp. 17492-17501 2022.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 DISCUSSION
While graph models can effectively capture complex relationships between multiple representation spaces, it is well-known that the computational complexity of adding removing nodes in graph structures is high. This becomes especially challenging in open-set scenarios, where tasks or data points are continuously evolving. Developing efficient methods for dynamically adding and removing nodes while maintaining the integrity of the graph remains an important research question.
Another critical point is the representation compatibility between different tasks. Ideally, the represen- tation form and model architecture should be customized to the specific requirements of each task and application scenario. Until truly unified model emerges, representations and architectures might not always be fully compatible across tasks. In our current work, we used shared model architecture and representations across tasks, which, while ideal for controlled experimentation, may not reflect the diversity seen in real-worid applications. However, our experiments successfully validated the feasibility and effectiveness of using a causal graph model based on conditional entropy for multi-task compression under these controlled conditions.
A.2 MORE DETAILS OF EXPERIMENTAL SETTINGS
We applied and measured 6 tasks in Taskonomy which is listed below:
Semantic Segment:
The annotations include 18 unique labels, with 16 object classes, "background" class, and an "uncertain" class. For this task, we evaluate compression performance at different compression rates using cross-cntropy loss.
This task involves 2D keypoint heatmaps. We assess compression perfor- mance different compression rates using the LI loss.
Keypoint 2D:
Edge Texture: This task involves detecting 2D edge textures. Similar to Keypoints2D, we evaluate performance using LI loss at different compression rates.
Surface Normal: This task includes surface normal images, centered at 127. To evaluate performance under different compression rates, use the rotate__oo loss, which is commonly applied in image processing or volume rendering tasks. The loss computes the LI difference between the output and target and compares the result across 9 different orientations to find the minimal loss. This ensures that the model's depth predictions remain consistent under rotational and translational transformations, which is crucial when dealing with real-world noise and variations.
Depth Z-buffer: This task involves Z-buffer depth images, measured units of 1/5 12m with maximum range of 1 128m. Similar the Normal task, we use rotate_ loss different compression rates, first calculating the I.1 difference between the output and target, then comparing across orientations find the minimal loss.
This task reconstructs RGB images at resolution of 512x512. We evaluate compression performance at different compression rates using the L.1 loss. Additionally, we assess fidelity using PSNR/SSIM (Wang et al. 2003) and perceptual quality using
Autoencoder:
LPIPS (Zhang et al. 2018) / KID (Bińkkwski et al. 2018) FID (Heusel et al. 2017).
VTM-17. ELIC R and MLIC++ ? For downstream tasks, we uniformly used the Taskonomy pre-trained Xception encoder and task-specific decoder. TAMC directly performs coherent task grouping and
For the compression baselines, we used the open source codes, i.e., JPEC WebP c
by Zamir et al. 2018
>5k52usp-sharing
15

---

Published as a conference paper at ICLR 2025
causal graph construction in the latent space, so its encoder and decoder follow the Xception structure of the pre-trained model. For the image compression task, better model architecture designs are already available to optimize performance.
A.3 SUPPLEMENTARY ABLATION STUDIES
In this section, we extend our analysis to E2E compression involving multiple supervised tasks as auxiliary tasks. We also examine single-task groups, where each task is treated as an independent group rather than grouping tasks together. Additionally, we explore whether the conherence from Task to Task B can be used to predict the conherence from Task B to Task A. Furthermore, we investigate the impact of prohibiting the same task from appearing in multiple clusters and assess whether this restriction leads to better or worse performance. Finally, we compare our approach with VQ-GAN-based compression method to evaluate its overall effectiveness.
A.3.1 ADDITIONAL RESULTS OF E2E COMPRESSION WITH MULTIPLE TASK AUXILIARY LOsS
Semanticc Segmentation Performance [A]
Surface Normal Performance [4]
Edge Texture Performance [1]
Ancner
Anchar
Ascheer
vuc Dure
MLC
MUC
Durs
Duts
ftututt Itast
Dr.Ride
St.uutt bapo
(a)
(b)
(c)
Zbuffer Parformance
Keypcint Performance
Autoencoder Nerformance
Acchar MuCh MUCH+
Task Aud Maaco NuC++
Aa0t0 Lo0
VUC++
itred
2000
(d)
(e)
(f)
Figure 8: Performance-rate curves for 6 tasks: Semantic Segmentation, Depth Z-buffer, Surface Normal, Keypoint 2D, Edge Texture, and Autoencoder, on the Taskonomy dataset. The comparison includes baseline compression methods MLIC++, Task AuxiliaryMMLI++) and our proposed TAMC. Results highlight the efficiency of our method achieving superior task performance at various bit rates, demonstrating the necessity of task grouping and scalable encoding.
To evaluate the impact of task grouping and the DAG on performance, we conducted an ablation study where both components were removed. In this alternative setup, multiple auxiliary tasks were integrated directly into the compression framework, as illustrated in Fig. 7 optimiz- ing the following combined loss function:
T
X
L Ccompression +
Σ W; Crask
(14)
T
Here, w; represents the weight assigned to each task, we set equal weights for tasks in our ablation studies. We tested this approach on 6 tasks: Semantic Segmentation, Surface Normal, Edge Texture, Depth Z-buffer, Keypoint 2D, and Autoencoder. Fig.&presentt the performance-rate curves for each task, comparing our method to baseline compression methods (c.g., MLIC++) and an auxiliary task- variant (Task Auxiliary+MI.ICH The results demonstrate the advantages of
Figure 7: End-to-end compression with Multiple Task Auxiliary Loss.
16

---

Published as a conference paper at ICLR 2025
approach in both compression efficiency and task performance, well the complex interplay between task collaboration and conflict, which highlights the significance of task grouping and scalable encoding.
Our method consistently outperforms baseline compression methods, especially for tasks such as Semantic Segmentation, Surface Normal, Edge Texture, and Depth Z--ufffrr. Even at lower bit rates, our framework achieves notable improvements over MLIC+ and Task Auxiliar+MMLIC++ underscoring its robustness and adaptability in multi-task settings. These results confirm the efficacy of task grouping and the integration of causal DAG in preserving task performance under constrained compression conditions.
The Task Auxiliary+MLICC variant, which replaces task grouping and DAG with auxiliary tasks, provides useful insights into task-level interactions. For tasks like Edge Texture, Semantic Seg- mentation, Surface Normal, and Depth Z-buffer, the auxiliary task approach yields substantial improvements compared to end-to-end compression methods, suggesting enhanced task collaboration and feature sharing. However, for Autoencoder, the auxiliary task approach performs similarly to end-to-end methods, indicating limited benefits for tasks with strong self-supervised structures.
In contrast, the Keypoint 2D task experiences performance degradation with the auxiliary task ap- proach, likely due to task interference. This highlights the potential conflicts between task objectives, emphasizing the importance of careful task grouping mitigate such issues.
The observed interplay of task collaboration and conffict further validates the need for task grouping By grouping tasks based on gradient coherence, our framework minimizes inter-task interference and promotes effective task collaboration, explaining its superior performance relative to the auxiliary task-based approach. Moreover, these results show that uncoordinated task interactions negatively impact specific tasks, such as Keypoint 2D.
Fig. Jalso reveals variations in bit-rate efficiency across tasks. For instance, Semantic Segmentation and Surface Normal maintain strong performance even at lower bit rates, while tasks like Edge Texture require higher bit rates due to the need for detailed feature representation. These findings highlight the importance of scalable encoding to accommodate the varying bit-rate needs of different tasks, By enabling task grouping and scalable encoding, our method addresses these challenges while optimizing compression efficiency.
A.3.2 PERFORMANCE COMPARISON OF SINGLE TASK VS. GROUPED TASKS
Semantic Seg.
Depth Z-buffer Surface Normal
Keypoint 2D
Method Bitrate Test Loss
Bitrate Test Loss Bitrate Test Loss Bitrate Test Loss
0.0014
0,0852
0.0008
0.2925
0.0006
0.1315
0.0017
0.2439
Single Task
0.0055
0.0680
0.0051
0.2648
0.0052
0.0963
0.0645 0.1115
0.0069
0.0674
0.0063
0.2643
0.0069
0.0938
0.0940
0.0954
0.0015 0.0704
0.0015
0.2615
0.0015
0.1378
Group
0.009
0.0598
0.0096
0.2419
0.0096
0.1079
0.0139
0.0574
0.0139
0.2385
0.0139
0.1045
0.0018
0.1528
10.0018
0.2412
Group 2
0.0623
0.1
0.0623
0.0944
0.0843 1080
0.0843 0.0936
Table 2: Performance-Bitten comparing task grouping with single task. Group represents the task grouping of Semantic Segmentation, Depth 7-bufffr, and Surface Normal. Group 2 represents the task grouping of Surface Normal and Keypoint 2D.. 2D.
To further examine the effectiveness of task groups, we trained model based on single-task com- pression, where each task is treated as an independent group rather than grouping tasks together. Additionally, we set up Grouping I compression model that jointly optimizes the task grouping of Semantic Segmentation, Depth Z-buffer, and Surface Normal, and a Group 2 compression model that jointly optimizes the task grouping of Surface Normal and Keypoint 2D. The results are shown in Tab2] and further visualized in the performance curves in Fig. 9 As observed, Semantic Seg- mentation and Depth Z-buffer benefit from a more compact and high-precision representation when
17

---

Published as a conference paper at ICLR 2025
Semantic Performance
Depth Zouffer Performance
Surface
Genip Socaric Sna Eoeptt Wwmmatieen Sepi
Gropp Senaine Sep pepejoommiiionn Seyht
Groul Semondi (Dopth NorrakSurthis Nerral
Singre Sux Narmal
(a)
(b)
(c)
Normal Performence
Keyooint
crusp Nimral Respoetifurtree tema]
Greas Nemral Repaiintespve
Segle Sorrs
Shnge despoly
(d)
(e)
Figure 9: Performance-Bitaae curves comparing task grouping with single task. The purple curves represent the bitratepeeffrrmacce performance of task grouping, while the red curves represent the performance of the single-task model.
Method
Semantic Depth Z-buffer/Surface Normal Texture Keypoint 2D 2otata Test Loss
Group 1
0.0535
0.2515
0.1070
Setting
0.5279
Group 2
0.0234
0.0925
Group Group 2
0.0535
0.2515
0.1070
Setting
0.1086
0.0274
0.0743
0.5223
Table Number of Task Grouping-2. Tasks are not allowed appear in multiple groups in Setting 1. Tasks can appear in multiple groups in Setting 2.
Method
Semantic Depth Z-buffer[Surface Normal Edge Texture Keypoint 2D Total Test Loss
Group 1
0.0531
0.2624
Setting Group 2
0.0234
0.0925
0.5298
Group 3
0.0984
Group
0.0535
0.2515
0.1070
Setting Group 2
0.0495
0.1034
0.0244
0.4866
Group 3
0.1100
0.0573
Table 4: Number of Task Groupings=3.
grouped in Group 1. Similarly, Keypoint 2D shows improved representation with higher compactness and precision in Group 2. Surface Normal's performance remains comparable between task grouping and the single-task approach.
One additional benefit of task grouping is the shared encoder and shared representations across multiple tasks. For example, in Group 1, a single encoder feature extraction is used for inference. consuming 0.0015 bpp for Semantic Segmentation, Depth Z-buffer, and Surface Normal. In contrast, treating each task independently requires three separate feature extraction inferences, with a total bitrate of 0.0014 bpp + 0.0008 bpp + 0.0006 bpp = 0.0028 bpp to serve the three tasks.
A.3.3 IMPACT OF TASK EXCLUSIVITY ACROSS GROUPS ON PERFORMANCE
From performance ceiling perspective, allowing the same task to appear in different clusters maximizes the potential for task collaboration. As shown in Tab. Bad | we conducted experiments under two settings: Setting 1, where tasks allowed appear in multiple clusters, and Setting
18

---

Published as a conference paper at ICLR 2025
2, where tasks appear in multiple clusters. Setting 2 achieves superior performance ceiling, demonstrating the advantages of task interdependence. Notably, since each task is only inferred once, Setting 2 does not introduce additional inference complexity. However, it does result in significant increase in GPU memory consumption during training. This highlights the trade-off between performance and resource utilization when task exclusivity is relaxed.
A.3.4 IMPACT OF TASK ORDER TASK GROUPING AND ADDRESSING VARIABILITY
Regarding whether the order of tasks affects the cost cal- culation and how to address potential variability in the method using coherence scores: In Sec. 3.2 the number of possible groupings for n tasks is given by the Bell num- bers. To quickly estimate the similarity between tasks, after calculating Cu-ru. Cw-ses we estimate the higher- order costs for (Tu: Tus Twr) which significantly reduces the computational complexity. We mitigate the impact of task order with the following operations:
Tel arriig Ma-i
The coherence score is measure based on the impact of gradient updates between tasks on the loss function. It is a relative measure that reduces the impact the absolute order of task execution. 2. By calculating the coherence scores throughout the entire training process and taking the average, can mitigate the impact caused by specific stages of training, thus reducing the potential variability brought about by changes in the order of tasks. 3. To validate the reasonableness of this operation, in Fig TO experimentally demonstrate that although the coherence symmetric (Cn-po A Co..n), exhibits strong symmetry trend in practice. This allows to approximate the values while maintaining accuracy and efficiency.
Figure 10: Inter-task coherence on Taskonomy. Red color signify higher inter-task affinities.
between task pairs is not strictly
A.3.5 COMPARISON WITH VQ-GAN COMPRESSION
Semantic Segmentation Performance [
Surface Normal Performance [
Edge Texture Performance
Arhhv
Andor
Or:
cuer
Dur
Et-Rula
(a)
(b)
(c)
Death Zbuffer Performance (6]
Keypoint Performance
Autocncoder Performacce
Anthor VaciA
POGAA
28
ViSW
MUC-
Duts
Itpe?
(d)
(c)
Figure 11: Comparison with VQGAN Mao et al. 2024 in multi-task compression: Our method outperforms VQGAN compression efficiency at low bitrates and closely matches the supervision anchor at higher bitrates, particularly in tasks like Keypoint 2D, Semantic Segmentation, and Depth Z-buffer. VQGAN, however, shows performance degradation in fine-grained tasks, highlighting the advantages of our task-aware semantic compression approach.
19

---

Published as a conference paper at ICLR 2025
While VQGAN-based compression methods (Esser al. 2021 Mao et al. 2024) achieve perceptual compression at low bitrates through discretized codebooks, our optimization goal diverges signifi- cantly. Unlike VQGAN (Mao et al. 2024), which prioritizes image reconstruction and perceptual quality, our approach focuses on compact, multi-task semantic compression. Specifically, we opti- mize for efficient semantic representation sharing across tasks, reducing redundancy in encoding. This contrasts with VQGAN's generative approach. Comparative experiments in Fig. [1 reveal key differences in performance and efficiency.
Although VQGAN (Mao et al., 2024) demonstrates significant improvements in compression rates over VAE backbone models, especially for perceptual tasks, it still introduces bias compared the optimal supervision anchor, even with sufficient bitrate. This is due to its generative nature, which cannot fully eliminate reconstruction errors, leading to discrepancies from the ground truth. Moreover, VQGAN struggles with tasks requiring precise, sparse local feature detection (e.g.. Keypoint 2D), a limitation common in generative models that fail capture fine-scale features. contrast, our method is specifically designed to encode and preserve task-specific features, achieving superior performance in these tasks.
In multi-task settings (e.g., Semantic Segmentation. Surface Normal, Edge Texture, Depth Z-buffer), our approach demonstrates superior compression efficiency at low bitrates and near-optimal per- formance at higher bitrates, closely matching the supervision anchor. VQGAN, while effective for perceptual compression, struggles to leverage task-specific semantic information, leading to inferior performance, particularly at low bitrates.
A.4 COMPLEXITY ANALYSIS
A.4.1 COMPLEXITY ANALYSIS OF COMPRESSION FOR MULTIPLE TASKS: ATC vs. CTA PARADIGMS
MLIC++ Encoder/Deeoder Madale
9o
So
Paramss
12033..
4396.3
MMACCs
194556.9
296377.8
Table 5: Parameters and Forward Maes of Encoder/Decoder of MLIC++ on 512 x 512 images.
Task Xception EncoderDeoooer Module
La
KParams
16467.2
525.
MMAC's
25708.0
4968.1
Table Parameters and Forward Macs Task Encoder/Decoder of Xception on 512 x 512 images.
In the task of compression for multiple downstream tasks, we investigate two compression paradigms Analysis and Then Compression (ATC) and Compression and Then Analysis (CTA). ATC pipeline consists of two main phases: the downstream task analysis phase, where the input image is used for specific tasks such as keypoint detection, segmentation, and depth estimation, and the feature compression phase, which includes the encoding and decoding steps. The CTA pipeline also consists of two main phases: the image compression phase, which includes encoding and decoding, and the downstream task analysis phase, where the compressed image is used for tasks such as keypoint detection, segmentation, and depth estimation. To better understand the computational costs involved Operations)
in each module, we summarize the parameters and forward MACs (Multiply-Acuuuule of the different components in the ATC and CTA pipelines. These values are presented the tables 5 6 Below, present the complexity analysis for both approaches.
Our proposed method belongs to ATC paradigm, and the total computational complexity of this pipeline results from both the downstream task analysis and the feature compression phases. The number downstream tasks N and the number of task groups K directly affect the computational cost of the task analysis phase. The overall complexity is expressed as:
Total Complexity of ATC N 1, K (Ia + Ond + Oen+ 0go)
(15)
N 4968. 1 + K 40088.0,
20

---

Published as a conference paper at ICLR 2025
Context Modile
Phd
9cm
8y0
KPaaras
5810.1
766.2
1487.7
2264.4
MMACa
8925.1
1148.2
2264.4
7100.5
Table Parameters and Forward Macs of entropy context modules on 512 x 512 images.
where Ond. Oem+. and 0ge represent the complexities of the context modules. N is the number of downstream tasks, is the number of task groups, and l, represent the complexities the Task Xception Encoder/Decoder for each individual task.
This formulation indicates that the task analysis phase (which involves both la and (2) and the context module complexities contribute to the total computational cost.
The total computational complexity in the CTA pipeline arises from both the compression and downstream task analysis phases. The number of tasks N directly affects the computational cost of the task analysis phase. The overall complexity is expressed as:
Total Complexity of CTA Yo Js + Ond + 0em 0ge + N (la + +(,)
(16)
500008.6 + N 300666.1,
where Ia and Is are the complexities of the MLIC+ Encoder/Decoder. Ona. Owm and 0ge represent the complexities of the context modules. is the number downstream tasks. la and l, represent the complexities of the Task Xception Encoder/Decoder for each individual task.
This formulation emphasizes the contribution of the encoding/decoding processes (represented by 9e and gs) as well as the task-specific encoding/decoding complexities L, and I, in the CTA pipeline.
A.4.2 COMPLEXITY ANALYSIS OF OUR LOOKAHEAD MODULE
Our Lookahead module consists of two steps: task grouping and DAG construction. In the task grouping step, we first compute the coherence score between tasks using joint training on N N task pairs. Then, we use pairwise coherence scores to estimate the coherence scores of higher-order task groupings. Finally, based all task groups, task coherence scores and budget multitask networks to maximize the overall task performance. This is an NP-hard problem. A brute
select
force approach would take O(T [Co minnECy < ), which is exponential in the maximum number of groups that fit within the budget. Here, [T] is the number of tasks, |Col is the number of candidate networks. b is the budget, and min,ec, C, is the smallest cost among the networks. This can be solved using the branch-am and-bound- bound-like algorithm provided prior work (Zamir et al., 2018) as detailed in A.5
21

---

Published as a conference paper at ICLR 2025
Algorithm 2 Get Task Grouping Strategy
Input: C,, running set of candidate groups, each with associated cost e E IR and a performance score for each task the network solves. Initially, C, Co
Input: S, C Co. running solution, initially
Input: by E I,, the remaining budget, initially b
1: function GETBBESTNETWOKKSSC......
2:
C, FILTER(C,, Sr, b,)
3:
C, S
b Most promising groups first
4:
Best + S
5:
for n E C, do
6:
C, + C, \ S,
\ set subtraction.
7:
8:
b br Gn
9:
Child +- GETBESTNATWORKSC,, b;
10:
Best + BETtER(Best, Child)
11: end for
12: return Best
13: end function
14: function FILTER Sr, by)
15:
Remove groups from C, with Cn bb
Remove groups from C, that cannot improve Sy's performance on any task.
17: return C₁
end function
19:
function BETTER(S,, S2)
20:
if ((i) C(22) then
21:
return S
22: else
23:
return S
24:
end if
25:
end function
The overall time complexity of the DAG construction algorithm dominated by the nested loops that compute conditional and independent entropies for each latent group. The algorithm iterates all K groups, leading to outer loop complexity of O(K). For each group, the algorithm computes the conditional entropy between pairs of latent variables. This operation takes ((n) time, where n is the size of the dataset (number of samples). Additionally, the algorithm computes the independent entropy for each group, which takes ((m) time, where m represents the complexity of entropy calculation for single group Thus, the overall time complexity is:
O(K². O(K .m).
where K is the number of groups, the time complexity for calculating conditional entropy. m is the time complexity for calculating independent entropy for group.
The process of computing task coherence scores, grouping tasks, and constructing the corresponding Directed Acyclic Graph (DAG) is computationally intensive. These steps share conceptual similarity with the lookahead stage traditional video encoding (Li) 2003 He & Mitra 2002] Wang & Kwong 2008 Ma et al.] 2005). In video encoding, the lookahead stage performs a preliminary analysis of the video content optimize the encoding process, ensuring that the final compression achieves the best trade-off between quality and bitrate. Specifically, it evaluates potential encoding decisions ahead of time to minimize redundancies and improve efficiency, while adhering to bitrate constraints.
Similarly, in multi-task compression, the task grouping and causal relationship modeling steps aim to optimize the encoding of multiple tasks by leveraging the inherent interdependencies among them. However, due the high computational complexity of calculating coherence scores between tasks and determining the optimal task groupings, an efficient pre-analysis phase is essential. Our contribution lies in exploring the effectiveness of task grouping and cross-task causal relationship
22

---

Published as a conference paper at ICLR 2025
modeling, demonstrating that these techniques can significantly enhance the performance of multi-task compression.
To make this process feasible and scalable, further exploration involve using downsampled low-rcsolution images as inputs for the pre-analysis phase. This strategy, when combined with a low-complexity feature extraction backbone (landola (016), provides an efficient means of assessing task grouping performance under bitrate consumption constraints.
A.5 IMPLEMENTATION DETAILS OF TASK GROUPING
As mentioned Sec. 3.2 we adopt a branch-and-boune method, as in prior works (Standley et al. 2020; Zamir et al., 2018). to search for locally optimal grouping strategies under the given bitrate budget b. Here we provide pseudo-algorithm for clarity, and the implementation is provided in Githut Consider the situation in which we have an initial candidate set Co {MI, .i.,..., 70m of fully trained networks that each solve some subset of our task set T. Our goal is to choose a subset of Co that solves all tasks in budget b and the lowest overall loss. More formally, we want to find solution S, = arg minscCo (CC:: C(S). It can shown that solving this problem is NP-hard in
general (reduction from SET-COVER). A brute-force approach would take O(TT) |Co E en ) which is exponential in the maximum number of groups that fit in our budget. This would be computationally challenging even for small problems.
However, many techniques exist that can optimally solve most instances of problems like these in reasonable amounts of time. All of these techniques produce solutions that perform equally well. We chose to use branch-and www--nndd--ddiiie Algorithm 2) but in principle the same solution could be achieved by other optimization methods, such encoding the problem
algorithm for finding this optimal solution (shown in
binary integer program (BIP) and solving it in a way similar to Taskonomy (Zamir et al., 2018). 8). Algorithm 2 chooses the best subset of groups in our collection, subject to the inference time budget constraint. The algorithm recursively explores the space of solutions and prunes branches that cannot lead to optimal solutions. The recursion terminates when the budget exhausted, at which point C, becomes empty and the loop body does not execute. The sorting step on line 3 requires heuristic upon which to sort. We found that ranking models based on how much they improve the current solution, S, works well. It should be noted that this algorithm always produces an optimal solution, regardless of which sorting heuristic is used. However, better sorting heuristics reduce the running time because subsequent iterations will more readily detect and prune portions of the search space that cannot contain an optimal solution.
A.6 MORE EXPERIMENTAL RESULTS
In Fig. [2] we present examples showing competitive qualitative results from our method compared to VTM and end-to-end compression methods. Fig. provides detailed comparison of PSNR-Bitras and LPIPS-Bitrate performance.
23

---

Published as a conference paper at ICLR 2025
JPEG 0.359 hpp. 33.01dB, 0.9624
VTM 0.282 hpp. 38.42dB, 0.9903
Ground Truth
ELIC 0.370 bpp. 40,98dB, 0.9932
MLIC 0.352 hpp. 41dB,dB, 0.9946
Ours 0.346 hpp. 37.85 1d, 0.9890
JPEG 0.479 bpp. 33.60dB, 0.9753
VTM 0.389 bpp. 37,24dB, 0.9888
Ground Truth
11
3
5 3
6 5
5
ELIC
MLIC 0.472 hpp. 39.65dB, 0.9934
Ours 0.436 hpp. 36.013dB, 0.9840
0.370 hpp, 40.98dB, 0.9978
Figure 12: Subjective comparisons.
FES
PCS
WEEP
TRaP
(a)
(b)
Figure 13: Comparison of PSNR-Bitrate performance and LPIPS Zhang et al. (018)-Bitran performance for image compression on the Taskonomy dataset, using baseline methods and our proposed TAMC.
24


Reviews:
[{"review_id": "nJRfw14Qft", "reviewer": "Reviewer_63Jo", "summary": "In this paper, the authors focus on learning methods for image compression. To this end, they propose a method that optimizes image compression for a number of different downstream visual tasks. They consider the potential redundancy of encodings for similar groups of visual tasks, and they utilize directed acyclic graphics to learn causal relationships between tasks. This approach leads to a better multi-task representation. They evaluate their approach on a number of different visual tasks and demonstrate convincing quantitative results.", "questions": "1. Would it be possible to incorporate more details of the graph learning into the main paper?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The problem has multiple applications and is very sensible. Image compression has direct implications, and it is sensible to think image compression in terms of its final use (object detection, classification, etc) as opposed to only pixel reconstruction.\n\n2. The proposed approach has a strong mathematical foundation and makes intuitive sense (conditional entropy should help resolve redundancy between certain tasks). \n\n3. Quantitative results are thorough and show promise.", "weaknesses": "1. I believe the description of the DAG learning should have more detail. Specifically, some of the details of the DAG based algorithm (described in the appendix as Algorithms 1 and 2) should be incorporated into the main paper. \n\n2. There are some typos and grammatical errors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NF4AAIBuGu", "reviewer": "Reviewer_mF5B", "summary": "This paper proposes a framework, Taskonomy-Aware Multiple Task Compression (TAMC), for lossy compression where the distortion is defined by multiple downstream tasks. First, TAMC  groups tasks into clusters where tasks within a cluster are mutually supportive and a share representation is learned for each cluster. Then, it leverages causal discovery to identify dependencies between groups. This results in a directed acyclic graph (DAG), which can be used for further compression of the representation.  Experiments on the Taskonomy dataset demonstrate that TAMC achieves superior bitrate reduction and task performance compared to baseline compression methods.", "questions": "1. Could you state the number of parameter sets used and explain how they are allocated across groups and individual tasks? The paper seems to suggest it's per group, but it also needs a set of task parameters per task for computing the pairwise gradient coherence.\n2. How is the forecast loss in Eq.3 used? \n3. Line 287, could you clarify if task order impacts the cost calculation and, if so, how do you address this potential variability in their method?\n4. Is there any constraint on the subsets used for set cover?\n4. Is it better or worse when the same task is not allowed to appear in different clusters? \n5. How is bitrate controlled in the proposed framework? How do you determine $\\lambda_i$ for each task?\n6. Is the causal discovery step required? Could the minimum description length principle be used for causal discovery and, thus, unifying step 2 and 3 as finding a DAG structure that minimizes the bit-rate?\n7. Could you provide a complexity analysis showing how computational requirements change as the number of tasks increases?\n8. A simple baseline would be multi-task learning naively combined with end-to-end compression, i.e. without the grouping, just use them auxiliary tasks with well-tuned weights. Why would you not include such a baseline? How would this baseline compare theoretically to the proposed method?\n9. Figure 3, could you add a legend for what the color shade represent?  Could you define the \"Anchor\" baseline in the figure caption or main text?\n10. How does the method perform compare to more familiar baselines such as VQGAN?\n11. Github link is not provided?\n12. L137 typo \"Hu\"?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "The paper discusses an important but less studied aspect of representation learning - how to leverage different supervised signals to learn a better representation, where \"better\" is defined as lower bitrate with higher downstream performance. This principle is sound. The experimental results demonstrate that the proposed framework achieves better performance comparing to end-to-end compression methods using the same bitrate.", "weaknesses": "**Clarity**: the description of the framework lacks clarity. Given that the paper proposes a fairly complex system, clarity is even more important for the readers to understand. For example, although there is a space constraint, the description of step 3 (taskonomy-based compression) is too brief. Other more specific questions are raised below.\n\n**Lack of ablations**: there are at least 2 ablations the authors could provide. 1. an end-to-end compression with multiple supervised tasks as auxiliary tasks; 2. single task groups, where instead of grouping tasks together in phase 1, simply treat each task as a group and carry out the rest of the learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CuBpeJdO6y", "reviewer": "Reviewer_5H7f", "summary": "In this paper, the authors propose a data compression technique using representations that can be applied to multi-tasking in computer vision. They propose a methodology for task-aware representation learning by using multi-task learning approaches and grouping similar tasks based on the alignment of gradient vectors during the training phase. To efficiently utilize the learned representations, they calculate the causal relationships between task groups using conditional entropy and construct a directed acyclic graph (DAG). The proposed method was validated using the Taskonomy dataset, and the baselines included traditional data compression methods (e.g., JPEG, WEBP) as well as some recent methods (e.g., ELIC, MLIC++, etc.). The authors confirmed the robustness of the proposed method at relatively lower bit-rates compared to the baselines across multiple tasks.", "questions": "- Although the paper states that tasks were grouped based on gradient coherence, the justification for task grouping is not clearly stated in LN 470. Could the task grouping process be further explained?\n- In Appendix A.5.2, the authors state that they select the parent node for a chosen child based on the node with the highest mutual information and claim that this is equivalent to the optimal choice for conditional entropy. While this claim seems correct, in the partial ordering of Algorithm 5, the child node is not explicitly defined, so the claim does not appear to be directly related to the implementation of the proposed method. It seems there might be an implicit assumption that the task with the larger entropy in its representation distribution is assigned the parent node. Would this be an accurate interpretation?\n- It is mentioned that the total bit-rate of task-wise representations was used as a regularization for model training, but there is not enough explanation regarding the bit-rate for each task in the trained models. It would be helpful to include information on the observed bit-rates for each task and to explain how these bit-rates may be related to the tasks.\n- Although theoretical approaches are mentioned in Appendix A.5.4 and A.5.5, they do not seem to be fully utilized or proven within the logic of the paper. Is there a plan to elaborate on these? In addition, could further explanations be provided in relation to the above Weaknesses and Questions?\n- (minor) Uniform upper and lower case conventions in subsection titles (especially in Section 5)\n- (minor) Typo in Equation 3 (\\theta_s -> \\theta_s^t)\n- (minor) Typos in Algorithm 1, 2 in Appendix (end for\"\\n\"return)", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- Hierarchical structure of multiple tasks: The authors propose a methodology that learns a universal representation for computer vision tasks by utilizing techniques from multi-task learning and uses these representations to form a hierarchical structure of the tasks. I believe this methodology could aid research on general representations in areas beyond data compression.\n- Robustness of the trained representation: At lower bit-rates, the performance on downstream tasks consistently outperforms trained compression methods (such as MLIC++, ELIC).\n- Ablation with random graphs: The authors compared the performance of the proposed causal DAG structure, which exploits causal relationships, with that of a random graph structure to verify its effectiveness. This demonstrates the existence of inter-task relationships and shows that efficient learning and inference, which take these relationships into account, have a considerable impact on performance.", "weaknesses": "- Lack of clarity, not well-organized components: The figures in the paper are generally not well-organized, and the font size within the figures is excessively small, causing visibility issues. In particular, in Figure 2, the font size within the figure is less than half the line height. Even if it results in a reduction of the amount of information in each figure, it seems necessary to increase visibility by focusing on the most important information.\n- Relatively low data decoding accuracy: As mentioned by the authors, the results in subsection 4.3.2 show that the proposed method performs similarly or relatively worse than existing methods in terms of image compression. In order to show that this difference is not a significant problem for the application of the method, it seems necessary to qualitatively compare the decoded images, at least with some examples.\n- No results with time and space complexity: One of the critical aspects of a data compression algorithm is minimizing the time and space cost of its execution. Since this work focuses on data compression techniques using multi-task learning methods rather than representation learning methods, I believe it is necessary to analyze such costs to justify the applicability of the proposed method.\n- Lack of baselines for comparison: While I think it seems meaningful that the paper uses traditional data compression techniques (e.g., JPEG, WEBP) as baselines, there is still a lack of comparison with more recent trainable methods. Even if the compared baselines are considered state-of-the-art models, I believe that additional comparisons are necessary to clarify the consistent robustness of the proposed method compared to other approaches.\n- To summarize, the paper lacks completeness in explaining the methodology and convincing the readers through the results, and the experiments are not sufficiently thorough. Regarding the experiments, it may be helpful for the authors to refer to the analytical techniques used in the MLIC++ paper [1], which they have cited as a key reference. Through a comprehensive revision of the content and additional analyses, the paper’s clarity can be improved, and its novelty can be further emphasized.\n\n[1] Wei Jiang et al. MLIC++: Linear Complexity Multi-Reference Entropy Modeling for Learned Image Compression. https://arxiv.org/abs/2307.15421", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)