Multi-task Learning with 3D-Aware Regularization
Authors: Wei-Hong Li
Steven McDonagh
Ales Leonardis
Hakan Bilen
Venue: ICLR 2024 poster (2024)
Keywords: Multi-task learning
3D-aware
dense prediction
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We proposes a 3D-aware regularizer that maps the input image to a structured space where multiple geometrically inconsistent cross-task correlations can be eliminated and improves the performance.

Abstract: Deep neural networks have become the standard solution for designing models that can perform multiple dense computer vision tasks such as depth estimation and semantic segmentation thanks to their ability to capture complex correlations in high dimensional feature space across tasks. However, the cross-task correlations that are learned in the unstructured feature space can be extremely noisy and susceptible to overfitting, consequently hurting performance. We propose to address this problem by introducing a structured 3D-aware regularizer which interfaces multiple tasks through the projection of features extracted from an image encoder to a shared 3D feature space and decodes them into their task output space through differentiable rendering. We show that the proposed method is architecture agnostic and can be plugged into various prior multi-task backbones to improve their performance; as we evidence using standard benchmarks NYUv2 and PASCAL-Context.

Published as a conference paper at ICLR 2024
MULTI-TASK LEARNING WITH 3D-AWARE REGULARIZATION
Wei-Hong Li', Steven McDonagh' Ales Leonardis", Hakan Bilen'
University of Edinburgh. UUniversity of Birmingham github. www//onc//oo.. UoE/ UoE/ reMW
ABSTRACT
Deep neural networks have become the standard solution for designing models that perform multiple dense computer vision tasks such as depth estimation and semantic segmentation thanks to their ability to capture complex correlations in high dimensional feature space across tasks. However, the cross-task correlations that are learned in the unstructured feature space can be extremely noisy and susceptible to overfitting, consequently hurting performance. We propose to address this problem by introducing a structured 3D-aware regularizer which interfaces multiple tasks through the projection of features extracted from an image encoder shared 3D feature space and decodes them into their task output space through differentiable rendering. We show that the proposed method architecture agnostic and can be plugged into various prior multi-task backbones to improve their performance; we evidence using standard benchmarks NYUv2 and PASCAL--Contxt..
INTRODUCTION
Learning models that perform multiple tasks coherently while efficiently sharing computation across tasks is the central focus of multi-task learning (MTL) (Caruana 1997). Deep neural networks (DNNs), which have become the standard solution for various computer vision problems, provide at least two key advantages for MTL. First, they allow for sharing a sigmificant portion of features and computation across multiple tasks, hence they are computationally efficient for MTL. Second, thanks to their hierarchical structure and high-dimensional representations, they can capture complex cross-task correlations at several abstraction levels (or layers).
Yet designing multi-task DNNs that perform well in all tasks is extremely challenging. This often requires careful engineering of mechanisms that allow for the sharing of relevant features between tasks, while also maintaining task-specific features. Many recent multi-task DNNs (Vandenhendel et al. 202T) can be decomposed into shared feature encoder across all tasks and following task- specific decoders to generate predictions. The technical challenge here is to strike balance between the portion of the shared and task-specific features to achieve good performance aefformance-coopputtiioo trade- off. To enable more flexible feature sharing and task-specific adaptation, Liu al. (2019) propose to use 'soft' task-specific attention modules appended to the shared encoder that effectively shares most features and parameters across the tasks while adapting them each task through fight-weight attention modules. However, these attention modules are limited to share features across tasks only within each layer (or scale). Recent works (Vandenhende et al. 2020b; Bruggemann et al. 2021 propose to aggregate features from different layers and to capture cross-task relations from the multi-scale features. More recently, capturing long-range spatial correlations across multiple tasks has been shown to further improve MTL performance through use of vision transformer modules (Ye) Xu 2022).
Our central hypothesis is that the high-dimensional and unstructured features, shared across tasks, in the recent MTL models are prone to capturing noisy cross-task correlations and hence hurt perfor- mance. To this end, we propose regulating the feature space of shared representations by introducing a structure that valid for all considered tasks. In particular, we look at dense prediction computer vision problems such as monocular depth estimation, semantic segmentation where each input pixel is associated with a target value, and represent their shared intermediate features in a 3D-aware feature space by leveraging recent advances in 3D modeling and differentiable rendering (Niemeyer et al.

---

Published as a conference paper at ICLR 2024
2020: Mildenhall et al. 2020 Chan et al. 2022
)023). Our key intuition is that the physical 3D world affords us inherent and implicit consistency between various computer vision tasks. Hence, by projecting diteeeinnnii features to a structured 3D-aware space, our method eliminates multiple commerrically-nnooiiiin
cross-task correlations.
To this end, propose a novel regularization method that can be plugged into diverse prior MTL architectures for dense vision problems including both convolutional (Vandenhende et al. 2020b) and transformer (Ye & Xu 2022) networks. Prior MTL architectures are typically composed of shared feature extractor (encoder) and multiple task-specific decoders. Our regularizer, instantiated as a deep network, connects to the output of the shared feature encoder, maps the encodings to three groups of feature maps and further uses these to construct a tri tri-plane representing planes x-- y, 2-2, y-2 as in Chan et al. (2022). We are able to query any 3D position by projecting onto the tri-plane and retrieve corresponding feature vector through bi-linear interpolation across the planes, passing them through light-weight, task-specific decoders and then rendering the outputs as predictions for each task by raycasting, as in Mildenhall al. (2020). Once the model has been optimized by minimizing each task loss for both the base model and regularizer, the regularizer is removed. Hence our method does not bring any additional inference cost. Importantly, the regularizer does not require multiple views for each scene and learns 3wwawe representations from a single view. Additionally, our model generalizes to unseen scenes, as the feature encoder is shared across different scenes.
Our method relates to both MTL. and 3D modeling work. It is orthogonal to recent MTL contributions that focus rather designing various cross-task interfaces (Vandenhende et al.. 2020a Liu et al.
2019). or optimization strategies that may obtain more balanced performance across tasks (Kendall al. 2018 Chen al. 2018). Alternativlly, our main focus is learn better MTL representations by entorcing 3D structure upon them, through our 3D-aware regularizer. Our method can be incorporated to several recent MTL methods and improve their performance. Most related to ours, Zhi al. (202I) and Kundu et al. (2022) extend the well-known neural radiance field (NeRF) (Mildenhal et al. 2020)
to semantic segmentation and panoptic 3D scene reconstruction, respectively. Unlike them, our main focus is to jointly perform multiple tasks that include depth estimation, boundary detection, surface normal estimation, in addition to semantic segmentation. Uniquely, our method does not require multiple views. Finally, method is not scene-specific, can learn multiple scenes in single model and generalizes to unseen scenes.
To summarize, our main contribution is novel 3D 3waawere regularization method for the MTL of computer vision problems. Our method is architecture agnostic, does not bring any additional computational cost for inference, and yet can significantly improve the performance of state-of-the-art MTL models as evidenced under two standard benchmarks; NYUv2 and PASCAL-Context.
2 RELATED WORK
Multi-task Learning MTL (Caruana 1997)
commonly aims to learn single model that can
accurately generate predictions for multiple desired tasks, given an input (see Fig. Fig. (a)). We refer to Ruder (017)); Zhang& & Yang (201)); Vandenhende et al. (2021) for comprehensive literature review. The prior works in computer vision problems can be broadly divided into two groups The first group [ookkinos] 2017 Ruder et al.
2019 Vandenhende et al. 2020a Liang et al.) 2018
Bragman et 2019 Strezoski et al. 2019
Xu et al.] 2018 Zhang et al. 2019 Bruggemann et al. 2018 Xu al. 2018) focuses on improving network
2021 Bilen & Vedaldi 2016 Zhang et al.
architecture via more effective information sharing across tasks by designing cross-task attention
mechanisms Misra et al. (2016), lask-specific attention modules (Liu et al.
2019; Bhattacharjee et al.,
2023), cross-tasks feature interaction (Ye & Xu] 2022]] Vandenhende et al.
(022b). gating strategies
or mixture of experts modules (Bruggemann et al.] 2020: Guo et af.] | [0200] Chen et al. 20023] Fan
et al. 2022 Ye & Xu (023b), visual prompting (Ye & Xu 2023a] Liu et al. (023). The second group aims to address the unbalanced optimization for joint minimization of multiple task-specific loss functions, where each may exhibit varying characteristics. This is achieved through either actively
changing loss term weights (Kendall et al. [0118] Liu et al. 2019: Guo et al., 2018 Chen et al. 2018
Lin et al. 2019:; Sener& & Koltun| 2018. Liu et al. 2021b). modifying the gradients of loss functions
w.r.t, shared network weights to alleviate task confiits ( (Yu et al. 2020: Liu et al.) 2021a Chen et al.
2020 Chennupati al. 2019 Suteu & Guo. 2019) or knowledge distillation (Li & Bilen 2020
2

---

Published as a conference paper at ICLR 2024
Thok Tod-spenifcc ceroodera
decodars
Crooder staned balks ocanns
P
NJP
20
mooss laoko
3-8irttioonal VIe diriciio
3-dimaie
Vew foction
3D CO-anar regUiriO
InPaamtile modti-knk Learning
Standard rediurnoe lonkd (Nerths)
3D NTL [0an]
Figure 1:
Illustration of (a) vanilla multi-task learning, (b) standard neural radiance fields (NeRFs) and (c) our 3D-aware multi-task learning method.
al. 2023). Unlike these methods, our work aims improve MTL performance by regularizing DNNs through the introduction of 3D-aware representations (see Fig. (c)).
Neural Rendering Our approach also relates to the line of work that learns 3D scene represen-
tations for performing novel view synthesis (Lombardi et al. [2019] Meshry et al.
2019: Sitzmann
ct al.
2019; Thies et al. [2019] Mildenhall et al. 2020; Chan et al. 2021] Cai et al.
2022
Gu et al.
2021 D. creating explicit 3D object (Lin et al. 2023), view editing (Zhang et al. 2023 Prior methods
with few exceptions can represent only a single scene per model, require many calibrated views, or
not able to perform other tasks than novel view synthesis such as semantic segmentation, depth
estimation (see Fig. (b)). PixelNeRF Yuu et al. 202T) conditions a NeRF (Mildenhall et al.) (0200) on image inputs through an encoder, allows for the modeling of multiple scenes jointly and generalizes to unseen scenes, however, the work focuses only on synthesizing novel views. ZZhi et al. (2021) extend the standard NeRF pipeline through parallel semantic segmentation branch jointly encode semantic information of the 3D scene, and obtain 2D segmentations by rendering the scene for given view using raycasting. However, their model is spen--se and does not generalize to unseen scenes. Panoptic Neural Fields (Kundu et al. 2022) predict radiance field that represents the color, density, instance and category label of any 3D point a scene through the combination of multiple encoders for both background and each object instance. The work was designed for predicting those tasks only on novel views of previously seen scenes, hence it cannot be applied to new scenes without further training them and is also limited to handle only rigid objects (c.f. non-rigid, deformable). In contrast, our method can be used to efficiently predict multiple tasks in novel scenes, without any such restrictions object type, can be trained from single view and is further not limited to fixed architecture or specific set of tasks. Finally, our work harnesses efficient triplane 3D representations from (Chan et al., (2022) that is originally designed to generate high-quality. 3D-aware representations from a collection of single-view images. Our method alternatively focuses on the joint learning of dense vision problems and leverages 3D understanding bring a beneficial structure to the learned
representations.
3 METHOD
We next briefly review the problem settings for MTL and neural rendering to provide required background and then proceed to describe our proposed method.
3.1 MULTI-TASK LEARNING
Our goal is learn a model ý that takes an RGB image I as input and jointly predicts ground-truit truth labels Y "ui....,yy'" for T tasks. In this paper, we focus on dense prediction problems such as semantic segmentation, depth estimation where input image and labels have the same dimensionality. While it is possible to learn an independent model for each task, a more efficient design involves sharing a large portion of the computation across the tasks, via a common feature encoder f. Encoder takes in image as input and outputs nigh-dimensional feature map which has smaller width and height than the input. In this setting, the encoder is followed by multiple specifcc decoders hy that each ingests f (T) predict corresponding task labels iee,, (T)), depicted in Fig. (a).
3

---

Published as a conference paper at ICLR 2024
feature
In-plane
tri slane anooder
seile,))
Handaring
Neund Tesk
30
Pkg
Encodings
reprosentations
from shared encoder
Figure 2: Diagram of aw-awere regularizer g. The regularizer takes as input the features from the shared encoder and transforms it to tri- tri-plane using tri tri-plane encoder Given 3D point (I, y, on a given ray, we project the coordinates onto three planes and aggregate features from three planes using summation obtain the 3D representations, which are then fed into light-weight MIPP n, to estimate predictions of each task or the density of the 3D point. Finally, in volume rendering we integrate the predictions over the ray render the predictions of each task.
Given labeled training set D with N image-label pairs, the model weights can be optimized as:
min S.N.
Σ Σ c, o 2()).
(LYY)D
where L. is the loss function for task I, i.e, cross entropy loss for semantic segmentation, I, loss for depth estimation. We provide more details in Sec.
3.2 3D-AWARE MULTI-TASK LEARNING
An ideal feature extractor f is expected to extract both task-agnostic and task-specific information, towards enabling the following task-specific decoders solve their respective target tasks accurately. However, in practice, the combination of high-dimensional feature space and highly non-linear mappings from input to output is prone to overfitting to data and learning of noisy correlations. To mitigate these issues, we propose a new 3D-aware regularization technique that first maps extracted features 3D neural codes, projects them to task-specific fields and finally renders them to obtain predictions for each target task through differentiable rendering (see Fig. (2). In the regularization, outputs for all tasks are conditioned on observations that lie on a low-D manifold (the density (Mildenhall et al. 2) enforcing 3D consistency between tasks.
Training the recent MTL models (e.g. Vandenhende et al. 2020b); Ye & Xu (2022) on high resolution input images for multiple dense prediction tasks simultaneously is computation and memory intensive. Hence, naively mapping their multi-scale high-resolution features to 3D is not feasible due memory limitations in many standard GPUs. Hence, adopt tri- tri-plane representations of Chan et al. (2022). In particular, we first feed
3D representations.
the hybrid explicit-implle into a shared encoder and obtain a W H x xC C-dimensional feature map where and W are the height and width. Then, through tri-plane encoder e, we project the feature map to three explicit W H C' dimensional feature maps, ery: eyz: Cazz that represent axis aligned orthogonal feature planes. We can query any 3D coordinate (æ, y, 2) by projecting it onto each plane, then retrieve the respective features from three planes via bi-linear interpolation and finally aggregate features using summation obtain the 3D representation (e(z, y, 2) Exy (I, y) +ey 2) +ez (x, 2)) in Chan et al. 2022).
Neural task fields. For each task, we use an additional weight network n,, implemented as a small MLP, to estimate both a density value and task-specific vector, where this element pair can be denoted as a neural task field for the aggregated 3D representation. We are then able to render these quantities via neural volume rendering Max (1995); Mildenhall et (2020) through a differentiable renderer r to obtain predictions for each task. In particular, for the tasks including semantic segmentation, part segmentation, surface normal estimation, boundary detection, saliency prediction, we estimate prediction for each point of a given ray (e.g. logits for segmentation) and integrate them the ray. We normalize the predictions after rendering for surface normal and apply softmax after rendering for segmentation tasks. For depth estimation task, we use the raw prediction depth maps.
4

---

Published as a conference paper at ICLR 2024
The sequence of mappings can be summarized as: firstly mapping the shared feature encoding fI) to tri tri-plane features through e, further mapping it to neural task fields through n,, finally rendering these to obtain predictions for task t, i.e. 9t f(I) where 9: ny is the regularizer for task t.
Discussion. While novel view synthesis methods such as NeRF require the presence of multiple views and knowledge of the camera matrices, here assume single view extract the corresponding 3D representations and to render them as task predictions. For rendering, we assume that the camera orthogonal to image center here, and depict not the viewpoint as input. In the experiments, we show that our model consistently improves the MTL performance, even when learned from single view per scene, thanks the 3D structure of representations imposed by our regularizer.
function that takes only the output of n. but
Optimization. We measure the mismatch between ground labels and the predictions obtained from our 3D-aware model branch, and use this signal to jointly optimize the model along with the original common task losses found in Eq. (T:
Σ Σ fe(h (()), + an Li o (I)) (LY)ED
min N
3D-aware regularizer
where is hyperparameter balancing loss terms.
Cross-view consistency. Though our 3D-aware regularizer does not require multiple views of the same scene to be presented, it can be easily extended to penalize the cross-view inconsistency on the predictions when multiple views of the same scene are available, e.g. video frames. Given two views of scene I and I' with their camera respective viewpoints V and V", we compute predictions for I' but by using I as the input and render it by using the relative camera transformation A V from V to V'. Then we further penalize the inconsistency between this prediction and ground-truth labels of I':
min S.
1
C.(he o I(T), vt) + an Lelg. o J(T). v.) +o Lelot I(T), (T (Y)'")E
Σ (U.Y), MYYY
E
N
3D-aware regulatizer
cross-view regularizer
(3)
where at and af are hyperparameters balancing loss terms. We set a, af. Note that in this case AV is a function of AV, as the relative viewpoint AV is used by the renderer
4
EXPERIMENTS
Here we first describe the benchmarks used and our implementation details, then present quantitative and qualitative analysis of our method.
4.1 DATASET
It contains 1449 RGB-D images, sampled from video sequences from a variety of indoor scenes, which we use to perform four tasks; namely 40-class semantic segmentation, depth estimation, surface normal estimation and boundary detection in common with prior work (Ye & Xu 2022 Bruggemann et al. (202I). Following the previous studies, we use the true depth data recorded by the Microsoft Kinect and surface normals provided in the prior work (Eigen) & Fergus 2015) for depth estimation and surface normal estimation tasks.
NYUv2 Silberman et al. 2012):
NYUv2 video frames: In addition to the standard data split, NYUv2 (Silberman et al.) 2012) also provides additional video frames" which are labeled only for depth estimation. Only for the cross-view consistent regularization experiments, we merge the original split with video frames, and train MTL models by minimizing loss available labeled tasks, i.e. all four tasks the original data and only the depth on video frames. To estimate the relative camera pose AV between the frames, we use COLMAP (Schönberger & Frahm 2016 Schönberger al. 2016).
https:
5

---

Published as a conference paper at ICLR 2024
PASCAL-Context (Chen et al. 2014): PASCAL (Everingham et al., 2010 is commonly used image benchmark for dense prediction tasks. We use the data splits from PASCAL-Contex (Chen et al. 2014) which has annotations for semantic segmentation, human part segmentation and semantic edge detection. Additionaly, following 4
(Vandenhende et al.) 2021] Ye & Xu 2022), we also consider surface normal prediction and saliency detection using the annotations provided by Vandenhende
et al. (2021).
4.2 IMPLEMENTATION DETAILS
Our regularizer is architecture agnostic and can be applied to different architectures. In our exper- iments, it is incorporated into two state-of-the-art (SotA) MTL methods; MTI-Net (Vandenhende)
et al. 2020b) and InvPT YYe & Xu 2022) which builds the convolutional neural network (CNN),
HR (Wang et al.) 2020) and transformer based ViT-L (Dosovitskiy et al. (2020) respectively. In
all experiments,
follow identical training, evaluation protocols (Ye & Xu 2022). We append our
3D-aware regularizer to these two models using two convolutional layers, followed by BatchNorm and ReLUU, to project feature maps the tri-plane space resulting in common size and channel width (64). A 2-layer MLP is used to render each task as in Chan et al. (2022). We use identical nyper-parameters; learning rate, batch size, loss weights, loss functions, pre-trained weights, opti- mizer, evaluation metrics as MTI-Net and InvP',, respectively. We jointly optimize the task task-specific losses and losses arising from our 3D regularization. During inference, the regularizer is discarded. We refer to the supplementary material for further details.
4.3 RESULTS
Comparison with SotA methods. We compare our method with the SotA MTL methods on
NYUv2 and PASCAL-Conted datasets and report results in Tab. and Tab. 2 respectively. Follow- ing Bruggemann al. (202T), we use HRNet-48 (Wang et al.)
2020) as backbone when comparing to
CNN based methods;
(Misra et al. 2016), PAP
ZZhang et al. 2019), PSD (Zhou et al.
2020 PAD-Net (Xu et al. 2018), ATRC (Bruggemann et
2021) MTI-Net (Vandenhende et al.
2020b We use VIT-L (Dosovitskiy et al. 2020) as backbone when comparing to InvPT (Ye & Xu
2022
In NYUv2 (see Tab. I. when using HRNet-48 as backbone, we observe that ATRC (Bruggemann jt al, 2021) and MTI-Net (Vandenhende et al. 2020b) obtain the best performance. By incorporating our method to MTI-Net /Vandenhende et al. 2020b), we improve its performance on all tasks and outperform all CNN based MTL methods. comparison, the InvPT approach (Ye & Xu 2022)
achieves superior MTL performance by leveraging both the ViT-L (Dosovitskiy et al. (2200) backbone and multi-scale cross-task interaction modules. Our method is also able to quantitatively improve upon the base InvPT by integrating our proposed 3D-aware regularizer, e.g. +1.31 mloU on Seg. The results evidence that the geometric information is beneficial for jointly learning multiple dense prediction tasks.
Backioue
(aloo))
Dopti (RMSE)
Nomal (mEr)
Bounday (odaF)
Croes-Sric Mtl
2015
HRNGs448
20.7
76.43
HRNis-48
0.6700
20855
HRNer
0,3333
20.15
77.94
MTT-Nv Vindenberd
2000
HRNe-44
45.97
0.5965
20
77 86
Curs
HRN
46.67
IL5210
19.93
76.10
In Yo & X022 2022
VIL.L
$3.56 5487
0.5153 0.5000
19.04
78.10
Cues
WIT-
1855
78.30
Table 1: Quantitative comparison of our method to the SotA methods; NYUv2 dataset.
Tab. depicts experimental results on the PASCAL-Context dataset where previous method results are reproduced from Ye & Xu (2022). We also report results from our implementation of the MTI-Net, denoted by 'MTI-Net**, where we found that our implementation obtains better performance. We
observe that the performance of existing methods is better than in the previous NYUv2 experi- ment (Tab. D. as PASCAL-Context has significantly more images available for training. From Tab. 2 observe that method, incorporating our proposed regularizer MTI-Net (Vandenhende et al. 2020b), can improve the performance on all tasks with respect to our base MTI-Net implementation, e.g. +2.29 mloU Seg, and obtains the best performance most tasks compared with MTL.

---

Published as a conference paper at ICLR 2024
methods that use the HRNett-48 backbone. As in NYUv2, the InvPT model (Ye & Xu 2022) achieves better performance on a majority of tasks over existing methods. Our method with InvPT again obtains improvements Boundary. This result further suggests that our method is effective for enabling the MTL network to learn beneficial geometric cues and that the technique can be incorporated with various MTL methods for comprehensive task performance improvements.
all tasks over InvPT, e.g. +1.51 mloU on PartSeg and +1.00 odsF on
Method
Backbone
Seg. (mloU))
PunSeg (mlol)) |
Sal (maxF)
Normal (mLir)
Boundary (odsf)
ASTMT
Mananss et al. 2019
HRNee-45
68.00
61.10
65,70
14.70
72.40
PAD-NG Nureeaa bnih)
HRNer-48
$3.60
52.60
65.80
15.30
72.50
MTT Net
Vardehhnnk v al 2000
IRNNa-48
61.70
60.18
$4.78
14.23
70.50
AIRC
B
Huggemans
HRNer-4s
62.99
59.42
$4,70
14,20
70.96
MIL.Net
Vandenlends
2026
HRNer-as
64.42
64.17
34.56
13822
74,30
Ours
HRNe1-48
66.71
65.20
84.59
1371
74.50
InxPT Yo & Xn 2022
WTHL WFL
79.03
67.61
84.81
14.15
73,00
Ours
69.12
84.94
74,00
Table 2: Quantitative comparison of our method to the SotA methods; PASCAL-Context dataset.
3D regularizer with multiple views. Here we investigate whether learning stronger 3D consistency across multiple views with our regularizer further improves the performance in multiple tasks. To this end, we merge the NYUv2 dataset with the additional video frames possessing only depth annotation and train the base InvPT, our method and our method with cross-view consistency on the merged data. For InvPT, we train the model by minimizing losses over the labeled tasks, We train our method by minimizing both the supervised losses and the 3D-aware regularization loss. We further include the ciew consistency loss as in Eq. (3)
Method
Seg. (mloU) ?
Depth (RMSE)
Normal (mErr) |
Boundary (odsF)
InvPT (ee & Xu 2022
53.44
0.4927
18.78
77.90
Ours
54,93
0.4879
18.47
77.90
Ours cross-view consistency
54.99
0,4850
Table 3: Quantitative comparison of our method on NYUv2 dataset + extra video frames with multiple views.
Results of the three approaches are reported in Tab. 3 Compared with the results in Tab. | we can see that including video frames for training improves the performance of InvPT on depth and surface normal tasks while yielding comparable performance on remaining tasks. We also see that our method obtains consistent improvement over the InvPT on four tasks with applying 3D-aware regularization using only a single view. Adding the cross-view consistency loss term to our method, we can observe further performance improvement beyond using only single view samples. This suggests that better 3D) geometry learning through multi-view consistency is beneficial, however, the improvements are modest. We argue that coarse 3D scene information obtained from single views can be sufficient to learn more structured and regulate inter-task relations.
We also note that this experimental setting also related to the recent MTL work (Li et al. 2022) that can learn from partially annotated data by exploiting cross-task relations. However we here focus on an orthogonal direction and believe our complementary works have scope to be integrated together. We leave this promising direction for future work.
Comparison with auxiliary network heads. Prior work suggests that the addition of auxiliary heads performing the same task with identical head architectures yet with different weight initializa- tions can be further helpful to performance (Meyerson & Miikkulainen 2018). To verify whether the improvements obtained by our regularizer not due to the additional heads solely but introduced 3D structure, conduct comparison with our bascline and report results in Tab. 4 The results show that adding auxiliary heads ('InvPT+ + Aux. Heads") does not necessarily lead to better performance on all tasks; e.g. Seg, whereas our method can be seen to outperform this baseline on all tasks suggesting the benefit of introducing 3D-aware structure across tasks.
Method
Seg. (mlol)) 1
Depih (RMSE) I
Normal (mErr)
Boundary (oddF) 1
In Yo & Xu. 2022
$3.56
0.5183
19.04
78.10
Henus
0.5131
Ours
54.87
0.5006
18.55
78.30
Table 4: Quantitative comparison of our method to the baseline of adding auxiliary heads to InvPT; NYUv2 dataset.
7

---

Published as a conference paper at ICLR 2024
3D-aware regularizer predictions. Though we discard the regularizer during inference, the regu- larizer can also be used to produce predictions for the tasks. To investigate their estimation utility, we report task performance using the default task specific heads hy. the regularizer output (regularizer) and finally using the averaged predictions over two in Tab. [ We observe that the regularizer alone estimations worse than the task-specific heads, however, the performance of their averaged output yields marginal improvements to the boundary detection task. The lower performance of using the regularizer alone may be explained by the fact that the rendering image size is typically small (e.g. we render 56x 72 images for NYUv2). The addition of a super-resolution module, similar to previous work (Chan al. 2022). can further improve the quality of the related predictions. We leave this to future work.
outputs
Sog. (mnolU)
Depth (RMSE) | Normal (mErr)
Boundary (odsF) 1
task- specific heads segularizer
0.5006
$1.79
0.5282
18,90
74,80
avg
54.68
0.5062
18:70
78.50
Table 5: Quantitative results of the predictions from the task-specific heads, regularizer or the average of both the tpec-fpe heads and regularizer in our method; NYUv2 dataset.
Tasks for 3D-aware regularizer. Our regularizer renders predictions for all learning tasks by default. We further study the effect of isolating different tasks for rendering with the regularizer in Tab. [6 Specificallyy we jointly optimize the MTL network with regularizer that renders only one individual task predictions. From Tab. 6 we observe that rendering different individual tasks in the regularizer leads to only marginally differing results and yet using all tasks for rendering help to better learn the geometric information for MTL, i.e. 'All tasks' obtains the best performance on the majority of tasks.
render tasks
Neg. [mlol]]
Depth (RMSF) J
Nontal (mFFr))
Boundary (odsF) 1
Seg.
55.04
0,5038
18.78
77.90
Depth
54
0.5044
18.93
77.50
Edge
53.97
0,5022
18.97
77.50
All tasks
51.87
0.5006
18.55
78.30
Table 6: Quantitative results of our method isolating different tasks for rendering with the regularizer;
NYUv2 dataset.
Using less data. We further investigate the performance gain obtained by our method when trained with fewer training samples. To this end, we train the baseline InvPT (Ye & Xu 2022) and our method 25% and 50% of the NYUv2 data after randomly subsampling the original training set. The results are reported in Tab. 7 As expected, more training samples result in better performance in all cases. Our method consistently outperforms the baseline on all tasks in all label regimes with higher margins when more data is available. As the full NYUv2 training set is relatively small, contains only 795 images, our regularizer learns better 3D consistency across tasks from more data too, hence resulting enhanced task performance.
Ensages
Method
Seg. (mboo))
Depth (RMSE)
Nanual (mEn)
Beuralayy (odsF)
795 (100%)
ImPT
Ne 2022
$3.56
0,5185
0.500
18.55
1904
78.10
ImPT
& 2002
0,5741
20.60
24.90
0.565h
198 (25%)
ImP[ Ours
Ne X2
2002
44.79
43.83
0,5972
21.57
74800
24.80
Table 7: Quantitative comparison of the baseline InvPT and our method in the NYUv2 dataset for varying training set sizes.
4.4 QUALITATIVE RESULTS
We visualize the task predictions for both our method and the base InvPT method NYUv2 sample in 3 Our method can be observed to estimate better predictions consistently for four tasks. For example, our method estimates more accurate predictions around the boundary of the refrigerator, stove and less noisy predictions within objects like curtain and stove. The geometric information learned in method helps distinguish different adjacent objects, avoids noisy predictions within
8

---

Published as a conference paper at ICLR 2024
SKy mID 28-16
Deat Frbe 5421
Narraleder: 17.73
Barader IC' kes 0163
Image
Sog 26.79
Eepin FMSE 2013
Norrllntrr: 10.4
Doundey 9C4 los 10046
Reguis riser
mr
Say HMM 30.50
Dopin FobE 2000
Norrulnthn 17.40
Boankey BC2 fox 3o00
mg
Our
Sep:
Soandayy
GroundTruth
my
Figure 3: Qualitative results on NYUv2. Each column shows the image predictions and per- formance for each task. The last row shows the ground-truth of four tasks. The first to the third row shows the predictions of InvPT, the regularizer in method and tpsk-specific decoders of our method, respectively.
object boundaries and also improves the consistency across tasks as in the regularizer, all tasks predictions are rendered based on the same density.
We then visualize the predictions of method's regularizer and the task-specific decoder NYUv2 in Fig. As shown in the figure, our regularizer can also render high quality predictions for different tasks it was observed obtain worse quantitative performance than the task specifc decoders. As discussed, this is due to the rendering image size being usually small (e.g. we render 56x 72 images for NYUv2).
5 CONCLUSION AND LIMITATIONS
We demonstrate that encouraging 3D-aware interfaces between different related tasks including depth estimation, semantic segmentation and surface normal estimation consistently improves the multi- task performance when incorporated the recent MTL techniques in two standard dense prediction benchmarks. Our model can be successfully used with different backbone architectures and does bring any additional inference costs. Our method has limitations too. Despite the efficient 3D modeling through the triplane encodings, representing 3D representations for higher resolution 3D volumes still expensive in terms of memory or computational cost. Rendering specular objects will require different rendering or objects with high frequency 3D details may require more accurate modeling. Furthermore, we balance loss functions with fixed cross-validated hy perparameetrss, while it would be more beneficial to adaptive loss balancing strategies (Kendall et al. (0088) discarding conflicting gradients (Liu et al., 2021a)). Finally, the cross-view consistency experiments where only portion of images are labeled for all the tasks, method does not make of semi-supervised learning or view-consistency for the tasks with missing labels which can further improve the performance of model.
Acknowledgement. HB is supported by the EPSRC programme grant Visual AI EP/T028572/1. We thank Octave Mariotti, Changjian Li, and Titas Anciukevicius for their valuable feedback.

---

Published as a conference paper at ICLR 2024
REFERENCES
Deblina Bhattacharjec, Sabine Süsstrunk, and Mathieu Salzmann. Vision transformer adapters for generalizable multitask learning. arXiv preprint arXiv:2308, /2372, 2023.
Hakan Bilen and Andrea Vedaldi. Integrated perception with recurrent multi-task neural networks. In Advances in Neural Information Processing Systems, pp. 235-243, 2016.
Felix JS Bragman, Ryutaro Tamno, Sebastien Ourselin, Daniel C Alexander, and Jorge Cardoso. Stochastic filter groups for multi-task cnns: Learning specialist and generalist convolution kernels. In ICCV, pp. 1385-1394, 2019.
David Bruggemann, Menelaos Kanakis, Stamatios Georgoulis, and Luc Van Gool. Automated search for resource-efficient branched multi-task networks. arXiv preprint arXiv:2008. 10292, 2020.
David Bruggemann, Menelaos Kanakis, Anton Obukhov. Stamatios Georgoulis, and Luc Van Gool. Exploring relational context for multi-task dense prediction. ICCV, 2021.
Shengqu Cai, Anton Obukhov, Dengxin Dai, and Luc Van Gool. Pix2nerf: Unsupervised conditional p-gan for single image to neural radiance fields translation. In CVPR, pp. 3981-3990, 2022.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In CVPR, pp. 5799-5809, 2021.
Eric R Chan, Connor Z Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-awant 3d generative adversarial networks. In CVPR, 2022.
Eric R Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. Generative novel view synthesis with 3d-aware diffusion models. In CVPR, 2023.
Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. Detect what you can: Detecting and representing objects using holistic models and body parts. In CVPR, pp. 1971-1978, 2014.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In ICML, pp. 794 803.
PMLR, 2018.
Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. NeurIPS, 2020.
Zitian Chen, Yikang Shen, Mingyu Ding, Zhenfang Chen, Hengshuang Zhao, Erik G Learned-Millee and Chuang Gan. Mod-squad: Designing mixtures of experts modular multi-task learners. In CVPR, pp. 11828-11837, 2023.
Sumanth Chennupati, Ganesh Sistu, Senthil Yogamani, and Samir A Rawashdeh. Multinet+4: Multi- stream feature aggregation and geometric loss strategy for multi-task learning. In CVPR Workshop, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 11929, 2020.
David Eigen and Rob Fergus. Predicting depth. surface normals and semantic labels with common multi-scale convolutional architecture. In IEEE International Conference on Computer Vision, pp. 2650-2658, 2015.
10

---

Published as a conference paper at ICLR 2024
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJV,, (8(2):303-3333 2010.
Zhiwen Fan, Rishov Sarkar, Ziyu Jiang, Tianlong Chen, Kai Zou, Yu Cheng, Cong Hao, Zhangyang Wang, et al. vit: Mixture-of-expers vision transformer for efficient multi-task learning with model-accelerator co-design. Neurips, 35:28441-28457, 2022.
Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: A syle-based 3d-aware generator for high-resolution image synthesis. arXiv preprint arXiv:2110.08985, 2021.
Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. Dynamic task prioritiza-
tion for multitask learning. In ECCV, pp. 270-287, 2018.
Pengsheng Guo, Chen-Yu Yu Lee, and Daniel Ulbricht. Learning to branch for multi-task learning. In ICML, pp. 3854-3863 PMLR, 2020.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In CVPR, pp. 7482-7491, 2018.
lasonas Kokkinos. Ubernet: Training a universal convolutional neural network for low-, mid- and high-level vision using diverse datasets and limited memory. In CVPR, pp. 1138, 2017.
Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas J Guibas,
Andrea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser. Panoptic neural fields: A semantic object-aware neural scene representation. CVPR, pp. 12871-12881, 2022.
Wei-Hong Li and Hakan Bilen. Knowledge distillation for multi-task learning. ECCV Workshop on Imbalance Problems in Computer Vision, pp. 163-176. Springer. 2020.
Wei-Hong Li, Xialei Liu, and Hakan Bilen. Learning multiple dense prediction tasks from partially annotated data. In CVPR, 2022.
Wei-Hong Li, Xialci Liu, and Hakan Bilen. Universal representations: A unified look at multiple task and domain learning. I/CV, pp. 1-25, 2023.
Jason Liang, Elliot Meyerson, and Risto Mikkulainen. Evolutionary architecture search for deep multitask networks. In Proceedingy of the Genetic and Evolutionary Computation Conference, pp. 466-473, 2018.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung-Yi Yi Lin. Magic3d: High-resolution text-to-3d content creation. In CVPR, pp. 300-309, 2023.
Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong, Pareto multi-task learning. NeurlPS, 32: 101222000-12777 2019.
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. NeurlPS, 2021a.
Liyang Liu, Yi Li, Zhanghui Kuang, Jing- Hao Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Towards impartial multi-task learning. In ICLR, 2021b.
Shikun Liu, Edward Johns, and Andrew Davison. End-to-end multi-task learning with attention. In
CVPR, pp. 1871-1888 2019.
Yajing Liu, Yuning Lu, Hao Liu, Yaozu An, Zhuoran Xu, Zhuokun Yao, Baofeng Zhang, Zhiwei Xiong, and Chenguang Gui. Hierarchical prompt learning for multi-task learning. In CVPR, pp. 10888-10898, 2023.
Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. ACM Trans. Graph., 2019.
Maninis, Ilija Radosavovic, and Iasonas Kokkinos, Attentive single-tasking of multiple tasks. In CVPR, pp. 1851-1860, 2019.
11

---

Published as a conference paper at ICLR 2024
Nelson Max. Optical models for direct volume rendering. IEEE Transactions Visualization and Computer Graphics, 1(2):99-108, 1995.
Moustafa Meshry, Dan B Goldman, Sameh Khamis, Hugues Hoppe, Rohit Pandey, Noah Snavely, and Ricardo Martin-Brualla, Neural rerendering in the wild. CVPR, pp. 6878-6887, 2019.
Elliot Meyerson and Risto Mikkkulanen. Pseudo baeudo-task augmentation: From deep multitask learning to intratask sharing-and back. ICML, pp. 3511-3520. PMLR, 2018.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In CVPR, pp. 3994-4003, 2016.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In CVPR, pp. 3504-3511 3515, 2020.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05009, 2017.
Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. Latent multi-task architecture learning. AAAI, volume 33, pp. 4822-4829, 2019.
Shunsuke Saito, Zeng Huang. Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In ICCV, pp. 2304-2314, 14, 2019.
Johannes Lutz Schönberger and Jan-Michael Frahm. Structure- cootiee-frommmtiinin motion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm. Pixelwise view selection for unstructured multi-view stereo. In European Conference on Computer Vision (ECCV), 2016.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. NewrIPS, 2018.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In European conference on computer vision, pp. 746-760. Springer, 2012.
Vincent Sitzmann, Justus Thics, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer. Deepvoxels: Learning persistent 3d feature embeddings. In CVPR, pp. 2437-2446, 2019.
Ivan Skorokhodov, Sergey Tulyakov, Yiqun Wang, and Peter Wonka. Epigraf: Rethinking training of 3d gans. Neurips, 2124447-245011 2022.
Gjorgji Strezoski, Nanne van Noord, and Marcel Worring. Many task learning with task routing. In ICCV, pp. 1375-1384, 2019.
Mihai Suteu and Yike Guo. Regularizing deep multi-task networks using orthogonal gradients. arXiv preprint ArXiv:/9/2.066444 2019.
Justus Thies, Michael Zollhöfer, and Matthias Nießner. Deferred neural rendering: Image synthesis using neural textures. TOG, 38(4):1-12, 2019.
Simon Vandenhende, Stamatios Georgoulis, Bert De Brabandere, and Luc Van Gool. Branched multi-task networks: deciding what layers to share. In bmve. 2020a.
Simon Vandenhende, Stamatios Georgoulis, and Lue Van Gool. Mti-net: Multi-scale task interaction networks for multi-task learning. In ECCV, pp. 527-543. Springer, 2020b.
12

---

Published as a conference paper at ICLR 2024
Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Mare Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A survey. PAMI, 2021.
Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, al. Deep high-resolution representation learning for visual recognition. TPAMI, 43(10))3349-336, 2020.
Dan Xu, Wanli Ouyang, Xiaogang Wang, and Nicu Sebe. Pad-net: Multi-tasks guided prediction-and- distillation network for simultaneous depth estimation and scene parsing. In CVPR, pp. 675 684, 2018.
Jiawei Yao, Chuming Li, Keqiang Sun, Yingjie Cai, Hao Li, Wanli Ouyang, and Hongsheng Li. Ndc-scene: Boost monocular 3d semantic scene completion in normalized device coordinates space. In ICCV, pp. 9455-9999 1555-9465,40 2023.
Hanrong Ye and Dan Xu. Inverted pyramid multi-task transformer for dense scene understanding. In ECCV, pp. 514-530. Springer, 2022.
Hanrong Ye and Dan Xu. Taskprompter: Spatial-channel multi-task prompting for dense scene understanding. In ICLR, 2023a.
Hanrong Ye and Dan Xu. Taskexpert: Dynamically assembling multi-task representations with In ICCV, pp. 21828 228837,,,, 2023b.
memorial mixture-of
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one or few images. In CVPR, pp. 4578-4587, 2021.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. NeurIPS, 2020.
Jingbo Zhang. Xiaoyu Li, Ziyu Wan, Can Wang, and Jing Liao. Text2nerf: Text-driven scene generation with neural radiance fields. arXiv preprint arXiv:2305. 11588, 2023.
Yu Zhang and Qiang Yang. A survey on multi-task learning. arXiv preprinl
14, 2017.
Zhenyu Zhang, Zhen Cui, Chunyan Xu, Zequn Jie, Xiang Li, and Jian Yang. Joint task-recursive learning for semantic segmentation and depth estimation. In ECCV, pp. 235-251, 2018.
Zhenyu Zhang, Zhen Cui, Chunyan Xu, Yan Yan, Nicu Sebe, and Jian Yang. Pattem-affinitive propagation across depth, surface normal and semantic segmentation. CVPR, pp. 4106-4115, 2019.
Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew Davison. In-place scene labelling and understanding with implicit scene representation. In ICCV, pp. 15838-15847, 2021.
Ling Zhou, Zhen Cui, Chunyan Xu, Zhenyu Zhang, Chaoqun Wang, Tong Zhang, and Jian Yang. Pattern-strun structure diffusion for multi-task learning. CVPR, pp. 4523,4523, 2020.
13

---

Published as a conference paper at ICLR 2024
A IMPLEMENTATION DETAILS
We implement our approach in conjunction with state-of-the-art multi-task learning methods; MTI- Net (Vandenhende et al. 2020b) and InvPT (Ye & Xu 2022) while following identical training, evaluation protocols (Ye & Xu 2022). We use HRNet-48 Wang et al. 2020) and ViT-L (Dosovisskiy al. 2020) serve as shared encoders and append our 3D-aware regularizer to MTI-Net and InvPT using two convolutional layers, followed by BatchNorm, ReLU, and dropout layer with dropout rate of 0.15 to transform feature maps the tri- tri-plane dimensionality. resulting in a common size and channel width (64). A 2-layer MIP with 64 hidden units as in Chan et al. (2022) and a LeakyRel.U non-linearity with the negative slope of -0.2 as in |Skorokhodov et al. (2022). used to render each learning rate, batch size, loss
task as in Chan et al. (002)). We use identical weights, loss functions, pre-trained weights, optimizer, evaluation metrics as MTI-Net and InvPT, respectively. We jointly optimize task-specific losses and losses arising from our 3D regularization. During inference, the regularizer is discarded. We use the same task-specific loss weights as in Ye & Xu (2022). We train all models for 40K iterations with batch size of 6 for experiments of using InvPT as in Ye & Xul(2022) and a batch size of for experiments of using MTT--Net as in (Vandenhende al. (2020b). We ramp up the or from 0 to 4 linearly in 20K iterations and keep a₁ 4 for the rest 20K iterations. In the regularizer, we assume that the camera is orthogonal to image center, and depict r as a function that takes only the output of n, but not the viewpoint as input. In a 3D coordinates (z, y, z), the x and y coordinates are aligned with pixel locations and is the depth value. We further use two-pass importance sampling as in NeRF (Mildenhall et al., 2020p. For the majority of the experiments in the manuscript, we use 128 total depth samples per ray. We render 56x 72 predictions for NYUv2 and 64x64 for PASCAL-Context and resize the predictions via bilinear interpolation to the groundtruth resolution. Our code and models will be made public based upon acceptance.
B TRAINING COST ANALYSIS
Method
Time
Memory
Params.
FLOPS
MTI-Net (Vandenhende et al. 2020b
1.000
1.000
1.000
1.000
Ours
1.489
1.638
1.005
1.263
InyPT (Ye & Xu 2022
1.000
1.000
1.000
1.000
Ours
1.318
1.397
1.016
1.114
Table 8: Training Cost Comparisons to MTI-Net and InvPT; NYUv2 dataset. Note that our method has no additional inference cost as the regularizer is discarded during testing.
C RESULTS OVER MULTIPLE RUNS
Here, we report the results of our method over 3 runs on NYUv2 and PASCAL-Context and report the results in Tabs. g and 10 From the results, we can see that our method is stable (i.e. the std very small on each task) and improves over the baseline consistently on all tasks.
Method
Seg. (mlotT) 1 Depth (RMSE)
Normal (mFrr)
Boundary (odsF)
IovPT Yo & Xn 2022
$3.56
0,3183
19.04
78.10
54.86
0.5000
0.0010
18.49
78.17 0.09
Table 9: Quantitative comparison our method to the InvPT over 3 runs; NYUv2 dataset.
Method
Seg. (mloU)) f
PartSeg (mloU) t
Sal (maxP)
Normal (mEr)) I
Boundary (odsF)
InvPT Ours
79.03
67.61
84.81
14.15
79.92 0,32
69.08 + 0.15
84,85 0.06
13.70 0.14
73,83 + 0.17
Table 10: Quantitative comparison of our method to the InvPT over 3 runs; PASCAL-Context dataset.
Here, we analyze memory and computational cost during training for tackling four tasks in NYUv2 and report them in Tab. As shown in Tab, 8, our method that incorporates the regularizer the MTL baseline slightly increases the number of parameters (Ours vs InvPT: 1.016 vs 1) and FLOPS (Ours InvPT: 1.114 1) during training, training time (Ours vs InvPT: 1.318 vs 1), and training memory
14

---

Published as a conference paper at ICLR 2024
(Ours vs InvPT: 1.397 vs 1). We highlight that there is *NO additional cost during inference, since the regularizer will be discarded during inference.
D COMPARISONS WITH MORE RECENT SOTA
Method
(mloU)
Depth (RMSE)
Normal (mEr)) 1
Boundary (odsF)
TaskPromper (Ye 2023g
$5.30
0.5152
TaskExpert & All 202.0
$5.35
0.5157
18:54
78.40
IosPT (Yo Xo 2022
53.56 54.87
0,5183
19.04
78.10
0.5006
78.30
Table 11: Quantitative comparison of our method to more SotA methods; NYUv2 dataset.
Method
Seg. (iloUU)
PartSog (inloU) f
Sal (maxF)
Normal (mEr))
Boundary (odsF)
TaskPrompter Y: & Xu 20233
80.89
68.3 09.89
84.83
13.72
73.50
TaskExpent (Yo & Xu 20236
80.64
69.42
84.87
13.56
73.30
InvPT (oo 2022
67.61
84.81
14.15
73.00
Ours
79.53
69.12
84.94
13.53
74.00
Table 12: Quantitative comparison of our method to the SotA methods; PASCAL-Contex dataset.
We include the comparisons of our method incorporated with InvPT to more recent satte-of-the-asd methods, including TaskPrompter YYe & Xu| (023a) and TaskExpert Ye & Xu 2023b) and report the results in Tabs. I1 and T2 Methods from Liu et al. (2023) and[Chen et al. (2023) are not compared as they did not reported results NYUv2 and PASCAL benchmarks with the same backbone. Note that TaskExpert (Ye et al., 2023b) is published after we submitting the manuscript. From the results shown in Tab. []] we can see that, our method incorporated with InvPT achieves much better result on Depth while comparable results on the rest of tasks in NYUv2 compared with TaskPrompter and TastExpert. In PASCAL benchmark, from Tab. [2 we can see that our method obtains much better results saliency, surface normal and boundary estimation while obtaining comparable result on Human part segmentation and slightly worse on semantic segmentation. TaskPrompter adds learnable prompts for refining the feautures and the TaskExpert emsembles task-specific features from multiple task-specific experts for final task predictions and they all increase the capacity of the network to achieve better results. Also, they potentially be complementary to our method and we believe incorporating our method with them can further improve the performance in multi-task learning by regulating the shared features be 3D-aware with no additional cost during inference.
E DISCUSSION
Camera parameters. In our paper, the 3D coordinates and strategy of projecting the 3D coordinates onto the feature planes are similar to the ones in PiFU (Saito al., 2019) and (Yao et al. (003).. The feature planes are generated by the feature encoder and it pixel- wise feature map instead of
global pooled feature vector. The x and y coordinates are aligned with pixel locations and = the depth value. We follow Chan et al. (2022) that projects the coordinates (z, V, 2) onto three planes Exy: Eye, Cuz, retrieving the features via bilinear interpolation, and aggregates features from three planes instead of taking the 2D features and the values as representations in PiFU (Saito al.) 2019) or dividing the dimension of the feature map channel into D groups (D is the number of depth bins) in (Yao al.) 2023). So method has similar property as in PiFU (Saito et al. 2019 and (Yao et al. (023) and does not overfit to the camera parameters.
Also, as first feed the image into the feature encoder, which should be scaling the 3D coordinates accordingly and the coordinates will not be absolute but at the right scale for rendering. After training, the 3D-aware regularizer is discarded and we only use the multi-task learning branch for generating predictions for different tasks. We also visualize multiple images' predictions of the regularizers on PASCAL in Fig. The PASCAL dataset consists of annotated consumer photographs collected from the flickr shoto-sharing web-site, taken by various cameras with different intrinsics. From Fig. 4 we can see that the regularizer can render good quality predictions for all tasks on all images which also indicates that it does not overfit to the camera parameters.
Limitations and future work. Despite the efficient 3D modeling through the triplane encodings, representing 3D representations for higher resolution 3D volumes is still expensive in terms of
15

---

Published as a conference paper at ICLR 2024
Nelbyy
Figure 4: Qualitative results on PASCAL. Each column shows the image or predictions of our method's regularizer branch or the groundtruth for each task, respectively.
memory or computational cost. Some common efficient sampling strategies such as random sampling and pixel binning can be useful for reducing the cost. The tri-plane generated from the feature encoder can be relatively small resolution due to the feature downsampling and requires upsampling strategies for generating higher resolution feature planes for better rendering while it will inevitably increase the training cost. Additionally, rendering specular objects will require different rendering or objects with high frequency 3D details may require more accurate 3D modeling. Though our proposed method obtains performance gains consistently over multiple tasks, we balance loss functions with fixed cross-validated hyperparameters, while would be more beneficial to use adaptive loss balancing strategies (Kendall et al. 2018) discarding confiicting gradients (Liu et al. 2222aa). Finally, in the cross-view consistency experiments where only some the images are labeled for all the tasks, method does not make missing labels which can be further improve the performance of our model. We believe that more advanced techniques in 3D modeling can further improve our method for rendering higher resolution predictions with higher efficiency and better regulating the cross-task correlations and cross-view consistency.
of semi-supervised learning or view-consistency for the tasks with
16


Reviews:
[{"review_id": "dUHwQoWM9z", "reviewer": "Reviewer_aMPL", "summary": "This paper proposes to improve multi-task dense prediction by introducing a 3D representation branch during training. This 3D-aware branch(a triplane representation with volumetric render) shares the same backbone as a conventional multi-task learning setup. This NeRF branch encourages the feature extracted from the backbone to contain rich 3D information and to follow the physical imaging process. \n This 3D-aware head also enables supervising with multi-view consistency. Experiments show that this auxiliary 3D aware branch helps improve the performance of the conventional branch during inference when the 3D aware branch is dropped.", "questions": "My main concerns are (1) the potential camera overfitting problem, and (2) The extra training cost introduced by the NeRF head.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) Interesting idea, the auxilary 3D aware branch  forces the backbone to be truly 3D-aware. This method also improves the alignment between different prediction heads and the depth/density head as they will be rendered following the physical imaging process.\n\n(2) Using 3D aware representation as an auxiliary branch only for regularization also enables fast inference, which is a new and interesting idea to me.", "weaknesses": "(1) The 3D aware branch uses Triplane representation and volumetric render, which needs a specific camera model (intrinsic matrix). So it is suspicious that the model can somehow overfit to the specific camera parameters.  As a comparison, pixel-aligned scene representation (i.e., PiFU) can resolve this problem and it uses NDC representation.\nFor scenes with different camera parameters, the performance is unclear.\n\n(2) Although the tri-plane representation has significantly reduced the memory cost and rendering of NeRF. Rendering the whole image and calculating the gradient is still a heavy burden. Common practice includes random sampling or pixel binning.  In the supplement material, there is an incomplete explanation. It is important to clarify the NeRF sampling implementation and extra training cost(memory footprint and FLOPS)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FieGFJNYrT", "reviewer": "Reviewer_girT", "summary": "This paper proposed a structured 3D-aware regularizer that interfaces multiple tasks through the projection of features extracted from an image encoder to a shared 3D feature space and decodes them into their task output space through differentiable rendering. The experiment results are relatively good and promising.", "questions": "For example, \n1. how is the 3D quality of the learned nerf? \n2. Any geometric insights about the performance improvement? otherwise, it will just be like a regularization paper (1 point improvement, that's all).\n3. I think MTL might not be the best task to show the benefit of your method, do you have any ideas on that?\n\nOverall, I think this is a decent paper to accept, and I know the questions I made above might not be easy to answer. I will increase my score if the authors can offer some convincing and motivated answers.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Overall, I like this idea pretty much. For me, this work is like creating a new research line that makes most recognition methods 3D aware, which is dual to another research line that makes 2D image generation tasks 3D-aware. Although the introduced 3D-awared encoder does not explicitly give the 3D representation (I mean this method could not output decent 3D mesh), it gives way to investigating the 3D properties in most recognition tasks.", "weaknesses": "I paid a lot of expectations on the experiments after reading the Abstract and Introduction sections, but the experiments were not strong enough to match my expectations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xitnNgElI2", "reviewer": "Reviewer_boi3", "summary": "The paper tackles the problem of multi-task learning for dense tasks like depth prediction, semantic segmentation, normal estimation. Cross-task correlations are noisy and there is no 3D regularization. The paper proposes introducing a latent that is structured and 3D-aware (by outputting a K-planes representation) and uses projection with differentiable rendering and a task specific decoder. The 3D awareness combined with view dependent rendering can be used to both regularize the 3D structure and multi-view consistency on the same 3D latent. Results are shown in NYUv2 and PASCAL-Context datasets.", "questions": "1. Its unclear if the results are statistically significant compared to the baseline. Since the method can be applied on any baseline MTL architecture, it is unclear if the improvements (especially in Table 2) are just due to randomness of the SGD training dynamics or is an actual improvement. Results over 3-5 trials should be reported if the improvements are marginal/inconsistent.\n\n2. Table 3 shows that multiview consistency may not help in a MTL scenario considered in the paper. What is the significance of this section?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is quite well-written and easy to follow. The main idea of the paper is clear - to output a 3D feature representation from an image (using K-planes), which can then be used as part of a NeRF-like neural rendering pipeline to project the feature into a 2D feature image which can then be used by a task-specific decoder.", "weaknesses": "Predicting 3D representations (to use in neural rendering) from unlabeled 2D images and/or text is a relatively common idea [1, 2, 3, 4, 5]. The paper applies this idea in a multi-task setting, and adds a multi-view consistency for multi-view dataset. The results are not highly compelling compared to the baselines in the paper, which may internally learn some 3D structure too. So the explicit formulation of the 3D latent structure is not well motivated, unless the task requires some view editing or novel view synthesis, or an explicit 3D object representation, etc.\n\n____\n[1] Chan, Eric R., et al. \"pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.\n\n[2] Cai, Shengqu, et al. \"Pix2nerf: Unsupervised conditional p-gan for single image to neural radiance fields translation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n[3] Lin, Chen-Hsuan, et al. \"Magic3d: High-resolution text-to-3d content creation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[4] Zhang, Jingbo, et al. \"Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields.\" arXiv preprint arXiv:2305.11588 (2023). \n\n[5] Gu, Jiatao, et al. \"Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis.\" arXiv preprint arXiv:2110.08985 (2021).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zN6GMcz4kW", "reviewer": "Reviewer_4RSv", "summary": "The paper proposes a structured 3D-aware regularizer for multi-task learning (MTL) in computer vision. The regularizer interfaces multiple tasks by projecting features from an image encoder to a shared 3D feature space and decoding them into task output space through differentiable rendering.", "questions": "See weakness section for more details.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper provides clear explanations of the proposed method, the problem it addresses, and the evaluation process. \n\n* The paper's contributions have significant implications for multi-task learning in computer vision. The structured 3D-aware regularizer can be integrated into existing models, improving their performance and reducing noise in cross-task correlations.", "weaknesses": "* The paper could benefit from a more extensive analysis of the proposed method's computational efficiency. While it is mentioned that the regularizer does not introduce additional computational cost for inference, a more detailed analysis or comparison with other methods in terms of computational efficiency would strengthen the paper.\n\n* It would be beneficial to discuss the limitations or potential failure cases of the proposed 3D-aware regularizer. It would be valuable to address any potential drawbacks or scenarios where the regularizer may not be as effective.\n\n* The paper lacks a thorough comparison with existing state-of-the-art methods in multi-task learning. It would be beneficial to include comparisons with other regularization techniques or approaches that address the problem of noisy cross-task correlations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)