Universal Few-shot Learning of Dense Prediction Tasks with Visual Token Matching
Authors: Donggyun Kim
Jinwoo Kim
Seongwoong Cho
Chong Luo
Seunghoon Hong
Venue: ICLR 2023 notable top 5% (2023)
Keywords: few-shot learning
dense prediction tasks

Abstract: Dense prediction tasks are a fundamental class of problems in computer vision. As supervised methods suffer from high pixel-wise labeling cost, a few-shot learning solution that can learn any dense task from a few labeled images is desired. Yet, current few-shot learning methods target a restricted set of tasks such as semantic segmentation, presumably due to challenges in designing a general and unified model that is able to flexibly and efficiently adapt to arbitrary tasks of unseen semantics. We propose Visual Token Matching (VTM), a universal few-shot learner for arbitrary dense prediction tasks. It employs non-parametric matching on patch-level embedded tokens of images and labels that encapsulates all tasks. Also, VTM flexibly adapts to any task with a tiny amount of task-specific parameters that modulate the matching algorithm. We implement VTM as a powerful hierarchical encoder-decoder architecture involving ViT backbones where token matching is performed at multiple feature hierarchies. We experiment VTM on a challenging variant of Taskonomy dataset and observe that it robustly few-shot learns various unseen dense prediction tasks. Surprisingly, it is competitive with fully supervised baselines using only 10 labeled examples of novel tasks ($0.004\%$ of full supervision) and sometimes outperforms using $0.1\%$ of full supervision. Codes are available at https://github.com/GitGyun/visual_token_matching.

Published as a conference paper at ICLR 2023
UNIVERSAL FEW-SHOT LEARNING OF DENSE PREDIC- TION TASKS WITH VISUAL TOKEN MATCHING
Donggyun Kim², Jinwoo , Seongwoong Cho', Chong Luo", Seunghoon Hong!
1 School of Computing, KAIST
2 Microsoft Research Asia [kdgyun425, jinwoo-kim, seongwoongjo, Beanghoon.hongJekaisttt chong. www.wuicccooootccomm
ABSTRACT
Dense prediction tasks are a fundamental class of problems in computer vision. As supervised methods suffer from high pixel-wise labeling cost, few-shot learning solution that can learn any dense task from few labeled images is desired. Yet, current few-shot learning methods target restricted set of tasks such as semantic segmentation, presumably due to challenges in designing a general and unified model that is able to flexibly and efficiently adapt arbitrary tasks of unseen se- mantics. We propose Visual Token Matching (VTM), a universal few-shot learner for arbitrary dense prediction tasks. It employs non- non-parametric matching patch- level embedded tokens of images and labels that encapsulates all tasks. Also, VTM flexibly adapts any task with tiny amount of task-specific parameters that modulate the matching algorithm. We implement VTM as powerful hierar- chical encoder-decoder architecture involving ViT backbones where token match- ing is performed at multiple feature hicrarchies. We experiment VTM on a chal- lenging variant of Taskonomy dataset and observe that it robustly few-shot learns various unseen dense prediction tasks. Surprisingly, it is competitive with fully su- pervised baselines using only 10 labeled examples novel tasks (0.004% of full supervision) and sometimes outperforms using 0. of full supervision. Codes are available at httttt:://ggthhii www./iityynnvvvsaaaaaaaoooo token matching
INTRODUCTION
Dense prediction tasks constitute fundamental class of computer vision problems, where the goal is to learn a mapping from an input image to a pixel-wise annotated label. Examples include semantic segmentation, depth estimation, edge detection, and keypoint detection, to name few (Zamir al., 2018 Cai & Pu. 2019). While supervised methods achieved remarkable progress, they require sub- stantial amount manually annotated pixel-wise labels, leading to a massive and often prohibitive per-task labeling cost (Kang et al.) 2019 Liu et al. 2020 Ouali et al.) 2020). Prior work involving transfer and multi-task learning have made efforts to generally relieve the burden, but they often assume that relations between tasks are known in advance, and still require fairly large amount of labeled images of the task of interest (e.g., thousands) (Zamir et al. 2018 (ttnnleyy et al. 2020) O Pinheiro et al. 2020; Wang al. 2021). This motivates to seek few-shot learning solution that can universally learn arbitrary dense prediction tasks from a few (e.g., ten) labeled images.
However, existing few-shot learning methods for computer vision are specifically targeted to solve restricted set of tasks, such as classification, object detection, and semantic segmentation (Vinyals) et al. 2016 Kang et al.) 2019) Min et al. [202T]. As a result, they often exploit prior knowledge and assumptions specific to these tasks in designing model architecture and training procedure, therefore not suited for generalizing to arbitrary dense prediction tasks (Snell et al.) 2017, Fan et al.) 2022] Iqbal et al. 2022, Hong al., 2022). To our knowledge, no prior work in few-shot learning provided approaches to solve arbitrary dense prediction tasks in a universal manner.
We argue that designing universal few-shot learner for arbitrary dense prediction tasks must meet the following desiderata. First, the learner must have a unified architecture that can handle arbitrary tasks by design, and share most of the parameters across tasks that it can acquire generalizable knowledge for few-shot learning of arbitrary unseen tasks. Second, the learner should flexibly adapt its prediction mechanism solve diverse tasks of unseen semantics, while being efficient enough to prevent over- fitting. Designing such a learner is highly challenging, as it should be general and unified while being able flexibly adapt to any unseen task without over-fitting few examples.

---

Published as a conference paper at ICLR 2023
In this work, we propose Visual Token Matching (VTM), universal few-shot learner for arbitrary dense prediction tasks. We draw inspiration from the cognitive process of analogy making (Mitchell 202I]; given few examples of new task, humans can quickly understand how to relate imput and output based on a similarity between examples (i.e., assign similar outputs to similar inputs), while flexibly changing the notion of similarity to the given context. In VTM, we implement analogy- making for dense prediction as patch-level non non-parametric matching, where the model learns the similarity in image patches that captures the similarity in label patches. Given a few labeled exam- ples of novel task, it first adapts its similarity that describes the given examples well, then predicts the labels of an unseen image by combining the label patches of the examples based on image patch similarity. Despite the simplicity, the model has a unified architecture for arbitrary dense predic- tion tasks since the matching algorithm encapsulates all tasks and label structures (e.g, continuous or discrete) by nature. Also, we introduce only a small amount of task- parameters, which makes our model robust to over-fitting as well as flexible.
Our contributions are as follows. (1) For the first time to our knowledge, we propose and tackle the problem of universal few-shot learning of arbitrary dense prediction tasks. We formulate the prob- lem episodic meta-learning and identify two key desiderata of the learner unified architecture and adaptation mechanism. (2) We propose Visual Token Matching (VTM), a novel universal few- shot learner for dense prediction tasks. It employs non-parametric matching on tokenized image and label embeddings, which flexibly adapts to unseen tasks using tiny amount of task-specific param- eters. (3) We implement VTM as a powerful hierarchical mcoder-decoder architecture, where token matching is performed at multiple feature hierarchies using attention mechanism. We employ ViT image and label encoders (Dosovitskiy et al., 2020) and convolutional decoder (Ranftl et al., (02T). which seamlessly works with our algorithm. (4) We demonstrate VTM on challenging variant of Taskonomy dataset (Zamir et al.] 2018) and observe that robustly few-shot learns various unseen dense prediction tasks. Surprisingly, it is competitive to outperforms fully supervised baselines given extremely few examples (0.1%), sometimes using only 10 labeled images (< 0.004%).
PROBLEM SETUP
We propose and tackle the problem of universal few-shot learning of arbitrary dense prediction tasks. In our setup, we consider any arbitrary task T that can be expressed as follows:
T Hx + #xWxCT Cr C N.
(1)
This subsumes a wide range of vision tasks including semantic segmentation, depth estimation, surface normal prediction, edge prediction, to name few, varying in structure of output space, e.g., dimensionality (CT) and topology (discrete or continuous), as well as the required knowledge.
Our goal is to build universal few-shot learner F that, for any such task T, can produce predictions ÿra for an unseen image (query) X given a few labeled examples (support set) Sy:
y = =(X:(X) ST = ((XttY"))ss
To build such universal few-shot learner F, we adopt the conventional episodic training protocol where the training is composed of multiple episodes, each simulating a few-shot learning problem. To this end, we utilize meta-training dataset Prnin that contains labeled examples of diverse dense prediction tasks. Each training episode simulates a few-shot learning scenario of a specific task Tuaain in the dataset- the objective is to produce correct labels for query images given a support set. By experiencing multiple episodes of few-shot learning, the model is expected to learn general knowledge for fast and flexible adaptation to novel tasks. At test time, the model is asked to perform few-shot learning on arbitrary unseen tasks TrE not included in the training dataset (Dmiin)
An immediate challenge in handling arbitrary tasks in Eq. is that each task in both meta-training and testing has different output structures (i.e., output dimension CT varies per task), making it dif- ficult to design single, unified parameterization of model for all tasks. As a simple yet general so- lution, we cast a task T IR H W x3 - x CT into CT single-channel sub-tasks - .oT H x1 independently using
of learning each channel, and model each sub-task Te Hx the shared model F in Eq. Although multi-channel information is beneficial in general, we ob- serve that its impact is negligible in practice, while the channel-wise decomposition introduces other useful benefits such as augmenting the number of tasks in meta-training, flexibility to generalize to arbitrary dimension of unseen tasks, and more efficient parameter-sharing within and across tasks. Without loss of generality, the rest of the paper considers that every task is of single-chama label.
2

---

Published as a conference paper at ICLR 2023
2.1 CHALLENGES AND DESIDERATA
The above problem setup is universal, potentially benefiting various downstream vision applications. Yet, to our knowledge, no prior work in few-shot learning attempted to solve it. We attribute this to two unique desiderata for a universal few-shot learner that pose challenge to current methods.
Task-Agnostic Architecture As any arbitrary unseen task Tiest can be encountered in test-time, the few-shot learner must have a unified architecture that can handle all dense prediction tasks by design. This means we cannot exploit any kind of prior knowledge or inductive bias specific to certain tasks. For example, we cannot adopt common strategies in few-shot segmentation such as class prototype or binary masking, as they rely on the discrete nature of categorical labels while the label space of Ties can be arbitrary (continuous or discrete). Ideally, the unified architecture would allow the learner to acquire generalizable knowledge for few-shot learning any unseen tasks, as enables sharing most of the model parameters across all tasks in meta- training and testing.
Adaptation Mechanism On top of the unified architecture, the learner should have a flexible and efficient adaptation mechanism address highly diverse semantics of the unseen tasks Tistt This is because tasks of different semantics can require distinct sets of features depth estimation requires 3D scene understanding, while edge estimation prefers low-level image gradient. As a result, even with a unified architecture, learner that depends on a fixed algorithm and features would either underfit to various training tasks in Dirain. or fail for an unseen task Trest that has a completely novel semantics. Thus, our few-shot learner should be flexibly adapt its features to a given task T based on the support set Sy. e.g., through task-specific parameters. At the same time, the adaptation mechanism should be parameter-efficien (e.g.. using a tiny amount of task-specific parameters) to prevent over-fitting to training tasks Tuain or test-time support set ST
3 VISUAL TOKEN MATCHING
We introduce Visual Token Matching (VTM), a universal few-shot learner for arbitrary dense prediction tasks that is designed to flexibly adapt to novel tasks of diverse semantics. We first discuss the underlying motivation of VTM in Section 3.1 and discuss its architecture in Section 3.2
3.1 MOTIVATION
We seek for general function form of Eq. 2 that produces structured labels of arbitrary tasks with a unified framework. To this end, we opt into non- non-parametric approach that operates on patches, where the query label is obtained by weighted combination of support labels. Let X {x, Js denote an image (or label) on patch grid of size M = h x w, where x, is j-th patch. Given query image X = {x]] J and a support set {(X', H') = {(x f JeggMMs for task T, we project all patches to embedding spaces and predict the query label Y {y bis patch-wise by,
9(19))
Σ Σ (x g(vi). ISN KSM
where fr(x) = f(x: 8, 07) E Rdd and g(y) g(y: E Rd correspond to the image and label encoder, respectively, and o: : Rd x Rd + [0, 1] denotes a similarity function defined on the image patch embeddings. Then, each predicted label embedding can be mapped to a label patch y3 =
(((()))) by introducing a label decoder h R g
Note that Eq. 3 Sggeneralizes the Matching Network (MN) (Vinyals et al. 2016). and serves as a gen- eral framework for arbitrary dense prediction tasks]] First, while MN interpolates raw categorical labels y for classification (i.e. g is an identity function), we perform matching on the general em- bedding space of the label encoder g(y); it encapsulates arbitrary tasks (e.g.. discrete or continuous) into the common embedding space, thus enabling the matching to work in consistent way agnostic to tasks. Second, while MN exploits a fixed similarity of images (f((x), f(xx)), we modulate the similarity a (ff((x)), fo (x')) adaptively to any given task by switching the task-specific parameters 07. Having adaptable components in the image encoder fr is essential in our problem since it can adapt the representations to reflect features unique in each task. Finally, additional decoder h to project the predicted label embeddings to the output space.
method employs an
Once trained, the prediction mechanism in Eq 3 can easily adapt to unseen tasks at test-time. Since the label encoder g is shared across tasks, we can use it embed the label patches of unseen
While MN performs image-level classification, we consider its extension to patch-level emheddings.
3

---

Published as a conference paper at ICLR 2023
Image Encoder
Label Encoder
Marching Modules
Label Decoder
Dlock
WiT Blece
VIT Black
Sapeot idedd
Macciig
Paaic D3)
Blok
Block
vit Block
Scatec ai aa
Block
ViT Black
Maxching
Suppci W0ge000
Maching
Bock
Qaeyy
Quary Mtd
Taken Matching Module (Ea. 4,
paramettes
Emited
Emioed
Embbed
Head
Inicca orn LIFE
Telro
Quary imga
Support Imsage and label
Prodicted query libel
Figure 1: Overall architecture of VTM. Our model is a hierarchical encoder-decoder with four main components: the image encoder fr, label encoder g, label decoder h, and the matching module. See the text for more detailed descriptions.
tasks with frozen parameters é. Adaptation novel task is performed by the image encoder JT. by optimizing the task-specific parameters 07 which take a small portion of the model. This allows our model to robustly adapt to unseen tasks of various semanties with small support In experiments, our model becomes competitive to supervised method using less than 0. % of labels.
3.2 ARCHITECTURE
[igurr/[illlusrattee our model. Our model has hierarchical encoder-decoder architecture that imple- ments patch-level non-parametric matching of Eq. 3 in multiple hicrarchies with four components: image encoder fr, label encoder g. label decoder h, and the matching module. Given the query image and the support set, the image encoder first extracts patch-level embeddings (tokens) of each query and support image independently. The label encoder similarly extracts tokens of each support label. Given the tokens at each hierarchy, the matching module performs non-parametric matching of Eq. to infer the tokens of query label, from which the label decoder forms the raw query label.
Image Encoder We employ a Vision Transformer (ViT) (Dosovitskly et al, 2020) for our image encoder. The VIT is applied query and each support image independently while sharing weights, which produces tokenized representation of image patches at multiple hierarchies. Similar to Ranftl al. (002I) extract tokens at four intermediate ViT blocks form hierarchical features. To aid learning general representation for wide range of tasks, we initialize the parameters from pre- trained BEiT (Bao al. 2021). which is self-supervised thus less biased towards specific tasks.
As discussed in Section 3.1 we design the image encoder to have two sets of parameters and θr, where 0 is shared across all tasks and 07 is specific to each task T. Among many candidates to design an adaptation mechanism through fT, we find that bias tuning (Cai et al. 2020 Zaken al.) 2022) provides the best efficiency and performance empirically. To this end, we employ separate sets of biases for each task both meta-training and meta-testing, while sharing all the other parameters.
Label Encoder The label encoder employs the same ViT architecture the image encoder and extracts token embeddings of the support labels. Similar to the image encoder, the label encoder applied to each support label independently, with major difference that it sees channel at time since we treat each channel as an independent task as discussed in Section 2 Then the label tokens are extracted from multiple hierarchies that matches the image encoder. Contrary to the image encoder, all parameters of the label encoder are trained from scratch and shared across tasks.
Matching Module We implement the matching module at each hierarchy as a multihead attention layer (Vaswani et al. 2017). At each hierarchy of the image and label encoder, first obtain the tokens of the query image X as 1qibss and support set {(Xi, Y'))ss as {(k] v wi)kks.isM from the intermediate layers of image and label encoders, respectively. We then stack the tokens to
4

---

Published as a conference paper at ICLR 2023
row-matrices, q RMxd and v E NMxd Then, the query label tokens at the hierarchy inferred as output of a multihead attention layer, as follows:
MHA(q. k, v) = Concat(o1, o.,,,,Hw,
(4)
qui Q (
where o₁, = Softmax
vw
(5)
VdH
where H is number of heads, dy is head size, and wh w w/ E ERaxdi wo E &RHduxd,
Remarkably, each attention head in Eq. 5 implemennt the intuition of the non non-parametric matching in Eq. [] This is because each query label token is inferred as a weighted combination of support label tokens based the similarity between the query and support image tokens q. k. Here, the similarity function a of Eq. 3 is implemented as scaled dot-product attention. Since each head involves different trainable projection matrices wh wh , wy, the multihead attention layer in Eq. able to learn multiple branches (heads) of matching algorithm with distinct similarity functions.
Label Decoder The label decoder h receives the query label tokens inferred at multiple hicrar- chies, and combines them predict the query label of original resolution. We adopt the multi-scale decoder architecture of Dense Prediction Transformer (Ranftl et al., 2021) as it seamlessly works with ViT encoders and multi-level tokens. At each hierarchy of the decoder, the inferred query label tokens are first spatially concatenated to a feature map of constant size (M + h x w). Then, (trans- posed) convolution layers of different strides are applied each feature map. producing feature pyramid of increasing resolution. The multi-scale features are progressively upsampled and fused by convolutional blocks, followed by a convolutional head for final prediction.
Similar to the label encoder, all parameters of the label decoder are trained from scratch and shared across tasks. This lets the decoder to meta-learn generalizable strategy of decoding a structured label from the predicted query label tokens. Following the channel split Section 2 the output of the decoder is single-channel, which allows it to be applied to tasks of arbitrary number of channels.
3.3 TRAINING AND INFERENCE
We train model labeled dataset Prain of training tasks Tuain following the standard episodic meta-lcarning protocol. At each episode of task T, we sample two labeled sets ST, 27 from Dusin- Then we train the model to prediet labels in Qr using Sy support set. We repeat the episodes with various dense prediction tasks in Dirain so that the model can learn a general knowledge about few-shot learning. Let F(X:; ST) denote the prediction of the model on X using the support set Sr. Then the model (/T, g, h, σ) is trained by the following learning objective in end-to-end:
min fTih
Es
Σ F(X9, (XX,Y9)ERT
(6)
[QT|
where L is the loss function. We cross-entropy loss for semantic segmentation task and L1 loss for the others our experiments. Note that the objective (Eq. 6 does not explicitly enforce the matching equation (Eq. 3 in token space, allowing some knowledge for prediction to be handled by the label decoder h, since we found that introducing explicit reconstruction loss on tokens dete- riorates the performance in our initial experiments. During training, training tasks Timin. we keep and train separate sets of bias parameters of the image encoder f' for each training task (which are assumed be channel-splitted)
have a fixed number of
After training on D the model is few-shot evaluated on novel tasks Tes given support set STue- We first perform adaptation of the model by fine-tuning bias parameters of the image encoder fT using the support set ST.. For this, we simulate episodic meta-learning by randomly partitioning the support set into sub-support set and sub-query set 2, such that ST... = SU
Σ ((Y9, (X
min E.. S 07
1
where 0y denotes bias parameters of the image encoder fr. The portion of parameters to be fine- tuned is negligible so the model can avoid over-fitting the small support set ST... After fine-tuned, the model is evaluated by predicting the label of unseen query image using the support set ST..
5

---

Published as a conference paper at ICLR 2023
4 RELATED WORK
To the best of our knowledge, the problem of universal few-shot learning of dense prediction tasks remains unexplored. Existing few-shot learning approaches for dense prediction are targeted to spe- cific tasks that require learning unseen classes of objects, such as semantic segmentation (Shaban ct al. 2017 Wang al. 2019 Iqbal et al. 2022). instance segmentation (Michaelis et al. 2018 Fan
et al. 2020b). and object detection (Fan et al.) 2020a Wang et al., 2020), rather than general tasks. As categorical labels are discrete in nature, most of the methods involve per-class average pooling of support image features, which cannot be generalized to regression tasks as there would be infinitely many "classes" of continuous labels. Others utilize masked correlation between support and query features (Min et al. [202T] Hong et al. (0022). learn a Gaussian Process on features (Johnander et al.) 2021 or train a classifier weight prediction model (Kang et al.) 2019). In principle, these architec- tures can be extended to more general dense prediction tasks with slight modification (Section (5), yet their generalizability to unseen dense prediction tasks, rather than classes, has not been explored.
As our method involves task-specific tuning of a small portion of parameters, it is related transfer learning that aims to efficiently fine-tune pre-trained model to downstream tasks. In natural lan- guage processing (NLP), language models pre-trained on large-scale corpus (Kenton & Toutanova
2019 Brown et al. 2020) show outstanding performance on downstream tasks with fine-tuning minimal amount parameters (Houlsby et al. 2019 Zaken et al. [022]] Lester et al,, 202T). Follow-
ing the emergence of pre-trained Vision Transformers (Dosovitskiy al. (2020), similar adaptation approaches have been proven successful in the vision domain (Li et al. 2021 Jia et al. [002 Chen
et al.] 2022). While these approaches reduce the amount of parameters required for wate-of-the-an performance on downstream tasks, they still require a large amount of labeled images for fine-tuning (e.g., thousands). In this context, our method can be seen as a few-shot extension of the adaptation methods, by incorporating a general few-shot learning framework and a powerful architecture.
5 EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Dataset We construct a variant of the Taskonomy dataset (Zamir et al.] 2018) to simulate few- shot learning of unseen dense prediction tasks. Taskonomy contains indoor images with various annotations, where we choose ten dense prediction tasks of diverse semantics and output dimensions: semantic segmentation (SS), surface normal (SN). Euclidean distance (ED), Z-buffer depth (ZD), texture edge (TE), occlusion edge (OE), 2D keypoints (K2), 3D keypoints (K3), reshading (RS), and principal curvature (PC) P We partition the ten tasks to construct 5-fold split, in each of which two tasks are used for few-shot evaluation (Tres) and the remaining eight are used for training (Tainn). To perform evaluation on tasks of novel semantics, we carefully construct the partition such that tasks for training and test are sufficiently different from each other e.g., by grouping edge tasks (TE. ОΕ) together as test tasks. The split is shown in Table We process some single-channel tasks (ED, TE, ОЕ) to multiple channels to increase task diversity, and standardize all labels to [0, 1]. Additional details are in Appendix A
Baselines
We compare our method (VTM) with two classes of learning approaches. Fully supervised baselines have an access the full supervision of test tasks Tiss during train- ing, and thus serve as upper bounds of few-shot performance. We consider two state-of-the- art baselines supervised learning and multi-task learning of general dense prediction tasks DPT (Ranftl al. 2021) and InvPT (Ye & Xu 2022), respectively, where DPT is trained on each single task independently and InvPT is trained jointly on all tasks.
Few-shot learning baselines do not have an access to the test tasks Tisst during training, and are given only few labeled images at the test-time. As there are no prior few-shot method developed for universal dense prediction tasks, we adapt state-of -het-rr-rrt few-shot segmentation methods to our setup. We choose three methods, DGPNet (Johnander al. (021). HSNet (Min al. (02T), and VAT (Hong et al. 2022). whose architectures are either inherently task-agnostic (DGPNet) can be simply extended (HSNet, VAT) handle general label spaces for dense prediction tasks. We describe the modification on HSNet and VAT in Appendix B
We choose all dense prediction tasks defined RGB images with pixel-wise loss functions.

---

Published as a conference paper at ICLR 2023
Table 1: Quantitative comparison on Taskonomy dataset. Few-shot baselines are 10-shot evaluated on each fold after being trained on the tasks from the other folds, where sully-supe baselines are trained and evaluated on tasks from each fold (DPT) or all folds (InvPT).
Tasks
Fold 1
Fold 2
Fold
Fold
Fold
Supervision
Model
SS
SN
ED
ZD
TE
OE
K2
K3
RS
PC
minU1
1
mFrr L
RMSF
1
RMSF L
RMSF
RMSF
1
RMSE
RMSF
RMSF, J. RMSF
DPT
0.4449
6.4414
0.0534
0.0268
0.0188
0.0689
0.0358
0.0357
0.0860
0.0347
Full
InvPT
0,3900
12,9249
0,0589
0.0298
0.0517
0.0788
0.0456
0.0384
0.0949
0.0370
HSNet
0,1069
24.9120
0.2375
0.0748
0.1746
0.1643
0.1056
0.0651
0.2627
0.0610
10-Shot
VAT
0.0353
25.8134
0.2718
0.0779
0.1719
0.1655
0.1450
0.0678
0.2709
0.0796
(< 0.004%)
DGPNet
0,0261
29.1668
0,4579
0.2846
0.1881
0.2130
0.1104
0.1308
0,3680
0.3574
Ours
0.4097 11.4391
0.0741
0.0316
0.0791
0.0912
0.0639
0.0519
0.1089
0.0420
For models based on ViT architecture (Ours, DPT, InvPT), we use BEiT-B (Bao et al. (222)) backbone as image encoder, which is pre-trained on ImageNet-22k (Deng et al. 2009) with self-supervision. For the other baselines (DPGNet, HSNet, VAT), as they rely on convolutional encoder and it is nontrivial to transfer them to ViT,
Implementation
ResNet-101 (He et al. (016) backbone pre-trained on ImageNet-I k with image classification, which is their best-performing configuration. During episodic training, we perform task augmentation based on color jittering and Mix (Zhang) et al. 2018) to increase the effective number of training tasks. For all few-shot learning models, we further apply label channel split described in Section 2
Further details are in Appendix B
Evaluation Protocol We use the train/val/test split of the Taskonomy-ting partition provided by Zamir et al. (011)). We train all models on the train split, where DPT and InvP'' are trained with full supervision of test tasks Tre and the few-shot models are trained by episodic learning of training tasks Train only. The final evaluation
test tasks Trest is done on the test split. During the evalua- tion, all few-shot models are given a support set randomly sampled from the train split, which also used for task-specific adaptation of VTM as described in Section 3.3 For evaluation on semantic segmentation (SS). we follow the standard binary segmentation protocol in few-shot semantic seg- mentation (Shaban et al. (017)) and report the mean intersection over union (mloU) for all classes. For tasks with continuous labels, we use the mean angle error (mErr) for surface normal prediction (SN) (Eigen & Fergus 2015) and root mean square error (RMSE) for the others.
5.2 RESULTS
In I we report the 10-shot performance of our model and the baselines on ten dense prediction tasks. Our model outperforms all few-shot baselines by a large margin, and is competitive with supervised baselines many tasks. In Figure 2 we show qualitative comparison where the few- shot baselines catastrophically underfit to novel tasks while our model successfully learns all tasks. We provide further qualitative comparisons of ours and the baselines in Appendix C.3
The large performance gap between the few-shot learning baselines and our model can be explained by two factors.
(1) The core architectural component of HSNet and VAT (feature masking) implic- itly relies on the discrete nature of labels, and thus fails to learn tasks with continuous labels whose values are densely distributed. (2) Since the baselines are designed solve tasks without any adap- tation of their parameters, the core prediction mechanism (e.g.. g.. hypercorrelation of HSNet and VAT, kernel of DGPNet) is fixed and cannot adapt to different semantics of tasks. Unlike the baselines, our model is general for all dense prediction tasks, and has a flexible task adaptation mechanism of the similarity matching. Our model can also be robustly fine-tuned on few-shot support, thanks to the parameter efficiency of adaptation (0. 28% of all parameters; see Table 7 for comparison with su- pervised baselines). To demonstrate how VTM performs task adaptation, we visualize the attention of the matching module (Eq. 4. Figure f shows that, when adapted for different tasks, the model flexibly changes the similarity to attend to support patches appropriate for the given task.
Surprisingly, with < 0.004% of the full supervision (10 labeled images), our model even performs better than fully-super InvPT on some tasks (e.g., SS and SN). This can be attributed to the robust matching architecture equipped with a flexible adaptation mechanism that enables efficient knowledge transfer across tasks. In ablation study, we show that our model can be further improved by increasing the support size, reaching or sometimes surpassing supervised baselines in many tasks.

---

Published as a conference paper at ICLR 2023
Image
SS
ED
OE
K3
RS
Ground Truth
DPT (sup.)
InvPT (sup.)
Ours
HSNet
VAT
DGPNet
Figure 2: Qualitative comparison of few-shot learning methods 10-shot evaluation for ten dense prediction tasks in Taskonomy. While other approaches fail, our model successfully few-shot learns all novel tasks with diverse semantics and different label representations.
Ours
DPT
InvPT
0.125
SN (nVEi 43
6:67
(nloll
ZD (RMSE
0.05
0.00
0.005
0.02
30
275
20
10
275
20
0.042
(RASS 47
(RMSE L)
IRMSE
0.0RD
0.075
X2
0.000
0.035
0.070
30
215
20
0.035
205
275
shot
shot
shat
shot
shot
Figure 3: Performance of VTM on various shots. In general, VTM consistently improves perfor-
mance as more supervision is given, and even surpasses fully supervised baselines on many tasks.
5.2.1 ABLATION STUDY
Component- Wise Analysis To analyze the effectiveness of our design, we conduct an ablation study with two variants. (1) Ours w/o Adaptation does not adapt the similarity for each task in the non-parametric matching. This variant shares all parameters across all training tasks Train as well as test tasks Trsa and is evaluated without fine-tuning. (2) Ours w/o Matching predicts the query label tokens directly from the query image tokens, replacing the matching module with parametric linear mapping at each hierarchy. This variant retains the task-specific parameters of ours, thus identical in terms of task adaptation; it utilizes the separate sets of task-specific biases in the image encoder learn the training tasks Tiains and fine-tune it the support set of Ties in the test time.
The results are in Table 2 Both variants show lower performance than ours in general, which demonstrates that incorporating non-parametric matching and task-specific adaptation are both ben- eficial for universal few-shot dense prediction problems. It seems that the task-specific adaptation is a crucial component for our problem, as Ours w/o Adaptation suffers from the high discrepancy be- tween training and test tasks. For some low-level tasks whose labels synthetically generated by applying a computational algorithm on an -mmgee (e.g., TE, K2), Ours w/o Matching achieves slightly better performance than Ours. Yet, for tasks requiring high-level knowledge of object
8

---

Published as a conference paper at ICLR 2023
Table 2: Ablation study on matching and task- adaptation.
Tasks
Fold 1
Fold 2
Fold 3
Fold 4
5
Model
SS
SN
ED
ZD
TE
OE
K2
K3
RS
PC
mloU f mErr 4
RMSE L RMSE
RMSE 4 RMSE 4
RMSE
RMSE 4
RMSE
RMSE
Ours w/o Matching
0.2681
13.0704
0,1111
0.0404
0.0778
0.1061
0.0613
0.0537
0.1559
0.0445
Ours w/o Adaptation
0.0002
23.4212
0.1515
0.0641
0.1513
0.1152
0.1110
0.0625
0.2033
0.0632
Ours
0.4097
11.4391
0.0741
0.0316
0.0791
0.0912
0.0639
0.0519
0.1089
0.0420
Query
Support
Support
Support
Support 4 Support Support
Support
Support Support 9 Support
Image
ED
SN
Figure 4: Attention maps of selected level and head for each task given query patch (red box in the 1st column). Our model flexibly adapts its similarity to attend to appropriate regions for each task (e.g., chairs for semantic segmentation, all planes orthogonal to camera view for surface normal).
mantics (e.g., SS) or 3D space (e.g., ED, OE, K3, RS), the introduction of non-parametric matching fairly improves the few-shot learning performance. Qualitative comparison is in Appendix C.4
Impact of Support Size As our method already performs well with ten labeled examples, a natural question arises: can it reach the performance of gilly-supervised approaches if more examples are given? In Figure 3] we plot the performance of VTM by increasing the size of support set from 10 (< 0.004%) to 275 (0.1%). Our model reaches or surpasses the performance of the supervised methods on many tasks with additional data (yet much smaller than full), which implies potential benefits in specialized domains (e.g., medical) where the number of available labels ranges from dozens to hundreds.
Sensitivity to the Choice of Support Set As our model few-shot learns a new task from a very small subset of the full dataset, we analyze the sensitivity of the performance to the choice of support set. Table 3 in Appendix reports the 10-shot evaluation results with 4 different support sets disjointly sampled from the whole train split. We see that the standard deviation marginal. This shows the robustness of our model to the support set, which would be important in practical scenarios.
Impact of Training Data The amount and quality of training data is an important factor that can affect the performance of our universal few-shot learner. In Appendix CC.1.4 we analyze the effect of the number of training tasks and show that few-shot performance consistently improves when more tasks are added to meta-training data. Also, in more practical scenario, we may have an incomplete dataset where images are not associated with whole training task labels. We investigate this setting in Appendix C.I.5 and show that our model trained with an incomplete dataset still performs well.
CONCLUSION
In this paper, we proposed and addressed the problem of building a universal few-shot learner for arbitrary dense prediction tasks. Our proposed Visual Token Matching (VTM) addresses the chal- lenge of the problem by extending a task-agnostic architecture of Matching Networks to patch-level embeddings images and labels, and introducing a flexible adaptation mechanism in the image encoder. We evaluated our VTM in a challenging variant of the Taskonomy dataset consisting of ten dense prediction tasks, and showed that VTM can achieve competitive performance to fully super- vised baselines on novel dense prediction tasks with an extremely small amount of labeled examples (<0.004% of full supervision), and is able to even closer the gap or outperform the supervised methods with fairly small amount of additional data (~0.13 1% of full supervision).

---

Published as a conference paper at ICLR 2023
Acknowledgements This work was supported in part by the Institute of Information & commu- nications Technology Planning & Evaluation (ITTP) (No. 2022-0-00926, 2022-0-00959, and 2019- 0-00075) and the National Research Foundation of Korea (NRF) (No. 202IRICICI01253 and 2021R A4A3032834) funded by the Korea government (MSIT), and Microsoft Research Asia Col- laborative Research Project.
Reproducibility Statement To help readers reproduce our experiments, we provided detailed de- scriptions of our architectures in Section B.I and implementation details of the baseline methods in Table in Section Since our work proposes the new experiment settings for the few-shot learn- ing of arbitrary dense prediction tasks, we also provide details of the dataset construction process in the main text (Section 55) and the appendix (Section A), which includes details of the data splits, specification of tasks, data preprocessing, and evaluation protocols. We plan to release the source codes and the dataset to ensure the reproducibility of this paper.
Ethics Statement We have read the ICLR Code Ethics and ensures that this work follows it. All data and pre-trained models used in our experiments are publically available and has ethical concerns.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. 1Xiv:1607.004400, 2016.
arXiv preprint
Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung-Woo Ha, and Jonghyun Choi. Rainbow memory: Continual learning with a memory of diverse samples. In CVPR, 2021.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In ICLR, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. NeurlPS, 2020.
Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, not parameters for efficient on-device learning. NeurlPS, 2020.
Xin Cai and Yi-Fei Pu. Flattenet: simple and versatile framework for dense pixelwise prediction. IEEE Access, 2019,
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. arXiv preprint arXiv:2205.068: 088334, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. hierarchical image database. In CVPR, 2009.
Imagenet: A large-scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020.
David Eigen and Rob Fergus. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. ICCV, 2015.
Qi Fan, Wei Zhuo, Chi-Keung Tang, and Yu-Wing Tai. Few-shot object detection with attention-rpn and multi-relation detector. In CVPR, 2020a.
Qi Fan, Wenjie Pei, Yu-Wing Tai, and Chi-Keung Tang. Self-support few-shot semantic segmenta- tion. arXiv preprini arXiv:2207.11549, 2022.
Zhibo Fan, Jin-Gang Yu, Zhihao Liang, Jiarong Ou, Changxin Gao, Gui-Song Xia, and Yuanqing Li. Fgn: Fully guided network for few-shot instance segmentation. In CVPR, 2020b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
10

---

Published as a conference paper at ICLR 2023
Dan Hendrycks and Kevin Gimpel. AXiv:1600.00815.. 2016.
Gaussian
enror
linear units
(gelus).
arXiv preprint
Sunghwan Hong, Seokju Cho, Jisu Nam, Stephen Lin, and Seungryong Kim. Cost aggregation with 4d convolutional swin transformer for few-shot segmentation. arXiv preprint arXiv:2207 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In ICML, 2019.
Ehtesham Iqbal, Sirojbek Safarov, and Seongdeok Bang. Msanet: Multi-similarity and attention guidance for boosting few-shot segmentation. arXiv preprint (xXiv2220.00966) 2022.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. arXiv preprint urXiv:2203. /2/ 2022.
Joakim Johnander, Johan Edstedt, Michael Felsberg, Fahad Shahbaz Khan, and Martin Danelljan. Dense gaussian processes for few-shot segmentation. arXiv preprint arXiv:2110003674 2021.
Bingyi Kang, Zhuang Liu, Xin Wang, Fisher Yu, Jiashi Feng, and Trevor Darrell. Few-shot object detection via feature reweighting. In ICCV, 2019.
Nick Kanopoulos, Nagesh Vasanthavada, and Robert L Baker. Design of an image edge detection filter using the sobel operator. /SSC, 1988.
Jacob Devlin Ming-Wes Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep
bidirectional transformers for language understanding. In Proceedings of NAACCLIILL, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Brian Lester, Rami AI-Rfou, and Noah Constant, The power of scale for parameter-efffiient prompt tuning. In EMNLP, 2021.
Yanghao Li, Saining Xie, Xinlei Chen, Piotr Dollar, Kaiming He, and Ross Girshick. Benchmarking detection transfer learning with vision transformers. arXiv preprint arXiv:2111.11422 2021.
Guosheng Lin. Anton Milan, Chunhua Shen, and Ian Reid. Refinenet: Multi-path refinement net-
works for high-resolution semantic segmentation. In CVPR, 2017.
Wei Liu, Andrew Rabinovich, and Alexander C Berg. Parsenet: Looking wider to see better. arXiv preprint arXiv:1506.0455 2015.
Yongfei Liu, Xiangyi Zhang, Songyang Zhang, and Xuming He. Part-aware prototype network for few-shot semantic segmentation. In ECCV, 2020.
Claudio Michaelis, Ivan Ustyuzhaninov, Matthias Bethge, and Alexander S Ecker. One-shot instance segmentation. arXiv preprint 1rXiv:/811.I 1507, 2018.
Juhong Min, Dahyun Kang, and Minsu Cho. Hypercorrelation squeeze for few-shot segmentation. In CVPR, 2021.
Melanie Mitchell. Abstraction and analogy-making in artificial intelligence. Annals of the New York Academy of Sciences, 2021.
Pedro O O Pinheiro, Amjad Almahairi, Ryan Benmalek, Florian Golemo, and Aaron C Courville. Unsupervised learning of dense visual representations. NeurIPS, 2020.
Yassine Ouali, Céline Hudelot, and Myriam Tami. Semi-supervised semantic segmentation with cross-consistency training. In CVPR, pp. 12674-12684, 2020.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In ICCV, 2021.
11

---

Published as a conference paper at ICLR 2023
Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for
semantic segmentation. In BMVC, 2017.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. NeurIPS, 2017.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 2014.
Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In ICML, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. NeurlPS, 2016.
Kaixin Wang. Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot image semantic segmentation with prototype alignment. ICCV, 2019.
Xin Wang, Thomas Huang, Joseph Gonzalez, Trevor Darrell, and Fisher Yu. Frustratingly simple few-shot object detection. In /CML., 2020.
Xinlong Wang, Rufeng Zhang, Chunhua Shen. Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training, In CVPR, 2021.
Zhen Wang, Liu Liu, Yiqun Duan, Yajing Kong, and Dacheng Tao. Continual learning with lifelong vision transformer. CVPR, 2022.
Hanrong Ye and Dan Xu. Inverted pyramid multi-task transformer for dense scene understanding. arXiv preprint a7X1::2033.07997, 2022.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-efficient fine-tuning for transformer-basee masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Short Papers), 2022.
Amir R. Zamir, Alexander Sax, William B. Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In CVPR, 2018.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
12

---

Published as a conference paper at ICLR 2023
APPENDIX
A DATASET DETAILS
This section describes the details about the dataset we used in experiments (Section
We use "tiny" version of Taskonomy dataset provided by (Zamir al.) 2018), which consists of images and labels collected from 35 different buildings. We use the train and val split for training and early-stopping, respectively, and use the" "muleshoe" building included in the test split for evaluation.
To demonstrate our universal few-shot learner, we use ten dense prediction tasks in Taskonomy dataset (Zamir et al. 2018), which are semantic segmentation (SS), surface normal (SN), Euclidean distance (ED), Z-buffer depth (ZD), texture edge (TE), occlusion edge (OE), 2D keypoints (K2), 3D keypoints (K3), reshading (RS), and principal curvature (PC). All labels are normalized into [o. 1 with task-specific pre- processsss... For details the pre-procce pre-processing, (2018). Based on the annotations provided by Taskonomy, we preprocess some tasks to increase the diversity of tasks. Specifically, we modify three single-channel tasks that can be easily augmented: Euclidean distance, texture edge, and occlusion edge.
refer readers to Zamir et al.
1. Texture edge (TE) labels are generated by applying Sobel edge detector (Kanopoulos et al,) 1988) to RGB images, which consists of a Gaussian filter and image gradient computation. The Gaussian filter has two nyper-parameters, namely kernel size and the standard deviation, where adjusting those syper-parameters yield different thickness of detected edges. We use three different sets of hyper-parameters (3, 1), (11, (19,3) 3) - to produce 3-channel labels. We give example of each channel of TE task in Figure
2. Euclidean distance (ED) labels consists of pixel-wise depth map, where the depth is computed by the Fuclidean distance from each image pixel to the camera's optical center. As this task is very similar to the Z-buffer depth prediction (ZD) whose label pixels the distance from each image pixel to the camera plane, we augment the ED task by segmenting the depth range and re-normalizing within each segment. Specifically, we compute the 5-quantiles of the pixel- wise depth labels in the whole dataset, then use each quantile as different channels after re- noramlization into [0, 1]. Thus the objective of each channel of the augmented ED task is to predict Euclidean distance within a specific range, where the ranges are disjoint for different channels. We give an example of each channel of ED task Figure 6 To visualize 5-channel labels, we average the first and the second channels as "R"-channel, the third and the fourth channels "G" "G"-channel, and use the fifth channel as "B"-channel.
3. Occlusion edge (OE) labels are similar to texture edge, but they constructed depend only the 3D geometry rather than color or lighting (Zamir et al.; 2018). We observe that the channel augmentation by quantiles (that we apply Euclidean distance task) can fairly diversify the labels. Therefore, we augment the OE labels into 5-channel labels, where we visualize them similar to the ED labels. We give an example of each channel of OE task in Figure
Also, for semantic segmentation, exclude three classes ("bottle", "toilet", "book"), as little im- ages of the classes are included in the Taskonomy dataset. The 12 classes we used experiments are: "chair", "couch", "plant", "bed", "dining table". "tv", "mircrowave", "oven", "sink", "fridge", "clock", and "base".
13

---

Published as a conference paper at ICLR 2023
Kernel size 3, Std. =1
Kernel size 11, Kernel size 19. Std.
3 3-Channel
RGB Image
Std. =3
Visualization
Figure 5: Channel augmentation on texture edge prediction (TE) task. We apply three different sets of hyper-parameters (kernel size, standard deviation) in Sobel edge detector generate 3-channel edge task. Second to Fourth columns show the augmented channel with different kernel size and standard deviation, where the last column shows the 3-channel label visualized as RGB.
5 5-Channel Visualization
RGB Image
5-quantile
2nd 5-quantile
5-quantile
4th 5-quantile
5th 5-quantile
Figure 6: Channel augmentation on Euclidean distance prediction (ED) task. We compute 5- quantiles of the pixel-wise label distribution, and each p-th 5-quantile each channel after re-normalizing into [0, 1]. Second to Fifth columns show the augmented channel with different quantile, where the last column shows the 5-channel label visualized as RGB.
s-Channel Visualization
RGB Image
1st 5-quantile
2nd 5-quantile
3rd 5-quantile
4th 5-quantile
5th 5-quantile
Figure 7: Channel augmentation occlusion edge prediction (OE) task. We compute 5-quantiles of the pixel-wise label distribution, and use each p-th 5-quantile as each channel after re-normalizing into [0, 1). Second to Fifth columns show the augmented channel with different quantile, where the last column shows the 5-channel label visualized as RGB.
14

---

Published as a conference paper at ICLR 2023
B IMPLEMENTATION DETAILS
This section describes the implementation details in our experiments (Section
B.1 ARCHITECTURE DETAILS OF VTM
Encoders and Decoders We employ BEiT-B architecture (Bao et al. 2021 pretrained on Imagenet-22k dataset (Deng et al., 2009) with 224 x 224 resolution as our image encoder. For our label encoder and decoder, we follow the DPT-B architecture (Ranftl et al. 202 I. Specifically, we use randomly initialized ViT-B (Dosovitskiy et al. 2020) as label encoder g and extract features from 3, 6,9, 12-th layers of the encoder to form multi-level label features (label tokens). Similarly, we extract multi-level image features (image tokens) from 3, 6, 9, 12-th layers of the image encoder (BEiT). As the DPT-B architecture decodes four-level features using Refine et al. 2017), we pass the predicted query label features from matching module at each layer to the decoder. As the label values of tasks Taskonomy are normalized [0. 1]. we use sigmoid activa- tion function at the head of the decoder to produce values in [0, 1J. To predict semantic segmentation task whose label values are discrete (either 1). we discretize the predicted label with threshold 0.1.
decoder (Lin
Matching Modules In the implementation of the matching module with multihead attention, we adopt three conventions in vision transformer (Dosovitskiy al., 2020) which slightly modifies the equations described in Section 3.2 Recall that the matching module is computed on three input
matrices q E IR MRd and k, v P. M as follows:
MHA(q. k, v) Concat(o) Oz >n)
(8)
qu 2 (kw
where On Softmax
v
vdH
where H is number of heads, dy head size, and wh w w E Rdxda, wo E Hddu First, we perform layer normalization (Ba et al. 2016) before each input projection matrices w w wx and after the output projection matrix w where we share the layer normalization parameters for 2 and w Second, we add a residual connection with GELU non-linearity (Hendrycks & Gimpel 2016) after gathering the outputs from multiple heads follows:
MHA(q, k, v) = o + GELU(ou)
(10)
where Concat 01, 02, .H))
(11)
Finally, we apply Dropout (Srivastava et al. 2014) with rate 0. in the attention scores.
B.2 ARCHITECTURE DETAILS OF BASELINES
Encoders and Decoders For the supervised learning baselines based on transformer encoder (DPT and InvPT), we use the same encoder backbone with ours (BEiT pretrained on ImageNet- 22k). We use the decoder of DPT-B configuration in Ranftl et al. (2021) 1) for DPT as ours, and use the original multi-task decoder implementation provided by Ye & Xu (2022)
) for InvPT. For few- shot learning baselines (HSNet, VAT, DGPNet), we use ResNet-101 (He et al.) (016) pretrained on ImageNet- 1k (Deng et al.] (009) as their encoder backbones, which is their best configuration. For the other architectural details, we follow the original implementation of each method provided by
Min et al. (202T) (HSNet), Hong et al. (2022) (VAT), and Johnander et al. (202 1) (DGPNet).
Modification on Few-shot Baselines As HSNet and VAT are designed for semantic segmentation, we slightly modify their architectures to train them on general dense prediction tasks. Specifically, both models involve binary masking operation to filter out support image features using their labels (which are assumed to be binary), before computing 4D correlation tensor between support and query feature pixels. For continuous labels of general dense prediction tasks, the binary masking becomes pixel-wise multiplication with labels. However, as the correlation is computed by cosine similarity between feature pixels that is norm-invariant, all non-zero feature pixels with the same
15

---

Published as a conference paper at ICLR 2023
direction are treated in the same manner. This make them unable to discriminate different non- zero label values, e.g., correlation between query and support feature pixels would be the same move the masking operation to after
regardless of the assigned support label values. Therefore, computing the cosine-similarity, so that the models can recognize different non-zero label values through different norms of the masked features by (non-binary) labels.
We use the DGPNet without modification as is based on a regression method (Gaussian Processes)
which is inherently applicable general dense prediction tasks with continuous labels.
B.3 TRAINING DETAILS
Training We train all models with 300,000 iterations using the Adam optimizer (Kingma & Ba, 2015), and use poly learning rate schedule (Liu et al., 2015) with base learning rates 10 for pre- trained parameters and 10-4 for parameters trained from scratch. The models are early-stopped based on the validation metric. At each episodic training of iteration, we sample a batch of episodes with size 8. In each episode, splitting all channels training tasks and randomly sample channels among them. Then support and query
construct a 5 5-channel task from the training tasks Tuain by first
sampled for the selected channels, where Ours and DGP, while using for HSNet and VAT as they only supports 1-shot training. To train DPT, we construct batch of each target task Tiet whose channels are given at once, with batch size 64. To train InvPT, we construct batch of all ten tasks, whose channels are all given at once, while using batch size 16 due its large memory consumption.
support and query size of 4 for
Data Augmentation We apply random crop (from 256 x 256 resolution to 224 x 224) and random horizontal flip to images, where the random horizontal flip is applied except for surface normal labels as their values are sensitive to the horizontal direction (flipping images and labels together changes the semantics of the task). As we apply random crop during training, the resolution of test images (256 x 256) differs from the training images. To evaluate the models with consistent resolution, perform five-crop (cropping the four corners and center of an image) to test query images so that the model also predicts five-cropped labels, then aggregate them by averaging the overlapping regions to produce final prediction for evaluation of resolution (256 x 256). For few-shot models, we apply center crop to support images test-time.
Task Augmentation For episodic training of few-shot models, we further apply two kinds of task augmentation. First, for each channel of C-channel labels sampled at each episode (C 5 in our experiments), apply random jittering and gaussian blur each channel independently. Then we apply MixUp (Zhang et al. 2018) on the augmented channels and auxiliary channels which additionally sampled from the training tasks Timin to create a linearly interpolated label of two channels. We apply the task augmentation consistently in each episode to preserve the task identity.
16

---

Published as a conference paper at ICLR 2023
C ADDITIONAL RESULTS
This section provides additional results on our experiments (Section 5).
C.I ADDITIONAL RESULTS ON ABLATION STUDY
C.I.1 SENSITIVITY TO THE CHOICE OF SUPPORT SET
As discussed in Section 5 we evaluate the 10-shot performance of our VTM with four different support sets that are disjointly sampled from the training data Duain- We report the results in Table which shows that our model is robust to the choice of support set. We use the first support set (# in Table 3 for comparison with other baselines ablated variants in Section due the huge computational cost for evaluating few-shot baselines HSNet and VAT.
Table 3: Ablation study on the choice of support set. We disjointly sample four different support sets and report the 10-shot performance on each set, with the mean and standard deviation.
Tasks
Fold 1
Fold 2
Fold 3
Fold 4
Fold 5
Support set
SS
SN
ED
7D
TE
OF
K2
K3
RS
PC
mloU + mErr +
RMSE
RMSE 4
RMSE
RMSE
RMSE
RMSE
RMSE 1. RMSE
#1
0.4097
11.4391
0.0741
0.0316
0.0791
0.0912
0.0639
0.0519
0.1089
0.0420
#
0.4190
11,8860
0.0845
0,0338
0.0839
0.0926
0.0629
0,0497
0.1131
0.0437
#
3
0.3781
11.7418
0.0776
0.0343
0.0807
0.0944
0.0656
0.0494
0.1101
0.0425
#
0.4017
11.6203
0.0794
0.0362
0.0799
0.0908
0.0672
0.0502
0.1158
0.0424
Mcan
0.4021
11.6718
0.0789
0.0340
0.0809
0.0922
0.0649
0.0503
0.1120
0.0427
Std.
0.0152
0.1640
0.0038
0,0016
0.0018
0.0014
0.0016
0.0010
0.0027
0.0006
C.1.2 ABLATION STUDY ON TRAINING PROCEDURE
To understand the source of the generalization performance of our method more clearly, conduct an ablation study on training procedure. We compare four models based on DPT architecture with different training procedures follows.
M1: Randomly initialized DPT, 10-shot trained.
M2: DPT with BEiT pre-trained encoder, 10-shot fine-tuned.
M3 (Ours w/o Matching): DPT with BEiT pre-trained encoder, multi-task trained with task- specific bias tuning, and then 10-shot fine-tuned.
M4 (Ours): DPT with BETT pre-trained encoder, meta-trained with task-specific bias tuning, and then 10-shot fine-tuned.
We summarize the quantitative result in Table and qualitative comparison in Figure g First, as ex- pected, we observe that DPT with naive 10-shof training (M1) fails to generalize to the test examples most of the tasks, except for two 2D texture-related tasks (TE, K2). We conjecture that TE and K2 are "casy" cases in terms of few-shot learning, as they are defined as low-level computational algo- rithms on RGB images, while other high-level tasks require knowledge about semantics (SS) or 3D space (SN, ED, ZD, OE, K3, RS, PC). Second, we note that BEiT pretraining (M2) largely improves the few-shot generalization performance, allowing the model to produce coarse predictions of the dense labels. However, it still cannot capture object-level fine-grained details in many tasks. Third, we observe that multi-task training and few-shot adaptation, combined with an efficient parameter- sharing strategy of bias tuning (M3, M4). further improves the performance with a clear gap with M2 where the predictions are also qualitatively finer than M2's. Finally, as discussed in Section 5.2.I] M4 still further improves over M3 with clear gap. This shows that in few-shot learning setting.
17

---

Published as a conference paper at ICLR 2023
Table 10-shot learning performance of ablated variants of DPT and Ours.
Tasks
Fold
Fold 2
Fold 3
Fold 4
Fold 5
Model
SS
SN
ED
ZD
TE
OE
K2
K3
RS
PC
mloU
mErr
RMSE
J. RMSE
RMSE
RMSE J
RMSE
RMSE 4
RMSE
RMSE J
M1
0.0644
21.0976
0.1959
0.0711
0.0995
0.1842
0.0670
0.0600
0.2335
0.0431
M2
0.0582
15.8135
0.1615
0.0530
0.1136
0.1480
0.0948
0.0606
0,1858
0.0431
M3
0.2681
13.0704
0.1111
0.0404
0.0778
0.1061
0.0613
0.0537
0,1559
0.0445
M4
0.4097
11.4391
0.0741
0.0316
0.0791
0.0912
0.0639
0.0519
0.1089
0.0420
Image
SN
ZD
TE
OE
K3
PC
Ground-Truth
M1 (Naive DPT)
Pan
Ma BEIT)
(Ours w/o Matching)
(Ours)
Figure 8: Qualitative comparison Ours and its ablated variants in training procedure. All models use 10 labeled examples for each target task, where M3 and M4 observe additional labeled examples of training tasks (different from the target task) in each fold.
our matching framework and episodic training are more effective than simple multi-task pretraining employed in M3. In summary, we may conclude that the fast generalization of Ours is benefitted from episodic training various tasks followed by parameter-efficient few-shot adaptation as well as powerful pre-training of the encoder (BEiT).
C.1.3 FINE-TUNING WITH FULL SUPERVISION
To further explore how our method scales well when large labeled dataset is given, we also fine- tuned our VTM with full supervision of test tasks. For the fine-tuning, we used the same training dataset as the superviedd DPT and employed the episodic fine-tuning objective (Section 3.3). For evaluation, since providing the entire training data the support set for the matching module infeasible, we provide a random subset of the training data as the support set to the model. We summarize the result in Figure g which extends Figure in Section 5 In most tasks, model consistently improves when more supervision is given. With full supervision at test tasks, our model performs slightly worse than the DPT baseline in seven tasks and performs better similarly in the other three tasks. We conjecture that the performance degradation comes from two aspects: (1) the absence of direct input-output connection, i.e., the matching module serves bottleneck. and (2) negative transfer from meta-tran training tasks to test tasks.
C.1.4 EFFECT OF NUMBER OF TRAINING TASKS
The amount of meta-training tasks is important factor that can affect the performance of the universal few-shot learner. To verify this, we fixed two test tasks (SS. SN) and trained our VTM on five different subsets of the original eight training tasks (three different subsets with two tasks and two different subsets with five tasks). We summarize the results in the Table [] As expected, the performance consistently improves
increase the number of training tasks. We also note that the
18

---

Published as a conference paper at ICLR 2023
Ours
DPT
IrYFT
0.02
0.35
SM (mnCr 4)
59 (miou
2D LRMKS
(RMSS
0.05
0.065
0.208
0.45
0.000
9.01
0.40
01024
S1dt
shot
shot
5ho0
s7ot
0.090
IAMSE 11
0.050
(RASE
0.000
0.085
0.05
0.045
(AMSE
0.00
(RMIS
0.083
0.075
0.00
275
275
Figure 9: Performance of VTM various shots. In general, VTM consistently improves perfor- mance as more supervision is given, and even surpasses fully supervised baselines on many tasks.
few-shot performance becomes sensitive to the choice of training tasks when their number is small (two), presumably as the model becomes reliant on training tasks more correlated to tasks, while the variance decreases substantially when more training tasks are added. In addition, the experiment with incomplete training data (Appendix C.1.5) shows the potential ability of our methods more realistic settings where the training dataset is formed by a combination of different task-specific datasets. From these results, we expect that our model can further enhance its universality on few- shot learning by utilizing combined training dataset of much more diverse tasks, which we leave as future work.
Table 5: 10-shot learning performance of Ours with various number of training tasks.
Tasks
Fold
Number of Training Tasks
SS
SN
mloU 1
mErr 4
0.2878 + 0.0565
17.3947 = 4.8742
5
0.3919 + 0.0132
12.6769 0.1233
0.4097
11.4391
C.1.5 EPISODIC TRAINING WITH INCOMPLETE DATASET
It would make method more practical if the model could learn from incomplete dataset where images are not associated with whole training task labels. To see how our framework extends to such incomplete settings, we conducted an additional experiment. We simulate the extreme case of incomplete data by partitioning the training images, such that each image is associated with only single task out of 8 training tasks. Specifically, we partitioned the buildings Taskonomy into eight groups each corresponds to a different training task. As this reduces the effective size of training data by the number of training tasks (1/8 our case), we also train a baseline where we use complete data but use only 1/8 of the training images (for each building, we discard 7/8 of the images). The results are summarized in Table We can see that the performance degradation is marginal when we give incomplete data, which implies that method can be promising in handling realistic scenarios where the training data is collection of heterogeneous datasets with different label annotations.
C.2 FURTHER ANALYSIS
C.2.1 PARAMETER CORAMETER-EFFIIEENYY ANALYSIS
We report the number of task-specific and shared parameters of our VTM and two supervised base- lines, DPT and InvPT, to compare how task adaptation is parameter-efficient. As DPT is single-task learning model, no parameters are shared across tasks and the whole network should be trained independently for every new task. InyPT, which is multi-task learning model, shares large
19

---

Published as a conference paper at ICLR 2023
Table 6: 10-shot learning performance of Ours trained with incomplete and complete multi-task dataset.
Tasks
Fold
Training Data
mloU
mErr J
Incomplete (one task per building)
0.3559
13.6207
Complete (1/8 training data)
0.3980
12.1633
Complete (whole training data)
0.4097
11.4391
portion of its parameters across tasks (e.g., encoder backbone), still consumes many parameters for each task in the decoder.
Due to the extensive amount of parameter-sharing, our method is also promising in continual learn- ing setting. As all task-specific knowledge is included in the bias parameters of the image encoder, knowledge acquired from past tasks can be recalled without forgetting by keeping the corre- sponding bias parameters and switching to them whenever past model is needed. We especially note that the size of bias parameters is fairly small (288 KB, which amounts to keeping about beled images of 256x256 resolution for each task). This allows our model to retain past knowledge very efficiently by keeping the tuned bias parameters plus few-shot support set, whose external memory requirement is far less compared memory-based approaches in continual learning that keep hundreds of images (Bang et al. [2021] Wang et al. (002).. While the continual learning setting not our main focus, applying our method continual learning setting would be an interesting future direction.
Table 7: Number of task mask-speeific and shared parameters for a single-channel task (in million).
Model
Task Task-Specific
Shared
DPT (supervised learning)
110.55
InvPT (multi-tat -ask learning)
24.57
106.75
Ours (few-shot learning)
0.0703
202.95
C.2.2 COMPUTATION COST ANALYSIS
To analyze how our method is computationally efficient compared supervised DPT, we measured the MACs (multiply accuuullaee operations) of our model and DPT using an open-source python library thop Bl We report the results in Table 8 Having encoded the support set (e.g.. 10-shot), we can see that the computational cost of our model's inference on single query image is about 30% larger than the cost of DPT's, due to the Matching part.
Table 8: MACs of Ours and DPT on single-query inference for a single-channel task.
Model
MAC's (G)
DPT
30.15
Ours after encoding support (10-shot)
38.79
20

---

Published as a conference paper at ICLR 2023
C.2.3 ROLE OF ATTENTION HEADS
To analyze the role of attention heads, in Figure 10 we visualized the attention maps for each head over support images for a given query patch, feature level (3rd level in this example), and task (RS in this example). The figure shows that each head attends to different regions of the support images. Moreover, we can find some patterns in heads; for example, the first head tends to attend to flat areas of the scene, such as the floor or ceiling (low-frequency features), while the third head tends attend to objects, such as couch or plant (high-frequency features). To further verify the benefit of multi- head attention in the matching module, we also trained our VTM with single head the matching modules. The result is summarized in the table below and Table 9 We can see the performance drop both SS and SN tasks, which supports that exploiting multiple heads benefits our matching framework.
Query
Support
Head 1
Head 2
Head 3
Head 4
C BBB aDOG
E BOBO
BBB D ODDS
C OBBO
OBOS
ODOB
Figure 10: Visualization of multi-heaa attention maps of VTM. Here we visualize the matching module at 3rd level for reshading (RS) task.
Table 10-shot learning performance of Ours with different number of attention heads in Matching module.
Tasks
Fold
Number Attention Heads
SS
SN
mioU
mErr
0.3702
12.5936
4
0.4097
11.4391
21

---

Published as a conference paper at ICLR 2023
C.3 ADDITIONAL QUALITATIVE COMPARISON WITH BASELINES
We provide additional results on the qualitative evaluation of our model and the baselines. Figure [1- s4os visualizations different query image and support set, where vary the class of semantic segmentation task included in each support. The result shows consistent trends of that we discussed in Section ||
Ours competitive the fully supervised baselines (DPT and InvPT), while the other few-shot baselines (HSNet, VAT, DGPNe() fail to learn different dense prediction tasks.
In Figure [2] even the GT label for semantic segmentation ("couch" class) noisy is pseudo- label generated by pre-trained segmentation model (Zamir et al. 2018), our model successfully
segments two couches present the figure. This can attributed the task aggkosicc architecture of VTM based non-parametric matching.
Image
SS
SN
ED
ZD
TE
OE
K2
K3
RS
PC
Ground
Truth
DPT
(sup.)
InvPT
(sup.)
Ours
HSNett
VAT
DGPNet
Figure 11: Additional results of qualitative comparison between Ours and the baselines.
22

---

Published as a conference paper at ICLR 2023
Image
SS
SN
ED
ZD
TE
OE
K2
K3
RS
PC
Ground Truth
DPT
(sup.)
InvPT (sup.)
Ours
HSNet
VAT
DGPNet
Figure 12: Additional results of qualitative comparison between Ours and the baselines.
Image
SS
SN
ED
ZD
TE
OE
K2
K3
RS
PC
Ground Truth
DPT
(sup.)
InvPT (sup.)
Ours
HSNet
VAT
DGPNet
Figure 13: Additional results of qualitative comparison between Ours and the baselines.
23

---

Published as a conference paper at ICLR 2023
Image
SS
SN
ED
ZD
TE
OE
K2
K3
RS
PC
Ground Truth
DPT
(sup.)
InvPT (sup.)
Ours
HSNet
DGPNet
Figure 14: Additional results of qualitative comparison between Ours and the baselines.
24

---

Published as a conference paper at ICLR 2023
C.4 ADDITIONAL QUALITATIVE COMPARISON WITH OUR VARIANTS
We also provide additional results on the qualitative evaluation of our model and our ablated vari- ants, Ours w/o Matching and Ours w/o Adaptation. Figure [5 18 show visualizations different query image and support set. The results show a consistent trend with the quantitative results in Table ! Interestingly, our method without adaptation already exhibits some degree of adaptation to the unseen tasks even without fine-tuning and task-specific components, showing that the non- parametric architecture our model and the parameter sharing derived from is appropriate to learn generalizable knowledge to understand the novel tasks. On the other hand, adding task-specific component and adaptation mechanism to the model allows more dramatic improvement under- standing novel tasks from few-shot examples, showing the importance of the adaptation mechanism our task. Finally, we observe that equipping the matching mechanism with the adaptation module provides much sharper and fast adaptation to the unseen tasks, which verifies claims.
Image
sS
SN
ED
ZD
TE
OE
K2
K3
RS
PC
Ground Truth
Ours
Ours w/o Matching
Ours w/o Adaptation
Figure 15: Additional results of qualitative comparison between Ours and its ablated variants.
Image
SS
SN
ED
ZD
TE
OF
K2
K3
RS
PC
Ground Truth
Ours
Ours w/o Matching
Ours w/o Adaplation
Figure 16: Additional results of qualitative comparison between Ours and its ablated variants.
25

---

Published as a conference paper at ICLR 2023
Image
SS
SN
ED
ZD
TE
K2
K3
PC
Ground Truth
Ours
Ours w/o Matching
Ours w/o Adaptation
Figure 17: Additional results of qualitative comparison between Ours and its ablated variants.
Image
SS
SN
ED
ZD
TE
OE
K2
K3
RS
PC
Ground Truth
Ours
Ours w/o Matching
Ours w/o Adaplation
Figure 18: Additional results of qualitative comparison between Ours and its ablated variants.
26


Reviews:
[{"review_id": "k-2xQG_Sm_", "reviewer": "Reviewer_1qaw", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles few-shot dense prediction from image, i.e. tasks where the input is an image and the output is a per-pixel prediction. The authors propose a Transformer-based meta-learning approach: learning some dense prediction tasks in an episodic fashion helps the model generalize to novel tasks. Most of the parameters on the model are shared, so a small-scale finetuning is needed to learn a new task.\n\nThe architecture is simple: the target image and the support images are split into patches that are featurized using a Transformer. Then the target patches attend to the support patches. Based on their similarity, a target patch inherits the weighted label encoding of each support patch. The label encodings are summed over all patches into a single vector. Another network upsamples this vector to the full image resolution.\nThere are two important details here. First, all weights are shared across tasks, except for the bias terms in the image encoder. This forces us to finetune for a new task, but the actual number of parameters is small. Second, supposing that the output has C channels, the proposed model predicts a dense map for each channel separately. This allows generalization to an arbitrary number of channels at test time.\n\nThe authors test their approach on Taskonomy, by splitting 10 dense prediction tasks into 8 for training and 2 for testing (5-fold validation). They compare against fully-supervised and few-shot baselines and significantly outperform the few-shot ones while sometimes performing on par with fully-supervised approaches. Their ablation indicates that finetuning for new tasks is necessary and that most of the times their network benefits from context (attending to the support set) to improve performance.", "summary_of_the_review": "In general I like the motivation and technical aspects of this paper. I feel the problem is important and the approach seems to make a step towards the right directions. However, I feel that some comparisons that could make the arguments stronger are missing. In addition, I am concerned about depending on finetuning, although I recognize that continual learning is not the focus of this work. I'm looking forward to the discussion with the authors and will happily vote for accepting the paper if they address my questions.\n\nAfter the rebuttal, all my concerns were addressed and I strongly recommend this paper for acceptance.", "strength_and_weaknesses": "Strengths:\n\n(1) The problem of few-shot multi-tasking for dense prediction is important and unexplored.\n\n(2) The proposed approach is simple and intuitive, without task-specific inductive biases or extreme computational complexity. The experiments show that it works well.\n\n(3) The writing is clear and helps the understanding of the motivation, desiderata, solution.\n\n\nWeaknesses:\n\nThere are some discussions/comparisons that are missing.\n\n(1) Could you please explain the technical differences between the w/o Matching baseline and DPT? Equivalently, if we add a support set on DPT, how different is this from the proposed model?\n\n(2) The proposed architecture is not specifically designed for few-shot settings. In fact, if we compare the w/o Matching variant from Table 2 to the results of the baselines in Table 1, it does pretty well. This is good but also raises the question of how much the fast generalization is a result of the powerful architecture vs the multi-tasking meta-training. These additional experiments would shed light on this:\n(a) What is the performance of the proposed model if trained fully-supervised? Does it scale well when a lot of data is there? How does it compare to DPT? (b) How does out-of-the-box DPT work after few-shot finetuning?\n\n(3) From Table 2 it appears that fine-tuning is very important. This is fine as long as we care about the performance on the novel tasks only. But once finetuned, the model could forget the training tasks. This makes the approach less significant from the point of continual learning. Could you evaluate the forgetting, i.e. what is the performance on the training tasks after finetuning on the novel tasks?\n\n(4) Training for more tasks jointly is a harder job for the model but makes the dependency on the support set stronger. How important is that aspect for the proposed model? If you train on 5 tasks and test on 5 vs 8-2 vs 2-8, how does the performance change? It would be nice if you keep 2 novel tasks as fixed and then train the model on 2, 5 and 8 tasks and show how the performance on the two held-out tasks changes.\n\nQuestions:\n\n(a) Is the whole support set used in a forward pass? That's fine in 10-shot, but when you have 275 support images, do you still feed all of them at the same time? Would it make sense to finetune on 275 support images but use only a fixed set of them during inference? In that case this approach could scale to large datasets as well.\n\n(b) Why do you think that you need to finetune the model? Isn't the support set enough to provide the task description? Could somehow the support set predict the task-specific parameters to use?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "h30mvavXyBR", "reviewer": "Reviewer_arpe", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method for few-shot learning on wide varieties of novel of dense prediction tasks (not just semantic classes) on images.  The method applies meta-learning episodes to a vision transformer, trained to predict label tokens, which are decoded to final labels using a convolutional net.  The final attention step of the transformer is applied using features from support images as the keys, but encoded features (\"label tokens\") as the values; this effectively transfers labels from the support set to the query image, as combinations of support label tokens, which are subsequently decoded to final labels.  The system is evaluated using 10 tasks from the Taskonomy dataset, comparing against fully-supervised baselines as well as current few-shot semantic segmentation methods, finding excellent performance in both 10-shot and larger 200-shot settings, sometimes competitive with fully-supervised training in only the 10-shot setting.\n", "summary_of_the_review": "This is an excellent paper that describes a very effective approach to a new and interesting problem.", "strength_and_weaknesses": "This paper was clear, presented an interesting new problem and has excellent results.  I much enjoyed reading it.  Questions and comments I had are below:\n\n* What is the computational cost of the system, compared to a fully-supervised model (DPT) inference?  Given already-encoded support set features, what the cost to execute one query inference, including query image encoding and matching?\n\n* Was any loss or gradient balancing needed between task types (between the different loss measures), and how was this applied?\n\n* While the motivation of nearest-neighbor lookup is clear, I think the multiheaded attention is more sophisticated than simple NN on key to label tokens, particularly with multi-scale features, as it is here.  Since each head outputs a projection of values, which are linearly combined, based on a projection of keys and queries, different parts of different support images can be used for different components of the label token.  In particular, there can be differences in scale:  Low frequency parts of the token requiring wide contexts can match to one relevant set of support features, while high frequency edge transitions requiring local context *for the same query prediction and task* might match to other support features, even from support images having completely different global context or layout.  How much have you seen this occur, and how important is the number of heads in the MHA matching?  What other sorts of components have this behavior?  Some discussion of this would be nice.\n\n* intro sec 1:  \"the model learns the similarity in image patches that capture the similarity in label patches\":  This seems it would require a relatively large number of tasks of different types, in order to cover differently relevant ways of matching, and indeed the 80/20 5-fold cross-eval does seem to supply a fair number.  What about varying the number training task types --- in what ways does the method fail?\n\n* sec 5.2: discussion on why the method outperforms few-shot semantic segmentation baselines:  the two points here, while likely large factors for most tasks, don't explain the performance difference for semantic segmentation task, where the model described in the paper also performs better.  Do you have any ideas about this difference, and can it explain a large portion of the gains for other tasks as well?\n\n* Decomposing the problem to single-channel output tasks also may have a factorization effect, that stands to improve performance --- does it?\n\n* Are biases in the encoder tuned for each output channel separately, or jointly for all output channels in the test task (for multi-channel tasks like SN)\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "drydsWAO6qi", "reviewer": "Reviewer_hDLX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a few-shot multitask learner, Visual Token Matching (VTM), for dense prediction tasks. VTM is composed of four components including an image encoder which contains shared and separate parameters for different tasks, a label encoder which is kept the same across tasks, a matching module and a label decoder to read out desired dense predictions. The method is benchmarked on Taskonomy dataset and shows significant improvement over prior work of few-shot multitask learning. ", "summary_of_the_review": "Overall this is an inspiring paper with a well-designed and effective method for few-shot multitask learning on dense prediction tasks. Despite aforementioned limitations of the architecture and the training data, it is a work worth reading by a large community. I hope the authors can provide more thoughts regarding the weaknesses during the rebuttal phase. ", "strength_and_weaknesses": "Strengths:\n- The idea is novel and intuitive. I very much enjoy many designs of the architecture. Just to list a few:\n  - The image encoder has shared and separate portion of parameters across different tasks. Leveraging the bias-tuning idea, the few-shot learning optimizing biases should be very efficient.\n  - Multi-channel labels are split into single-channel ones. This design enables flexible joint learning of multiple tasks\n  - The token matching module provides a straightforward way of aggregating labels from the support set. It is also multi-resolutional to support learned aggregation on different abstraction levels.\n- The results are impressive. VTM outperforms few-shot learning baselines significantly. The qualitative results also appear to be better than baselines.\n- The paper is well-written and designs are well-motivated. \n\nWeaknesses:\n- Limitation on model architecture. The method requires a same architecture for all tasks. This design may lead to suboptimal architecture for different tasks. I don't see an easy way to enable flexibility of architecture for different tasks within the proposed paradigm.\n- Limitation on training data. I could be wrong on my understanding of the experimental setting but apparently in the meta training, labels of 8 training tasks are all used for each image example. Although Taskonomy supports such a setting with comprehensive labels, in the real world it could be too expensive to obtain so many dense labels for each image. Have you tried an alternative setting, where in the meta training stage, each training image only comes with one type of annotation? This setting will be more realistic and can be extended to real-world datasets by combining multiple datasets for different dense understanding tasks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%