TOIST: Task Oriented Instance Segmentation Transformer with Noun-Pronoun Distillation
Authors: Pengfei Li
Beiwen Tian
Yongliang Shi
Xiaoxue Chen
Hao Zhao
Guyue Zhou
Ya-Qin Zhang
Venue: NeurIPS 2022 Accept (2022)

Abstract: Current referring expression comprehension algorithms can effectively detect or segment objects indicated by nouns, but how to understand verb reference is still under-explored. As such, we study the challenging problem of task oriented detection, which aims to find objects that best afford an action indicated by verbs like sit comfortably on. Towards a finer localization that better serves downstream applications like robot interaction, we extend the problem into task oriented instance segmentation. A unique requirement of this task is to select preferred candidates among possible alternatives. Thus we resort to the transformer architecture which naturally models pair-wise query relationships with attention, leading to the TOIST method. In order to leverage pre-trained noun referring expression comprehension models and the fact that we can access privileged noun ground truth during training, a novel noun-pronoun distillation framework is proposed. Noun prototypes are generated in an unsupervised manner and contextual pronoun features are trained to select prototypes. As such, the network remains noun-agnostic during inference. We evaluate TOIST on the large-scale task oriented dataset COCO-Tasks and achieve +10.7% higher $\rm{mAP^{box}}$ than the best-reported results. The proposed noun-pronoun distillation can boost $\rm{mAP^{box}}$ and $\rm{mAP^{mask}}$ by +2.6% and +3.6%. Codes and models are publicly available.


TOIST: Task Oriented Instance Segmentation Transformer with Noun-Pronoun Distillation
Pengfei Li!
Beiwen Tian'
Yongliang Shi'
Xiaoxue Chen'
Hao Zhao Guyue Zhou' Ya-Qin Zhang!
AIR, Tsinghua University Pekknng University
3 IItel Labs
11-pf220mails
htt.:/mmass.sssssiiiiat.eee tsiighua, edu. cn
zhao-hao@pku edu hao zhao@intel.com
Abstract
Current referring expression comprehension algorithms can effectively detect segment objects indicated by nouns, but how to understand verb reference is still under-explored. As such, we study the challenging problem of task oriented detec- tion, which aims to find objects that best afford an action indicated by verbs like sit comfortably on. Towards finer localization that better serves downstream appli- cations like robot interaction, we extend the problem into task oriented instance segmentation. unique requirement of this task to select preferred candidates among possible alternatives. Thus we resort to the transformer architecture which naturally models pair-wise query relationships with attention, leading to the TOIST method. In order to leverage pre-trained noun referring expression comprehension models and the fact that we can access privileged noun ground truth during training, novel noun- noun-pronoun distillation framework is proposed. Noun prototypes are gen- erated in an unsupervised manner and contextual pronoun features are trained to se- lect prototypes. As such, the network remains noun-agnostic during inference. We evaluate TOIST on the large-scale task oriented dataset COCO-Tasks and achieve +10 +1.99% higher mA Pbox than the best-reported results. The proposed noun-pronoun distillation can boost m.A Pbox and mAPmaask by +2.8% and +3.8%. Codes and models are publicly available atthttss///gii www.bib-..//AR--DD..AA- wtt..////AAA-DSSSOVE//OEESEE...
Introduction
As benchmarked by the RefCOCO, RefCOCO+ referring expression comprehension models have seen tremendous progress, thanks to large-scale vision-language pre-training models like VL-BERT 57), VilBERT 43., OSCAR B3, UNITER [TO]
and RefCOCO-g datasets, noun
12-in-1 44 and MDETR [26]. As shown in the left top part of Fig. these algorithms take noun prompts like hatchback car as inputs and generate bounding box or an instance mask of that car. However, in real-world applications like intelligent service robots, system inputs usually come in the form of affordance (i.c., the capability to support an action or say a verb phrase). Whether modern vision-language model designs can effectively understand verb reference remains under-explored.
To this end, we focus on the challenging problem of task oriented detection, as introduced by the COCO-Tase benchmark [55]. As shown in the right top part of Fig. [ a task oriented detector outputs three boxes of forks as they can be used smear butter. We also extend the problem an upgraded instance segmentation version using existing COCO masks [40], as the masks can provide finer localization. When RGB-D pairs are available, instance masks can be used to obtain object point clouds. When image sequences are available, instance masks can be used to reconstruct objects using visual hull [3T 67]. As such, the newly proposed task oriented instance segmentation formulation (Fig. bottom) is useful for down-stream robot interaction applications.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

While noun referring expression comprehension datasets aim minimize ambiguity [45 an in-
hotchback car
teresting and challenging feature of task oriented the intrinsic ambigu- ity. For example, in the right top panel of Fig. [ the pizza peel can also be used to smear butter. If we have neither forks nor pizza peels at hand, is still possible to use the plate to smear but- ter. Another example is shown in Fig. bottom. When we consider an object to step on, the chair a better choice because the sofa is soft and the table is heavy to move. When the need switches to sit comfortably on, sofas are obviously the best candidates. In one word, objects that afford verb ambiguous and the algorithm needs to model preference.
detecction/semmenttioon
Rufuring experssion Grtorion@eegneetaaioo
Task otterned object detection
stop on someehun
comfortallyy someting
Prodiction
Groun Teush
To this end, current models 555 use two-stage pipeline, in which objects are firstly detected then relatively ranked. Inspired by the success of DETR-like methods [6] [4]] and the ad-
formulation
Figure 1: Left top: Noun referring expression com- prehension. Right top: Task oriented detection. Bottom: Task oriented instance segmentation.
vantage of the attention mechanism in revealing
the relationship between visual elements [34 64 we resort to the transformer architecture as imposes self-attention
object queries thus naturally models the pair-wise relative preference between object candidates. Our one-stage method is named as Task Oriented Instance Segmentation Transformer and abbreviated as TOIST. Transformers are considered be data hungry [1), but obtaining large-scale visually grounded verb reference data with relative preference (e.g.. COCO- Tasks (55)) is difficult. This inspires us explore the possibility of reusing knowledge in noun referring expression comprehension models. We propose to use pronouns like something as proxy and distill knowledge from noun embedding prototypes generated by clustering.
Specifically, we first train a TOIST model with verb-noun input (c.g., step on chair for the bottom panel of Fig. [ bottom), using the privileged noun ground truth. But during inference, access the noun chair, thus we train the second TOIST model with verb-pronoun input (e.g.. step on something) and distill knowledge from the first TOIST model. As such, the second TOIST model remains noun-agnosti during inference and achieves better performance than directly training model with verb-pronoun input. This framework is named nown-pronoun distillation. Although leveraging knowledge from models with privileged information has been used in robotics research like autonomous driving 7 and guadrupedal locomotion [32), the proposed paradigm of distilling privileged noun information into pronoun features is novel, to the best of our knowledge.
cannot
To summarize, this paper has the following four contributions:
We upgrade the task oriented detection task into task oriented instance segmentation and provide the first solution to it. Although this is natural extension, this new formulation is of practical value robotics applications.
Unlike existing two-stage models that firstly detect objects then rank them, we propose the first transformer mansformer-based method TOIST, for task oriented detection/segmmenttio.. It has only one stage and naturally models relative preference with self self-attention object queries.
In order to leverage the privileged information in referring expression understanding models, we propose a novel noun- coun-pronoun distillation framework. improves TOIST by +2.8% and +3.8% for mAphox and m.Pmask respectively.
We achieve new state-of-the-a (SOTA) results on the COCO-Tase dataset, out- out-performing the best reported results by +10.9% mAphoxx Codes and models are publicly available.
2 Related Works
Vision and Language. Connecting vision and language a long-existing topic for visual scene understanding. Barnard et al. [4] propose system that translates image regions into nouns. Babytalk

---

is an early method that turns images into sentences, based upon conditional random fields. Visual question answering [3] aims to answer questions about an image, with potential applications in
30
helping visually impaired people [1]] The CLEVR dataset [25] focuses on the reasoning ability of
question answering models, thanks to full control over the synthetic data. The Flickr30k benchmark
[5T] addresses the phrase grounding task that links image regions and descriptions. Vision-lane Sainnnlanguage
navigation [2][ aims to learn navigation policies that fulfill language commands, DALL-E [52]
shows impressive text-to-image generation capability. Video dense captioning [28) [TS] generates language descriptions for detected salient regions. Visual madlibs
[60] focuses on fill-the-blank question answering. Visual commonsense reasoning [62 proposes the more challenging task of
justifying an answer. Referring expression detection [1] 45] and segmentation (24] localize objects
specified by nouns. Although this literature is very large with many problem formulations proposed, the task of detecting objects that afford verbs (e.g.. COCO-Tasks($51 is still under-explored.
Action and A Affordance. Verbs, as the link between subjects and objects, have been extensively studied the both vision [58] 48 and language 23 40 communities before, while we focus on the
vision side. It is difficult to define standalone verb recognition tasks, so existing problem formulations depend on the focus on subjects or objects. A simple taxonomy can be considered as such: recognizing subjects and verbs is named as action recognition (56] 177; recognizing verbs and objects is named
as affordance recognition 666 [59 13 [9]; recognizing triplets is named as human-objie caenaan-ooject-inng object-interaction recognition 63] Task oriented object detection/segmmt serrrtiin/semmmtttioonnno as an affordance understanding task, is very challenging and this study explores the noun noun-pronoun distillation framework to borrow rich knowledge from more visually grounded noun targets.
Knowledge Distillation. This technique is proposed in the deep learning literature by Hinton et al. 22 to distill knowledge from large models to small models. The insight is that soft logit targets generated by large models contain richer information that better serves as a supervision signal than hard one-hot labels. Knowledge distillation has been extended to show effectiveness in other domains like continual learning 37 and object detection [8) The survey of Guo et al. [88 provides a comprehensive summary of knowledge distillation variants and applications. Most related to our
method is the privileged knowledge distillation methods in robotics research [7]],,, in which the teacher model has access to privileged information that the student cannot access during inference. Our noun- noun-pronoum pronoun distillation method is tailored for the task oriented snrection/segmentatiion problem which borrows rich knowledge from noun referring expression compression teacher models while still allowing the student model to be noun-agnostic.
3 Formulation
The problem is to detect and segment objects that are preferred to afford a specific task indicated by verb phrases, from input image. Yet clearly defining affordance and preference is actually challenging so we follow the existing annotation protocol of the COCO-Tasks dataset (55):
Affordance. Firstly, the target objects afford specific task. In an input image, it is possible that no objects or multiple objects afford the task. And in the latter case, the objects may belong to multiple classes. For example, in the right top panel of Fig I nothing affords the task sit confortably on. Instead, in the bottom panel, there are at least two sofas, a table and a chair that afford the task.
Preference. Secondly, we need to find the best ones from the objects which afford the task. In other words, the preference among multiple objects needs be understood. In Fig T the two sofas are obviously more suitable for the task sit comfortably on than other objects enumerated above. Thus the ground truth objects for this task are the two sofas (covered in blue).
Now we formally define the task. The input is an RGB image X, E x33xHaXW, (u represents visual) and a piece of text X, (1 represents language). X₁ describes specific task like sit comfortably on. The targets are bounding boxes Bgt = [b..... b E [0, 1 n2 x4 and instance segmentation masks Mg = [m,,... Mmg.] E I x Hly of target objects O₂t (Bgt. Mgx), where Ngat 2 0 is the count of targets. The four components of by E [0, 1] are normalized center coordinates, height and width of the i-th box. An algorithm f that addresses this problem works as:
f((X,, Xi)) predd Mpredd Spred),
(I)
3

---

Teacher
Eubsaling
Segeonn
FFN
Heed
Trarsfomee
Transformes Neaneer
hole
Tixil Enscater
TFN
1
with Skatibboard
Logi Hedd
Update
Text Feature Memory Bank
Soft Binary
Sclecur
Target Loss
Claster
Student
Cluster Loss
Dig hole with somsching
Toxt Inoonder
FFN
Teteet
Teanf
Transforme Decoder
COO
Segnonn
Eubeedling
Figure 2: TOIST network architecture and the noun-pronoun distillation framework. The cluster loss and soft binary target loss distill privileged noun knowledge and preference knowledge respectively.
where Bpred and M Mpred are predicted boxes and masks, respectively. Spred = [S1,..., , Snproal E [0, pped is the probability scores the predicted objects being selected, which reflects preference. We denote the predicted objects O pred
( pred: M Mpred, Spred)-
4 Method
We propose an end-to-end Task Oriented Instance Segmentation Transformer, abbreviated as TOIST (Section 4.1. Leveraging pre-trained referring expression comprehension models, we further adopt a teacher-student framework for noun-pronoun distillation (Section
4.1 Task Oriented Instance Segmentation Transformer
The SOTA method 55 uses a two-stage pipeline to solve the problem. Taking a single image X, input, it first detects the bounding boxes B Breed of all objects with Faster-RCNN [53]. Then it ranks Bpred with a GNN [36,, predicting the probabilities Spred of the objects being selected for a task. Our method differs from it in three ways: (1) We address the task with one-stage architecture, allowing joint representation learning for detection and preference modeling. (2) We specify tasks using the text X, (3) We predict instance masks M Mpred along with bounding boxes.
choose to build our method upon the transformer architecture [6] because the self eeffaattention operators in the decoder can naturally model pair-wise relative preference between object candidates. As shown in Fig 2 bottom, TOIST contains three main components: a multi-modal encoder (color brown) extract tokenized features, transformer encoder (color green) aggregate features of two modalities and a transformer decoder (color blue) to predict the most suitable objects with attention.
Two Input Forms. To find object that affords the task of dig hole, the first step is to construct a task description input X, To achieve this goal, we can extend the task name with the ground truth object category to the verb-noun form like dig hole with skateboard or with pronoun to the verb-pronout form like dig hole with something. While the former violates the noun-agnostic constraint during inference, it can be leveraged to improve the latter within the proposed noun-pronour distillation framework, which will detailed later. For plain TOIST, the verb-pronoun form is selected as task description X,, which is fed into the multi-mode encoder along with visual input X...

---

Multi-Modal Encoder. For X,,, pre-trained CNN-based backbone and a one-layer feed forward network (FFN) are leveraged to extract a low-resolution feature map F,, Rd Flattening the spatial dimensions of F,, into one dimension, we obtain sequence of tokenized feature vectors V = [ .., Un..] E R xd (light blue squares on the left of the transformer encoder in Fig 2 where n, = H x W. To preserve the spatial information, 2D positional embeddings are added to V. For X,, we use a pre-trained text encoder and another FFN to produce corresponding feature vectors L = [n,]] E R.uxdd (light orange squares where is the total count of language tokens. Among these ny features, we denotes the one corresponding to the pronoun (or noun) token as 4prou (or (noun) (dark orange squares ). We concatenate these vectors and obtain the final feature sequence [V, L] [V1,.... Un, 4 11..... 1ng] E R((
Transformer Encoder. The transformer encoder consists of fer sequential blocks of multi-head self-attention layers. Given the sequence of features [V, Ll,, it outputs processed feature sequence
[V", ["] = [u] ...., "H 11 ..... I E ((notm))xd The pronoun (or noun) feature 1pron (or (nou)) is encoded into 'pron (or 2 n.un), which will be used for noun-pronoun distillation (Section 4.2
Here for the plain TOIST, directly use the features ( L"] for later processing.
Transformer Decoder. The transformer decoder consists of Per blocks self-attention and cross- attention layers. It takes as input set of learnable parameters serving as an object query sequence Q = [q).-... Inpreal E "pee xd [v L are used as keys and values for cross-attention layers.
The outputs of the transformer decoder are feature vectors Q" = [gl" ..... "ef":....!9ppaa E "eeed xd which are projected to final results by three prediction heads. Specifically, the detect head predicts bounding boxes B pred and the segment head outputs binary segmentation masks Mpred- The logit head outputs logits Gpred g ................................ Enpreal E pred Xmmax where g, _oi..... : Irmman. ".f......9.. Ihyl corresponds text tokens L [l.-... bug E RRuxdd [iu+1>- -1 is used to pad g: to length maxx (by default mmax 2 and the last one gimnar stands for the logit of "no-object", With the output logits, we define the preference score 8; S pred of each predicted object as:
exp (
(2)
Tmaax j==1 exp (0)
During training, in DETR [6]. bipartite matching is computed between Mpred predicted objects O Opred and ground truth objects Ogt with the Hungarian algorithm [29]. The matched object pre- dictions supervised with L.1 loss and Generalized Intersection over Union (GIoU) loss [54] for
localization while Dice/F-1 loss [47) and Focal cross-entropy loss (39 for segmentation. We also adopt the soft-token prediction loss and the contrastive alignment loss used in MDETR [26). But different from them, we do not use a single noun or pronoun as the ground truth token span for matched object prediction. Instead, we use the whole verb-pronoun description token span such that the network can understand the verbs rather than noun/pronoun only. The total loss for TOIST is:
LTOIST = An + X2Liion + >aLdice + Aa + A5 + >gGaaignn
where >1-d6 are the weights of losses.
4.2 Noun-Pronoun Distillation
Motivation. As mentioned above, there are two possible input forms. Due to the privileged in- formation of target names (nouns), TOIST using verb-noun input out- performs its counterpart by 1.8% and +12.0% on mApir bbxx and mA maasi as demonstrated in Table. [ We use two pre- liminary experiments as motivation: directly replacing the pronoun feature Ipron or zrron in the verb-pronoun model with the correspond- ing noun feature Inown or fi the verb-noun model can boost performance (see TT) []. However, during inference, the noun of the ground truth
Table 1: TOIST quantitative results under several different settings related text.
Text related settings
mApbox
mApmmsk
verb-pronoun input verb-noun input
41.3
35.2
53.1 (+1188)
47.2 (+12.0)
replace 4pron with Inoun replace (tr (pron with from
43.7 (+2.4)
37.3 (+2.1)
41.9 (+0.6)
35.6 (+0.4)
noun-pronoun distillation
44.1 (42.8)
39.0 (43.8)
object is unavailable. Thus we believe a properly designed noun- pronoun distillation framework can leverage rich knowledge from the verb-noun model without violating the noun-agnostic constraint.
5

---

Distillation Framework Overview. Two TOIST models are trained simultaneously. The teacher (Fig[top) and the student [Fig(]bottom) take as input verb-noun and verb-pronoun descriptions. respectively. A clustering distillation method with a memory bank and tailored cluster loss is used to distill privileged object-centric knowledge from noun to pronoun (Fig middle left). Besides, we also use soft binary target loss imposed on G pred to distill preference knowledge (Fig 2 middle right), in which G pred are logits used to calculate preference scores Spred.
Clustering Distillation. Since one task can be afforded by objects of many different categories, we build text feature memory bank to store noun features, with which a prototype can be selected and used to replace pronoun feature and distill knowledge. We term this process as clustering distillation. Specifically, we use pron and naun instead of pron and houn for this process. The reason is that the former ones are conditioned on the image input and verb tokens of the task by self-attention layers, and thus it is meaningful to select a cluster center that suits the image and the task input.
Memory and Selector. The size of the memory bank is fttak X Amem X d. It consists of Deask queues of length mm for Reask tasks. During training, for each sample from task we update the j-th queue me [l...... 1ú I by adding the noun feature noun generated by the teacher model and removing the existing one closest to ftr noonn The updated queue is clustered with the K K-means clustering method, leading to K cluster centers L = {1e -ll,,... .......... L2 J. Then the student model uses cluster selector, which is implemented as the nearest neighbor classifier, prototype i L according to the pronoun feature 1 ppron and replace fpro with 12 Concate- nating other tokens and the selected prototype together, the output of student transformer encoder [o "......., ",, 1 " ..... 'rron.... s... is modified into [u " v 5. ................................ 12 ..... L and fed into the transformer decoder. To distill knowledge to the student transformer encoder, define cluster loss as:
select
Leluster pron 12
with which the privileged object-centric knowledge is distilled from clustered noun features to pronoun feature and further to the student TOIST encoder.
Preference Distillation. We use a soft binary target loss to distill preference knowledge from teacher to student. For an object query, first define binary query probabilities of being positive-query or negative-query. which denotes whether or not the query is matched to a ground truth object target, as p fppos, [eg]] E R1x2
The probabilities can be calculated by the softmax function:
Σ Rmaa exp (gs)
exp (g
ppos
E exp (9:) p neg
(5)
Timax
Thaa 2j=1 exp (9))
where Oj and Fomma represent the logits corresponding to i-th text token and "no-object" token, respectively. For all the object queries in teacher and student, the probability sequences denoted by P, [P...... pro...ca and P. = [P....... Pemprod- Then we use the Hungarian algorithm to
find a bipartite matching between the two sequences of object queries with P, and P.. Formally, we search for a permutation of Upred elements a E Grpred which minimizes the matching loss:
Alpred
arg min rE
Σ
Lmatch (Vt, Nem(e)) )
pree
where Vr, = =b Pt.) and b, is the predicted bounding box. Lmatth is a linear combination of box prediction losses (L1 & GloU) and KL-Divergence. The KL-Divergence LKL can be written as:
pass
nrg
(pt, ProCo) KL (pe.llPs.co)
poss
log
Pt.
+ Pt.
nag log
Pt.
With the optimal permutation o, we define the soft binary target loss as:
Tegieee
Diinary
LKL (Pe, Posco)-
(8)
It makes the binary query probabilities of the student model similar to the matched ones of the teacher model. And because the preference score & (Fq is defined the same way as the probability pPos (Eq 5) the preference knowledge is distilled from teacher to student as the loss decreases.

---

Table 2: Comparison of the proposed method to SOTA baselines on the extended COCO- Tasks dataset. The methods with tags and I are used to calculate mAP margins. TOIST with noun- pronoun distillation achieves best results: +10.9% and +6.6% mAP on detection and segmentation.
Object Detection
Instance Segmentation
Method
mApbox
Method
m.
m Amaak
Faster-RCNN
20.6
Mask-RCan RCNN
23.4
20.0
Faster-RCNN pick best
14.1
Mask-RCNN + pick best
18.8
16.8
Faster-RCNN ranker
Mask-RCNN + ranker
10.6
Faster-RCNN classifier
28.8
Mask-RCNN + classifier
33.7
29.2
Faster-RCNN GGNN
32.6
Mask-RCNN + GGNNT
37.4
32.4
Yolo + classifier
29.1
TOIST
41.3 (+8.1)
35.2 (+2.8)
Yolo + GGNN!
33.2
TOIST w/ distillation
44.1 (+10.99
39.0 (+6.6)
Summary. The final training loss function for TOIST with noun-pronoun distillation is:
LTOIST-NI C'OIST + LToIST + Ce As Aghiinary:
(9)
where '7 and As are the weights of losses. fTTOIST and LIOIST are separate TOIST loss terms defined by f teacher and student, respectively. Note that the cluster loss Latuster and the soft binary target loss Siniary are only used for supervising the student model.
As a reminder, during inference, we only use the student TOIST model and the fixed memory bank to find the most suitable objects, without violating the noun-agnostic constraint.
5
Experiments
Dataset, We conduct experiments on the COCO- Tasks dataset (55] which re-annotates the COCO dataset 440] with preference-awane affordance labels. This is the only dataset that involves instance- level preference in affordance. Though there are other datasets for affordance detection, they neither distinguish between instances involve preference, such ADE-Affordance [12] and IIT-AFF
[49] The COCO-Taske dataset contains 14 tasks. For each task, there are 3600 train images and 900 test images. In each image. the boxes of preferred objects (one or more) are taken as ground truth labels for detection. Using existing COCO masks, we extend the dataset to an instance segmentation version. In Appendix B and D we present more dataset details and show its diversity. Metric. We use the A.@0.5 metric for both detection and segmentation, where predicted preference scores Spred are used rank objects. Averaging the AP@0.5 values of all tasks leads mAP@0.5. The implementation details for TOIST and distillation can be found in Appendix A
5.1 Comparisons with State-of-the-art Methods
Table 2 shows
shows that TOIST with noun-pronoun distillation achieves state-of-the-arr results compared to existing methods on COCO-Taske For object detection, we use the results reported by (55] as baselines. For instance segmentation, following the same experiment settings of (55), we build new baselines using Mask-RCNN (20). The methods in the first row treat the problem as standard detection or segmentation task. All other baselines use two-stage pipelines, in which objects firstly detected or segmented then ranked. The proposed one-stage method achieves 41.3% mApbbox and 35.2% mApmask, which are +8.1% and +2.8% better than the previous best results Yolo+GGNM and
Noun- Noun-pronous pronoun distillation further boosts the performance of TOIST to 44.1% % (+10.9%) mApbox and 39.0% (+6.6%) pmask Our method also terrforrooms another
28
Ouste Number (b) Ablations for cluster number K. Figure 3: Experiments to study the impact of (a) self-attention and (b) cluster number.
Source Pralerncce Scores DDcooder Boo0
(a) Impuct self-attention.
7

---

baseline with the same backbone, as shown in Table of Appendix C These results demonstrate the effectiveness of the proposed method for the new problem of task oriented instance segmentation. Per-task quantitative results and precision-recall curves are provided in Appendix Caanndix(annn Cand D
5.2 Preference Modeling with Self-Attention
Protocol. Our design principle is that the self-attention layers in transformers can naturally model preference. But is this really the case? 3 (a) shows its effect. Two plain TOIST models are trained separately, with the only difference being that one model does not contain self-attention operators in the decoder. Note that the removal of self-attention does not impact the number of parameters. The mAP metric is impacted by two kinds of errors: inaccurate box/mask localization improper preference. To analyze the preference scores alone, for all object queries, we use the boxes and masks predicted by the last block of TOIST decoder, which are arguably the most accurate. We use the corresponding preference scores predicted by each block to calculate mAP values.
Interpretation. For the TOIST with self- the performance is gradually boosted as the source of preference scores becomes deeper: from 29.6% mAPbox and 25.0% mAPpaase to 41 41.3% and 35.2%. For the one without self-attention, the preference scores from the first block lead to the best performance: 33.9% mAPboxx and 28.7% mA which is -7.5% and -6.5% lower than another TOIST. The results demonstrate that the self-attention in TOIST decoder models pair-wise relative preference between object candidates. As the decoder deepens, the preference relationship between object candidates is gradually extracted by self-attention.
5.3 Effect of Clustering Distillation
Table 3: Ablations for distillation settings. CCR, CL and SBTL are short for cluster cen- ter replacement, cluster loss and soft binary target loss, respectively.
In Table we show the effects of using cluster loss and replacing pronoun features with cluster centers (noun prototypes). In (c) and (e), leveraging the two components alone brings an increase of +0.7% mAPbox, +1.9% mApmask and +0.7% mAPbox
Index
CCR
CL
SBTL
mApbox
mApinas
+1.8% mApmaak over baseline (a) respectively. In (g), the complete clustering distillation leads a performance improvement of +1.0% mAppbox and +2.3% mApmaskk These results show that the cluster-
(a)
X
41,3
35,2
(b)
X
V
(42.))))
38.0 (+2.8)
(c)
X
(2.0
37.1 (+1.9)
(d)
V
V
43.8
38.6 (+3.4)
(e)
X
X
(40.7
37.0 (+1.8)
(f)
42.3 (+1.0)
37.3 (+2.1)
ing distillation method can improve student TOIST
(g)
X
42.3 (+1.0)
37.5 (+2.3)
and enhance verb referring expression understanding.
(h)
44.1 (42.8)
39.0 (43.8)
In 4 we visualize the predicted results (filtered by a preference threshold of 0.9) and the attention maps of pronoun tokens. In the first row, when there is clustering distillation, TOIST wrongly prefers the flower to the cup, which is also confirmed by the attention map. But the TOIST with clustering distillation correctly selects the cup, and the attention on the flower is weakened. This shows that clustering distillation enables the student TOIST to reduce the ambiguity of verb-pronoun referring expression. In the second row, the bounding box of the knife is correctly detected by both two models. However, in the absence of the distillation, extra instance masks are predicted on the spoon and fork within the box. Instead, with the distillation, the masks predicted by TOIST are concentrated on the knife and the attention is more focused on it. This demonstrates that in the case of clustering distillation, TOIST can better ground the task into pixels within an object box.
Meanwhile, the fact that predicted masks may be inaccurate even if the box is correct makes it challenging for robot accurately grasp the preferred object when performing a specific task. This proves the importance of extending task oriented object detection to instance segmentation.
Input
TOIST wie Clustering Distillation
TOIST w/ Clustering Distillation
Ground Truth
place
open parcel
Figure 4: Visualization of the predicted results and attention maps of pronoun tokens.
8

---

5.4 Effect of Preference Distillation
TOIST w/o
TOIST w
Ground Truth
Preference Distilbation Preference Disiilation
In Table 3 (b), preference distillation with soft bi- nary target loss achieves +2.1 % mApbox and +2.8% mA pmpss higher results than baseline (a). This loss
pound
acts the preference probabilities of each object candidate in student TOIST. And the probabilities are used as scores sort the object candidates for the calculation of mAP values. Therefore, the result of Table 3 (b) strongly supports that the preference information is distilled to the student TOIST.
smear
butter
(3)
S Ole
S d Blub
Saxe
Cltb
parcel
A simple taxonomy differs three scenarios where preference distillation works. As shown in 5 the predicted results (filtered by preference threshold of 0.9) of the TOIST models w/ or w/o preference
Figure 5: Examples of three scenarios where preference distillation clearly works.
distillation are compared. (1) Preference distillation makes the preference score of the false positive object (the baseball in the left picture) lower than the threshold. (2) The preference score of the false negative object (the spoon in the middle) is raised above the threshold with the distillation. (3) When there is no distillation, the false positive object (fork) scores higher than the true positive object (knife) (0.9822 > 0.9808). Although the distillation fails lower the preference score of the false positive object below the threshold, its score is updated to be lower than the true positive one (0.9495 0.9680). These specific results demonstrate that the information of noun referring expression is distilled to the noun-agnostic student model in the form of preference scores.
5.5 Ablation Study and Qualitative Results
Distillation Methods. Instead of minimizing the distance between pron and 1/ a straightforward way is to directly minimize the distance between ftr pron and foom As shown in Table 4 this simplified method does not work well, which prompts to develop the distillation framework.
Table 4: Different distillation methods.
Table 5: Results without pre- tre-training.
Method
mApbux
mApamak
Method
mApbor
m.A pmaak
TOIST
41.3
35.2
verb-pronoun input
3.65
5.74
distill from C to pron
44.1 (+28)
39.0 (+3.8)
verb-noun input
11.19
12.67
distill from Inoun to Apeon
41.9 (40.6)
36.0 (40.8)
noun woun-pronoun distillation
7.43 (+3.78)
11.28 (+5.54)
Interaction of the Two Distillation Components. In Table 3 (d) and (f), we show the effects of cluster loss or cluster center replacement together with soft binary target loss. (d) achieves +2.5% mApbbx and +3.4% mApmask improvement, which demonstrates the two distillation losses collaborate well. (f) only achieves +1.0% mApbbox and +2.1% mApamakk improvement, slightly higher than (e) (using cluster loss only) but lower than (b) (using soft binary target loss only). This shows that preference distillation effectively improves object preference modeling. But solely replacing pronoun features to indicate target objects weakens the effect of preference distillation.
Ablations for Cluster Number K. Fig (b) shows the ablations for cluster number K. We perform dis- tillation experiments on different K values between I to 10 because increasing K to an even higher value makes the clustering task more difficult. All of the experiments yield better results than the plain TOIST (41.3% mbp 35.2% mApsask) and K 3 works the best. This demonstrates that a modest can better cluster the information of noun features and distill it to the student TOIST.
Table 6: Ablations for pronoun input.
Method
Pronoun
mAP
mA mApimall
something
41.3
35.2
41.3
35.2
TOIST
them
41.4
35.0
abed
39.0
33.2
something
44.1
39.0
TOIST w/ distillation
it 43.8
38.4
them
43.8
38.1
Ablations for Pronoun Input. Table 6 shows the results of TOIST with different pronoun input. the plain TOIST and TOIST with distillation, the usage of something, it or them leads to similar results,
abed
42.8
37.4

---

Table 7: Ablations for the task number Rrask on task oriented object detection.
Task
step on
sit comfortably
place flowers
get potatoes
water
Method
something
out of fire
plant
TOIST w/o dis
44.0
39.5
46.7
43.1
53.6
dis Plack = 14 46.2 (+2.2)
(+0.1)
49.9 (+3.2)
47.1 (+4.0)
54.5 (+0.9)
dis Prask = 5
46.4 (+2.4)
40.7 (+1.2
51.3 (+4.6)
46.8 (+3.7)
54.6
Ptask 1
47.0 (+3.0)
42.1 (+2.6)
50.8 (+4.1)
47.4 (+4.3)
(+1.6)
while meaningless string abed yields less improvement. Nevertheless, the proposed distillation framework can still work well in the last case, which demonstrates the robustness of our method.
Results without Pre-training. In our architecture, the pre-trained noun referring expression com- prehension models are leveraged. To investigate whether the noun-pronour distillation framework is standalone technical contribution, are trained from scratch on the COCO-Taske dataset and the results are shown Table [S] which demonstrates that the proposed distillation still improve performance even without pre-training.
conduct experiments without pre- tre-training. The models
A Ablations for Task Number. Table (oooss the ablation study of different task numbers, in which the first row corresponds to the plain TOIST without distillation and the others show the results with distillation under different Trask- The results demonstrate our proposed distillation works for different Tlakk even if Rrask = 1. And overall, smaller rtaak leads to better performance. We attribute this to the reduced problem complexity due to the less interaction between different tasks, which makes easier to improve the ability of the model to understand verbs through noun-pronoun distillation.
Qualitative Results. Fig.6 shows more qualitative results. In (a), two toilets are taken target objects and annotated partially or totally. But TOIST simultaneously predicts the two kinds of results for each toilet. (b). no object is annotated, while TOIST keenly detects two water bottles that afford the task. In (c). TOIST predicts more accurate mask result than ground truth. In (d), the table selected and interestingly does afford the task as the table edge can be used to open beer bottles. More qualitative results can be found in Appendix
Ground Truth Predietion
(6) sit comfortably
(b) place flwwess
(c) smair butter
(4) open beer
(c) pour sugar
(D dig hole
Figure Qualitative results proposed method for task oriented instance segmentation.
6 Conclusion and Discussion
We explore the problem of task oriented instance segmentation and propose a transformer- based method named TOIST with novel noun- coun-pronoun distillation framework. Experiments show method successfully models affordance and preference, achieving SOTA results on the COCO-Taske dataset. Limitations. Due the lack of large-scale datasets with more abundant tasks, TOIST is only evaluated on limited tasks. While this is sufficient for many robotics applications, it would be interesting to explore general verb reference understanding more tasks. Potential Negative Social Impact. Because TOIST not perfect, when is used in robotics applications, robots may have difficulty in selecting the most suitable object to carry out task or even cause damage.
References
[1] Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character-level language modeling with deeper self-attention. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3159-3166, 2019.
[2] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. likon-and-languuag navigation: Interpreting
10

---

visually-grounder grounded navigation instructions in real environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3674-3683, 2018.
[3] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitmick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433, 2015.
[4] Kobus Barnard, Pinar Duygulu, David Forsyth, Nando de Freitas, David M Blei, and Michael I Jordan. Matching words and pictures. Journal of Machine Learning Research, 3:1 107-1135, 2003.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference computer vision, pages 213-229. Springer, 2020.
[7] Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In Conference on Robot Learning, pages 66-75. PMLR, 2020.
[8] Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. Advances in neural information processing systems, 30, 2017.
[9] Xiaoxue Chen, Tianyu Liu, Hao Zhao, Guyue Zhou, and Ya-Qin Zhang. Cerberus transformer: Joint semantic, affordance and attribute parsing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19649-19658, 2022.
[10] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. European conference on computer vision, pages 104-120. Springer, 2020.
[11] German KM Cheung, Simon Baker, and Takeo Kanade. Visual hull alignment and refinement across time: A 3d reconstruction algorithm combining shape-from- from-silhouuttee with stereo. In 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings., volume 2, pages 11-375. IEEE, 2003.
[12] Ching- Yao Chuang, Jiaman Li, Antonio Torralba, and Sanja Fidler. Learning to act properly: Predicting and explaining affordances from images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 975-983, 2018.
[13] Thanh-Toan Do, Anh Nguyen, and Ian Reid. Affordancenet: An end-to-end deep learning approach for object affordance detection. In 2018 IEEE international conference on robotics and automation (ICRA). pages 5882-5889, IEEE, 2018
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2020.
[15) Xuguang Duan, Wenbing Huang, Chuang Gan, Jingdong Wang, Wenwu Zhu, and Junzhou Huang. Weakly supervised dense event captioning in videos. Advances in Neural Information Processing Systems, 31, 2018
[16] Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. Speaker- follower models for vision-and-languuge navigation. Advances in Neural Information Process- ing Systems, 31, 2018
[17] Rohit Girdhar and Deva Ramanan. Attentional pooling for action recognition. Advances in Neural Information Processing Systems, 30, 2017.

---

[18] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey, International Journal of Computer Vision, 129(6):189-11899 2021.
[19] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3608-3617, 2018.
[20] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[22] Geoffrey Hinton, Oriol Vinyals. Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint 1rXiv=1503. 150.022333.0 2(7), 2015.
02531,
[23] Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. Learning to solve arithmetic word problems with verb categorization. In EMNLP, volume 523533. Citeseer, 2014.
[24] Ronghang Hu, Marcus Rohrbach, and Trevor Darrell. Segmentation from natural language expressions. In European Conference Computer Vision, pages 108-124. Springer, 2016.
[25] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten. Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2901-2910, 2017.
[26] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulatee detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780-1790, 2021.
[27] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring objects photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787-798, 2014.
[28] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense- captioning events in videos. Proceedings of the IEEE international conference on computer vision, pages 706-715, 2017.
[29] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
[30] Girish Kulkarni, Visruth Premraj, Vicente Ordonez, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C Berg, and Tamara L Berg. Babytalk: Understanding and generating simple image descriptions. JEEE transactions on pattern analysis and machine intelligence, (5()))22891- 2903, 2013.
[31] Svetlana Lazebnik, Edmond Boyer, and Jean Ponce. On computing exact visual hulls of solids bounded by smooth surfaces. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, volume pages I-I. IEEE, 2001.
[32] Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47)xeabc5986, 2020.
[33] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for lasion-language tasks. In European Conference on Computer Vision, pages 121-137. Springer, 2020.
[34] Yang Li, Yucheng Tu, Xiaoxue Chen, Hao Zhao, and Guyue Zhou. Distance-aware occlusion detection with focused attention. IEEE Transactions Image Processing, 31:5661-5676. 2022.
12

---

[35] Yong-Lu Li, Xinpeng Liu, Xiaoqian Wu, Yizhuo Li, and Cewu Lu. Hoi analysis: Integrating and decomposing human-object interaction. Advances Neural Information Processing Systems, 33:5011-5022 2020.
[36] Yujia Li, Richard Zemel, Mare Brockschmidt, and Daniel Tarlow, Gated graph sequence neural networks. In Proceedings of ICLR' 16, 2016.
[37] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(1)):2 2947, 2017.
[38] Tsung-Y Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2117-2125, 2017.
[39] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[40] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[41] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. Dab-detr: Dynamic anchor boxes are better queries for detr. In International Conference Learning Representations, 2021.
[42] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint 190. 11692, 2019.
[43] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and -an---nd--aanuge tasks. Advances in neural information processing systems, 32, 2019.
[44] Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi- task vision and language representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10437-10446, 2020.
[45] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11-20, 2016.
[46] Diana McCarthy, Bill Keller, and John A Carroll. Detecting a continuum of compositionality in phrasal verbs. In Proceedings of the ACL 2003 workshop on Multiword expressions: analysis, acquisition and treatment, pages 73-80, 2003.
[47] Fausto Milletari, Nassir Navab, and Seyed-Ahmee Ahemad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourhh international conference on 3D vision (3DV), pages 565-571. IEEF, 2016.
[48] Tushar Nagarajan and Kristen Grauman. Learning affordance landscapes for interaction explo- ration in 3d environments. Advances in Neural Information Processing Systems, 33:2005-200 2015, 2020.
[49] Anh Nguyen, Dimitrios Kanoulas, Darwin G Caldwell, and Nikos G Tsagarakis. Object-based affordances detection with convolutional neural networks and dense conditional random fields. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1908-5915 IEEF, 2017.
[50] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
13

---

[51] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebmik. Flickr30k entities: Collecting eegion-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649, 2015.
[52] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. sero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[53] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[54] Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 658-666, 2019.
[55] Johann Sawatzky, Yaser Souri, Christian Grund, and Jurgen Gall. What object should use?-task driven object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recognition, pages 7605-7614, 2019.
[56] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. Advances in neural information processing systems, 27, 2014.
[57] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. V1-bert: Pre- training of generic visual-linguistic representations. In International Conference on Learning Representations, 2019.
[58] Heng Wang, Alexander Kläser, Cordelia Schmid, and Cheng-Lin Liu. Dense trajectories and motion boundary descriptors for action recognition. International journal of computer vision, 103(1):60-79, 2013.
[59] Xiaolong Wang, Rohit Girdhar, and Abhinav Gupta. Binge watching: Scaling affordance learning from sitcoms. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 1595,2605,,0 2017.
[60] Licheng Yu, Eunbyung Park, Alexander C Berg, and Tamara L Berg. Visual madlibs: Fill in the blank description generation and question answering. Proceedings of the ieee international conference on computer vision, pages 2461-2466 2015.
[61] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In European Conference on Computer Vision, pages 69-85. Springer, 2016.
[62] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6720-6731, 2019.
[63] Aixi Zhang, Yue Liao, Si Liu, Miao Lu, Yongliang Wang, Chen Gao, and Xiaobo Li. Mining the benefits of two-stage and one-stage hoi detection. Advances in Neural Information Processing Systems, 34, 2021.
[64] Hao Zhao, Ming Lu, Anbang Yao, Yurong Chen, and Li Zhang. Learning to draw sight lines. International Journal of Computer Vision, 128(5):1066-1100, 2020.
(65] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In International Conference on Learning Representations, 2020.
[66] Yixin Zhu, Yibiao Zhao, and Song Chun Zhu. Understanding tools: Task-oriented object modeling, learning and recognition. Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 2855-2866 2015.
14

---

[67] Xinxin Zuo, Chao Du, Sen Wang, Jiangbin Zheng, and Ruigang Yang. Interactive visual hull refinement for specular and transparent object surface reconstruction. Proceedings of the IEEE International Conference on Computer Vision, pages 2237-2245, 2015.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] See
htttps///iithbs Com/AAIR-DDSSOVVR//TSSSS
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix and and B
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See Appendix C
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix A..3
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]]
(b) Did you mention the license of the assets? [Yes] (a) MIT License for the COCO- Tasks dataset, (b) Creative Commons Attribution 4.0 License for the Microsoft COCO dataset. (c) Apache License 2.0 for MDETR, DETR and Mask-RCNN implemented by Detectron2.
(c) Did you include any new assets either in the supplemental material or as URL2 [Yes] See Appendix
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] See Appendix F
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [Yes] See Appendix F
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "jqXqWHtwgFj", "reviewer": "Reviewer_aXr4", "summary": "This paper aims at task oriented detection. Instead of specifying the type of object to detect, this problem requires detection of the objects that best fits the task description. The authors proposed a largely Transformer based model, TOIST, that outperformed the previous state-of-the-art. They then proposed two distillation techniques to distill the type of object into the student model, and performance is further boosted. Experiments are performed on COCO-Tasks. \n\nPOST-REBUTTAL UPDATE:\n\nI have read the authors' rebuttal. The authors added a lot of experiments to justify their design and showcase their big improvement over the previous baseline. However I don't think my non-result related questions are well-addressed, such as \"posing questions about the baseline and the dataset in general\". Overall I decided to slightly increase my score from 4 to 5. Regardless of the final result, I suggest the authors to simplify the proposed approach if possible, e.g. throwing away the notion of \"task\", which seems to deliver the best performance according to the new experiment.", "questions": "1. L169 mentioned that the text encoder is \"pre-trained\". What data is this text encoder pre-trained on? This is the reason why the pronoun in Table 4 makes a difference, right? What would the performance be if this part is trained from scratch? Does the distillation still work?\n\n2. The \"clustering distillation\" component requires the notion of \"number of tasks\", and a clustering algorithm is done for each task. However, I don't think the concept of \"task\" is well defined in this paper (e.g. in Section 3). Does \"task\" equal to verb, like \"dig hole\" is one task, and \"sit comfortably on\" is another? If so, how many training examples are in COCO-Tasks, and how many tasks? Dividing the former by the latter can give the reader a rough sense of how many training examples per task, and that will also inform how the number of clusters, K, ought to be chosen. \n\n3. Following the question above, Section 5.5 ablated the cluster number K. What about n_{task}? Does the distillation still work when n_{task} = 1, i.e. throwing away the notion of \"task\"?\n\n4. I do not understand why distillation has to be done the way it is in \"clustering distillation\". What about loading the same data for Teacher and Student, and simply distill l_{noun} into l_{pronoun} (say in the same way as Equation 4)? This gets rid of introducing \"number of tasks\" and \"memory bank\", which greatly simplifies the proposed method.", "limitations": "The authors used 3 lines in Conclusion to talk about limitations. I feel it can be expanded to talk about some of the angles in my Questions section above.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The base model, TOIST, is fairly well motivated and well described.\n- The performance gain over the previous state-of-the-art seems significant.\n- The idea of distilling the object type into \"something\", i.e. distilling noun into pronoun, is interesting and novel in my opinion.\n\nWeaknesses:\n- The distillation version of TOIST may be a bit overly complicated, and I do not understand why it has to be designed the way it is.\n- The baseline [48] is more than 3 years old, and as someone who is not extremely familiar with COCO-Tasks, it is concerning that there is no follow-up works in 3 years, posing questions about the baseline and the dataset in general.\n\nOverall, I think originality is fairly good; quality, clarity, significance is medium.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "htj_uPRyp7G", "reviewer": "Reviewer_mBEm", "summary": "In order to handle the affordance recognition task, this paper proposes a Task-Oriented Instance Segmentation Transformer (TOIST) to find objects that best afford an action indicated by verbs. The TOIST is a teacher-student knowledge distillation model, and such a model leverages the referring expression comprehension algorithm as the teacher module for guiding the student module to learn the noun-pronoun transformation. The experiments show the positive effect of the knowledge distillation mechanism.", "questions": "The tackled affordance recognition task is not a well-explored research topic; hence, the compared baseline method [48] is not advanced. In order to demonstrate the performance gain of the proposed TOIST, it is better to train the model without using the extra training data and comparing it with the baseline of an advanced backbone network, for example, ‘mdetr+GGNN.’ Please see [Weaknesses] for reference.", "limitations": "The authors described the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths] \n+ The idea of utilizing the referring expression comprehension algorithm as the teacher module is interesting.\n+ The manuscript is well organized and has several interesting analyses.\n\n[Weaknesses] \n- The contribution to upgrading the task-oriented detection into task-oriented instance segmentation upon the `existing’ transformer model is weak.\n- Though the existing method [48] is a two-stage model, yet the proposed TOIST needs to separately fine-turn the pre-trained student and teacher TOIST models with a final knowledge distilling. It seems that the pre-trained models employ the extra data for training, and hence the extra training data and the extra training procedure make the advantage of the claimed one-stage model somewhat weak.\n- Compared with the state-of-the-art methods, it is not fair to compare the methods extracting features with different backbones. It is interesting whether the ‘TOIST w/ distillation’ in table 2 can still surpass the baseline with the same backbone, i.e., ‘mdetr+GGNN’?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PDgf8lpXW4", "reviewer": "Reviewer_u6fT", "summary": "Authors propose a novel way to do task oriented object detection. Dataset used is COCO Task. Author modify the backbone used in DETR model to feed the transformer encoder with textual features along with image features so that better contextualized representations are obtained. The loss function is optimized to perform accurate bounding box localization along with instance segmentation. The paper leverages knowledge from a model trained with verb-noun captions (using the ground truth nouns) to train a student model with verb-pronoun caption. This way the model still remains noun-agnostic during inference time. But it can detect the noun from verb-pronoun if it was trained properly.", "questions": "1. How many unique pronouns are used in the captions to train the student model? How many objects in Coco Task dataset? Is the coco task dataset captions modified in any way (like replacing objects with pronouns) to train the student model?\n2. It would be good to see an analysis of what verbs are associated with what objects (comparing ground truth and model predictions). Something like a distribution plot that the verb \"sit on\" was associated with \"chair\" in 10 out of 20 times, it was associated with \"table\" in 5 out of 20 times etc. That will indicate if the model fails for any verbs more frequently than others.\n3. How is the score s_i is used in loss function in equation 3? Is it used in localization loss terms or segmentation loss terms? Or is it used in some other way?\n4. How is default value of n_max = 256 decided? What is the value of npred? I am assuming it should be greater than the total number of objects in CoCo dataset. Is that the case?\n5. Ablations for including vs not including the loss terms L_token and L_align?\n6. It is not clear what G_npred means in line 245. Please define what it stands for.\n7. L_match was a loss term in DETR model to encourage matching the class ad bounding boxes of ground truth and prediction. \nHowever it is not included in loss functions here (equations 3 and 9). Why is L_match not used in loss function? In line 247, authors mention KL divergence is also a part of L_match. However, the original DETR paper doesnt mention that.\n8. Instead of minimizing the distance between l_pron_tr and l_cs_j in equation 4, why can't one minimize the distance between l_pron_tr and l_noun_tr for knowledge distillation?", "limitations": "Limitations are briefly discussed in \"Conclusion\" section.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": " Strength:\n 1. Authors propose a novel way for task oriented detection by introducing verb pronoun captions and leveraging knowledge distillation to learn from noun ground truth.\n 2. The paper presents state of art results on the said task.\n 3. Ablation provided to show utility of the distillation components which is one of the novelties of the paper.\n\nWeakness: \n1. The memory bank is a queue and is updated in a FIFO fashion. However, this might lead to removal of a noun feature not adequately represented in the rest of the list. Wouldn't it make more sense to update the queue by removing elements in a smarter way to reduce occurrence of similar features? For example, for any new object feature remove one of the past object features whose representation is closest to it.\n2. Some ablation studies regarding why some loss terms are useful are missing.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept