Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Authors propose a novel way to do task oriented object detection. Dataset used is COCO Task. Author modify the backbone used in DETR model to feed the transformer encoder with textual features along with image features so that better contextualized representations are obtained. The loss function is optimized to perform accurate bounding box localization along with instance segmentation. The paper leverages knowledge from a model trained with verb-noun captions (using the ground truth nouns) to train a student model with verb-pronoun caption. This way the model still remains noun-agnostic during inference time. But it can detect the noun from verb-pronoun if it was trained properly.
strengths_and_weaknesses:  Strength:
 1. Authors propose a novel way for task oriented detection by introducing verb pronoun captions and leveraging knowledge distillation to learn from noun ground truth.
 2. The paper presents state of art results on the said task.
 3. Ablation provided to show utility of the distillation components which is one of the novelties of the paper.

Weakness: 
1. The memory bank is a queue and is updated in a FIFO fashion. However, this might lead to removal of a noun feature not adequately represented in the rest of the list. Wouldn't it make more sense to update the queue by removing elements in a smarter way to reduce occurrence of similar features? For example, for any new object feature remove one of the past object features whose representation is closest to it.
2. Some ablation studies regarding why some loss terms are useful are missing.

questions: 1. How many unique pronouns are used in the captions to train the student model? How many objects in Coco Task dataset? Is the coco task dataset captions modified in any way (like replacing objects with pronouns) to train the student model?
2. It would be good to see an analysis of what verbs are associated with what objects (comparing ground truth and model predictions). Something like a distribution plot that the verb "sit on" was associated with "chair" in 10 out of 20 times, it was associated with "table" in 5 out of 20 times etc. That will indicate if the model fails for any verbs more frequently than others.
3. How is the score s_i is used in loss function in equation 3? Is it used in localization loss terms or segmentation loss terms? Or is it used in some other way?
4. How is default value of n_max = 256 decided? What is the value of npred? I am assuming it should be greater than the total number of objects in CoCo dataset. Is that the case?
5. Ablations for including vs not including the loss terms L_token and L_align?
6. It is not clear what G_npred means in line 245. Please define what it stands for.
7. L_match was a loss term in DETR model to encourage matching the class ad bounding boxes of ground truth and prediction. 
However it is not included in loss functions here (equations 3 and 9). Why is L_match not used in loss function? In line 247, authors mention KL divergence is also a part of L_match. However, the original DETR paper doesnt mention that.
8. Instead of minimizing the distance between l_pron_tr and l_cs_j in equation 4, why can't one minimize the distance between l_pron_tr and l_noun_tr for knowledge distillation?
limitations: Limitations are briefly discussed in "Conclusion" section.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In order to handle the affordance recognition task, this paper proposes a Task-Oriented Instance Segmentation Transformer (TOIST) to find objects that best afford an action indicated by verbs. The TOIST is a teacher-student knowledge distillation model, and such a model leverages the referring expression comprehension algorithm as the teacher module for guiding the student module to learn the noun-pronoun transformation. The experiments show the positive effect of the knowledge distillation mechanism.
strengths_and_weaknesses: [Strengths] 
+ The idea of utilizing the referring expression comprehension algorithm as the teacher module is interesting.
+ The manuscript is well organized and has several interesting analyses.

[Weaknesses] 
- The contribution to upgrading the task-oriented detection into task-oriented instance segmentation upon the `existing’ transformer model is weak.
- Though the existing method [48] is a two-stage model, yet the proposed TOIST needs to separately fine-turn the pre-trained student and teacher TOIST models with a final knowledge distilling. It seems that the pre-trained models employ the extra data for training, and hence the extra training data and the extra training procedure make the advantage of the claimed one-stage model somewhat weak.
- Compared with the state-of-the-art methods, it is not fair to compare the methods extracting features with different backbones. It is interesting whether the ‘TOIST w/ distillation’ in table 2 can still surpass the baseline with the same backbone, i.e., ‘mdetr+GGNN’?

questions: The tackled affordance recognition task is not a well-explored research topic; hence, the compared baseline method [48] is not advanced. In order to demonstrate the performance gain of the proposed TOIST, it is better to train the model without using the extra training data and comparing it with the baseline of an advanced backbone network, for example, ‘mdetr+GGNN.’ Please see [Weaknesses] for reference.
limitations: The authors described the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper aims at task oriented detection. Instead of specifying the type of object to detect, this problem requires detection of the objects that best fits the task description. The authors proposed a largely Transformer based model, TOIST, that outperformed the previous state-of-the-art. They then proposed two distillation techniques to distill the type of object into the student model, and performance is further boosted. Experiments are performed on COCO-Tasks. 

POST-REBUTTAL UPDATE:

I have read the authors' rebuttal. The authors added a lot of experiments to justify their design and showcase their big improvement over the previous baseline. However I don't think my non-result related questions are well-addressed, such as "posing questions about the baseline and the dataset in general". Overall I decided to slightly increase my score from 4 to 5. Regardless of the final result, I suggest the authors to simplify the proposed approach if possible, e.g. throwing away the notion of "task", which seems to deliver the best performance according to the new experiment.
strengths_and_weaknesses: Strengths:
- The base model, TOIST, is fairly well motivated and well described.
- The performance gain over the previous state-of-the-art seems significant.
- The idea of distilling the object type into "something", i.e. distilling noun into pronoun, is interesting and novel in my opinion.

Weaknesses:
- The distillation version of TOIST may be a bit overly complicated, and I do not understand why it has to be designed the way it is.
- The baseline [48] is more than 3 years old, and as someone who is not extremely familiar with COCO-Tasks, it is concerning that there is no follow-up works in 3 years, posing questions about the baseline and the dataset in general.

Overall, I think originality is fairly good; quality, clarity, significance is medium.
questions: 1. L169 mentioned that the text encoder is "pre-trained". What data is this text encoder pre-trained on? This is the reason why the pronoun in Table 4 makes a difference, right? What would the performance be if this part is trained from scratch? Does the distillation still work?

2. The "clustering distillation" component requires the notion of "number of tasks", and a clustering algorithm is done for each task. However, I don't think the concept of "task" is well defined in this paper (e.g. in Section 3). Does "task" equal to verb, like "dig hole" is one task, and "sit comfortably on" is another? If so, how many training examples are in COCO-Tasks, and how many tasks? Dividing the former by the latter can give the reader a rough sense of how many training examples per task, and that will also inform how the number of clusters, K, ought to be chosen. 

3. Following the question above, Section 5.5 ablated the cluster number K. What about n_{task}? Does the distillation still work when n_{task} = 1, i.e. throwing away the notion of "task"?

4. I do not understand why distillation has to be done the way it is in "clustering distillation". What about loading the same data for Teacher and Student, and simply distill l_{noun} into l_{pronoun} (say in the same way as Equation 4)? This gets rid of introducing "number of tasks" and "memory bank", which greatly simplifies the proposed method.
limitations: The authors used 3 lines in Conclusion to talk about limitations. I feel it can be expanded to talk about some of the angles in my Questions section above.
review_title: null

Meta-review: metareview: The paper proposes a Task-Oriented Instance Segmentation Transformer (TOIST) approach for finding objects that best afford a verb-indicated action, to handle the affordance recognition task. TOIST proposes two approaches of teacher-student knowledge distillation — it leverages the referring expression comprehension algorithm as the teacher module for guiding the student module to learn the noun-pronoun transformation. Experiments on Coco tasks show the gains of the proposed approach.

All the reviewers accepted the paper, however there were multiple suggestions that would be good for the authors to address. Reviewer aXr4 recommended simplifying aspects of the approach further, and recommended using a more recent baseline. Reviewer u6fT suggested adding some more ablation experiments and asked for clarifications in the loss function formulation. Reviewer mBEm suggested using a different baseline, and had concerns about the proposed method using extra training data compared to the baselines.

Based on the feedback provided by the reviewers, we recommend this paper for publication at NeurIPS 2022. We thank the authors for addressing some of the comments of the reviewers in their original review and subsequent author feedback period. The authors seem to have reported new results and addressed the concerns/feedback from the reviewers in the rebuttal period -- it would be good to include these additional results and discussions as much as possible in the updated paper/supplemental materials.
confidence: Certain
Decision: Accept