Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes QAID, Question Answering inspired Intent Detection system, which models the intention detection classification as a question-answering task. The model uses two stages of training: a pretraining for better query representation and finetuning on few-shot labels of query and answers (name of intents). Certain choices of model, such as token-level similarity, batched contrastive learning, and inference with answers only, are verified by detailed ablation studies. The proposed method achieves SOTA  performance on three intention detection dataset from DialoGLUE.
strength_and_weaknesses: Strength: 

- The paper uses and adapts techniques from QA and answer retrieval to the task of ID. The method is effective and efficient on three benchmark datasets when compared with various baselines.
- The detailed ablation study verifies the necessity of stages and component of proposed framework.
- The paper is well-written and easy to follow.

Weakness:
- Given the number of intents being only a few hundreds at most, is Faiss really necessary for inference?
- n has two meanings in section 2.1: one for batch size and another for number of tokens.
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is clearly written.
Quality: The statements are well supported. The empirical results are solid.
Novelty: The paper adapts the method from question answering to intent detection. The adaptation itself is novel although the proposed methods are a combination of existing techniques.
Reproducibility: It requires some effort to reproduce the results.
summary_of_the_review: The paper is inspired by the development of ColBERT in QA task and propose a variant of the model for the task of intent detection. The proposed model makes several adaptations including batch contrastive loss and  signal from the intent names. The proposed method achieves SOTA performance on three few-shot ID benchmarks. And the ablation study proves the necessity of various components in the system.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper reformulates intent detection as a retrieval task, by treating utterances and intent names as questions and answers respectively. The authors leverage a dual-encoder based retrieval architecture for few-shot intent detection, with late-interaction scores (as used in ColBERT) and batch contrastive training. 
strength_and_weaknesses: Strengths:

Formulating as a retrieval task helps make use of current SOTA techniques in retrieval such as late interaction scores and batch contrastive loss. The idea of using a retrieval-based dual encoder for this task is interesting from an efficiency perspective too, since the intent names (which are the answers) can be pre-computed during inference. This overcomes the high latency of cross-attention based approaches for which the processing time scales with the number of intents.

Weaknesses:

1) It's not entirely clear why augmentation is needed and what is the motivation for / benefit from using the data augmentation module

2) Since CPFT seems to be most similar to this paper, the authors need to highlight better what the main differences are with CPFT. From reading the paper, the impression I get is that the authors just incorporate the few-shot intent detection methodologies used in CPFT (such as self-supervised contrastive pre-training and fine-tuning) into a dual-encoder based retrieval formulation (by using ColBERT)

3) It is not surprising to see benefits from using late-interaction scores and batch contrastive training (which is one of proposed contributions). ColBERT/ColBERT-V2, which are based on late-interaction scores, have already been shown to outperform cosine similarity based matching in dual encoder models. Further, batch contrastive loss is already a popularly used training technique for dual-encoder based retrieval models (such as DPR, RocketQA)
clarity,_quality,_novelty_and_reproducibility: The writing in the paper can be improved with a better description of the two-stage contrastive training subsection. (Some terminologies in there, such as anchor/pivot, are hard to understand). A more comprehensive schematic illustration (compared to Figure 1) will help. Further, adding some qualitative examples showing why the proposed changes show improvements can help strengthen the motivation.

The main originality in the paper is the reformulation intent detection as a retrieval task. Other proposed contributions are mainly about combining ideas from retrieval literature with those recently proposed for few-shot intent detection in the CPFT paper.

summary_of_the_review: Overall the paper shows good (and statistically significant) improvements over existing methods for few-shot intent detection. However, the novelty in the paper is limited as it mainly involves combining existing techniques in a retrieval-based formulation (such as batch contrastive loss and late-interaction scores) with methodologies for few-shot intent detection proposed in CPFT (such as self-supervised contrastive pre-training and fine-tuning)
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a Question-Answering inspired Intent Detection System (QAID). The paper treats the intent identification task as a Question-Answering retrieval task by treating the utterances and the intent names as queries and answers respectively. QAID adapts ColBERT architecture from prior work and replaces the loss function with batch contrastive loss. The novelty of this work is in converting intent id task to Question-Answering form and using the intent names as answer, and using supervised batch contrastive training for finetuning LM on this task. 
strength_and_weaknesses: Strengths: 

1. The paper reports SOTA results in three benchmark intent identification tasks for both 5-shot and 10-shot setting. 

2. The paper shows ablations to support the claims on supervised batch contrastive training and late interaction scores. 

3. The paper is well written and easy to understand 

 
Weakness: 

1. The ablations can be improved. The paper can show ablation to understand the component of each component and the impact it has on the intent id task. The missing ablations are: QAID – batch contrastive,  QAID – intent names (not sure if this is possible) and QAID – data augmentation. But it would be good to show results by removing one component at a time. 
clarity,_quality,_novelty_and_reproducibility: 1. Where are the ablation results for "data augmentation". The paper says that data augmentation results are inconsistent but the results are not shown in Table 3. 
summary_of_the_review: Overall, this paper builds on top of prior work (especially ColBERT and CPFT) but uses intent names for retrieval. The paper shows the benefit of combining these techniques and presents SOTA results and shows good ablations.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this work, the authors reformulate the problem of intent detection as a question-answering task by treating utterances and labels as questions and answers. A two stage training schema is employed by utilizing question-answering retrieval architecture and batch contrastive loss. The first of the training stages is for learning better query representations, while the second stage is for optimizing the contextualized token-level similarity scores between queries and answers from the same intent. State-of-the-art results are reported on three few-shot intent detection benchmarks.

strength_and_weaknesses: Strengths

S1: It is an interesting idea to treat ID problem as a question-answering task.

S2: Utilization of Token level domain knowledge in a self-supervised fashion along with queries adaptation to dialogue domain is interesting.

S3: Masked language modeling has also been employed as auxiliary loss.

S4: Intent names have been employed for generating extra supervision signals.

S5: There are detailed experiments in this work, with an interesting insight that ColBERT triplet loss with batch contrastive loss leads to a considerable improvement.

S6: The paper is well written and easy to follow.


Weaknesses
W1: Why a balanced dataset is the requirement for this work. If the dataset is imbalanced, how will the proposed method behave?

W2: Typos: In abstract: “self supervise manner.”

clarity,_quality,_novelty_and_reproducibility: The paper is well written and the results are good. This work is for balanced few-shot datasets. It will be interesting to see if the proposed techniques work on imbalanced datasets as well.

summary_of_the_review: Overall, this is an interesting paper and addresses the challenging intent detection task in a question-answering setting.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors propose "Question-Answering inspired Intent Detection" (QAID), which frames (few-shot) intent detection as a QA pair of [utterance=question] and [answer=intent names] and builds on recent methodological advances from the QA community to achieve state-of-the-art few-shot intent classification results on widely-used benchmark datasets. Specifically, they adapt the ColBERT architecture by replacing triplet loss with batch contrastive loss, first augmenting the data with widely-used corruption techniques (MLM and dropping representation/inner-neuron elements) and using this data to perform LM fine-tuning and supervised training on the intent names -- noting that the encoder is used on the intent names during supervised training to take advantage of 'dataless' classification advantages. Since this is a QA inspired system, they also use the common approach of a dual encoder with 'late interaction scores' via FAISS-based retrieval. Using the DialoGLUE intent detection data (CLINC150, BANKING77, HWU64) and several recent baseline systems, they show that they establish a new SoTA on these datasets. Secondly, they perform ablation studies to show the relative contribution associated with each subcomponent.

The consensus strengths of this work include:
- QAID builds on recent notable improvements to build a few-shot intent detection algorithm, proposing non-trivial modifications specific to aspects of intent detection (e.g., intent name, suitable data augmentation to support contrastive learning).
- The primary experiments exhibit SotA few-shot intent detection on the widely-used benchmark datasets for this task and the ablation studies support the claimed improvements in the paper.
- The paper is well-written, well-contextualized, and easy to follow (especially when reading some fo the background work).
- The proposed solution is methodological in nature, but ostensibly would easily scale to commercial solutions if desired.

The weaknesses discussed in the reviews (and my own reading) of this work include:
- Since this work relies heavily on recent QA work and conceptually uses methods similar to CPFT for training, there were some concerns regarding novelty. I believe this was addressed in the rebuttal and encourage the authors to include these in the final revision.
- There were some detailed technical questions in the reviews -- that were sufficiently answered in the rebuttal.
- The modifications are somewhat specific to intent classification (or at least to 'dataless classification' settings where the class label can be projected into a suitable embedding space). Thus, it isn't clear that this will 'back-translate' to improve QA nor inspire similar works. This isn't a criticism as it isn't the point of the paper, but does indicate limited scope -- and that it may be more suitable for a NLP-specific venue as opposed to a more ML venue.

Overall, the consensus is that this is a work that is 'correctly inspired' by recent QA methods adapted using recent intent classification works to achieve SotA results that are likely scalable all the way to industry settings. Overall, this is a elegant, easily understood approach and the paper is well-executed. As an intent detection work, this will very likely be a notable baseline regarding future work. Thus, we lean toward acceptance. That being said, the results are likely unique to intent classification and might be of lesser interest to the broader ML community.
 
justification_for_why_not_higher_score: The primary concern is that this work is very specific to intent classification, which is an important problem, but not necessarily of wide interest. In principle, this work could be expanded to other settings where scale is important and labels have meaning -- which would likely increase the potential impact of the work.
justification_for_why_not_lower_score: Intent classification is a NLP problem of significant interest and potential impact -- and the solution is easy to understand and performs well. Additionally, the ablation studies do a good job at supporting the claims of this paper.
Decision: Accept: poster