Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a method to learn closed boundaries on top of the final layer of a classification model. The authors propose to optimize the classifier with an appropriate loss adjusted by the miss-classified examples within and outside the boundaries. The authors, extend state-of-the-art approaches in three datasets. The results show some improvement in all the cases.
strength_and_weaknesses: - The paper is easy to follow and intuitive. Explanations of the different parts is adequate.
- The authors present several related works also and do some comparisons.
- Results seem to be ok even though only on three datasets.

Regarding weakness one of the first comments is the lack of other approaches that treat the same problem. The authors cite several related works in related cases (OOD detections, anomalies etc) but do not do any comparisons. I mean, the fact that they enhance few SOTA approaches is already good but I cannot assess how this pushes the current state-of-the-art methods in the specific problem.

Lack of statistical tests to evaluate significance. In some cases improvements are quite small. Maybe you could add them?

I am curious if any other baseline like for example to consider the universum class as a side task would work here.
clarity,_quality,_novelty_and_reproducibility: Quite good presentation of the material and as well as the motivation. There is some novelty in the loss function and the authors provide enough details even though I would encourage them to add more material regarding the implementation/setting in the Appendix.
summary_of_the_review: Generally the paper is easy to read and flows well. I have some difficulty to assess the improvements that this approach brings. Based on the presented results the proposed approach seems to help improving the models that it extends.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies how to appropriately handle the universum (or miscellaneous) class in several common NLP classification problems, such as the "non-entity" label in NER, "no-relation" label in RE. The motivation of the paper is that these universum classes can have a wide coverage of cases whose semantics vary a lot and thus should not be treated as one category as a whole. That is to say, the decision boundary of the universum class should be in different shapes from other target classes with specific semantics. The solution proposed by the paper is to learn closed boundaries for target classes, and regard the remaining points that lie out of the boundaries of any target class as the universum case. The concrete implementation is to leverage GMM to estimate the class boundaries, and use thresholds over class-conditioned generation probabilities to classify whether a data point belongs to any target class or it should be put into the universum label. The method is evaluated on three different tasks instantiated with six SOTA models and demonstrates improvements over conventional open-boundary learning.
strength_and_weaknesses: Pros:
* The motivation of the paper is clear. The proposed solution is simple and intuitive.
* The method is empirically shown to improve different models on several different tasks.

Cons:
* The first major concern is regarding the correctness of the motivation. While the motivation for special treatments of the universum class is clear and intuitive, it's not convincing to me whether the limitations of conventional multi-class classification as pointed out in this paper actually matter that much. Figure 1 illustrates that the decision boundaries learned by conventional classification models are "open", which makes it hard to distribute the universum class instances accurately. However, I doubt if this is really the case -- in the context of using deep neural nets (DNNs) as the feature encoder, the data representations (i.e., $\boldsymbol{h}$) and decision boundaries ($f(\boldsymbol{h})$) are jointly learned, so even if the universum class instances have quite different semantics, it should not be hard for DNNs to map them into one compact cluster in the last layer's representation space. In other words, the limitations illustrated in Figure 1 (c) could be a problem for simple models like logistic regression or SVM, but do not seem to be a challenge for DNNs which can learn arbitrary non-linear functions for data representations.
* The second major concern is the additional parameters and hyperparameters introduced by the proposed algorithm. Firstly, the GMM parameters need to be fit using target-class data -- when the amount of labeled data is small for some classes (e.g. few-shot learning or imbalanced learning), the GMM obtained may not be accurate/stable, and this limitation has not been discussed in the paper. Secondly, a threshold value ($\xi$) needs to be set for each class. It seems to be a hyperparameter that cannot be automatically learned, but the paper also didn't discuss how it can be appropriately set. The authors also didn't show how sensitive the model is to these threshold values. This is quite concerning because it's impossible to manually set threshold values for each class, especially when the number of classes is large (e.g., in fine-grained NER/RE). 
* The paper is unclear in many technical details. For example, Section 3.3 mentions using N-pair loss for pretraining, but does not explain the motivation/reason. Table 4 seems to show that the pretraining step is not useful, making the purpose for pretraining very confusing. Figure 3 studies the impact of last layer dimension and reports that the optimal value is 7. This is quite confusing because the common dimensionality of pretrained models are over 700; a dimension of 7 seems way too small for the last layer dimension. Did you use another linear layer on top of the last BERT/RoBERTa encoder layer to convert the high-dimensional vector into lower-dimenisonal ones (e.g., 7-10 dimensions) before performing classification?
* The performance improvements of the proposed method over conventional classification setting are quite marginal (in most cases less than 1 point). The paper also didn't report error bars/stds or significant testing results to validate whether the improvements are meaningful. 

---
**Post-Rebuttal Updates**:  
I'd like to thank the authors for their provided responses and paper updates to address my concerns raised above. The original paper version had many unclear technical details which made assessing the contribution of the paper quite difficult. However, in the authors' rebuttal and updated paper version, many details have been further clarified, and I found them largely strengthened the contribution. Detailed comments are as below:
* The difficulty of learning with heterogeneous samples in the universum class has been clarified. I see that the miscellaneous patterns from the universum class that do not appear in training but in test time could be an issue for traditional classification models. However, I feel that some concrete case studies would help (e.g., some embedding plots of the actually learned classification boundaries by the proposed method); Figure 1 does not seem to be the actual plots but looks more like conceptual illustrations.
* The learning process of GMM (especially how the threshold values are learned) and its limitations (e.g., hard to adapt to few-shot learning) have been better articulated.
* The motivation for pretraining looks clearer to me.
* Glad to see the added significant tests and the new OECC baseline.

Overall, I found the paper to have studied an interesting problem (i.e., handling the universum class in classification problems), and the method proposed to be reasonable, resulting in good effectiveness. However, I feel that the application scenarios of the method are somewhat narrow (i.e., there are many classification problems in NLP that do not have universum classes at all) and the method also has inherent limitations of not being applicable to low-data regimes (e.g., few-shot learning). Although it does not appear to be a very strong and suitable paper for ICLR, I also don't want to gatekeep it. Hence, I'm recommending a weak accept.
clarity,_quality,_novelty_and_reproducibility: * Clarity: The motivation is clear, but many technical details are not explained sufficiently/clearly (see cons #3 above).
* Quality: The major assumption made by the paper regarding why/how conventional treatments of the universum class could be an issue is not convincing (see cons #1 above) 
* Novelty: The proposed approach is relatively new under the studied topic.
summary_of_the_review: While the paper studies an interesting issue in several NLP problems regarding how to appropriately handle the universum (or miscallaneous) class, the major motivation seems not well-justified in the context of using DNN encoder models. There are also concerns regarding the stability/generalization ability of the model due to the additionally introduced parameters and hyperparameters, and the lack of studies for these. Many technical details have not been sufficiently discussed and thus remain confusing. The empirical advantage over standard classification baseline is quite marginal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper addresses the universum class (negative class) problem in NLP tasks, by modelling the class boundaries with Gaussian mixture models, and learning thresholds with a boundary learning loss on misclassified examples. They showed that their approach outperforms vanilla classifiers for 3 different NLP tasks.



strength_and_weaknesses: Strengths:
- the approach shows that it improves performance on 3 different NLP tasks, by simply learning GMM and threshold values of the positive classes.

Weaknesses:
- There is little analysis to show how the learned boundaries can better model class boundaries. A detailed analysis of precision/recall would be useful, and whether the approach improves multi-class decision boundaries, or only the boundary with the negative class.
- The proposed approach emphasises on misclassified positive samples, which could have a similar effect on work that optimises F1-measures, see e.g., [1]. 

Update after author rebuttal:
- The authors have addressed some of my concerns mentioned above. Hence I have increased the score to 6.

[1] Ye, N., Chai, K. M., Lee, W. S., & Chieu, H. L. (2012). Optimizing F-measures: A tale of two approaches. In Proceedings of the 29th International Conference on Machine Learning (pp. 289-296). Omnipress.
clarity,_quality,_novelty_and_reproducibility: Clarity: 
- I am unable to follow the justification in section 3.5 that arrives at the proposed p(h_k,U) in Equation (4). 
- In section 3.6, the boundary learning loss is optimised over wrongly classified samples. Are these misclassified examples of the training set, or validation set? An analysis of how this loss function improves F1 while also optimising for cross-entropy could be useful to help understand the benefits of the approach.

Quality:
- the arguments are hard to follow, and there is little analysis on the results to support the arguments.

Novelty:
- the work is novel in addressing the universum boundary.

Reproducibility:
- the authors mentioned they will release the code on GitHub.



summary_of_the_review: I would not recommend acceptance as I think the arguments made by the authors are not supported by sufficient analysis of the results. The authors could also compare against approaches that directly optimises for f1-measures.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: To specifically address the universum class for NLP tasks (i.e., the outside/rejection/miscellaneous class), the authors propose to learn closed boundaries for the known classes on top of the final layer of an (arbitrary) classification model by optimizing the decision boundary with a GMM-based solution for each class with a targeted misclassification loss. Based on the resulting model, the authors improve over SotA results on the six datasets by explicitly mitigating the standard open-boundary assumption.

The consensus strengths regarding this work include:
- The universum class is widely used in NLP tasks (and many general ML tasks), but hasn’t received significant explicit attention in these contexts. Thus, the authors are addressing an understudied problem.
- The proposed method is intuitive and can be easily applied to many settings to achieve a ‘free’ empirical gain in many cases.
- The paper is well-written overall (and the paper significantly improved during the rebuttal).
- The authors provided significant new results during the rebuttal period to strengthen the paper and address open issues.

The reviewer (and my own) identified weaknesses for this work include:
- As the authors are aware, closed boundary learning approaches appear in many ML settings (e.g., OOD, anomaly detection, outlier detection). It also appears in class imbalance settings (and other cases). It would strengthen the paper to survey these some and make a clear contrast to the proposed work. For example, another approach is to perform k-NN on the universum class and treat this as the closed set (assuming mixtures of spherical-based distributions) — and I expect some such ideas would surface in such a survey that may introduce more closely comparable baselines. Specifically, section 2 should be more concise, precise, and complete.
- Some of these existing works in other setting include analytical analysis that seems translatable to this problem to help with a theoretical analysis. As it stands, the paper is largely empirical and makes some claims that the reviewers didn’t feel were well-supported (i.e., the motivation is in 2D spaces).
- There remain open questions around some technical details and sensitivity analysis of hyperparameters.
- While the empirical gains are ‘free’ in that a practitioner can apply the approach without requiring significant problem specialization, they aren’t universally notable (even if they are statistically significant).
- The new results improve the technical strength of the paper significantly, but the interpretation of the overall empirical results is more disjoint now as the new results should be better integrated.

Overall, the reviewers agree that this is an understudied problem (at least within the NLP community) and there is an opportunity to modify existing algorithms to get consistent improvements with minimal effort — thus, potentially impactful. Additionally, the reviewers appreciated how much better the paper became during the rebuttal period. However, there remain concerns regarding contextualizing this work within non-NLP techniques for this problem (including possible theoretical analysis) and more interpretation/discussion of all the empirical results in the paper.

Additional note: This submission was also discussed with the SAC during final decisions. My assessment is that the original submission was a confident reject and the improvements during the rebuttal period make it a borderline case. The technical content is now strong enough to warrant consideration for acceptance, but the analysis/interpretation/contextualization would need to be improved to make it a clear accept. The SAC, program chair, and myself agreed upon this decision after reading all of the reviews, rebuttals, and discussion -- and then discussing amongst ourselves.
justification_for_why_not_higher_score: Basically, the paper needs another round of work to improve contextualization wrt existing work, discussion of empirical results, tying up a few loose ends, and overall integration of these results into the narrative.
Decision: Reject