Kati8/ICLR_2023_2025 huggingface dataset:
authors: Mostafa Mushsharat
authors: Nabeel Mohammed
authors: Mohammad Ruhul Amin
reviews: 
Comments: This paper proposes a “sandbox” framework for text classification. In this framework, the LLM is kept frozen. The input texts contain both the problem input and the textual descriptions of labels (”concept operator words”, or cop-words). This frozen LLM computes the cosine similarities between the input texts and the cop-words. These similarity scores are then passed into an aggregate function, leading to the prediction result.

With experiments on multiple text classification datasets and LLM models, this paper finds that this sandbox network generally improves the accuracy, compared to a baseline (direct text classification without cop-words).

This paper shows evidence that the model’s secondary incentives do not match human decisions.
Strengths: - The explanation and alignment of models is an important problem, and the idea to do model explanation and/or alignment by routing the models through text descriptions of concepts is an interesting one.
- Through experiments, this paper shows the proposed approaches have better performances than baselines.
- There are some ablation studies showing the effects of different configurations of cop-words.
Weaknesses: - How the proposed method is relevant to behavioral alignment is unclear. As far as I understand, model alignment requires adapting/training the models in some ways, but the proposed framework in this paper keeps the LLM frozen.
- The experiments are not strong enough to support the goal indicated in the title of this paper, i.e., “discovering spurious concepts”. It is unclear to me how the experiments are relevant to *spurious* concepts (the cop-words only describe the concepts).
- The experiments are also not strong enough to reveal the additional benefits in model explainability. I was expecting some quantitative or qualitative results comparing the “model’s secondary incentives” and “human decisions”, but these are not clear from the experiments.
- Crucial components in the framework are not described. How the cop-words are determined is not elaborated (I can see some sporadic descriptions, e.g., “defined using description documents”, but what documents are these description documents?)
    - Relatedly, this paper spends the space lavishly describing some details that are not central to the methodological contributions, e.g., how the components of the frameworks can be written in equations.
- There are multiple typographical errors throughout this paper. Many sentences can’t be parsed. Several citations have incorrect formats (e.g., citep vs citet). I think the readability can significantly benefit from a round of careful proofreading.
Recommendation: 1: strong reject
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 1 poor
Clarity: 1 poor
reviews: 
Comments: The paper introduces a neural sandbox framework designed for text classification. The framework is supplied with specific concept words (cop-words) related to the classification labels, alongside the input text. The system then optimizes an operator that classifies the text based on the relevance of these concept words with the input text.

The paper highlights the framework's potential as an evaluative tool to uncover any biases or spurious behaviors. This is achieved by introducing "foreign" or unrelated concept words and observing the model's reactions, allowing for an understanding of the model's underlying learned concepts and potential biases. Comparing the model’s chosen concepts for a specific input text with human-selected lower-level labels, it also offers insight into the model’s decision-making based on the provided concept words.

In addition, the paper demonstrates that introducing this neural sandbox network improved text classification accuracy and macro f1 scores across multiple datasets when compared to a standard baseline.
Strengths: 1.The evaluation method involving "foreign" cop-words is an insightful approach to understanding and identifying spurious correlations, biases, or other undesired behaviors in the model.

2.The framework’s ability to not only classify text but also provide insights into the model's decision-making process using cop-words is a notable advantage. This could bridge the gap between model interpretability and performance.
Weaknesses: 1.The framework's overview can be described as a retrieval methodology. In this method, databases comprise concept words or descriptions that match high-level labels. Models then infer by measuring the similarity between the input text and these database entries. From this point, the proposed method seems to have limited novelty.

2.The presentation of mathematical formulas lacks clarity, making it challenging to understand and replicate the research. I recommend revising the mathematical sections with clearer definitions and examples for better comprehension and accessibility.
Recommendation: 3: reject, not good enough
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: This work introduces a neural sandbox framework for text classification using LLMs by leveraging defined label concepts.
The framework takes a frozen LLM as a vector embedding generator for input text and label concepts and then optimizes an operator to classify the input text based on the relevance scores to the label concepts. The experiments on IMDB, GoEmotion and Amazon Hierarchical Text  Classification show that the framework outperforms three simple classifier baselines.
Strengths: The approach attempts to provide more conceptual explanation, which differs from traditional inherent methods used for explaining text
classification models via attention scores.
Weaknesses: * The major issue to me is the unpair comparsion with the LLM classifier baselines.
For current LLMs, prompts and in-context learning play a key role in their success. Baselines do not have access to the information about label concepts, while the proposed approach has access to them. It is suggested to use those label concepts as at least system prompts along with the input text to feed into the baselines.
* The presentation requires improvement in terms of writing quality and organizational structure.
Recommendation: 3: reject, not good enough
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 1 poor