Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: Great Start (GS): a submission which meets some of the reviewing criteria but has room for improvement
review_content: summary_of_contributions: This paper presents a method for combining a human lexical-semantic model with LLM-generations to generate high-quality semantic feature norms
strengths_and_weaknesses: ### Strengths ###
- Follows basic requirements: The submission meets formatting requirements and the page limit.

### Weaknesses ###
- Clarity: The methods mentioned in this paper are similar to retrofitting embeddings, but instead of retrofitting to a word sense or task, the LLM embeddings/feature space is retrofitted to a human-generated feature. It would be useful to mention prior retrofitting literature on this and show additional baselines to compare this method to prior techniques.
- Correctness: The claims and conclusions are partially justified by the findings. They can be reinforced if a comparison with other baselines is added.

suggested_changes: - It would be useful to mention prior retrofitting literature on this and show additional baselines to compare this method to prior techniques.
review_title: null

final_score: rating: Clear, Correct, and Reproducible (CCR): a submission which meets the reviewing criteria
review_content: summary_of_contributions: The paper proposes a method to improve on raw LLM-generated features by doing matrix completion. Results show that the proposed method mostly improve over the raw features.
strengths_and_weaknesses: Strengths:

* This is a very timely idea. Since human annotation is often the most costly step of a learning process, using LLMs in innovative ways to help with the process is an interesting problem.
* The SVD-based approach seems to be robust in the sense that even with high levels of data withholding, it gives reasonable improvement over the raw LLM-generated features.

Improvement scopes:

* It's not reported how the number of features vs number of concepts affect this result. For example, if #features < #concepts, how well would the logistic regression, and in turn, the full process perform?
* I understand why the current SVD + logistic regression based method is a great start. But it's well worth thinking of improved methods for getting the revised features. In particular, maybe use of somewhat deeper models and techniques for self-supervised learning to get embeddings. Specially since the output of the logistic regression seems akin to embeddings already.

suggested_changes: The paper seems well-written and self-contained for the most part, as a tiny paper should be. The only change I would suggest is to add some definition/intuition about the measure of agreement, $d'$, in the main paper.
review_title: null

final_score: rating: High Potential (HP): a submission which meets the reviewing criteria and has potential to make an impact on the field
review_content: summary_of_contributions: The authors propose a technique to combine the powerful but less accurate LLM generated norms with the restrictive but highly accurate human lexical-semantics to efficiently generate high-quality feature norms. The authors tested this combination on 129 animals and 166 tools concepts. 
strengths_and_weaknesses: - Clarity: The paper was easy to follow and relevant related works were cited. 

- Correctness: 
    - There is a huge difference in performance between concepts. What does -ve delta d mean in figure 1.c?
    - It is unclear whether the proposed techniques work only for one type of concept. Delta d with animals concept is much higher. 

- Reproducibility: No code is provided but I see enough details in Appendix to reproduce the analysis.

- Follows basic requirements: yes

suggested_changes: - Specify what is deltaD and what values it can take and which way is better.
- I would also like to see some examples of the LLMs-generated concepts using the proposed technique make sense.
review_title: null

Meta-review: metareview: This work studies an interesting problem of combining human effort and LLM behavior. Overall, this paper is timely and the idea is technically sound. The presentation is good. The reviewers have provided various suggestions to make the experiments and analysis more solid.







confidence: 4: The area chair is confident but not absolutely certain
Decision: Invite to archive