Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: High Potential (HP): a submission which meets the reviewing criteria and has potential to make an impact on the field
review_content: summary_of_contributions: The paper introduces a new method for learning contextual word embeddings (CWE) that uses a simple and transparent attention mechanism derived from the SVD-based Syntagmatic Word Embeddings. The authors test their model on the Word-in-Context dataset and show that it outperforms the simple but tough-to-beat bag-of-words baseline by a substantial margin.
strengths_and_weaknesses: Strengths:

The paper proposes a new method for learning contextual word embeddings that uses a simple and transparent attention mechanism derived from syntagmatic word embeddings.
The paper provides a clear and concise explanation of the proposed method.
Their approach to the use of SVD-based Syntagmatic Word Embeddings for obtaining word associations shows promise for future research.  

Weaknesses:  

The paper does not compare the proposed method with state-of-the-art contextual word embedding models like BERT, RoBERTa, and GPT-3.
The paper does not provide any insight into the computational complexity of the proposed method or how it scales with the size of the input corpus.
The paper lacks thorough experimentation with different hyperparameters and evaluation of their impact on the performance of the proposed method.
suggested_changes: Further experiments and results are needed to solidify their approach.
review_title: null

final_score: rating: Clear, Correct, and Reproducible (CCR): a submission which meets the reviewing criteria
review_content: summary_of_contributions: The paper presents a method to create contextual word embeddings using syntagmatic embedding based attention. The CWEs are evaluated on the WiC dataset and shown to perform better than the BOW baseline.
strengths_and_weaknesses: ### Strengths ###
- Clarity: The authors have clearly described their methodology of embedding creating using syntagmatic embedding association-based attention and shown preliminary results on the WiC dataset with improvements over the BOW baseline
- Follows Basic Requirements: The paper meets formatting requirements and the page limit.

### Weaknesses ###
- Correctness: Although the authors claim that their method is 'effective' for learning contextual embeddings, they do not evaluate their model on additional WSD datasets to find whether their method can capture a coarser difference in word meaning, although this is a minor weakness for a tiny paper.


suggested_changes: none
review_title: null

Meta-review: metareview: A new method for learning contextual word embeddings (CWE) using Syntagmatic Word Embeddings based attention is introduced. The CWEs are evaluated on the WiC dataset and shown to perform better than the BOW baseline.

Strengths: New approach, clear and concise explanation, and preliminary results show improvement over baseline. 

As a “Tiny” paper, this work fits well. Looking forward to seeing the next stage of this research, especially when compared to transformers etc. 


confidence: 4: The area chair is confident but not absolutely certain
Decision: Invite to present (notable)