Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: High Potential (HP): a submission which meets the reviewing criteria and has potential to make an impact on the field
review_content: summary_of_contributions: This work investigates combining different embeddings from PLMs to generate semantically rich text representations, towards the goal of improved hate speech detection. The authors experiment with various embeddings and fusion techniques; present findings that combining embeddings yields marginal improvements with computational tradeoffs.
strengths_and_weaknesses: ### Quality:
(+) This study is clearly motivated with a thorough literature review; the technical details of this study are detailed, sound and easily reproducible.  
(-) The authors have obtained interesting results but have not elaborated on why they came to be. While this could be due to the page limits, I would be interested in e.g. why the performance benefit of combined representations is only "marginal"? Is it due to internal structures (within embeddings) being destroyed, or perhaps redundant information?  

### Clarity:
(+) The paper is well-organised with a coherent, logical flow. Figures and tables are well-formatted and nicely support the message of the work.

### Significance: 
(+) This paper applies techniques from representation learning to tackle an important societal problem tied to NLP. The proposed method is very well suited to the current state of NLP research - where we have numerous pretrained LLMs each producing meaningful representations - I appreciate the idea of trying to make better use of these "off-the-shelf" embeddings.
suggested_changes: I enjoyed reading this paper and would like to get the authors' perspectives on the following:  
1. Have you explored non-linear or learning based (projecting to a shared latent space before fusion) approaches to combining embeddings?  
2. How do you think combining embeddings (embedding space) compares to ensembling approaches?
review_title: null

final_score: rating: Great Start (GS): a submission which meets some of the reviewing criteria but has room for improvement
review_content: summary_of_contributions: The goal of the study is to ascertain what combination of pretrained language models is most effective in hate speech detection. The paper experimentally tests various combinations such as addition, interleaving, multiplication, concatenation, and random interleaving. The paper concludes that all these combinations, except for random interleaving, perform roughly at the same capacity. 
strengths_and_weaknesses: Strengths: 

Hate speech detection is an important and timely topic of study. The methodology used in the paper is quite reasonable. 

Weaknesses: 

I think the paper could greatly benefit from attempting to explain the experimental findings. Is there any hypothesis the authors might have that supports the observations? What obstacles do the authors foresee in scaling up their experiments? Do the current techniques work on larger datasets (or, what are the size limitations)? 
suggested_changes: Please see the list of weaknesses above.
review_title: null

final_score: rating: Clear, Correct, and Reproducible (CCR): a submission which meets the reviewing criteria
review_content: summary_of_contributions: In this paper, the authors focus on the problem of hate speech detection. To this end, this paper analyzes various combination techniques for pre-trained language models and critically evaluates their effectiveness. The accuracy has been shown to marginally increase for all methods except random interleaving.
strengths_and_weaknesses: **Strengths**
* S1: The idea, experiments and results have been described clearly
* S2: Experiments have been thoroughly performed and presented in order to evaluate the combination techniques.
* S3: The hyperparameters used for experimentation, data preprocessing steps as well as the description of embedding combination methods have been provided making the work reproducible.

**Weaknesses**
* W1: The reason for conducting experiments with the chosen five embeddings has not been explained. Why were some combination methods, such as taking average of the two vectors, not considered? 

suggested_changes: Minor grammatical errors can be improved. For example,in the abstract, there should be a comma after the word "However."
review_title: null

Meta-review: metareview: Overall, the paper investigates the effectiveness of combining different embeddings from pre-trained language models (PLMs) for hate speech detection. The authors experiment with various embeddings and fusion techniques and present findings that combining embeddings yields marginal improvements with computational trade-offs. While the paper is well-organized with a coherent flow and sound technical details, there are some weaknesses that could be addressed. The authors do not elaborate on why the performance benefit of combined representations is only marginal and have not explained the reason for conducting experiments with the chosen five embeddings. Additionally, the paper could benefit from attempting to explain the experimental findings, such as exploring non-linear or learning-based approaches to combining embeddings and comparing embedding space to ensemble approaches. Despite these weaknesses, the paper presents a clear, correct, and reproducible study that has the potential to make an impact on the field.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Invite to present