Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: Great Start (GS): a submission which meets some of the reviewing criteria but has room for improvement
review_content: summary_of_contributions: The author proposes a way to modify transformer architecture by replacing key and query in transformer architecture with MLP
strengths_and_weaknesses: Strength:
* Author proved through loss function that his theory performed well compared to transformer loss for train and test set\
 Weaknesses:
* Though the good performance of the model on train and test set the author has to perform analysis to different datasets to come to conclusion or use the same dataset that were used to train the transformer in order to come up sustainable theory.
suggested_changes: * More metrics need to be put in place other than loss function
* More description of the dataset used
* More description of the used transformer
review_title: null

Meta-review: metareview: This paper proposes a modification to the Transformer architecture by replacing the dot product of query and key vectors with an MLP to compute attention weights directly from the embeddings. The author demonstrates that this modification improves the performance of the model on NLP tasks.

However, the work has several weaknesses that need to be addressed. The lack of context and details make it difficult to understand the proposed method, and the author needs to revise the paper to improve its clarity and reproducibility. The author needs to provide a detailed description of the proposed method, including how to calculate the attention weights using an MLP, which dimensionality the MLP applied to the representation, how to causally mask the output of an MLP, and the computation/FLOPs used in the new MLP attention, etc. The author also needs to explain the dataset and training settings to ensure the reproducibility of the work.

Overall, this work has potential but requires significant revision to meet the Clarity, Correctness, and Reproducibility (CCR) criteria. Therefore, the paper is recommended as ``Invite to Revise''.
confidence: 3: The area chair is somewhat confident
Decision: Invite to revise