In-Context Learning at Representation Level via Unlabeled Texts
Authors: Yunpeng Tai, Juntao Li, Yibin Chen, Min Zhang


Reviewer paper summaries: This paper proposes a representation-based method to improve the in-context learning performance of language models. The proposed method compresses the input representations as prefixes and pool representations to do the prediction.
Summary of strengths: The aspect of exploring the ability of in-context learning is interesting: This paper proposes that there are two gaps between pretraining and inference, and then defines two gaps and proposes methods to tradeoff these gaps for better performance.
Summary of weaknesses: My concerns are mainly about the motivation.

First, the validity of the two gaps. Even if this work provides some new insights, I'm not sure if this is valid. For example, to introduce the two gaps, the authors say "The goal of LLM pretraining is to maximize the likelihood of each next token given its preceding context tokens, while ICL inference forces LLMs to predict the output of downstream tasks conditional on the given demonstrations that are not involved in pre-training.". However, recent ICL explanation papers show that ICL works mainly because the models recognize the task and use the pretrained knowledge to perform the task. It may not be appropriate to assume that ICL could work for newly proposed tasks when given demonstrations. Thus, I'm not sure if these two proposed gaps are valid. And if the following analysis/improvements are meaningful.

Second, the motivation of the proposed representation-based ICL paradigm. People use ICL since: it’s hard to handle parameters/representations of LLMs, and there are only a few labeled examples. However, the proposed method handles the representations directly, making it difficult to adopt this method to LLMs such as GPT models. Second, the method needs to get representations for all data in the training set (instead of just using several ICL examples). This further undermines the advantages of ICL.  Therefore, the meaning and usefulness of the proposed method remain unclear.
Comments, suggestions and typos: See weakness
Overall assessment scores: 2.5