Iterative Context Vectors: Boost In-Context Learning within Activations
Authors: Yiting Liu, Zhi-Hong Deng


Reviewer paper summaries: The paper proposes iterative context vectors  (ICVs) that improve in-context learning performance by modifying model activations on classification tasks.
Summary of strengths: The proposed method achieves good results in Table 1
Summary of weaknesses: - First, I'm not sure if it's a naming problem or a writing thing, but 2.2 Theoretical Foundation wasn't as theoretical as I expected and it feels more like a related work section to me.
On the other hand, I'm not familiar with Appendix B and Dai et al. 2023 and I can't figure it out based on the descriptions in 151~161. Are these crucial to your method? If so, I feel like it's probably better to make it a related work section and clearly define some terminologies, e.g., "meta-graidents".
- The descriptions of the proposed method (2.3) are unclear. What do you mean by extracting the meta-gradients?  Why simulate the gradients? What do you do in each iteration?
- Another concern is about novelty: the proposed method seems very similar to prior work: Hendel et al. 2023, Todd et al. 2023, and Liu et al. 2023. I know you've cited them in the paper, but I'm not very convinced how this paper stands out from prior methods
- Most datasets used in this paper (Table 1, Table 3) are homogeneous (from tweet_eval)
Comments, suggestions and typos: - (minor) Is there any reason you call the demonstrations an episode E? It seems uncommon in the ICL context. Or you're using the term from meta-learning?
- hyperparameters are not discussed when they first occurred in Eq4; I only know what it means after line 223
- What's the data size of the Train and Val set?
Overall assessment scores: 2