ICL CIPHERS: Quantifying "Learning'' in In-Context Learning via
  Substitution Ciphers
Authors: name: Zhouxiang Fang, name: Aayush Mishra, name: Muhan Gao, name: Anqi Liu, name: Daniel Khashabi

Recent works have suggested that In-Context Learning (ICL) operates in dual
modes, i.e. task retrieval (remember learned patterns from pre-training) and
task learning (inference-time ``learning'' from demonstrations). However,
disentangling these the two modes remains a challenging goal. We introduce ICL
CIPHERS, a class of task reformulations based on substitution ciphers borrowed
from classic cryptography. In this approach, a subset of tokens in the
in-context inputs are substituted with other (irrelevant) tokens, rendering
English sentences less comprehensible to human eye. However, by design, there
is a latent, fixed pattern to this substitution, making it reversible. This
bijective (reversible) cipher ensures that the task remains a well-defined task
in some abstract sense, despite the transformations. It is a curious question
if LLMs can solve ICL CIPHERS with a BIJECTIVE mapping, which requires
deciphering the latent cipher. We show that LLMs are better at solving ICL
CIPHERS with BIJECTIVE mappings than the NON-BIJECTIVE (irreversible) baseline,
providing a novel approach to quantify ``learning'' in ICL. While this gap is
small, it is consistent across the board on four datasets and six models.
Finally, we examine LLMs' internal representations and identify evidence in
their ability to decode the ciphered inputs.