SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We observed that ICL emergence was more sensitive to model initialization when using LLaMa token embeddings as exemplars. An example seed where ICL did not emerge at all (for 3,200 classes, 5 exemplars per class) is shown in Figure 12. However, in all cases where ICL did emerge (even to a small degree), we found it to be transient (Figures 7, 13). Furthermore, the results presented in Section 4.3 are averaged over two different clustering seeds (see step 2 above), which yield different output datasets. ICL is transient across all 8 datasets tested.
At 1,600 classes, with 10 exemplars per class, we found mild ICL emergence, as shown in Figure 13. We found that 3,200 classes were necessary to get any strong ICL, which is shown in Figure 7.
Figure 13: Mild emergence and subsequent transience of ICL on a model trained on the LLaMa derived datasets with 1,600 classes and 10 exemplars per class.
D Additional regularization results
(b) In-weights learning.
Figure 14: For extreme values of L2 regularization ( $>1\mathrm{e}-04$ ), the model seems to be over-regularized: We observe a return to ICL transience, as well as poor performance on IWL.
Figure 15: Total train loss (including L2 penalty) for a sampling of experiments. Regularization seems to make the training loss smoother, as well as causing train loss to plateau at a higher value (rather than continuing to drop).
(c) In-weight learning, decay  $= 1\mathrm{e} - 5$
(d) In-weight learning, decay  $= 1e - 4$
Weight decay applied to: ResNet embed MLP layers self attention
Figure 16: Selectively applying weight decay to different sets of network weights reveals that IWL relies on MLP layers. When we apply weight decay only to self-attention layers (blue), ICL is still transient. When we only apply weight decay to MLP layers, ICL transience is mitigated (red). These results can be interpreted in light of prior work indicating that in-weights information is stored in MLP layers [31, 32]. By selectively penalizing this behavior, we enable ICL to persist, thus providing convergent evidence that – when no weight decay is applied – ICL fades due to competition with IWL circuits.