SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To tune hyperparameters, for each experiment we tested a range of learning rates from 1e-8 to 1e-4 - specifically, 1e-8, 5e-8, 1e-7, 5e-7, 1e-6, 5e-6, 1e-5, 5e-5, 1e-4. We used an iterative procedure to hone in on optimal learning rates. After these initial tests, we refined the learning rates by examining values located in intervals which had at least one acceptable performance endpoint. For example, if both learning rates of 5e-5 and 1e-4 yielded divergent results (e.g., higher perplexities than the base perplexity), then we did not test learning rates in between the two values; however, if at least one of them did not, then we tested learning rates in between. Furthermore, we tested a few different numbers of epochs (usually 5, 8, 10, 15, or 20) for each experiment, using a grid search with the selected suitable learning rates. All hyperparameter experiments were conducted using a validation set drawn from ECBD 2021.
Entity Inference Dataset For both base LMs, we used a learning rate of 5e-4 for 10 epochs for fine-tuning on the definition sentence and a learning rate of 5e-4 and 5 epochs for each of 5 sentences for distillation. For fine-tuning on the definition and transfer set, we use a smaller learning rate of 4e-5.
ECBD Dataset For GPT-Neo-1.3B and GPT2-XL, we trained for 5 epochs with a learning rate of 3e-6 for fine-tuning. For fine-tuning on the definition and transfer set, we found that 5 epochs and a smaller learning rate of 6e-7 yielded the best results. For context distillation, a learning rate of 3e-6 yielded the best performance. For all distillation experiments involving GPT2-XL and GPT-Neo-1.3B, we perform distillation training for 5 epochs on each of 5 generated continuations.
For LLaMA-2-7B, we found that a learning rate of 5e-6 yielded best performance for both finetuning on the definition sentence and distillation. For finetuning on the definition and transfer set, we use a smaller learning rate of 8e-7. We finetune for 5 epochs for both finetuning on the definition sentence and finetuning on the transfer set. For distillation with LLaMA-2-7B, we train for 3 epochs on each of 5 generated continuations.
B.2 Compute
All experiments were run using Quadro RTX 8000 GPUs with 48GB RAM. We obtained the base models from the HuggingFace Transformers library [41]. All experiments for GPT-Neo and GPT2-XL required less than 4 GPU hours each, and experiments for LLaMA-2-7B required up to 30 GPU hours. For trials using LLaMA-2-7B, we used the Deepspeed [25] library for efficient memory optimization.
B.3 Additional Results: Diversity of Transfer Set
Figure 5: Perplexity for using  $n$  distinct transfer sentences during distillation, with number of updates standardized to 10. We see the benefit of having a diverse transfer set compared to repeating the same transfer sentence 10 times.
B.4 Additional Results: Results on ECBD Popular Set
GPT-NEO-1.3B
Pre-Edit PPL (↓)	37.0	26.1
Target (Δ) ↓	Spec. (Δ)
Finetuning on de (full)	36.6 (-0.5)	26.0 (-0.1)
Finetuning on de + Te (full)	37.2 (+0.2)	26.1 (+0.0)
Distillation (Mg = Ms)	34.5 (-2.5)	25.5 (-0.6)
Prepend Def.	31.7 (-5.3)	26.1
Prepend Random Def.	58.4 (+21.4)	26.1
Table 10: Results on ECBD Popular, a dataset of popular entities such as SpaceX and Eminem dated before the pretraining date of GPT-Neo-1.3B. Notably, the entities in the dataset should be well-known to GPT-Neo-1.3B.
B.5 Experimental Details on CounterFact Evaluation
CounterFact [21] consists of edit statements formatted as subject-relation-object triplets. For a given factual triplet  $(s, r, o)$ , the goal is to edit the counterfactual triplet  $(s, r, o*)$  into the model, where  $o*$  is a counterfactual object. For example, given the fact "The Eiffel Tower is in Paris", the subject  $s = \text{Eiffel Tower}$ , the relation  $r = \text{is in}$ , and the object  $o = \text{Paris}$ . One counterfactual edit might be 'The Eiffel Tower is in Rome'; here,  $o* = \text{Rome}$ .
The Efficacy score measures the percentage of instances where  $P(o_{*}) > P(o)$  post-edit, when given the prompt  $s + r$  ('The Eiffel Tower is in'). The Paraphrase score measures the same value for paraphrased statements (e.g., 'The location of the Eiffel Tower is'). The Neighborhood score measures the percentage of instances where  $P(o) > P(o_{*})$  for unrelated entities. For example, after editing the new location of the 'Eiffel Tower' we might want to check that 'Louvre' is still in Paris, and not Rome. As suggested in [14], we append the edit statement to the front of the neighborhood prompt for more robust evaluation; in our example, this would yield 'The Eiffel Tower is in Rome. The Louvre is in'.
C Transfer Set Generation
To generate transfer set sentences using GPT-3.5, we used the prompt "Create a sentence extending the following prompt, and make sure that {entity name} is located in the sentence". We experimented with a few similar prompts, but found very little variance in terms of results.
For smaller base models, we generated continuations of the definition by sampling from the model with entity definition as a prefix. Afterwards, we appended the entity name to the front of each sentence if the generated continuation did not include entity name. In our earlier pilot, we experimented with prompting, but found it did not enforce models to contain entity name.
Table 11 contains example continuations generated by each base model.
Table 11: Randomly sampled continuations from different generation LMs. See Appendix C for prompting methods for each LM.