SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We use HuggingFace [48] implementations of the BART and T5 models. For the decoding method, we adopt the standard beam search with a beam size of 4 for all baseline models. As for checkpoint selection, we save a checkpoint for each epoch and select the checkpoint with the highest ROUGE-2 on the validation set. Other default hyperparameters are shown in Table 7.
Table 8 shows an example of GPT prompt format, consisting of a fixed instruction ("Generate a pair of contrastive sentences with the given set of keywords.") and a few in-context demonstrations ("Keywords:  $c_1$ , ...,  $c_k$  \n Sentences:  $s_1$ $s_2$ ").
We use the standard implementation of BLEU, ROUGE, METEOR, CIDEr, and SPICE in pycocoevalcap<sup>13</sup>. As recommended, we adopt the Recall score of BERTScore<sup>14</sup> and the hash code for evaluation setting is "roberta-large_L17_noidf_version=0.3.12(hug_trans=4.21.3)-rescaled_fast-tokenizer". In addition, we design and implement MATCH to evaluate how well the machines solve
<sup>13</sup>https://github.com/salaniz/pycocoevalcap
$^{14}$ https://github.com/Tiiger/bert_score
Table 7: Hyper-parameter settings for all baseline models.
Parameter	Value
epoch	10
batch size	32
beam size	4
max input length	64
max output length	128
learning rate	3e-5
warm-up steps	500
Table 8: An example of InstructGPT prompt format. We only show two in-context demonstrations here for brevity.
Generate a pair of contrastive sentences with the given set of keywords.
Keywords: Kansas, steakhouses, New York City, city, pizzerias
Sentences: Kansas city is known for its steakhouses. New York City is known for its pizzerias.
Keywords: seven days, one day, 1,440 minutes, a week
Sentences: There are 1,440 minutes in one day. There are seven days a week.
Keywords: axis, one day, one month, Earth, Moon
Sentences:
the challenge of situated semantic matching (Section 3.2). We now define the keyword matching accuracy MATCH based on mathematical notations introduced in Section 3.1.
$t = (t_{1},\dots ,t_{k}),t_{i}\in \{0,1\}$  indicates that each keyword  $c_{i}$  appears in which sentence in the answer pair  $y^{true} = \{s_1^{true},s_2^{true}\}$ . In other words, if  $c_{i}$  should appear in  $s_1$ , then  $t_i = 0$ ; if  $c_{i}$  should appear in  $s_2$ , then  $t_i = 1$ .  $p = (p_1,\dots ,p_k)$ ,  $p_i\in \{-1,0,1\}$  indicates that each keyword  $c_{i}$  appears in which sentence in the output pair  $y^{pred} = \{s_1^{pred},s_2^{pred}\}$ . In other words, if  $c_{i}$  actually appear in  $s_1$ , then  $p_i = 0$ ; if  $c_{i}$  actually appear in  $s_2$ , then  $p_i = 1$ ; if  $c_{i}$  does not actually appear in both  $s_1$  and  $s_2$ , then  $p_i = -1^{15}$ . We define the matching accuracy of a sentence pair  $\mathbf{match}(y^{true},y^{pred})$  as the proportion of correctly matched keywords, which is calculated as  $\frac{1}{k}\max (\sum_{i = 1}^{k}\mathbb{1}_{t_i = p_i},\sum_{i = 1}^{k}\mathbb{1}_{1 - t_i = p_i})\in [0,1]$ . Here  $\mathbb{I}$  is the indicator function. The formula includes both  $1 - t$  and  $t$  in a symmetric way because the sentence pair is unordered. For the whole test set, we take the average matching accuracy of all examples as MATCH.
We illustrate the computing process of matching accuracy with a simple example. Given [July, China, winter, Australia, summer, July], the answer could be "July is summer in China. July is winter in Australia." So  $t = (0, 0, 1, 1, 0, 1)$ . If the generated output is "July is summer in Australia. July is winter in China.", then  $p = (0, 1, 1, 0, 0, 1)$ . As a result, the matching accuracy is  $4/6 = 0.67$ .
As for the implementation, we utilize  $\mathsf{NLTK}^{16}$  to split the output into two sentences. In particular, if there is only one sentence in the output, we append an empty string as the second one; if there are more than two sentences, we only take the former two sentences into consideration. We lemmatize the sentence before determining keyword appearance.
<sup>15</sup>By defining  $p_i = -1$ , MATCH can also reflect the coverage of keywords in the output.
$^{16}$ https://www.nltk.org/