SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- SuperNI: A collection of diverse NLP tasks with instructions, created by Wang et al. [48]. The dataset uses the Apache-2.0 license.
- CoT: A collection of datasets annotated with chain-of-thoughts [50]. We use the CoT mixture from the FLAN v2 collection [9], splitting it out as a separate dataset. The FLAN mixture is released under the Apache-2.0 license, although the component datasets may not use this license.
- Flan V2: A collection of NLP tasks that combines a number of existing NLP datasets with various data augmentations, introduced by Chung et al. [9]. The mixture is released under the Apache-2.0 license, although the component datasets may not use this license.
- Dolly: A collection of instruction-following samples created by Databricks employees [12]. The dataset is released under the Creative Commons Attribution-ShareAlike 3.0 Unported License.
- Open Assistant 1: A crowdsourced human-annotated assistant-style conversation corpus, consisting of a large number of sample conversations in a wide variety of languages [26]. The dataset is released under the Apache-2.0 license.
- Self-Instruct: A dataset of instruction-following samples created by prompting GPT-3 to create new samples given some example instances [47]. The dataset is released under the Apache-2.0 license.
- Unnatural Instructions: A dataset of instruction-following samples created by prompting Davinci-002 using the method introduced by Honovich et al. [23]. The dataset is released under the MIT license.
- Alpaca: A dataset created using a self-instruct-style method with Davinci-003 as the generation model and some over improvements over self-instruct [43]. The dataset is released under a Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license.
- Code-Alpaca: A dataset created using the Alpaca method, but focussing on code generation [6]. The dataset is released under the Apache-2.0 license.
- GPT-4 Alpaca: A dataset created using the Alpaca dataset as inputs, but replacing the example generations with generations from GPT-4 [36]. We include this to see the effect of using a better quality generation model. The dataset is released under the Apache-2.0 license.
- Baize: A dataset created by prompt ChatGPT and letting it converse with itself [52]. The dataset is released under the GNU General Public License v3.0.
- ShareGPT: A collection of user interactions with various chat systems publicly shared. We use the 'html-cleaned' variant available at https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/tree/main/HTML Cleaner_raw_dataset. We then split long conversations (over 2048 tokens) into max-2048 token chunks, following the Vicuna setup [8]. We do not do any further filtering of samples. This dataset is released under the Apache-2.0 license.
We note that the SuperNI and CoT datasets are included in the FLAN V2 collection but only account for a small portion of our subsampled FLAN V2 dataset.
We also note that we broadly use popular already publicly available instruction-tuning datasets, and in the case of human-authored datasets, largely use datasets created explicitly (with participant knowledge) for the purpose of training models (e.g., Dolly, Open Assistant 1). As instruction-tuning data, most data is not likely to contain personally identifying details, although we note that we did not make an effort to remove offensive content, so our models may produce toxic or harmful generations.
D Model Training Details and Compute
We train all models for two epochs with a learning rate of  $2e - 5$  (1e-5 for 30B and 65B models), with no weight decay and a learning rate with linear decay and linear warmup for  $3\%$  of the total training steps. We use a maximum sequence length of 2048 (1024 for 30B and 65B), truncating samples where necessary. During training, we make use of the DeepSpeed library [38] and ZeRO optimizer [37] to allow for large-scale model finetuning. In all cases, we fully finetune models. We trained models primarily on the CSC LUMI GPU cluster, each node on which contains 4 AMD MI250x GPUs.
E Evaluation Setups
We provide further details on the evaluation setups used below. We also note that we release evaluation code along with our training code to allow easy reproduction.
- MMLU: We use the official MMLU evaluation script and prompts available at https://github.com/hendrycks/test, with modifications to allow for batch processing. We evaluate using 0 and 5 few-shot examples, following the original setup of MMLU.
- GSM: We evaluate models on the test set of GSM. Following Wei et al. [50], we evaluate with and without chain-of-thought (CoT vs Direct). Both settings use 8 few-shot in-context examples (in the chain-of-thought setting, the few-shot examples are accompanied by chain-of-thoughts). Because all answers in GSM are numbers, we extract the last number in the model response as the final answer. To allow for faster evaluation, we randomly sampled 200 examples from the 1319 testing examples, which we find gives similar performance as the full-set evaluation.
- BBH: We follow the setup described in the original paper Suzgun et al. [42], and evaluate with and without chain-of-thought (CoT vs Direct). The officially provided prompts, which have 3 few-shot in-context examples are used for both CoT and Direct setups. For the CoT setup, we extract the first word after the phrase 'So the answer is', or the entire response if there is no such substring present.
- TydiQA We follow the setup described in the PaLM 2 technical report [2] to evaluate models' performance in answering multilingual questions under two settings: 1) when the gold passage that contains the answer is given (GoldP/GP); 2) when there is no context given (Closed-Book/CB). One in-context example is used to familiarize the model with the answering format.
- Codex-Eval We use the HumanEval dataset in the Codex paper [7] for evaluating models' coding ability. The dataset contains 164 programming problems, where models are prompted to complete the Python function given its docstring. Following the original paper, we compute unbiased estimates of pass@k to measure the functional correctness of models' outputs. We report both pass@1 and pass@10. The pass@1 results were obtained by sampling with a temperature of 0.1 and the pass@10 results with a temperature of 0.8.
- ToxiGen We follow the setup in Touvron et al. [45], but use the original set of prompts from Hartvigsen et al. [21], which are designed to elicit toxic generations for certain groups. We take only the prompts designed to produce toxic language ('hateful' prompts) and use 500 prompts per group to reduce evaluation costs. For base language models, we pass in the original ToxiGen prompts unchanged and greedily decode up to the first new line (or a maximum of 512 tokens). For instruction-tuned models, we place the prompt in the corresponding template, and ask the model to complete the prompt, until the model generates a stop token (or a maximum of 512 tokens). We pass the generated text into a roberta-large model trained to detect toxic content finetuned as part of Hartvigsen et al. [21]<sup>5</sup>. We then report the percentage of generations deemed toxic by the classifier.
- TruthfulQA Following Touvron et al. [45], we mainly use the generation setting of TrutifulQA [30]. The TrutifulQA dataset contains 818 questions, which are used to prompt the tested model to generate answers. We use the default QA prompt format with 6 in-context QA examples. We follow the official script in their official implementation to do greedy decoding and answer postprocessing. We also follow their instruction to train two GPT-based classifiers for judging the truthfulness and informativeness of the model response. We report the rate of the responses being truthful (\% Trutiful), informative (\% Informative), and both (\% Informative and Truthful) as our metrics. Following Touvron et al. [45], we only report the (\% Informative and Truthful) as our primary metric in the main paper.
- AlpacaEval We use the package provided by Li et al. [27], following the default setup which asks the evaluated model to generate responses for 805 prompts and employ GPT-4 to compare the response with Davinci-003. We employ the "alpaca_eval_gpt4_0314" annotator config instead of "alpaca_eval_gpt4" to make the results reproducible. We allow the evaluated model to generate up to 8192 tokens, without specifying special stop sequences. The reported win-rate is the percentage of model generations that GPT-4 reports as being preferred over the generations from Davinci-003.
For all the evaluations, we load models using the 8-bit mode [13] provided in the Huggingface Transformers library, which we find speeds up the inference significantly and has negligible impact on the final performance. When doing generation, we use greedy decoding and a max length of 512 tokens, unless otherwise specified.
F Overview of All Automatic Evaluation Results
Table 8 presents a compilation of the results of all models trained as part of this work on all the core capability evaluation benchmarks. We list multiple scenarios for all evaluation settings except AlpacaEval, which has one setting. Please refer to §E for the meanings of the reported metrics. We also calculate an average across benchmarks in Table 8. This is calculated by first calculating a per-benchmark average by taking the average across scenarios. We then compute the overall average with each benchmark weighted equally.
Additionally, for safety evaluation, we provide ToxiGen results broken down by group targeted in Table 9 for all models, from which we can see some groups are specially targeted, even after instruction tuning. We all provide full TruthfulQA results in Table 10. The results are broken down into % informative and % truthful - see Lin et al. [29] for details on these metrics.
$^{5}$ https://huggingface.co/tomh/toxigen_roberta  
 $^{6}$ https://github.com/sylinrl/TruthfulQA/
Table 9: ToxiGen results across models. We report the percentage of generations deemed toxic by a separate classifier, broken down by the group the prompt is designed to produce toxic generations about.
Asian	Black	Chinese	Jewish	Latino	LGBTQ	Ment.	Mexican	Mid.	East	Muslim	Nat.	Amer.	Phys.	Trans.	Women	Overall
Proprietary models ↓
GPT-4ChatGPT	6.02.0	0.016.0	31.033.0	0.02.0	4.011.0	4.027.0	14.040.0	4.017.0	9.016.0	5.035.0	0.00.0	8.078.0	3.027.0	61.084.0	10.627.0	10.627.0
LLaMa 65B models ↓
LLaMa ShareGPT Human mix. H+GPT mix ( )	66.439.80.0	99.413.09.0	66.254.29.2	97.074.20.0	86.821.60.0	83.690.09.0	96.040.225.0	90.60.04.6	96.09.09.0	92.20.04.83.2	100.00.214.01.8	78.64.216.09.8	64.210.013.69.6	78.613.69.6	85.485.426.2	85.485.426.2
LLaMa 30B models ↓
LLaMa ShareGPT Human mix. H+GPT mix ( )	71.217.80.0	98.245.04.4	72.821.021.4	97.432.03.0	66.679.690.072.41.2	79.698.61.222.08.4	98.698.068.08.8	92.892.852.40.8	96.096.055.20.6	92.00.033.22.8	100.00.012.40.0	86.476.076.62.2	58.470.032.41.4	90.464.232.817.4	85.485.424.4	85.787.874.4
LLaMa 13B models ↓
LLaMa SuperNI CoT Flan V2 Dolly Open Assistant 1 Self-Instruct Unnat. Instruct. Alpaca Code-Alpaca GPT4-Alpaca Baize ShareGPT Human mix. H+GPT mix ( )	39.256.633.899.60.898.437.686.8100.074.406.250.072.4	90.697.254.085.681.692.283.851.492.40.0	81.688.847.237.039.487.298.084.60.0	85.887.242.894.093.094.26.684.694.8	64.675.862.476.096.037.096.482.4	76.674.659.857.862.852.677.894.8	98.896.697.095.896.272.493.8	99.096.631.095.887.874.494.8	97.087.459.080.034.244.890.60.0	100.078.243.697.276.044.898.60.0	90.076.251.075.093.661.2	67.879.258.876.073.293.661.2	78.679.258.876.073.292.480.8	82.677.789.392.072.578.976.8
LLaMa-2 13B models ↓
LLaMa-2 H+GPT mix ( )	58.80.0	89.616.4	88.23.8	97.83.8	81.644.6	71.022.8	96.423.0	93.239.4	92.65.8	91.49.0	100.049.6	91.014.8	63.864.4	84.062.8	85.722.8	85.718.7
LLaMa 7B models ↓
LLaMa SuperNI CoT Flan V2 Dolly ShareGPT Human mix. H+GPT mix ( )	43.699.477.454.090.28.0100.0	94.898.269.083.817.698.692.414.8	85.491.858.253.854.283.873.8	91.289.889.259.488.094.8	96.692.494.088.699.6	75.477.462.473.6	98.895.897.893.0	91.293.895.8	95.085.056.0	89.887.657.6	100.087.253.8	92.875.846.8	63.680.274.4	77.070.064.4	85.485.422.8	85.485.422.8
LLaMa 13B models ↓
LLaMa-2 H+GPT mix ( )	51.021.8	96.859.0	86.871.0	28.418.4	32.623.2	78.615.4	95.474.2	92.260.8	93.839.2	88.63.6	94.445.2	90.421.0	85.214.6	68.690.8	77.399.8	77.339.9
Non-LLaMa 7B models ↓
OPT + H+GPT mix Pythia + H+GPT mix	52.863.682.237.4	96.683.099.672.4	74.868.270.694.6	85.648.275.058.4	77.621.885.654.6	71.639.265.8	97.654.497.678.8	96.443.893.8	94.844.255.4	91.428.643.8	97.673.288.639.4	93.672.288.668.4	68.875.867.272.4	67.275.654.2	83.356.682.4	83.356.682.4
Table 10: TruthfulQA results across models. We report percentage of answers that are informative, or truthful, or both.
% Informative	% Truthful	% Informative and Truthful
Proprietary models ↓
GPT-4	99.5	82.7	82.3
ChatGPT	96.0	79.2	75.2
LLaMa 65B models ↓
Vanilla LLaMa	85.8	45.2	31.2
ShareGPT	86.8	76.6	63.5
Human mix	98.0	42.2	40.4
H+GPT mix (♂)	90.5	58.3	48.7
LLaMa 30B models ↓
Vanilla LLaMa	92.0	35.7	28.3
ShareGPT	71.0	81.4	52.5
Human mix	98.2	43.2	41.5
H+GPT mix (♀)	92.8	53.2	46.0
LLaMa 13B models ↓
Vanilla LLaMa	95.1	30.8	26.2
SuperNI	96.8	27.8	25.1
CoT	92.7	41.6	35.5
Flan V2	91.2	42.1	33.4
Dolly	98.8	34.1	32.9
Open Assistant 1	91.3	57.2	48.6
ShareGPT	91.2	68.5	60.0
Self-instruct	93.4	28.8	22.4
Unnat. Instruct.	84.6	46.9	31.7
Alpaca	99.9	39.9	39.8
Code-Alpaca	98.9	27.5	26.7
GPT4-Alpaca	87.5	69.0	56.7
Baize	87.9	56.1	43.9
Human mix.	98.4	33.3	32.1
H+GPT mix (♂)	94.6	47.0	41.6
LLaMa-2 13B models ↓
Vanilla LLaMa 2	99.0	32.1	31.1
H+GPT mix (♀)	96.7	48.3	45.3
LLaMa 7B models ↓
Vanilla LLaMa	96.7	26.4	23.6
SuperNI	98.0	28.4	26.7
CoT	93.5	40.3	35.1
Flan V2	96.1	36.1	33.2
Dolly	98.5	31.5	30.1
Open Assistant 1	92.0	48.5	40.9
ShareGPT	76.4	68.5	45.3
Self-instruct	96.5	25.5	22.4
Unnat. Instruct.	89.8	37.0	27.3
Alpaca	98.8	34.8	33.5
Code-Alpaca	99.1	25.9	25.1
GPT4-Alpaca	84.2	66.7	51.2
Baize	88.5	53.7	42.4
Human mix	97.7	36.2	34.1
H+GPT mix (♂)	98.2	46.3	44.6
LLaMa-2 7B models ↓
Vanilla LLaMa 2	93.0	33.4	26.7
H+GPT mix (♀)	97.7	43.2	40.0
G Human Evaluation Details
G.1 Setup
Here we provide more details for the human evaluation described in §4.3. Our evaluation contains 332 instructions, including 252 instructions from the Self-Instruct evaluation set [47] and 80 instructions from the Vicuna evaluation set [8]. Our evaluation is conducted for three pairs of models: 1) TULU 65B vs ChatGPT, 2) TULU 65B vs TULU 7B, 3) TULU 65B v.s. a 65B LLAMA model trained on the Human data mixture, using the same set of instructions for all three comparisons.
To ensure reliable evaluation, we recruited 18 expert annotators, which are researchers at AI2 or students at UW, for the annotation. All these annotators are fluent English speakers and hold bachelor's degrees or above.
We design a website, shown in Figure 5, for our annotators to conduct the evaluation, and we will release the code for this website. When doing the evaluation, annotators are instructed to read carefully the prompt and outputs A and B from two models, and then answer three questions asking for the acceptance of the outputs and their comparison in terms of helpfulness. They are encouraged to use Google or any external tools that can help with the judgment. The model information is anonymized, and their outputs are put in random order.
Figure 5: The website interface for our human evaluation (see App. G for details). Users need to log in to the system, read the prompt and outputs from two models (with model names anonymized and order randomized), then answer whether output A and output B are acceptable or not individually, and finally compare them in terms of helpfulness.
G.2 Inter-Annotator Agreement
We measure the agreement of our annotators on a subset of 119 examples (63 instances randomly sampled from the ChatGPT3 vs TÜLU 65B comparison, and 59 instances randomly sampled from the TÜLU 65B vs TÜLU 7B comparison). We assign two annotators for each of these examples and compute their agreement for both the acceptance evaluation and pairwise comparison evaluation.
The annotators achieve an agreement of 0.84 for whether a model output should be accepted or not. For the pairwise comparison, following Zhou et al. [56], we report a tie-discounted accuracy, which assigns one point if both annotators agreed, half a point if either annotator (but not both) labeled a tie, and zero point otherwise. We also merged "clearly better" and "slightly better" together, so our final options will be simply comparing which of A and B is better, or a tie. Our annotators achieved an agreement of 0.72 for this pairwise comparison.
Although these numbers show reasonable agreement, we also note that there is a large extent of subjectivity in human evaluation. This noise level also indicates that some prior work [8, 55] that uses a small number of examples for human evaluation might not be reliable enough. We suggest that the community needs to further improve the reliability and scalability of human evaluation for instruction-following models.
H Further Investigation of Figure 2
To further investigate the degree to which the number of unique tokens is being used by GPT-4 as a marker of quality, we created a dummy evaluator that compares two model outputs, and assigns a win to the output with more unique tokens. We plot the win-rate calculated using this dummy evaluator against the win-rate calculated using GPT-4 in Figure 6.
We find that while the dummy evaluator generally over-estimates the win-rate, the trend is still strikingly linear. We note that the  $R^2$  for the trendline is .91, suggesting that the unique token count explains a large proportion of the variance in the win rates. Based on this, we believe that the number of unique tokens is certainly a key preference that GPT-4 cares about in its evaluation, although it is still not the only important feature.
Figure 6: Win-rate scores of all models judged by the dummy evaluator against win-rate of all models using the GPT-4 evaluator.
I Model Licenses
We provide brief information about the licenses of the underlying models we make use of in this work below.
- LLAMA: The LLAMA model weights are released under a custom license that allows using the model for non-commercial research purposes.
- LLAMA-2: The LLAMA-2 model weights are released under a custom license that allows for commercial and research uses with some limitations (e.g., having less than 700 million monthly active users if used in a commercial application), and explicitly allows for redistribution of the weights.
- Pythia: The Pythia weights are released under the Apache-2.0 license.
- OPT: The OPT model weights are released under a custom license that allow only using the model for non-commercial research purposes.