reviews: 
Comments: The paper studies a special method that mixes various supervised finetuning (SFT) data together with the pretraining data for language models. This special method requires fitting existing SFT data from various tasks into a new binary classification format, where the supervision can be added as an extra term in the loss function. With this approach the pretraining and modified finetuning is conducted in iterations. 

The authors claimed in the abstract that their method is "finetuning free" and a small GPT2 sized model trained with this approach "out performs large models e.g. GPT-3 and InstructGPT".
Strengths: The original motivation of this work is on the right general direction. It is important to study methods that can improve a general purpose LLM beyond what is possible with pretraining along, leveraging smaller but high quality datasets. Many fruitful methods has been proven along these lines, such as FLAN, Instruction finetuning, self-instruct, RLHF, etc. 

The paper also studies whether the auto-regressive language modeling abilities has been impacted by adding SFT signals, and find that with their method this impact has been minimal. This is an encouraging sign.

The paper has set out to study cross-task generalizations, which is another important topic. They find some interesting generalization effect on COPA.
Weaknesses: Although the paper has many good motivations, it also has serious flaws.

1) The central claim is miss leading. They claims that their method is "finetuning free" and they compare their results to zero-shot and few-shot results. However, this is not the case. In particular, they still require gradient steps on the training split data of their eval tasks (or of a similar eval task). This is effectively finetuning, albeit with a slightly different loss function. In particular, using their method does not alleviate the user from collecting the finetuning set, which is importantly not the case for typical zero-shot and few-shot settings with a foundation LLM. Thus the claims involving comparison to GPT-3 and InstructGPT are unfair because those models did not explicitly have access to those finetuning training sets.

2) The proposed method lacks novelty. It is essentially doing multi-task finetuning with the pretraining task mixed in. They did not show whether doing the same scheduling mixing pretraining tasks with vanilla finetuning tasks would have a similar effect, or whether multi-task parallel training with both the pretraining task and the vanilla finetuning tasks would have a similar effect. This is a very important baseline that is completely missing. It is possible that a much simpler approach - simply mixing in some pretraining data/task at finetuning stage would have all the same effect.

3) The proposed formulation of the "unified data" seems narrow and classification heavy. It may not capture well generation heavy LLM use cases such as "write me a poem" that are very important and popular these days.

4) The evals are too narrow. For comparison with GPT3 and InstructGPT, the author should test few-shot and zero-shot performances on unseen tasks.
Recommendation: 1: strong reject
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 1 poor
Clarity: 2 fair