Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper investiagtes the application of pre-trained language models on the task of classifying non-textual data, without any architecture changes.
The input features are linearized into a text-like sequence and given as context, the output is then collected as a language model prediction.
While it doesn'g perform best overall, it does achieve surprisingly competitive performance.
strengths_and_weaknesses: The paper is well written and clear.
The direction of investigation is interesting.
There are many experiments presented, analysing different aspects of the set-up.
The findings are interesting and non-intuitive. For example, the model performing well on MNIST by describing each pixel in a textual sequence. 
While the setup is clearly not optimal for the tasks, the findings could lead to some interesting future research.

Previous work is mentioned, which replaces the input and output layers of the LM, whereas the current work keeps entirely the same architecture. 
The performance of that system should be reported as well. 
At the moment it is unclear whether keeping the original input or output layers actually helps or hurts the performance on different tasks.

Furthermore, the replacement of the input and output layerst should be investigated separately.
While giving the input as text might actually have some benefits (as experiments in Section 5.2 also indicate), using only language model predictions as output could be hurting performance. Even models that work with natural language generally fine-tune pre-trained language models by replacing the LM output layer with a task-specific version.

An open question left by the paper is whether these results are down to the architecture (and the large number of parameters) or the fact that the model is pre-trained on large amounts of textual data.

There should also be a discussion on the difference in the number of parameters.
As I understand, all the baseline models are quite simple, with perhaps a few hundred trainable parameters. In contrast, GPT-3 has billions of parameters.
How much does model capacity affect performance? Are there no baselines with more comparable numbers of parameters that could have been also reported?

questions: ^
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper fine-tunes pre-trained language models to perform non-language tasks, like regression of functions and classification. They test a large variety of tasks, and test against several standard algorithms ( like KNN, SVM, RF). Additionally, they investigate sample efficiency, ability to extrapolate, and robustness of these models. 
strengths_and_weaknesses: ### Strengths

Section 5.2 is the most interesting part of this paper to me, and to me, this is one of the best motivations for fine-tuning a large language model to perform this task. I haven't seen this idea before, and it's intriguing as a method for understanding the pre-training dataset and also harnessing contextual information. I'd like to see more focus directed here and maybe additional tasks explored or an expansion on experiments in this direction. 


* I appreciate the summary listed in table 1!
* I appreciated figure 4 (comparing # training samples vs. accuracy for different classes of tasks).

### Weaknesses
I don't doubt the results of this work, but after having read the paper, I'm still unsure what advantage we gain by finetuning a pre-trained language model to perform non-language tasks. It currently reads like the paper makes the argument that finetuning pre-trained language models can perform well on non-language tasks. However, these models are orders of magnitude more costly to train and run inference on, so when and why should we prefer it? Could the authors better motivate this?

Additionally, please explain what non-language tasks are earlier in the paper. Specifically, also say that the paper will test both classification and regression tasks. Please give an example of each in the abstract and also in the intro.
Also, the sentence: "We note that though achieving tremendous success with natural data, deep learning still faces difficulties with standard machine learning tasks, especially on tabular data..." seems to be at the core of this paper. I would include this idea earlier in the paper (abstract or intro).


This paper covers a lot of ground. I think it could be more focused by moving additional experiments to the appendix (for example, sec 6.2). 

### Additional Feedback
* Suggestion to use "what should the y be" as opposed to "what should be the y" in the template. Language models are still fairly sensitive to prompts, and wording may change results. 
* Include the sizes of GPT-J (6b) and GPT-3 (175b) in the section on "pretrained language models" as a reference for the reader. 
* Figure 4 text should be larger 
    * Also table 3 should highlight interesting results as the text is too small to read
* It would make the paper easier to read if the authors could include the names of the tasks (in addition to the IDs) in the text to better understand the context of the task. For example, in section 5.2, which DID is the Medical Insurance dataset?

questions: * Did the authors try other templates? How did they perform? What was the sensitivity of the model performance on different prompts?
* From figure 4, it looks like this might not work for few-shot prompting. Did you run that experiment?
* I would include additional information about how adversarial examples were generated. 
limitations: Currently the social impacts section says that "LIFT might have embedded bias targeting certain social groups." This seems possible to measure. In particular, for tasks where text is used in the prompt, the authors could compare LIFT with all other models they compare with to see if the examples that are mis-classified are systemically biased in some way. Additionally, this could be an interesting experimental setup to evaluate bias in the pre-trained model.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies an interesting and important topic: can we use large pre-trained language models to machine learning tasks where the inputs are unstructured real vectors? Through a series of experiments on real/synthetic classification/regression tasks, the author concludes that
- fine-tuning LLMs have performance on par with many traditional algorithms.
- LLM is performing bayesian prediction
- two-stage fine-tuning improves the performance
- feature name helps LLM perform better.
strengths_and_weaknesses: Strength:
 - The paper is generally very well-written, with clear figures and a lot of experimental details in the appendix. It also addresses its scope of study and societal impact properly.
- The topic is interesting, potentially impactful, and novel. 
- There are a lot of experimental results, which could help practitioners gain intuitions.

Weakness:
- The performance by LIFT is not impressive.
- Most of the important claims in the paper are not well-supported by the empirical evidence. See questions below.

**I am very willing to increase the score to 8 if the author can either 1) justify why their claims are correct, or 2) re-position the paper as a "negative empirical finding", rather than advertising "LIFT" as a very promising approach for real-value-input ML tasks.**

minor: 
- Non-language can also refer to image, sound, or haptic information, and I was confused what non-language task the paper was discussing when reading the title and the abstract. Probably rename "Non-Language Machine Learning Tasks" to "Real-Valued Machine Learning Tasks" (or something similar). For example, Flamingo (from DeepMind) also uses LM as an interface, but they used it for image + text.
questions: - Table 2: why does the paper claim that LIFT performs relatively well compared to other methods, even if in many or even most cases it cannot outperform 3-layer MLP? It does not seem that the algorithm is successfully making use of the LLM internal representation.
- Table 3 caption tries to conclude that larger models generally perform better -- but it seems that GPT-3 Davinci is overall worse than GPT-3 Curie, and the performance increase w.r.t. scaling is extremely unclear. 
- Section 5.1 tries to claim that LLM is performing bayesian inference. I thought it meant "LLM has a prior about what the function is look like and it performs posterior inference based on true labels". However, the paper only experimented with "trying sampling a lot of time and see whether the variance is correlated with noise level". This only implies that LIFT is calibrated (where prediction logits/samples reflects the confidence), rather than that LLM is performing bayesian inference. 
- Table 5 tries to claim that LLM can successfully use the feature name -- for only 2 out of 3 classification dataset this seems to hold, and when it holds, the margin is very small (compared to the noise) and it is unclear whether they are noise. In particular, what is dataset 48 and 54, and what are their intuitions of LIFT can help? One experiment you could potentially run is to find a dataset where we have very strong. prior notion of what features are good and test the models' behavior on that as well.  

limitations: Yes, the author successfully addressed their limitations.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper aims to answer whether pre-trained language models (PLMs) can be used for fine-tuning on non-language downstream tasks without changing the input and output format of the original PLM. To answer the above question, this paper introduces a framework, Language-Interfaced FineTuning (LIFT), which formats non-language classification and regression tasks into natural language-like sentences by predefined templates. By extensive experiments on various datasets, they show that fine-tuning autoregressive PLMs on those downstream tasks can achieve relatively well performance. This paper also presents a series of analyses on the properties of LIFT.
strengths_and_weaknesses: Update after rebuttal
===========
I have read all the comments by the authors, and they mostly clear up my questions on the experiment settings and formatting issues. While I agree that this paper has contribution to the ML community, I cannot lean strongly toward acceptance due to its massive content. I think the content is more suitable for a journal instead of a 10-page conference paper since a lot of details are placed in the appendix, making the main content not very self-contained. Also, I still feel like this paper is more of an empirical report and it is hard to know how LIFT can be used in real-life use cases.
***
***

### Strengths
1. This paper provides some interesting findings on directly fine-tuning PLMs on non-language downstream tasks.
2. The amount of downstream tasks evaluated in this paper is impressive.

### Weaknesses
(If I am misunderstanding any parts of the paper in the following parts, please let me know.)
1. **It is unclear whether the benefit of LIFT is just because the model is gigantic**
    * Since the goal of the paper is to answer "can fine-tuning PLMs work on non-language downstream tasks", it should be verified whether the exceptional performance of LIFT is really because *"LIFT fine-tunes a model **pre-trained on human language**"*. Chances are the good performance actually stems from *"LIFT uses a gigantic transformer model with billions of parameters, and it doesn't matter whether it is pre-trained on human language or not."* To exclude the latter case, the paper should include an experiment to train a transformer model with the same architecture of GPT-3/GPT-J from scratch on the downstream task, instead of initializing the weights from a PLM. Since the trained from-scratch model is not pre-trained with human language, if this model indeed fails to perform well on those non-language downstream tasks, we can rule out the hypothesis that "LIFT performs well simply because it uses a gigantic model". Next, one should also make sure that it is because of pre-training on **human language** that makes the success of LIFT. For example, one can use artificial datasets to train GPTs and fine-tune them on non-human language downstream tasks, as in [1], [2], and [3]. If training from scratch and fine-tuning from a model pre-trained with artificial datasets both result in bad non-language downstream performance, one will be able to arrive at the conclusion that "PLM fine-tuning" solves non-language downstream tasks. 
2. I am **not convinced by the motivation** of "why one will want to use a PLM to fine-tune on non-language downstream tasks". 
    * According to the paper, it seems that the goal is to achieve "no-code machine learning with language models". But I cannot see why that would be beneficial for two reasons. First, no-code machine learning can be achieved by the AutoML (Vertex AI by Google) without relying on PLM. It is unclear why one would want to rely on a bulky PLM to achieve no-code machine learning. Second, as illustrated in the experiment results, the performance of LIFT is still largely inferior to other simpler and smaller machine learning models; this will refrain LIFT from being used in real-world cases. It is unclear why one will resort to a larger and not always better model while one can just use smaller and better models (for example, always use logistic regression for tabular data and always use MLP for other classification tasks).
3. The input length of LIFT is upper-bounded by the maximum sequence length of the PLM (2048 for GPT-J and GPT3), making it hard to be used for high-dimensional data. Moreover, considering that the input of LIFT is not only the raw feature but also the words in the templates ( 
$\mathbf{x}_1 = v_1 $ 
what should be the 
$\mathbf{y}$ 
), the actual acceptable feature length will be significantly less than 2048. It will be even lesser if each feature $v_i$ is tokenized to multiple tokens by the model. For example, considering MNIST, which has 784 features per image, assuming that each feature will only be tokenized to one token. Then when encoding an image using the template $\mathbf{x}_i=v_i$,  it will require at least $784\times4=3136$ tokens, which is significantly longer than the maximum input length of the GPT models used in the paper. It is unclear how this is handled in the paper.
4. The memory of LIFT is quadratic to the input length, indicating that LIFT is highly memory inefficient compared to other models.  
4. This paper tries to include too many things in the main content, making the snippets included in the main content seem fragmented, incomplete, not self-contained, and hard to follow. The following lists are pieces of evidence from the paper to justify the above claim.
    1. The "non-deterministic predictions" that appeared in Line 18 and Line 70 do not appear in the rest of the paper, and it is unclear what "non-deterministic predictions" refers to? Is it related to the Bayesian prediction in Section 5.1? 
    2. How is the "increase the generation randomness" in Line 137 done? Perhaps by changing the temperature for sampling?
    3. What do the "inductive biases of language models" refer to in Section 4.2?
    4. What is a median kNN in Line 241?
    5. The paragraph of "Robustness to adversarial samples" in Line 244 includes no experiment results, making the paper not very self-contained.
    6. It is unclear how the mixup is done in Section 6.2, the details of this experiment is not presented neither in the main content nor the appendix.
    7. It is unclear how the in-context learning is done in Section 4.3. For example, how is the training/testing split determined? Is there an evaluation set? Where does the variance shown in Table 10 come from?
5. This paper is poorly formatted, and some of the figures are extremely small for reading.
    1. Ill-formatted parts
        * According to the formatting instructions for NeurIPS 2022, "the table number and title always appear before the table"; all caption of the table in the paper appears **after** the table.
        * The size of the caption is sometimes significantly smaller than what should be expected, for example, Table 3, Table 5, Table 6, and Table 7. I am not sure whether this violates the guideline of "not changing the font sizes except for the reference".
        * The bottom of Figure 7 on page 8 is occupying part of Line 329 on page 9.
        * The figures in Table 3 are extremely small.
        * The legends in Figure 4 and Figure 5 are too small to read when printed on A4 paper.
        * According to the formatting instructions for NeurIPS 2022, "Place one line space before the table title, one line space after the table title, and one line space after the table". But the margin of Table 3 is quite odd and seems to violate the instructions.


[1]Papadimitriou, Isabel, and Dan Jurafsky. "Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models." Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.

[2] Chiang, Cheng-Han, and Hung-yi Lee. "On the transferability of pre-trained language models: A study from artificial datasets." arXiv preprint arXiv:2109.03537 (2021).

[3] Ryokan Ri and Yoshimasa Tsuruoka. 2022. Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7302–7315, Dublin, Ireland. Association for Computational Linguistics.


questions: ### Questions 
1. Do the authors manually change the style file or change the font sizes of the table captions?
2. The sentence in Line 109 "For **tabular data**, ... non-language tasks" seems odd.
3. During inference, how is the answer generated? By argmaxing or sampling? If it is sampled, what is the sampling scheme? When will the generation end?
4. I am afraid that I will have to disagree with the claim on Line 219 that "LIFT/GPT outperforms in-context learning". This is because, in Table 10, the variances are very big. Considering the variance of both in-context learning and LIFT/GPT, a would rather say that there is no statistical significance between the performance of the two methods.
5. Why can the experiment results in Section 5.1 imply that LIFT "may" perform Bayesian inference? 
6. There is an important reference that is missing: Kao and Lee (2021) [4] have already fine-tuned BERT on non-language downstream tasks as standard fine-tuning on natural language downstream tasks and show that the performance is decent.


[4] Wei-Tsung Kao and Hung-yi Lee. 2021. Is BERT a Cross-Disciplinary Knowledge Learner? A Surprising Finding of Pre-trained Models’ Transferability. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2195–2208, Punta Cana, Dominican Republic. Association for Computational Linguistics.
limitations: The authors have addressed their limitations and potential social impacts.
review_title: null

Meta-review: metareview: The paper demonstrates that pre-trained language models can be competitive at classifying non-textual data, where the input features are linearized into a text-like sequence and used as the conditional prefix for the language model. While the method is still not competitive with supervised learning methods, the fact that LLMs are able to do the task is intriguing. During the rebuttal, the authors also provided convincing answers to when one might prefer this approach (which is computationally expensive) over traditional methods. This paper provides timely insights into empirical research on LLMs. Therefore, I recommend acceptance.
confidence: Certain
Decision: Accept