LIFT: Language-Interfaced Fine-Tuning for Non-language Machine Learning Tasks
Authors: Tuan Dinh
Yuchen Zeng
Ruisu Zhang
Ziqian Lin
Michael Gira
Shashank Rajput
Jy-yong Sohn
Dimitris Papailiopoulos
Kangwook Lee
Venue: NeurIPS 2022 Accept (2022)
Keywords: language interface
language-interfaced learning
fine-tuning
language models
non-language tasks
regression
classification.

Abstract: Fine-tuning pretrained language models (LMs) without making any architectural changes has become a norm for learning various language downstream tasks. However, for non-language downstream tasks, a common practice is to employ task-specific designs for input, output layers, and loss functions. For instance, it is possible to fine-tune an LM into an MNIST classifier by replacing the word embedding layer with an image patch embedding layer, the word token output layer with a 10-way output layer, and the word prediction loss with a 10-way classification loss, respectively. A natural question arises: Can LM fine-tuning solve non-language downstream tasks without changing the model architecture or loss function? To answer this, we propose Language-Interfaced Fine-Tuning (LIFT) and study its efficacy and limitations by conducting an extensive empirical study on a suite of non-language classification and regression tasks. LIFT does not make any changes to the model architecture or loss function, and it solely relies on the natural language interface, enabling "no-code machine learning with LMs."  We find that LIFT performs comparably well across a wide range of low-dimensional classification and regression tasks, matching the performances of the best baselines in many cases, especially for the classification tasks. We also report experimental results on the fundamental properties of LIFT, including inductive bias, robustness, and sample complexity. We also analyze the effect of pretraining on LIFT and a few properties/techniques specific to LIFT, e.g., context-aware learning via appropriate prompting, calibrated predictions, data generation, and two-stage fine-tuning. Our code is available at https://github.com/UW-Madison-Lee-Lab/LanguageInterfacedFineTuning.

LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks
Tuan Dinh; Yuchen Zeng. Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn, Dimitris Papailiopoulos, Kangwook Lee
University of Wisconsin-Madison. USA
Abstract
Fine-tuning pretrained language models (LMs) without making any architectural changes has become a norm for learning various language downstream tasks. However, for non-language downstream tasks, a common practice is to employ task-specific designs for input, output layers, and loss functions. For instance, it is possible fine-tune an LM into an MNIST classifier by replacing the word embedding layer with an image patch embedding layer, the word token output layer with (0-way output layer, and the word prediction loss with 10-way classification loss, respectively. A natural question arises: Can LM fine-tuning solve non-language downstream tasks without changing the model architecture loss function? To answer this, we propose Language-Interfaee (LIFT) and study its efficacy and limitations by conducting an extensive empirical study on a suite of non-language classification and regression tasks. LIFT does not make any changes the model architecture loss function, and it solely relies on the natural language interface, enabling "no-code machine learning with LMs." We find that LIFT performs comparably well across wide range of cow-dimensional classification and regression tasks, matching the performances of the best baselines many cases, especially for the classification tasks. We also report experimental results on the fundamental properties of LIFT, including inductive bias, robustness, and sample complexity. We also analyze the effect of pretraining on LIFT and a few properties/techniqee specific to LIFT, e.g., context-aware learning via appropriate prompting, calibrated predictions, data generation, and two-stage fine- tuning. Our code is available at httpt:://// ///ggittuub... com -on---o----e---aaaae Languagelnterfacedd ieniiinnTnnngg
Fine- Munnng
Introduction
Deep neural networks have been highly successful across multitude of domains, from computer vision [1] 2 and natural language processing 4| to game playing [5] K) Most advances in deep learning have come with a variety of domain-specific designs for network architectures, such as convolutional filters 7 g g for vision tasks, or recurrent modules (TO] [1l] and attention mecha-
nisms 12 13 in the context of natural language processing. A mommain-nndmmddllle agnostic model that can be adapted to solve tasks across different modalities and domains has become desidera- tum [4, motivating great efforts transfer learning [I5] and multi-modal learning 16 Recently, transformer-based language models (LMs) 13
I9) exhibited impressive versatility across different domains and modalities. They have shown great performances for various language-based tasks (20 such question answering 21 2 or commonsense reasoning 2 They have also been applied to non-language modalities
18
I& For instance, GPT-2 [7]] pretrained on language data can
be efficiently fine-tuned to perform image classification and numerical computation [18].
When downstream tasks are language-based tasks, adapting pretrained LMs can be achieved without modifying the models' architecture. Typically, this adaptation is enabled via simple fine-tuning [24
"qqual contribution. Emails: Tuan Dinh (tuan. dinh@uisc.edu) Yuchen Zeng (yzeng58@risc.edd)
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Training Data
Test Data
sopel longh
Sopal widh (omo
posa longth
potal dh
Sipal longt (om)
scoal wch (Cn)
posi lough (on)
pota wich
cas
(on)
(on)
(om)
(om)
3.5
4.1
2.8
47
12
Insvenscolor
Sentence Conversion
Sentence Corversion
Oution
An hiss plent with repal length Iem, sepal widh 3.5cm, pelal length petai with 0.2em
LM
Madel
Trompp
Inis plant with sepal langth 6.8cm, sepal width 3.0cm, petal langth 5.5cm. and potal width 2.1cm
names
An Hiss plant with sepal length sepal width petal ength 4.7 cm, and petal withh 1.2cm is ins-vericclo::
Ins-inginic..
1xx, -.1.1, -35. -14.x, 0.2,
LMI
Pronppt
IT - 68, 3.0, 5.5 2.,0hen y
Time Turnng
Vithou
Completion
1xx, = 611,,, 28, xy 4,7,x, 122 then y =it versicolor.
Inswinghic..
Training
Inference
Fine- Tuniin (LIFT) framework. LIFT has a two-phase procedure: (1) converting the dataset into sentences and (2) fine-tuning the pretrained language model (e.g., GPT) on the obtained sentences. This figure visualizes how LIFT can be applied to the Iris classification task. We first convert the Iris dataset into plain English sentences (left). Since feature names and the task description are available for this task, one could incorporate them as part of the prompt (as option in the figure). (In Sec. 4.1 we show that adding such contextual information to prompts helps LIFT achieve higher predictive accuracy.) One may also choose to use a simpler prompt with a generic naming convention (x1, ....... .aa for features (as option in the figure). After the sentence conversion step, LIFT fine-tunes a pretrained LM with the sentence set without making any changes to model architecture or loss. At inference time, we sentence form using the same prompt, excluding the label part. LIFT
Figure 1: A high-level illustration of the
convert the test samples performs surprisingly well in various non-language rggression/classifcatiin tasks, and we summarize our main findings Table g Note that to obtain model for given task, all we need here is to design proper sentence templates for LIFT and no changes to architecture or loss functions are needed.
However, not altering the architecture may pose a limitation for transferring to non-language tasks. As their input and output formats are not in some language form, adapting LMs to these domains may seem to require architectural changes. Indeed, it has been common practice re-design the input/output layers and loss functions to accommodate different predictive task. For instance, to adapt GPT-2 [2I] to other modalities, the frozen pretrained transformer [[]] adds new input/output layers to handle different types of input/output. To make such changes, one must have good understanding of the underlying principles of LMs and an ability make proper modifications at the code level.
25 26 27 or in-context few-shot learning methods 28
natural question that arises is whether such changes are necessary. In other words,
Does language model fine-tuning work for non-language tasks without changing the architecture or loss function all?
To answer this, we consider a simple fine-tuning procedure for LMs, referred to as Language- Interfaced Fine-Tuning (LIFT). This procedure can be used to learn predictors for any classification or regression task. LIFT runs in two phases: (1) converting labeled samples into sentences, and (2) fine-tuning pretrained LMs the sentence dataset without altering the architecture loss function.
Fig. T iillustrates how we fine-tune GPT with LIFT solve the Iris classification task 60) LIFT
first converts each labeled sample into sentence with two options. The first option is to incorporate feature names and the task description into the sentence template. this example, we could convert training sample into "An Iris plant with sepal length r.sepal length, sepal width r.sepal_ asepal_width, petal length r.petal length, and petal width r.petal width is rclass." Here, we use the dot notation, i... denotes the string conversion of the corresponding attribute of sample r. One may also adopt a simpler (and more generic) sentence template, such as "If =1=x,x1, x2=r.x2... .... xp=r.xp. then y=r.,"" if there are p features. We then fine-tune LMs without changing either architecture or loss function. Then, we perform inference as follows. LIFT first converts test samples into sentences using the same template while leaving the prediction part empty. It then feeds the converted sentences as prompts to the fine-tuned model. The output tokens are parsed to provide the final predictions.
Our work empirically shows that LIFT can provide high-accuracy solutions for variety of non- language tasks. Fig. 2 shows examples of real functions learned by GPT-J models (II fine-tuned using LIFT given 1000 samples. Recall that LIFT does not require any changes in the architecture

---

(A) Inreec
(bl quadoste
(d) exponeziial
(d) oosize
(G) IL DOURD
28 plecesiee
TYRUUE
0
0
RAE
RAB C.3E
RAE
RAE 6:31
FAE
LIT
10
8
0
0
8
0
Figure 2: Approximating various functions with LIFT using GPT-J. We visualize the target functions (first row) and the predictor functions learned by LIFT on GPT-J (second row). Blue dots show the 1000 training samples. One can observe that LIFT well approximates the target functions.
or loss function, Thus, our findings show that such changes architecture/loss function might nof be necessary, even when the target predictive task is not a language task. Thus. LIFT can be almost perceived "no-code machine learning" framework as the data-to-sentence conversion is extremely straightforward even without extensive programming skills and machine learning backgrounds.
Motivated by these intriguing properties, we investigate the efficacy and limitations of LIFT on non-language tasks by conducting an extensive empirical study on a suite of classification and regression tasks. First, we observe that LIFT performs well across wide range of low- oow-dinessional classification and regression tasks. In most cases, it nearly matches (or slightly outperforms) the best baselines' performance. To further understand LIFT, we conduct experiments testing the fundamental learning properties, e.g, its inductive bias, sample efficiency, ability to extrapolate, worst- and average-case noise robustness, and how the pretraining of LMs affects LIFT. Third, we study a few unique properties specific to LIFT, e.g., context-aware learning with task-specific prompting, prediction calibration, and the additional use of LIFT for data generation. Lastly, to improve upon the basic fine-tuning, we employ a few techniques: two-stage fine-tuning with synthetic pretext tasks and data augmentation. Both techniques improve the performance of LIFT. We finally provide discussions on limitations and future investigations of LIFT.
Scope of the study. Our work proposes the use of natural language interface for learning with LMs via LIFT. We emphasize that our goal is not to achieve the state-of-the-art performance, but to investigate thoroughly: (i) what LIFT can and cannot do, (ii) properties of models fine-tuned via LIFT, and (iii) whether we can improve LIFT with advanced techniques.
2 Methodology and Experimental Setup
LIFT training. To fine-tune a pretrained LM with LIFT on a target supervised learning task, we apply two steps: (1) convert each sample into sentence with a fixed template, and (2) fine-tune LMS with sentence datasets. We use the default cross-entropy loss for token prediction LMs. Our generic template (without feature names and task description) for sample r is
When we have xl=rx1, x2=rx2, xp=r.xp, what should be y?
#
r.y
@@@
question
a/a separator
answer end of
if r has p attributes. Here, we use the separator convention recommended by OpenAl
32
"*H#"
for question/answer separation, and "600" for end of generation. When attributes names and task deseriptions are available, can use more informative prompt (shown in Fig. M with all actual prompts are provided in Sec. 4.1 We report learning curves of LIFT on several tasks in Appendix E.5
LIFT inference. For inference, we use the same prompt template except for the answer and end-of-answer parts. Once the fine-tuned LM completes the test prompt, we simply parse the output tokens. For classification, we simply compare the generated text with the string representation of the class names. For regression, we convert the generated string into a number. For instance, with the output being "y=10.35 10w.0.55000exttatkeeee into two parts. Taking the first part 1y=10,,"", we parse the value "10.35" as our final prediction.
we split the output sentence by the stop separator "@ec"
The generated output might be invalid. For classification tasks, the output string may not match any actual class, which we declare as misclassification. Note that one may obtain better accuracy by returning its closest class using string metrics. For regression tasks, we consider output invalid if the string-to-numin parsing fails, these cases, we adjust the generation randomness by increasing the decoding temperature [3 R4 35 from 0 (deterministic mode) to 0.75 (random mode). We repeat
3

---

the inference up to five times, then return the average value of the training set if all attempts fail. Note that invalid output occurs very rarely (less than or around 1% in most tested cases).
For evaluation metrics, we use accuracy for classification tasks, and RMSE, RAE errors for regression Mal/ >i 1. J Mj yil and RMSE vE.- (0: 3:)2/n
tasks, where RAE >im 1i
on each dataset D { 36) }6 i==1 with samples, features E X' C RP, and outcome y.
Pretrained LMs. We apply LIFT on two pretrained LMs: GPT-J 31 and GPT-3 [9]. To fine-tune GPT-J, we use LoRA low-rank. For experiments on GPT-J, we used p3 8xlarge and p3. 2xlarge instances from AWS and RTX3090 GPUs in the local server. Since GPT-3 is not fully publicly available, we use the API provided by OpenAI to perform black-box GPT-3 fine-tuning. More details are in Appendix C.2.1
244 parameter-effiient method that constrains weight matrix updates to be
Datasets. We design and select a wide range of datasets to better understand the behavior of LIFT. For classification, we three types of non-language data: low-dimensional synthetic datasets, real tabular datasets in OpenML 36]. and vision datasets (MNIST 377 Fashion-MNIST B8 and their permuted variants (39 For regression, we use both synthetic and real datasets. For synthetic datasets, we defined samples (æ,, 34) of input-output pair as ~ f(x + (O, o²), where o² > is the noise level. Unless otherwise stated, we sample the feature x uniformly from a hypercube [L, U7P.
where L and U are minimum/maximmn
feature values, and is the number of features. Following the suggestion by 40,, we consider various functions f for regression tasks: (i) linear function, (ii) quadratic function, (iii) exponential function, (iv) cosine function, (v) é1-norm function, and (vi)
piece-wise linear function. Their 2D visualizations are provided in the first row of Fig. 2 We also use four real datasets: Medical Insurance (Insurance) 41, Combined Cycle Power Plant (CCPP) (42
Servo and Student Performance (Student)
More details are included in Appendix
Baselines. We consider standard learning algorithms 45 46.. For classification, we use logistic regression (LogReg). decision tree (DT), k-nearest neighbor (KNN), support vector machine with Gaussian kemnel (SVM), four-layer ReLU neural network (MLP) with 200 neurons per hidden layer, random forest (RF), and XGBoost (XG). We also use the majority class classifier (MCC) that outputs the most dominant class. For regression, we use polynomial regression (PR), kernel ridge regression (KR) with radial basis function kernel, k-nearest neighbors (KNN), three-layer ReLU neural network (MLP) with 50 hidden neurons per each layer, Gradient Boosting Trees (GBT), random forest (RF), and Gaussian process (GP). For hyperparameter selection, we apply the grid search on a set of parameters' values and use cross-validation on the training set (see details in Appendix +2
3 Basic Findings of LIFT
Table summarizes our main findings. We also study sample complexity (Sec. 3.2). comparison
(Sec.B.4). with in co-context learning (Sec. 3.3). models' decision boundaries (Sec. B.4). and the effect LMs'
pretraining LIFT (Sec. 3.6 Appendix E provides additional results, including the effect of input and output layers model size and LIFT for Ridge regression
3.1 How Well Does LIFT Perform on Standard ML Tasks?
Classification. Table compares classification accuracies between algorithms on a wide range of tasks. We observe that LIFT achieves comparable performance to most baselines. In most cases, LIFT/GPT ranks highly in the top three best- pest-performing methods. We find that LIFT can learn non-linear relationships between features and the responses: LIFT/GPT-3 achieves 81.17% accuracy on the circle dataset,, while logistic regression failed to perform better than the MCC (50%). As the difficulty of tasks varies, which can be estimated by the average performance of baselines, LIFT also suffers from performance degradation. LIFT can perform comparably well even when the number of features is large hundreds, though the limited number of tokens inputs to LMs restricts the number of features LIFT can input. However, when the number of classes is large (say 100s), both LIFT/GPT models have lower accuracies than many baselines, though they manage to be better than MCC. For instance, on the 100-class Margin dataset, the accuracy gap between LIFT/GPT-J and the best algorithm (RBF-SVM) is nearly 30%. Note that LIFT can directly raw data while most baselines require feature scaling and normalization for good performance. More results are provided in Appendix D.I.I including comparisons with methods leveraging larger models.
Regression. Tables and 20 present our function approximation comparison. For the low- dimensional cases, LIFT 18 comparable to baselines. Still, it fails to beat the strongest basclines, such

---

Table 3: Summary of the main findings.
Topic
Findings
Overall performance
On various classification tasks, LIFT achieves accuracies comparable to strong baselines (Table 4. For regression, LIFT well approximates different types of low-dimensional functions (Fig. but does not perform well for high- high-dimensional cases (Table 19
Robustness
For regression, LIFT is robust to outliers in training data (Fig- 28 For classification, LIFT is comparable to baselines under label corruption on training data (Fig. 20 but more vulnerable to feature corruption on test data (Table. 33
Context-aware learning
We can improve LIFT on classificatio tasks by designing prompts to specify feature names and the target task. The improvement is sigmificant when the description of the feature names and the target task can be interpreted with common knowledge (Table P).
Two-stage training
Warming up LIFT with pretext tasks using synthetic data improves the prediction performance, especially in the low-data regime (Fig. 40
Data
For classification tasks, training with augmented data significantly improves the tolerance
augmentation
of LIFT against perturbed test data (Table 12
Table 4: Accuracies (t) on classification datasets. We evaluate LIFT/GPTss on 2D synthetic data, tabular data OpenML [6), and image data, varying number of features (p) and data classes (c). Overall, LIFT/GPTs perform comparably well across tasks, adapting to non-linear datasets (circles, two circles) beyond the capacity of logistic regression. For OpenML datasets, they achieve competitive performances with the best methods, e.g., XGBoost). The performance degrades when more classes are given, e.g., c=100. They achieve competitive accuracies on both MNIST and Fashion MNIST. Note that MNIST's classes are not fully balanced; thus, MCC achieves 1 1..5% instead of 10%. Table 17 provides the full comparison with all baselines (KNN, MLP, Random Forest).
Dataset (ID)
ple
MCC
LogReg
DT
RBF-SVM
XG
LIFT/GPT-J LIFT/GPT:3
Synthetic Data
circles (3)
2/2
50.00
48.58+1.94
77.42+0.24
83.08+0.59
81.42-+0.11
79.95=1.53
81.17=0.42
two circles (6)
2/2
50.00
49.83:4.18
75.50=0020
80.0010.54
79.25+#0.35
75.92=1.65
81.42:0.82
blobs
2/4
25.00
96.7510.00
96.0810.82
96.75.10.00
96.17.0.12
96.17.0.59
96.66
moons (4)
2/4
50.00
88.58=0.12
99,25#::41
100.00:00.00
99.83-#0.12
99 99.58:20.42
100.00
9Clusters (1)
2/9
11.25
100.0010.000
100.001..000
100.0010.00
100.001..000
99.7510.00
100.00.0.000
Tabular Data (OpenML)
Customers (1511)
8/2
68.18
07.12:0054
85, 0.:50
86.36:20.00
85.23-+0.00
85.23
84.85 1.42
Pollution (882)
15/2
50.00
58.33111.79
77.7813393
58.33.16881
63.8917.86
63.8913393
63 89.8.687
Spambase (4-4)
57
60.59
93.27::0000
90.7:10.14
93. 93.70:10.00
95.87:10.00
94. 03 33:5
94 90 0.36
Hill- Valley (1479)
100/2
49.79
77.78=000
56.38+0.89
68.72+40.00
99.26+00.00
10..00+0.20
99.73-4.19
IRIS (61)
4/3
33.33
96.67=00.00
97.77+3.85
100.00=0.00
100.00::0.00
96. 96.67=00.00
97.00 97.0=-0.00
TAE (48)
5/3
35.48
45.16114.56
65.3911549
53.76.16.63
66.67.88.0
61. 169697
65, 59.693
CMC (23)
9/3
42.71
49.49910.83
56.72:10.32
56,50.10.97
52.43-10:12
83 00.28
0..74
Wine (187)
13/3
38.89
100.00::00 0.000
93.52:+2222
100.00
97.22:+0.00
93. 12..52:1133 1.51
92.59 1.31
Vehicle (54)
18/4
25.88
80,39+1.000
63.924-2.37
81.18+00.88
73.14++0.28
70. 20.20-22.73
LED (41496)
7/ 7/10
11.00
6..67
66.33-42.87
688 0.800010.82
66.00::082
65.33:10.47
69.33-:2.05
OPT (28)
64/10
10.14
96.53110.22
89.8.11.09
97. 37.05
97.48.10.17
98.22-10.11
98.9910.30
Mieat (12)
216 10
10.00
97.67 12
87.67 6:0:::00
98.83:10.44
96.75+00.00
94.17=
93.08:00.24
Margin (1491)
64/ 100
0.94
81.35
43. 1..6111.21
81.9810.30
70.2110.29
10.23311.33
59.37.0.92
Texture (1493)
64/ 100
0.94
81.67:00.97
46.88:11.93
83,44:10.99
70.73::1.41
50.32:22118
67. 67.50::1A43
Image Data
MNIST Permuted MNIST Fashion MNIST
11.35
91. 31.95:00.69
87.42:10.61
97.7011.97
97.69.10.04
97. 97.01:11.15
98.15=0.67
784/ 10
11.35
92.58=n004
87.87++069
98.06:10.31
97. 97.62++0.99
95.80- 0.07
96. 96.25=00.35
10.00
85.59:00.09
80.52++0.0 40
90.59:1.02
90. 90.19:#0.04
85. + 0.19
90.18 #:0.12
Permuted MNNIST
10.00
84.9510.44
79.9110093
18.04:11.99
89.93:10.44
82.25
88. 88.9220.71
as GP, as GPT models measure the error by comparing tokens instead of measuring how close the prediction values are to true values. We conjecture that we can improve our performance by level encoding, i.e., representing numerical values as binary values. We also investigate the interpolation and extrapolation of LIFT and defer the details to Sec. D.I.I All methods fail to extrapolate and interpolate well for all functions, and the interpolation performance of LIFT is only good in the linear regression case. Interestingly, LIFT tends to output seen values (from training data) for extrapolation.
3.2 How Many Samples Does LIFT Need?
We investigate whether LIFT is sample efficient. Fig. 25 in Appendix shows the sample complexity evaluation on classification and regression tasks. We find that GPT models can be quickly fine-tuned to learn new tasks with LIFT. For classification, as the number of classes increases (left to right columns in Fig. 25 b), LIFT does need more samples for adaptation, probably because the data input
5

---

Table 5: Comparison accuracies (1) between ICL and fine-tuning with LIFT on MpenML datasets. "LIFT/Full-Datt and "LIFT/Subset" represent LIFT the full dataset and and its subset used correspondingly in the ICL setting (number of prompts). Here, the size of subset is chosen to satisfy the LMs' context length. Overall, LIFT/GPTs on full data achieve the best performances. However, when using the same number of samples, LIFT and ICI. are more comparable in most cases. Note that both methods may be worse than MCC due the limited training data in some
GPT-I
GPT-3
Dataset (ID)
iProrppes
MCC
In-Context
LIFT//
In-Context
LIFT//ubset
LIFTT/ull-dat
Breast (13)
35
70.69
16.90=119951
58.62=244
64.95211.97
62.07=1.41
7..69=0.00
71,26=1.02
TAE (48)
35.48
34433+1.47
32 52.26=9950
61.29-4.56
37. 37.64:4.02
33.33=1-52
65.59-6.63
Vohiclo (54)
14
25.88
25.49:0.55
26.04=1669
64.31=2.37
28.8222.10
23.73=2.27
70.20:2.73
Hamster (893)
43
53.33
48 8943,11
60.00=100.88
16.55=16633
67.78=66229
53.33=0.00
53.33=0.00
Customers (1511)
56.06+117.14
85.23+1.61
60.61+1442
63.26
84.85-1.42
LED (40496)
33
68.67
10.00=0082
13.04=3.27
63.33=2047
8,00+1.63
11.33=2.62
69,33=2.05
and output spaces are more complex to learn. For regression tasks, we find that 1000 samples are sufficient for LIFT to have small RMSE, similar to other baselines. There exist some functions (e.g., cosine and piecewise) where LIFT has lower sample complexity than popular baselines.
3.3 Language-Interfaeed Learning: LIFT versus In-Context Learning (ICL)
Beyond fine-tuning (with LIFT)), our language-interfaeed learning framework can be used for other
learning methods for LMs, including in in-context learning (ICL) 47 48 [9] that performs inference
on new tasks without fine-tuning by conditioning on few training examples. Table compares the
classification performances between (a) ICL, (b) LIFT trained on subset with samples, and (c) LIFT trained on the full dataset. Note that the number of training samples (ni) used for ICL depends on the context length of given LMs. As we can see, LIFT using the full dataset always achieves the best performances. However, LIFT/Subset and ICL are more comparable in most cases when they use the same number of training samples, which sufficiently small for ICL methods to fit in LMs.
Remark. One can replace fine-tuning with ICL in our lunguage-interfee procedure when the target tasks require fewer truining samples.
3.4 Can We Understand the Inductive Biases of Language Models via LIFT?
To better understand LIFT/GPT'' inductive biases, we investigate their classification deci- sion boundaries varying the boundaries' com-
True Funeioo
LIFT/eTT
LIFT/PPT
plexity, as shown Fig. 6 We first train a binary-class neural network and use its snap- shots at different training epochs to construct datasets having decision boundaries at differ- ent complexity levels (first column in Fig. (). We observe that LIFT/GPT models adapt well to three boundaries and capture their rough shapes. Furthermore, their boundary shapes are axis- sais-larllll,, similar to the bound- ary of tree-based classifiers. They also show a lot of fractals similar to the observations on some convolution neural networks Appendix D.1.3 for results of 3-class and 5- class datasets and quantitative measurements.
Figure Decision boundary visualization. We use three snapshots of trained network to construct datasets having labels as their predictions (the first column). Top bottom: snapshots with more training epochs, corresponding to more complex boundaries. LIFT/GPTs adapt well on different boundaries.
40 See
3.5 How Robust LIFT?
We investigate the robustness of LIFT against the outlier samples in training data and the feature corruption on test data. Appendix D.1.4 provides additional experimental results, including the robustness for the case of label corruption training data and iaas-immallnceed data.
Robustness to outliers in training data. We consider regression tasks where we have outliers whose outcome y is not consistent with the majority of samples in terms of fitting (æ, y). Fig. 28 compares RAE values of methods with and without outliers (2% outliers in the training set). LIFT7GPT models among the most robust ones: their performances are almost unaffected, while basclines suffer

---

Table 8: Accuracies (1) of LIFT with different LMs. We compare variants of LIFT with different LMs: LIFT/GPTs using GPTs pretrained on natural language data (our models), LIFT/Rand using a randomly initialized GPT-J, LIFT/CodeGen and LIFT/CodeParro using LMs pretrained on programming language data, and LIFT/Gibberish using GPT-J fine-tuned on gibberish data.
Datasei (ID)
MCC
LIFTAGPT-3
LIFT/GPT
LJFT/Rand-GPL. LIFT/Gibberssh
LIFT/CodeGen
LIFTT/CodePPrrot
Blabs (2)
25.00
96.67 0.24
96.174 0.59
25.65 1.58
96.42 0.24
93.671: 0.72
93.391 1.82
Two Circles (6)
50,00
$1,42= 0.82
75.924 1.65
49.88 5.01
68.671 1.50
53,021 0.66
50.081 2.47
97.0+
27.78
43.314
Oustomers (1511)
68.18
84.85
1.42
85.23+ 1.61
52.47+ 7.15
63 1.42
45.96+ 8.96
43.11+ 3.54
Wine (187)
38.89
92.59= 1.31
93.524 1.31
22,22 15.71
84.26:2 3.46
77.781 0.00
33.881 3.87
LED (40496)
69.33 2.05
65.334
11.68
72.67 1.25
11.004
23.46
huge performance drops. Furthermore, we evaluate models under various percentages of outliers (1%, 2%, 5%, 10%, 20%). as shown in Fig. 28p. Compared to the robust baselines (median-3ND) and median-SMi 55N)) [50], LIFT/GPT-3 comparably robust, while LIFT/GPT-J is more vulnerable when more outliers are present.
Robustness to feature corruption test data. Given clean test data (z. y) having feature x and label y. we explore whether adding small perturbation ò on the feature changes the performance; we measure the accuracy of LIFT on perturbed data (ar + y. We apply transfer attack [5T] since we do not have full access to the GPT-3 model, and finding adversarial
Table 7: Accuracies (f) under the perturbation the input feature of MNIST data. See the full results
in Table 33 in Appendix D.1.4
Source
PGD attack LeNet-5
examples in the discrete input space complex [52]. Table 7 reports robustness results on MNIST classification under PGD
Target
LcNet-5
MLP LIFT/GPT-3
attacks transferred from LeNet-5. The perturbation radius is
0
99.22
98.09
98.15
0.01
97.27
97.77
44,88
set to E [0, 0.01, 0.1, 0.3] where MNIST pixel value is within [0.1]. We compare three networks: LeNet-5, MLP (2 hidden layers with 300 and 100 neurons), and LIFT/GPT-3. When
0.1
26.80
93.99
33.66
0.3
0.00
36.62
20.31
e E (0.01, 0.1], LIFT/GPT-3 tolerates random noise (as in Table
33 but cannot tolerate transferred
adversarial attack, implying that the adversarial attack on LeNet-5 IS transferred to LIFT/GPT-3.
3.6 Does LIFT Need Large-Scale Models Pretrained on Natural Language Data?
We investigate the requirement of pretrained LMs for which LIFT performs well. We compare variants of LIFT under different types of LMs: GPTss pretrained on natural language data (our models), a large LM without pretraining BRand-GPT-JJ and LMs pre-trained on non-human language data, including CodeParrot [53] and ModeGen-2B-moone data, and a GPT-J fine-tuned on Gibberish data [5 See Appendix D.1.5 for the detailed setting.
trained mainly programming language
Does LIFT only need large pretrained model? To answer this question, we compare performances of LIFT when GPTs are pretrained (LIFT/GPTs) and when GPT-J have weights being randomly More specifically, for HFFT/Rand-GP.. we randomly initialized a GPT-J model and fine-tuned the whole model (instead of LoRA), As shown in Table g accuracies of LIFT/Rand-GPT-J are much lower than those of our models (LIFT/GPTss, across all datasets. These results indicate that LIFT benefits from pretraining, not just from the large-scale design of LMs.
initialized (LIFT/Rand-GPT:
Does LIFT need model trained on natural language data? As shown in Table 8 LIFT/GPTs perform much better than LIFT/CodeGen and LIFT/CodeParrot for all datasets. This implies that LIFT may perform better with LMs pretrained on natural language data. When the pretrained GPT-J is fine-tuned on gibberish data (53), the accuracies drop for few tasks and are lower than LIFT/GPTs overall. However, LIFT/Gibberish still achieves comparably good performance and its small performance gaps to LIFT/GPT-J can be attributed to the relatively light impact of fine-tuning on large pretrained LMs. Thus pretraining on natural language data is necessary for LIFT.
4 Evaluation of LIFT-Specific Learning Properties
In this section, we study the behavior of LIFT in a more fine- fine-grained manner.
4.1 Does LIFT Benefit from Incorporating Feature Names?
Unlike standard machine learning algorithms, LIFT can be provided context information by incor- porating the feature names and task descriptions in the prompts. Intuitively, this incorporation may improve the sample complexity of LIFT as the prior knowledge already learned in the pretraining
7

---

Table 9: The effect of using feature names on LIFT. We compare classification accuracy (1) of LIFT/GPT-3 when feature names provided in the target dataset are and are not incorporated into the prompts. We provide four versions of LIFT when feature names are conrectly incorporated (Correct-Named columns) and when feature names are randomly shuffled (Shuffle--Names columns). We evaluate models on three OpenML datasets, including CMC (23), TAE (48), Vehicle (54), and German. We also compare our models with two baselines: the majority class classifier (MCC) and XGBoost. As result, all LIFT models achieve better performance than MCC. Among the evaluated models, LIFTss with correct feature names achieve the best accuracies on both TAE, Vehicle, and German datasets while achieving the comparable accuracies the best model on the CMC dataset. *Two designs of the format result in the same template for the Vehicle dataset.
LIFT
Dataset (ID)
MCC
W/o
Names
Shuflled-Names
Shuflled Namees
Corret-Names Names
Correct- Names
CMC (23)
42.71
$7.74.1.889
$7.40 1.37
56.27
4..06114224
$7.40.1 1.09
$6.27 2.2227
TAB
35.48
65.5911.63
66.6715.48
60.22:16.72
64.5228.53
69.8919.31
69.89.16.72
Vehielo (54)
25.88
2.720
71.96=3.09
70.20:25,34
69.22-2.72
2..29
German 70.00
71.33
5.20
2.72
73.00 1.87
71.67 0.94
4 1.70
74.171 1.25
phase may help LIFT predict better. We design seven prompt templates to assess how incorporating feature names affects the performance of LIFT (see more details in Appendix p.2.I). We empirically verify this intuition and show our results in Table for several classification tasks using pretrained GPT-3 models. We first observe that all LIFT models outperform MCC with significant accuracy gaps, indicating that they are all properly trained. Second, we observe that correctly incorporating feature names helps boost the performances of LIFT for datasets except for CMC. Third, if we use similar prompts with shuffled feature names (Shuff Schuffeed-oe -Nd-e I, II), then the performance of LIFT drops by a significant margin. These results imply that the aforementioned performance improvements are indeed due proper prompting with correct feature/value association.
4.2 Is LIFT Well Calibrated?
We investigate whether LIFT is calibrated, i.e., the prediction reflects the confidence, by exploring how LIFT performs under various noise levels, as shown in Fig. 36 in Appendix. We conduct experiments on six synthetic regression datasets, each consisting of 1,000 noisy training samples shown as blue markers in the first row. To be specific, we generate (1) the input x following the guideline in Sec. C.1 for regression tasks and (2) the noisy outcome y where the standard deviation
of noise a (x) = ( + 10) increases along the æ-axis (from x = -10 to x = 10), and study set the
how different noise level affects the predictive behavior of LIFT. In the inference phase, decoding temperature T = 1 for LIFT to make random predictions. For visualization purposes, we generate an additional 103 samples uniformly in [-10, 10] for each task and plot the standard deviation of 20 LIFT/GPT-J predictions on each sample in the bottom row of Fig. 36 Note that the bottom row of Fig. 36 shows that the standard deviation of LIFT/GPT-J's prediction nearly matches that of noisy training samples (observations) across different tasks. These results imply that LIFT/GPT-J is calibrated. Similarly, Fig. 37 of Sec. D.2.2 shows that LIFT/GPT-3 is calibrated.
4.3 Can We Use LIFT for Data Generation?
Generative models have been widely used in computer vision [50 57 [58. Beyond classifi- cation and regression tasks, we study whether LIFT can be used for generative tasks, i.e, learn- ing the underlying data distribution and gener- ating realistic data samples. In particular, we consider two image generation tasks on MNIST dataset: (a) generating an image given digit number, and (b) completing an image given digit number and its pixels on the top half of the image. Fig. [0 and Fig. 10p show our gener- ated images for the two tasks respectively. We observe that the generated images have the cor- rect digit shape and reasonably high quality in most cases, especially for the image completion (Fig. l0b). See Appendix D.2.3 for more details.
8
(a) Given only the digit number.
(b) Given the digit number and half of image pixels. Figure 10: Generating MNIST images using LIFT/GPT-J. We observe that LIFT/GPT-J can generate images of comparably high quality. The temperature is set to 1.
8

---

5
Improving LIFT with Existing Techniques
We improve LIFT with advanced techniques: two-stage fine-tuning and data augmentation.
5.1 Two-Stage Fine-Tuning for LIFT with Synthetic Pretext Tasks
In Sec. 32
we observe that LMs need a suffi-
Twosttge LFT/CPTT
UFT/CPTTJ
Pretect only
cient number of samples to start adapting. We suspect that LMs' adaptation to non-language tasks contains two phases: (1) learn the task de-
Arroracy
Accunazy
RMSE
scription, i.e., input space, label space, and sen-
tence templates 47 59, and (2) learn the target task. Thus, we consider utilizing synthetic data to describe the task for LMs in the first phase, thus reducing the sample complexity. This re- sults new two-stage training procedure for LIFT In particular, for any given dataset, we first generate two pretext tasks with simple syn- thetic Gaussian datasets (discussed in C.I) sharing the same number of features and the label space (for classification tasks) or the range responses' values (for the regression tasks) to the actual data. We apply LIFT on pretext tasks for a few (2 or 3) epochs, then continue LIFT with the target (given) dataset. For GPT-3, it is unclear how to keep the order of samples not shuffled with the current black-box API during the fine-tuning stage. Hence, we only provide the experimental results of GPT-J. Fig. M shows that two-stage fine-tuning improves LIFT over the original fine-tuning when the number of training samples is small on both classification and regression tasks.
Number training samples
Figure 11: Two-stage fine-tuning. The two-stage method (blue) applies LIFT first on synthetic pre- text data before the real datasets, outperforming fine-tuning (green) when training data is small. The full experiment results are presented in Fig. 40
5.2 Data Augmentation
Table 12: Accuracies (1) of LIFT with/without data aug- mentation (DA), as well as baselines (LeNet-5, MLP) on MNIST. Each row represents differrent ways of training, and each column means different test data. Data augmentation (DA) means that we are using noisy version of MNIST training data by adding Gaussian noise. Given an MNIST image having range [0,1], the noise is added in the L ball with radius e. One can confirm that the data augmentation significantly improves the tolerance of LIFT/GPT-J against perturbed test data in both Gaussian and signed constant noise. For each column, we boldfaced the highest value among baselines and the highest value among LIFT/GPT-J.
Data augmentation [6 is a simple tool for improving the generaliza- tion performance for various classi- fication problems. Here, we investi- gate whether data augmentation ben- efits LIFT. Table [2] shows the ef- feet of adding random noise in the training data on the performance of LIFT/GPT-J for the MNIST classifi- cation problem. Here, we test each model on three settings: (1) clean data, (2) Gaussian noise, and (3) signed constant noise. We allow each noise can perturb up to the magnitude of e E [0, 1] at each dimension (i.e., each pixel) when the black/white pixel of MNIST is represented in the [o, 1] range. We defer the generation procedure of random Gaussian noise to Sec. D.I.4 in Appendix.
Clean
Gaussian noise
Signed const. noise
0.01
0.1
0.01
LoNer-5 MLP
99.22
99.25
99.20
99,26
99.06
98.09
98.05
97.70
98.08
97.39
LIFTIGPTJ
96.88
95.27
56.14
55.83
27.73
LIFTIGPTH (Gaussian, 0.05)
93.
94.39
93.40
93.46
61.24
LIFTXGPTH DA (Gaussiaa, < 0.1)
93.78
94.31
94.98
94.12
75.25
One can observe that LIFT/GPT-J without any data augmentation (DA) is vulnerable to random noise, unlike existing baselines (LeNet-5 and MLP). However, when we apply data augmentation, ie., train LIFT/GPT-J with noisy training data, the accuracy improves significantly for the perturbed (either adding Gaussian noise Signed constant noise) test data. This shows the effectiveness of simple data augmentation in LIFT. Exploring the effect of other data augmentation schemes, e.g., is remained interesting future work.
mixup (2) and its variants 63 64
6 Related Works
Fine-tuning for adapting LMs to non-language tasks. Fine-tuning (66 pretrained LMs is the standard practice for learning downstream tasks, which may involve simple architecture modifi-
'The recent work 600 demonstrates the usefulness of the intermediate fine-tuning method for LMs. However, they focus self supervidd objectives for fine-tuning pretrained LMs for shot co-textex learning.

---

cations, such as adding linear layers 67 63
The recent progress techniques for reducing trainable parameters, including adapter- based 27 that trains additional small residual blocks between layers, freezing-ban -aaed
or freezing layers [8 69 71
focuses on parameter-efficienn
fine-tuning 22
26
fine-tuning 71
72 that freezes most of the pretrained parameters, and distillation-basee fine-
tuning [
Our LIFT/GPT-J is fine-tuned with LoRA 24 parameter-effiiiet method approxi- mating the weight updates using low-rank matrices.
To directly adopt existing fine-tuning methods of LMs for non-language tasks, it is common practice to modify the input/output layers and the loss functions, which may cause undesired behaviors like catastrophic forgetting [66] I74]. Our work is highly motivated by Frozen Pretrained Transformer
(FPT) [ that directly fine-tunes GPT-2 21 pretrained on language tasks for other modalities by freezing most pretrained parameters and adding only input and output layers for the modality adaptation. Unlike FPT, our method requíres no such changes in the architecture and objective function. Several works also extend the existing LMs to handle different input data types, such as
images 7 76, audio 777, tabular data 78. and knowledge base (7] by updating the pretraining phase with these data and their corresponding tasks or using general-purpose architecture [8)] Our work is based on GPT language models trained only on textual data.
Analyzing the adaptability of LMs. Similar to ours, recent works understand and quantify the adaptability 2 and capacity of large LMs, such as Big. Bench with new benchmark of more than 200 tasks on a diverse set of topics.
8
33
attempt to
General-purpose models. A primary goal of our work is push the limit of the existing generalist language models (e.g., GPT-3 (19] to other modalíties and domains, supporting the idea of building Note that LIFT can Furthermore, our
domain-and-men mnd-mod agnostic generalist model 19
35 86
89
be applied to any generalist model with LM-like architectures, such as GATO 80
work shares the general goal with automated machine learning (AutoML) 90 91 in improving the
usability of machine learning, though LIFT uses only a single pretrained LMs for all tasks while AutoML automates the standard machine learning pipeline from a set of existing algorithms.
7 Discussion and Conclusion
We propose the use of language-interfaced framework. via Languuge-Interfaeed Fine-Tuning (LIFT), for using LMs to solve non-language downstream tasks without changing the models' architecture or loss function. LIFT first converts labeled samples into sentences and then fine-tunes pretrained LMs on the sentence dataset using the standard fine-tuning method and loss function. Via an extensive empirical study, we show that LIFT/GPT performs relatively well low-dimensional classification and regression non-language tasks. Furthermore, LIFT/GPTs are robust in several practical settings, and can properly calibrate the predictions and generate realistic data samples. LIFT can be improved using in-context feature names, two-stage fine-tuning, and data augmentation. Moreover, our work is arguably one of the first to thoroughly study the efficacy of language-interfueer learning framework with pretrained language models on standard regression and classification tasks, paving the way for enabling "no-code machine learning with language models."
Limitations and open questions. Despite promising performances on various tasks and settings, we observe some limitations of LIFT to basic learning tasks. LIFT/GPT do not perform well the features have high dimensions (for regression) or when the number of classes is large (for classification). In addition, the context length of LIFT is restricted to the context length of LMs and LIFT/GPT is memory-inefficiene Lin Tinrrssffrrerr future works. First, do LMs and LIFT//GPT have behaviors similar to ensemble methods or decision tree since they have similar decision boundaries? Secondly, are LMs universal models that adapt well to any modalities and domains? Lastly, can LIFT/GPTs adapt better for regression tasks using more sophisticated encoding schemes for numeric features?
One can combine LIFT with memory-efficien LMS such as [92 address this issue. Besides, our works open some interesting questions for
Social impacts. Future research should also investigate potential fairness issues of applying LIFT. Based on large language models, LIFT might have embedded bias targeting certain social groups Especially when feature names are included in the training prompts, the models may be more sensitive to social biases and thus might make unfair and harmful predictions. We leave measuring the embedded bias in LIFT as one of the interesting future directions.
10

---

References
[1] David Forsyth and Jean Ponce. Computer vision: A modern approach. Prentice hall, 2011.
[2] Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold. Jakob Uszkor- eit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly, Thomas Unterthiner, and Xiaohua Zhai. An image is worth 16x16 words: Transformers for image recognition at scale. 2021.
[3] KR1442 Chowdhary. Natural language processing. Fundamentals of artificial intelligence. pages 603-649, 2020.
[4] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx. Michael S Bernstein, Jeannette Bobg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108. (tt.::2000002588, 07258, 2021.
[5] Fei- Wang, Jun Jason Zhang, Xinhu Zheng, Xiao Wang, Yong Yuan, Xiaoxiao Dai, Jie Zhang, and Liuqing Yang. Where does alphago go: From church-turing thesis to alphago thesis and beyond. IEEE/CAA Journal of Automatica Sinica, 3(2):113-120, 2016.
[6] Kai Arulkumaran, Antoine Cully, and Julian Togelius. Alphastar: An evolutionary compu- tation perspective. In Proceedings of the genetic and evolutionary computation conference companion, pages 314-315, 2019.
[7] Shih-Chung B Lo, Heang-Ping Chan, Jyh-Shyan Lin, Huai Li, Matthew T Freedman, and Seong K Mun. Artificial convolution neural network for medical image pattern recognition. Neural networks, 1201-114,,1 1995.
[8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[9] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pages 2961-2969, 2017.
[10] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal represen- tations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
[11] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 173: 1997.
[12] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need, Advances in neural information processing systems, 30, 2017.
[13] David So, Quoc Le, and Chen Liang. The evolved transformer. In International Conference Machine Learning, pages 5877-5886. PMLR, 2019.
[14] Lukasz Kaiser, Aidan N Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, and Jakob Uszkoreit. One model to learn them all. arXiv preprint arXiv:1706.05133 2017.
[15] Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big data, 3(1):1 40. 2016.
[16] Dhanesh Ramachandram and Graham W Taylor. Deep multimodal learning: A survey on recent advances and trends. IEEE signal processing magazine, 34(6):96-108, 2017.
[17] Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160-167, 2008
[18] Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Pretrained transformers as universal computation engines. arXiv preprint arXiv:2103.05247. 2021.

---

[19] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 1901, 2020.
[20] Belinda Z Li, Jane Yu, Madian Khabsa, Luke Zettlemoyer, Alon Halevy, and Jacob Andreas. Quantifying adaptability in pre-trained language models with 500 tasks. arXiv preprint arXiv:2112.03204, 2021.
[21] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[22] Dan Su, Yan Xu, Genta Indra Winata, Peng Xu, Hyeondey Kim, Zihan Liu, and Pascale Fung. Generalizing question answering system with pre-trained language model fine-tuning. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 203-211, Hong Kong, China, November 2019, Association for Computational Linguistics.
[23] Xuhui Zhou, Yue Zhang, Leyang Cui, and Dandan Huang. Evaluating commonsense in pre-trained language models. CoRR, abs/1911.11931, 2019.
[24] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 2766952200009685, 2021.
[25] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter--fffiient transfer learning for nlp. In International Conference on Machine Learning, pages 2790-2799. PMLR, 2019.
[26]
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. Advances in neural information processing systems, 30, 2017.
[27] Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Guihong Cao, Daxin Jiang, Ming Zhou, al. K-adapter: Infusing knowledge into pre-trained models with adapters. arXiv preprint arXiv::2022 1..002..18000 01808, 2020.
[28] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where find them: Overcoming sho- prompt order sensitivity. arXiv preprint
2021.
[29] Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heinz, and Dan Roth. Recent advances in natural language processing via large pre-trained language models: A survey, arXiv preprint arXiv:2111.01243. 2021.
[30] Kyle M. Monahan. Iris dataset for machine learning, 2020.
[31] Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. http:s///ggttabbb..oommiingg/g// otw.oo/////oooo-rrrrssrrrrrre jax May 2021.
[32] Openai fine-tuning documentation: Preparing your dataset. docs/
openai. com/
[33] Daphne Ippolito, Reno Kriz, Maria Kustikova, João Sedoc, and Chris Callison-Burch. Com- parison of diverse decoding methods from conditional language models. arXiv preprint arXiv:1906.06360 2019.
[34] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint 194.09190, 2019.
[35] Openai fine-tuning documentation: Create completion. https://beta. wwtet/aeoeaaaimmmemeeeeeeeeeeeee com/docs/ api -ww.feerenneeccoopeeeeeioot/rrre
[36] Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo Openml: Networked science in machine learning. SIGKDD Explorations, 15(2):49-60. 2013.
12

---

[37] Yann LeCun. The mnist database of handwritten digits. http://yam.lccu..comeeaammmm 1998.
[38] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mmin a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:/708.07710 2017.
[39] lan Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient- baade neural networks. arXiv preprint 1.Xiv:/312.6211, 2013.
[40] Keyulu Xu, Mozhi Zhang, Jingling Li, Simon Shaolei Du, Ken-Ichi Kawarabayashi, and Stefanie Jegelka. How neural networks extrapolate: From feedforward to graph neural networks. In International Conference on Learning Representations, 2021.
[41] Medical insurance dataset (kaggle).
https://www.ker
com/datasets/
[42] Pinar Tüfekei. Prediction of full load electrical power output of base load operated combined cycle power plant using machine learning methods. International Journal of Electrical Power & Energy Systems, 60:120::162. 160-140, 2014.
[43] John R Quinlan et al. Learning with continuous classes. In 5th Australian joint conference on artificial intelligence, volume 92, pages 343-348 World Scientific, 1992.
[44] Paulo Cortez and Alice Maria Gonçalves Silva. Using data mining to predict secondary school student performance. 2008
[45] Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
[46] Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. October 2021.
[47] Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors Computing Systems, pages 1-7, 2021.
[48] Taylor Shin, Yasaman Razeghi, Robert Logan IV, Eric Wallace, and Sameer Singh. Auto- prompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010. 15980, 2020.
[49] Gowthami Somepalli, Liam Fowl, Arpit Bansal, Ping Yeh-Chiang, Yehuda Dar, Richard Baraniuk, Micah Goldblum, and Tom Goldstein. Can neural nets learn the same model twice? investigating reproducibility and double descent from the decision boundary perspective. arXiv preprini arXiv:2203. 00133..8/4.... 2022.
[50] Peter J Huber. Robust statistics. In International encyclopedia of statistical science, pages 1248-1251. Springer, 2011.
[51] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning scale. arXiv preprint arXiv: 1.66/11011 1.0/266, 2016.
[52] Wei Emma Zhang, Quan Z Sheng, Ahoud Alhazmi, and Chenliang Li. Adversarial attacks on deep-learning models in natural language processing: survey. ACM Trunsactions on Intelligent Systems and Technology (TIST), 11(:)11-41, 2020.
[53] Loubna Allal, Leandro Werra, Thomas Wolf, and Li Jia. Codeparrot.
[54] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. A conversational paradigm for program synthesis. arXiv preprint
arXiv:2203.13474 2022.
[55] Better
gibberish detection with gpt-2. http::thh///eeelaaaaam/////////0
github. haaaubbio/bariiislappooogoggg/g
perterrgibberiss--eeeettto.....
13

---

[56] Ian Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11))139-144, 2020.
[57] Diederik P Kingma, Max Welling, et al. An introduction to variational autoencoders. Founda- tions and Trends@ in Machine Learning, 12(4):307-392, 2019.
[58] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey. arXiv preprint arXiv:2209.04447,2000 2022.
[59] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannanch Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202. 12837, 2022.
[60] Mingda Chen, Jingfei Du, Ramakanth Pasumuru, Todor Mihaylov, Srini Iyer, Veselin Stoyanov, and Zornitsa Kozareva. Improving in-context few-shot learning via self-supervised training. arXiv preprint 2r2iv:2205.0111 2022.
[61] Connor Shorten and Taghi M Khoshgoftaar, A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
[62] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09M4 2017.
[63] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023 6032, 2019.
[64] Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In International Conference on Machine Learning, pages 5275-5285, PMLR, 2020.
[65] Jy-yong Sohn, Liang Shang, Hongxu Chen, Jackyun Moon, Dimitris Papailiopoulos, and Kangwook Lee. Genlabel: Mixup relabeling using generative models. arXiv preprint urXiv:2201.02354. 2022.
[66] Zhizhong Li and Derek Hoiem. Learning without forgetting. JEEE transactions pattern analysis and machine intelligence, 40(12):2935-22) 2947, 2017.
[67] Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning, arXiv preprint arXiv:1605.09753 2016.
[68] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1697- PMLR, 2020.
[69] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[70] Ting Chen, Mario Lucic, Neil Houlsby, and Sylvain Gelly. On self modulation for generative adversarial networks. arXiv preprint 1110::8110001666 2018
[71] Mozhdeh Gheini, Xiang Ren, and Jonathan May. Cross-attention is all you need: Adapting pretrained transformers for machine translation. arXiv preprint arXiv:2 104.004..... (44.08771, 2021.
[72] Tuan Dinh, Daewon Seo, Zhixu Du, Liang Shang, and Kangwook Lee. Improved input reprogramming for gan conditioning. arXiv preprint arXiv:2201.0266 2022.
[73] Yen-Chun Chen, Zhe Gan, Yu Cheng, Jingzhou Liu, and Jingjing Liu. Distilling knowledge learned in bert for text generation. arXiv preprint arXiv:1911. 19t.::1991.038299 2019.
14

---

[74] Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. arXiv preprint arXiv:2004. 12651, 2020.
[75] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems, 32, 2019.
[76] Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200-212, 2021.
[77] Yung-Sung Chuang, Chi-Liang Liu, and Hung- Yi Lee. Speechbert: Cross-modal pre-trained language model for end-to-end spoken question answering. 2019.
[78] Qian Liu, Bei Chen, Jiaqi Guo, Zeqi Lin, and Jian-guang, Lou. Tapex: table pre-training via learning neural sql executor. arXiv preprint arXiv:2107.07665 2021.
[79] Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami AI-Rfou. Knowledge graph based synthetic corpus generation for nooweede-enhanneee language model pre-training. arXiv preprint arXiv:2010. 12688, 2020.
[80] Andrew Jaegle, Sehastian Borgeaud, Bean-Baptiste Alayrae, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14795, 2021.
[81] Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of large scale pre-training. arXiv preprint arXiv:2110.020955 2021.
[82] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2022.
[83] Yi Zhang, Arturs Backurs, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, and Tal Wagner. Unveiling transformers with lego: a synthetic reasoning task, 2022.
[84] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint 111.:2112.11446 2021.
[85] Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages I 11336-11344, 2020.
[86] Jean-Bay Saan-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204 14198, 2022.
[87] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, pages 8748 8763. PMIR, 2021.
[88] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le. Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904 PMLR, 2021.
[89] Scott Reed, Konrad Zoina, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06117. 2022.
15

---

[90] Matthias Feurer, Aaron Klein, Jost Eggensperger, Katharina Springenberg, Manuel Blum, and Frank Hutter. Efficient and robust automated machine learning. In Advances in Neural Information Processing Systems 28 (2015), pages 2962-2970, 2015.
[91] Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, and Frank Hutter. Auto-sklearn 2.0: Hands-free automl via meta-learning. arXiv:2007.04074 [cs.LG]] 2020.
[92] Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.0470 2020.
[93] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022.
[94] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners, arXiv preprint arXiv:2205.11911 2022.
[95] Zhijing Jin, Sydney Levine, Fernando Gonzalez, Ojasv Kamal, Maarten Sap, Mrinmaya Sachan, Rada Mihalcea, Josh Tenenbaum, and Bernhard Schölkopf. When to make excep- tions: Exploring language models accounts of human moral judgment. arXiv preprint arXiv:2210.01478, 2022.
[96] Karl Cobbe, Vincet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reichiro Nakano, Christo- pher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 1 2021.
[97] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.
[98] Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint 2xXiv2207,00222 2020.
[99] Devendra Singh, Siva Reddy, Will Hamilton, Chris Dyer, and Dani Yogatama. End-to-end training of multi-document reader and retriever for open-domain question answering. Advances in Neural Information Processing Systems, 34:25968-25988 2021.
[100] Dani Yogatama, Cyprien de Masson Autume, and Lingpeng Kong. Adaptive semiparametric language models. Transactions of the Association for Computational Linguisties, 9:362-375
2021.
[101] Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, al. Improving language models by retrieving from trillions of tokens. In International Conference on Machine Learning, pages
PMLR, 2022.
[102] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rXiv:1810.04805. 2018.
[103] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training 2018.
[104] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
[105] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self- supervisdd learning of language representations. arXiv preprint arxiv: /909. 10ee/1943 1942, 2019.
[106] Zhilin Yang. Zihang Dai, Yiming Yang. Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
16

---

[107] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 1901, 2020.
[108] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2 101. 03961, 2021.
[109] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts. Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXivv2204.02311, 2022.
[110] Junyi Li, Tianyi Tang, Wayne Xin Zhao, Yun Nie, and Ji-Rong Wen. A survey of pretrained language models based text generation. arXiv preprint arXiv:2201.05277 2022.
[111] Shuang Li, Xavier Puig, Yilun Du, Clinton Wang, Ekin Akyurek, Antonio Torralba, Jacob Andreas, and Igor Mordatch. Pre-trained language models for interactive decision-making. arXiv preprint arXiv:2202.01771, 2022.
[112] Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural lan- guage decathlon: Multitask learning as question answering. arXiv preprint arXiv: 1806.08730) 2018.
[113) Dan Su, Yan Xu, Genta Indra Winata, Peng Xu, Hyeondey Kim, Zihan Liu, and Pascale Fung. Generalizing question answering system with pre-trained language model fine-tuning. In Proceedings of the 2nd Workshop Machine Reading for Question Answering, pages 203-211, 2019.
[114] Dmitrii Aksenov, Julián Moreno-Schneider, Peter Bourgonje, Robert Schwarzenberg. Leon- hard Hennig, and Georg Rehm. Abstractive text summarization based on language model conditioning and locality modeling. arXiv preprint arXiv:2003. /3027, 2020.
[115] Inigo Jauregi Unanue and Massimo Piccardi. Pretrained language models and backtransla- tion for English-Baspee biomedical neural machine translation. In Proceedings of the Fifth Conference on Machine Translation, pages 926-832, Online, November 2020. Association for Computational Linguistics.
[116] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv preprint arXiv:2103.10338 2021.
[117] Lya Hulliyyatus Suadaa, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura, and Hiroya Takamura. Towards table-to-text generation with numerical reasoning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1451-1465, 2021.
[118] Qiaolin Xia, Haoyang Huang, Nan Duan, Dongdong Zhang, Lei Ji, Zhifang Sui, Edward Cui, Taroon Bharti, and Ming Zhou Xgpt: Cross-modal generative pre-training for image captioning. In CCF International Conference Natural Language Processing and Chinese Computing, pages 786-797. Springer, 2021.
[119] Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny. Visualgpt: Data- efficient adaptation of pretrained language models for image captioning. arXiv preprint 0rXiv:2102.10407 2021.
[120] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 7464-7473, 2019.
[121] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Unified laniunne pre-training for image captioning and vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13041-1304$ 2020.
17

---

[122] Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. Univl: A unified video and language pre-training model for multimodal understanding and generation. arXiv preprint arXiv: 2020.
[123] Jannis Born and Matteo Manica. Regression transformer: Concurrent conditional generation and regression by blending numerical and textual tokens. arXiv preprint arXiv:2202.01338 2022.
[124] John Jumper, Richard Evans. Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron-
neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-588, 2021.
[125] Kimia Noorbakhsh, Modar Sulaiman, Mahdi Sharifi, Kallol Roy, and Pooyan Jamshidi. Pretrained language models are symbolic mathematics solvers too! arXiv preprint arXiv:2110.03500 2021.
[126] Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4320-4333, Online, July 2020. Association for Computational Linguistics.
[127] Hiroshi lida, Dung Thai, Varun Manjunatha, and Mohit Iyyer. Tabbie: Pretrained representa- tions of tabular data. arXiv preprint arXiv:2105.0255: 2021.
[128] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140): 2020.
[129] Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. Visualisation and' diagnostic clas- sifiers' reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907-926, 2018.
[130] Alex Warstadt, Yian Zhang, Haau-Sing Li, Haokun Liu, and Samuel R Bowman. Learning which features matter: Roberta acquires a preference for linguistic generalizations (eventually). arXiv preprint arXiv:2010.05358, 2020.
[131] Charles Lovering, Rohan Jha, Tal Linzen, and Ellie Pavlick. Predicting inductive biases of pre-trained models. In International Conference Learning Representations, 2020.
[132] Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. Investigating the limitations transformers with simple arithmetic tasks. arXiv preprint arXiv:2102.13019 2021.
[133] Guillaume Lample and François Charton. Deep learning for symbolic mathematics. In International Conference on Learning Representations, 2019.
[134] Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner, Do NLP models know numbers? probing numeracy in embeddings. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP)) pages 5307-5315, Hong Kong, China, November 2019. Association for Computational Linguistics.
[135] Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jian- feng Gao, Jingjing Liu, and Bill Dolan. Dialogpt: Large-scale generative pre-training for conversational response generation. arXiv preprint arXiv:1911.00536 2019.
[136] Leonardo Ribeiro, Martin Schmitt, Hinrich Schüitze, and Iryna Gurevych. Investigating pretrained language models for graph-to-text generation. arXiv preprint arXiv:2007.0844 2020.
[137] Jason Phang, Thibault Févry, and Samuel R Bowman. Sentence encoders on stilts: Sup- plementary training on intermediate labeled-data tasks. arXiv preprint arXiv: 181000 2018.
18

---

[138] Nikita Moghe, Mark Steedman, and Alexandra Birch. Cross-lingual intermediate fine-tuning improves dialogue state tracking. arXiv preprint arXiv:2109. 2021.
[139] Huanru Henry Mao, Bodhisattwa Prasad Majumder, Julian McAuley, and Garrison W Cottrell. Improving neural story generation by targeted common sense grounding. arXiv preprint arXiv:7908. 000i:::9080095511, 2019.
[140] Alexander R Fabbri, Simeng Han, Haoyuan Li, Haoran Li, Marjan Ghazvininejad. Shafiq Joty, Dragomir Radev, and Yashar Mehdad. Improving zero and few-shot abstractive summarization with intermediate fine-tuning and data augmentation. arXiv preprint WxXiv20000.28..0 2020.
[141] Raphael Rubino and Eiichiro Sumita. Intermediate self-supervised learning for machine transla- tion quality estimation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4355 2020.
[142] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1 1-77, 2020.
[143] Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? arXiv preprint (rXiv:2101006800 2021.
[144] Teven Le Scao and Alexander M Rush. How many data points is prompt worth? arXiv preprint arXiv:2 08493, 2021.
[145] Han Guo, Bowen Tan, Zhengzhong Liu, Eric Xing, and Zhiting Hu. Text generation with efficient (soft) q-learning. arXiv preprint
2021.
[146] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.000
2021.
[147] Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint 104.02100,,,691,,, 2021.
[148] Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv: 10.07602, 2021.
[149] Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. Deep neural networks and tabular data: survey. arXiv preprint arXiv:21 2021.
[150] Inkit Padhi, Yair Schiff, Igor Melnyk, Mattia Rigotti, Youssef Mroueh, Pierre Dognin, Jerret Ross, Ravi Nair, and Erik Altman. Tabular transformers for modeling multivariate time series. In ICASSP 202/-202 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3565-3569. IEEE, 2021.
[151] Ravid Shwartz-Ziv and Amitai Armon Tabular data: Deep learning is not all you need. Information Fusion, 81:84-90, 2022.
[152] Hussein Hazimeh, Natalia Ponomareva, Petros Mol, Zhenyu Tan, and Rahul Mazumder. The tree ensemble layer: Differentiability meets conditional computation. In International Conference on Machine Learning, pages 4138-4148. PMLR, 2020.
[153] Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning on tabular data. arXiv preprint SEVV:1900.06312. 2019.
[154] Ira Shavitt and Eran Segal. Regularization learning networks: deep learning for tabular datasets. Advances in Neural Information Processing Systems, 2018.
[155] Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Regularization is all you need: Simple neural nets can excel on tabular data. arXiv preprint 1oXiv:2106.11788 2021.
[156] Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06677, 2020.
19

---

[157] Sercan O Ankk and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In AAAI, volume 35, pages 6679-6687, 2021.
[158] Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Gold- stein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.014 01....11332 342, 2021.
[159] Tin Kam Ho. Random decision forests. In Proceedings of 3rd international conference on document analysis and recognition, volume 1, pages 278-282. IEEE, 1995.
[160] Tianqi Chen, Tong He, Michael Benesty, Vadim Khotilovich, Yuan Tang. Hyunsu Cho, Kailong Chen, al. Xghoost: extreme gradient boosting. R package version 0.4-2, 14):1-44 2015.
[161] Scott Reed and Nando De Freitas. Neural petrrammer-intepprteer arXiv preprint arXiv:/511.06275 2015.
[162] Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:/909, (00i://9 2019.
[163] Yanyao Shen and Sujay Sanghavi. Learning with bad training data via iterative trimmed loss minimization. International Conference on Machine Learning, pages 5739-5748. PMLR, 2019.
[164] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
[165] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1706. 06083, 2017.
[166] Jonas Rauber, Wieland Brendel, and Matthias Bethge. Foolbox: A python toolbox to bench- mark the robustness of machine learning models. arXiv preprint
2017.
[167] David M Blei, Andrew Y Ng. and Michael Jordan, Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993-102) 2003.
[168) Melissa Roemmele and Andrew S Gordon. Automated assistance for creative writing with an mn language model. In Proceedings of the 23rd International Conference on Intelligent User Interfaces Companion, pages 1-2, 2018.
[169] Xikun Zhang, Deepak Ramachandran, Ian Tenney, Yanai Elazar, and Dan Roth. Do language embeddings capture scales? arXiv preprint urXiv:2010.05345. 2020.
[170] Aakanksha Naik, Abhilasha Ravichander, Carolyn Rose, and Eduard Hovy. Exploring numer- acy in word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3374-3380, 2019.
[171] Yuanhang Ren and Ye Du. Enhancing the numeracy of word embeddings: linear algebraic perspective. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 170-178. Springer, 2020.
[172] Guolin Ke, Di He, and Tin-Yan Liu. Rethinking positional encoding language pre-training. arXiv preprint arXiv:2006. 155555, 2020.
[173] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.0365: 2020.
[174] Benyou Wang, Donghao Zhao Christina Lioma, Qiuchi Li, Peng Zhang, and Jakob Grue Simonsen. Encoding word order in complex embeddings. arXiv preprint arXiv: 2019.
[175] Zhiheng Huang, Davis Liang, Peng Xu, and Bing Xiang. Improve transformer models with better relative position embeddings. arXiv preprint arXiv:2009.13658 2020.
20

---

[176] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré, Flashattention: Fast and memory-
exact attention with io io-awareness, 2022.
[177] Emily Sheng, Kai- Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked babysitter: On biases in language generation. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNI.P) pages 3407-3412, Hong Kong, China, November 2019. Association for Computational Linguistics.
[178] Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. A reductions approach to fair classification. International Conference on Machine Learning, pages 60-69, PMLR, 2018.
[179] Yuji Roh, Kangwook Lee, Steven Euijong Whang, and Changho Suh. Fairbatch: Batch selection for model faireess. In International Conference on Learning Representations, 2021.
[180] Michael Gira, Ruisu Zhang, and Kangwook Lee. Debiasing pre-trained language models via efficient fine-tuning. Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion, pages 59-69, Dublin, Ireland, May 2022. Association for Computational Linguistics.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]] The contribution and scope of this paper has been summarized in the abstract and the last part Introduction.
(b) Did you describe the limitations of your work? [Yes] We wrote the limitations in Sec.7.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] This has been discussed in Sec.7.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] Our paper conforms to the ethics guideline.
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] The code, data, and instructions are provided in the shared anonymous GitHub repository.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] All the training details are discussed in the paper and they can be found at the anonymous GitHub repository.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] All the experiments are run multiple times and at the tables mean and standard deviation of those runs are presented.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] This has been described in Sec.3.
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Our work uses public datasets/models, and has been cited properly, the paper and GitHub repo
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
21

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
22


Reviews:
[{"review_id": "x5t4o414sia", "reviewer": "Reviewer_igVi", "summary": "This paper aims to answer whether pre-trained language models (PLMs) can be used for fine-tuning on non-language downstream tasks without changing the input and output format of the original PLM. To answer the above question, this paper introduces a framework, Language-Interfaced FineTuning (LIFT), which formats non-language classification and regression tasks into natural language-like sentences by predefined templates. By extensive experiments on various datasets, they show that fine-tuning autoregressive PLMs on those downstream tasks can achieve relatively well performance. This paper also presents a series of analyses on the properties of LIFT.", "questions": "### Questions \n1. Do the authors manually change the style file or change the font sizes of the table captions?\n2. The sentence in Line 109 \"For **tabular data**, ... non-language tasks\" seems odd.\n3. During inference, how is the answer generated? By argmaxing or sampling? If it is sampled, what is the sampling scheme? When will the generation end?\n4. I am afraid that I will have to disagree with the claim on Line 219 that \"LIFT/GPT outperforms in-context learning\". This is because, in Table 10, the variances are very big. Considering the variance of both in-context learning and LIFT/GPT, a would rather say that there is no statistical significance between the performance of the two methods.\n5. Why can the experiment results in Section 5.1 imply that LIFT \"may\" perform Bayesian inference? \n6. There is an important reference that is missing: Kao and Lee (2021) [4] have already fine-tuned BERT on non-language downstream tasks as standard fine-tuning on natural language downstream tasks and show that the performance is decent.\n\n\n[4] Wei-Tsung Kao and Hung-yi Lee. 2021. Is BERT a Cross-Disciplinary Knowledge Learner? A Surprising Finding of Pre-trained Models’ Transferability. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2195–2208, Punta Cana, Dominican Republic. Association for Computational Linguistics.", "limitations": "The authors have addressed their limitations and potential social impacts.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Update after rebuttal\n===========\nI have read all the comments by the authors, and they mostly clear up my questions on the experiment settings and formatting issues. While I agree that this paper has contribution to the ML community, I cannot lean strongly toward acceptance due to its massive content. I think the content is more suitable for a journal instead of a 10-page conference paper since a lot of details are placed in the appendix, making the main content not very self-contained. Also, I still feel like this paper is more of an empirical report and it is hard to know how LIFT can be used in real-life use cases.\n***\n***\n\n### Strengths\n1. This paper provides some interesting findings on directly fine-tuning PLMs on non-language downstream tasks.\n2. The amount of downstream tasks evaluated in this paper is impressive.\n\n### Weaknesses\n(If I am misunderstanding any parts of the paper in the following parts, please let me know.)\n1. **It is unclear whether the benefit of LIFT is just because the model is gigantic**\n    * Since the goal of the paper is to answer \"can fine-tuning PLMs work on non-language downstream tasks\", it should be verified whether the exceptional performance of LIFT is really because *\"LIFT fine-tunes a model **pre-trained on human language**\"*. Chances are the good performance actually stems from *\"LIFT uses a gigantic transformer model with billions of parameters, and it doesn't matter whether it is pre-trained on human language or not.\"* To exclude the latter case, the paper should include an experiment to train a transformer model with the same architecture of GPT-3/GPT-J from scratch on the downstream task, instead of initializing the weights from a PLM. Since the trained from-scratch model is not pre-trained with human language, if this model indeed fails to perform well on those non-language downstream tasks, we can rule out the hypothesis that \"LIFT performs well simply because it uses a gigantic model\". Next, one should also make sure that it is because of pre-training on **human language** that makes the success of LIFT. For example, one can use artificial datasets to train GPTs and fine-tune them on non-human language downstream tasks, as in [1], [2], and [3]. If training from scratch and fine-tuning from a model pre-trained with artificial datasets both result in bad non-language downstream performance, one will be able to arrive at the conclusion that \"PLM fine-tuning\" solves non-language downstream tasks. \n2. I am **not convinced by the motivation** of \"why one will want to use a PLM to fine-tune on non-language downstream tasks\". \n    * According to the paper, it seems that the goal is to achieve \"no-code machine learning with language models\". But I cannot see why that would be beneficial for two reasons. First, no-code machine learning can be achieved by the AutoML (Vertex AI by Google) without relying on PLM. It is unclear why one would want to rely on a bulky PLM to achieve no-code machine learning. Second, as illustrated in the experiment results, the performance of LIFT is still largely inferior to other simpler and smaller machine learning models; this will refrain LIFT from being used in real-world cases. It is unclear why one will resort to a larger and not always better model while one can just use smaller and better models (for example, always use logistic regression for tabular data and always use MLP for other classification tasks).\n3. The input length of LIFT is upper-bounded by the maximum sequence length of the PLM (2048 for GPT-J and GPT3), making it hard to be used for high-dimensional data. Moreover, considering that the input of LIFT is not only the raw feature but also the words in the templates ( \n$\\mathbf{x}_1 = v_1 $ \nwhat should be the \n$\\mathbf{y}$ \n), the actual acceptable feature length will be significantly less than 2048. It will be even lesser if each feature $v_i$ is tokenized to multiple tokens by the model. For example, considering MNIST, which has 784 features per image, assuming that each feature will only be tokenized to one token. Then when encoding an image using the template $\\mathbf{x}_i=v_i$,  it will require at least $784\\times4=3136$ tokens, which is significantly longer than the maximum input length of the GPT models used in the paper. It is unclear how this is handled in the paper.\n4. The memory of LIFT is quadratic to the input length, indicating that LIFT is highly memory inefficient compared to other models.  \n4. This paper tries to include too many things in the main content, making the snippets included in the main content seem fragmented, incomplete, not self-contained, and hard to follow. The following lists are pieces of evidence from the paper to justify the above claim.\n    1. The \"non-deterministic predictions\" that appeared in Line 18 and Line 70 do not appear in the rest of the paper, and it is unclear what \"non-deterministic predictions\" refers to? Is it related to the Bayesian prediction in Section 5.1? \n    2. How is the \"increase the generation randomness\" in Line 137 done? Perhaps by changing the temperature for sampling?\n    3. What do the \"inductive biases of language models\" refer to in Section 4.2?\n    4. What is a median kNN in Line 241?\n    5. The paragraph of \"Robustness to adversarial samples\" in Line 244 includes no experiment results, making the paper not very self-contained.\n    6. It is unclear how the mixup is done in Section 6.2, the details of this experiment is not presented neither in the main content nor the appendix.\n    7. It is unclear how the in-context learning is done in Section 4.3. For example, how is the training/testing split determined? Is there an evaluation set? Where does the variance shown in Table 10 come from?\n5. This paper is poorly formatted, and some of the figures are extremely small for reading.\n    1. Ill-formatted parts\n        * According to the formatting instructions for NeurIPS 2022, \"the table number and title always appear before the table\"; all caption of the table in the paper appears **after** the table.\n        * The size of the caption is sometimes significantly smaller than what should be expected, for example, Table 3, Table 5, Table 6, and Table 7. I am not sure whether this violates the guideline of \"not changing the font sizes except for the reference\".\n        * The bottom of Figure 7 on page 8 is occupying part of Line 329 on page 9.\n        * The figures in Table 3 are extremely small.\n        * The legends in Figure 4 and Figure 5 are too small to read when printed on A4 paper.\n        * According to the formatting instructions for NeurIPS 2022, \"Place one line space before the table title, one line space after the table title, and one line space after the table\". But the margin of Table 3 is quite odd and seems to violate the instructions.\n\n\n[1]Papadimitriou, Isabel, and Dan Jurafsky. \"Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models.\" Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.\n\n[2] Chiang, Cheng-Han, and Hung-yi Lee. \"On the transferability of pre-trained language models: A study from artificial datasets.\" arXiv preprint arXiv:2109.03537 (2021).\n\n[3] Ryokan Ri and Yoshimasa Tsuruoka. 2022. Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7302–7315, Dublin, Ireland. Association for Computational Linguistics.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "stqW0ZaxOU", "reviewer": "Reviewer_iBBJ", "summary": "This paper studies an interesting and important topic: can we use large pre-trained language models to machine learning tasks where the inputs are unstructured real vectors? Through a series of experiments on real/synthetic classification/regression tasks, the author concludes that\n- fine-tuning LLMs have performance on par with many traditional algorithms.\n- LLM is performing bayesian prediction\n- two-stage fine-tuning improves the performance\n- feature name helps LLM perform better.", "questions": "- Table 2: why does the paper claim that LIFT performs relatively well compared to other methods, even if in many or even most cases it cannot outperform 3-layer MLP? It does not seem that the algorithm is successfully making use of the LLM internal representation.\n- Table 3 caption tries to conclude that larger models generally perform better -- but it seems that GPT-3 Davinci is overall worse than GPT-3 Curie, and the performance increase w.r.t. scaling is extremely unclear. \n- Section 5.1 tries to claim that LLM is performing bayesian inference. I thought it meant \"LLM has a prior about what the function is look like and it performs posterior inference based on true labels\". However, the paper only experimented with \"trying sampling a lot of time and see whether the variance is correlated with noise level\". This only implies that LIFT is calibrated (where prediction logits/samples reflects the confidence), rather than that LLM is performing bayesian inference. \n- Table 5 tries to claim that LLM can successfully use the feature name -- for only 2 out of 3 classification dataset this seems to hold, and when it holds, the margin is very small (compared to the noise) and it is unclear whether they are noise. In particular, what is dataset 48 and 54, and what are their intuitions of LIFT can help? One experiment you could potentially run is to find a dataset where we have very strong. prior notion of what features are good and test the models' behavior on that as well.  \n", "limitations": "Yes, the author successfully addressed their limitations. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n - The paper is generally very well-written, with clear figures and a lot of experimental details in the appendix. It also addresses its scope of study and societal impact properly.\n- The topic is interesting, potentially impactful, and novel. \n- There are a lot of experimental results, which could help practitioners gain intuitions.\n\nWeakness:\n- The performance by LIFT is not impressive.\n- Most of the important claims in the paper are not well-supported by the empirical evidence. See questions below.\n\n**I am very willing to increase the score to 8 if the author can either 1) justify why their claims are correct, or 2) re-position the paper as a \"negative empirical finding\", rather than advertising \"LIFT\" as a very promising approach for real-value-input ML tasks.**\n\nminor: \n- Non-language can also refer to image, sound, or haptic information, and I was confused what non-language task the paper was discussing when reading the title and the abstract. Probably rename \"Non-Language Machine Learning Tasks\" to \"Real-Valued Machine Learning Tasks\" (or something similar). For example, Flamingo (from DeepMind) also uses LM as an interface, but they used it for image + text.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "pO_xKfI8Dt_", "reviewer": "Reviewer_GQen", "summary": "This paper fine-tunes pre-trained language models to perform non-language tasks, like regression of functions and classification. They test a large variety of tasks, and test against several standard algorithms ( like KNN, SVM, RF). Additionally, they investigate sample efficiency, ability to extrapolate, and robustness of these models. ", "questions": "* Did the authors try other templates? How did they perform? What was the sensitivity of the model performance on different prompts?\n* From figure 4, it looks like this might not work for few-shot prompting. Did you run that experiment?\n* I would include additional information about how adversarial examples were generated. ", "limitations": "Currently the social impacts section says that \"LIFT might have embedded bias targeting certain social groups.\" This seems possible to measure. In particular, for tasks where text is used in the prompt, the authors could compare LIFT with all other models they compare with to see if the examples that are mis-classified are systemically biased in some way. Additionally, this could be an interesting experimental setup to evaluate bias in the pre-trained model.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\nSection 5.2 is the most interesting part of this paper to me, and to me, this is one of the best motivations for fine-tuning a large language model to perform this task. I haven't seen this idea before, and it's intriguing as a method for understanding the pre-training dataset and also harnessing contextual information. I'd like to see more focus directed here and maybe additional tasks explored or an expansion on experiments in this direction. \n\n\n* I appreciate the summary listed in table 1!\n* I appreciated figure 4 (comparing # training samples vs. accuracy for different classes of tasks).\n\n### Weaknesses\nI don't doubt the results of this work, but after having read the paper, I'm still unsure what advantage we gain by finetuning a pre-trained language model to perform non-language tasks. It currently reads like the paper makes the argument that finetuning pre-trained language models can perform well on non-language tasks. However, these models are orders of magnitude more costly to train and run inference on, so when and why should we prefer it? Could the authors better motivate this?\n\nAdditionally, please explain what non-language tasks are earlier in the paper. Specifically, also say that the paper will test both classification and regression tasks. Please give an example of each in the abstract and also in the intro.\nAlso, the sentence: \"We note that though achieving tremendous success with natural data, deep learning still faces difficulties with standard machine learning tasks, especially on tabular data...\" seems to be at the core of this paper. I would include this idea earlier in the paper (abstract or intro).\n\n\nThis paper covers a lot of ground. I think it could be more focused by moving additional experiments to the appendix (for example, sec 6.2). \n\n### Additional Feedback\n* Suggestion to use \"what should the y be\" as opposed to \"what should be the y\" in the template. Language models are still fairly sensitive to prompts, and wording may change results. \n* Include the sizes of GPT-J (6b) and GPT-3 (175b) in the section on \"pretrained language models\" as a reference for the reader. \n* Figure 4 text should be larger \n    * Also table 3 should highlight interesting results as the text is too small to read\n* It would make the paper easier to read if the authors could include the names of the tasks (in addition to the IDs) in the text to better understand the context of the task. For example, in section 5.2, which DID is the Medical Insurance dataset?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "aAiGrns56xR", "reviewer": "Reviewer_evpg", "summary": "The paper investiagtes the application of pre-trained language models on the task of classifying non-textual data, without any architecture changes.\nThe input features are linearized into a text-like sequence and given as context, the output is then collected as a language model prediction.\nWhile it doesn'g perform best overall, it does achieve surprisingly competitive performance.", "questions": "^", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and clear.\nThe direction of investigation is interesting.\nThere are many experiments presented, analysing different aspects of the set-up.\nThe findings are interesting and non-intuitive. For example, the model performing well on MNIST by describing each pixel in a textual sequence. \nWhile the setup is clearly not optimal for the tasks, the findings could lead to some interesting future research.\n\nPrevious work is mentioned, which replaces the input and output layers of the LM, whereas the current work keeps entirely the same architecture. \nThe performance of that system should be reported as well. \nAt the moment it is unclear whether keeping the original input or output layers actually helps or hurts the performance on different tasks.\n\nFurthermore, the replacement of the input and output layerst should be investigated separately.\nWhile giving the input as text might actually have some benefits (as experiments in Section 5.2 also indicate), using only language model predictions as output could be hurting performance. Even models that work with natural language generally fine-tune pre-trained language models by replacing the LM output layer with a task-specific version.\n\nAn open question left by the paper is whether these results are down to the architecture (and the large number of parameters) or the fact that the model is pre-trained on large amounts of textual data.\n\nThere should also be a discussion on the difference in the number of parameters.\nAs I understand, all the baseline models are quite simple, with perhaps a few hundred trainable parameters. In contrast, GPT-3 has billions of parameters.\nHow much does model capacity affect performance? Are there no baselines with more comparable numbers of parameters that could have been also reported?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept