Support  Vector  Generation: Kernelizing Large Language Models for Efficient Zero‑Shot NLP
Authors: Shohei Ohsawa
Venue: NeurIPS 2025 poster ()
Keywords: Zero-shot Learning
Kernel Machines
Large Language Models
Support Vector Generation
Resource-constrained NLP
Primary area: deep_learning

Abstract: We introduce Support Vector Generation (SVG), a kernel-based framework that converts a frozen language model into an interpretable, training-free classifier for zero- and few-shot learning. SVG operates by combining Metropolis–Hastings sampling with support vector machine optimization in the reproducing kernel Hilbert space (RKHS) induced by the language model's embedding. Each classification decision is based on a weighted combination of at most 32 natural-language sentences, which serve as explicit support vectors and provide faithful rationales. Our theoretical analysis proves that SVG minimizes the empirical hinge loss over the span of the supports and admits a generalization bound independent of the language model size. Experiments on the GLUE benchmark show that SVG matches or surpasses prompting-based zero-shot baselines in accuracy across multiple tasks—without any fine-tuning or GPU acceleration. Notably, our CPU-only implementation completes training in under three minutes per task, and maintains competitive inference speed. These results suggest that SVG offers a viable path toward efficient, interpretable NLP systems under compute constraints.

Support Vector Generation: Kernelizing Zero-Shot Classifiers from Pre-Trained Language Models
Shohei Ohsawa
shohei www.whhiys..cooomoo co. jp
Abstract
We introduce Support Vector Generation (SVG), kernel-based framework that converts a frozen language model into an interpretable, training-free classifier for zero- and few-shot learning. SVG operates by combining Aetropolis-Hastinne sam- pling with support vector machine optimization the reproducing kernel Hilbert space (RKHS) induced by the language model's embedding. Each classification decision is based on a weighted combination of at most 32 natural- -lnguuae sen- tences, which serve as explicit support vectors and provide faithful rationales. Our theoretical analysis proves that SVG minimizes the empirical hinge loss over the span of the supports and admits generalization bound independent of the lan- guage model size. Experiments on the GL.E benchmark show that SVG matches surpasses prompting- -aasd zero-shot baselines in accuracy across multiple tasks-without any fine-tuning or GPU accelcration. Notably, our CPU-only im- plementation completes training in under three minutes per task, and maintains competitive inference speed. These results suggest that SVG offers a viable path toward efficient, interpretable NLP systems under compute constraints.
Introduction
Lcrge-scale pre-trained language models (PLMs) [3, 6] achieve impressive zero-shot accuracy many natural-language understanding tasks. Despite this empirical success, two practical obstacles hinder their deployment at scale. First, the decision process is opaque: predictions emerge from billions of latent parameters, and token-level saliency methods provide only indirect, often disputed explanations [1, 2]. Second, inference is computationally costly: autoregressive decoding grows linearly with output length and typically requires GPU-class hardware, whereas many edge or privacy-sensitive applications must run on CPUs.
Fig. 1 contextualises these obstacles. Panel (a) depicts the canonical pipeline in which a labelled corpus is used to train parametric classifier; panel (b) reinterprets this teacher-student com- munication game. Panel (c) situates our contribution: a frozen PLM (teacher) communicates with a non-parametric kernel machine (student) over the Internet by exchanging natural-languee prompts, thereby obviating gradient- based training while retaining interpretability.
Let (ar) denote the probability frozen PLM with parameters 0 assigns to sentence The negative likeliholdodd induces positive-defintte language kernel
kig (r, x') = expl (o)))()) - log logpo(<')]
embedding all sentences in a reproducing-kent -ernel Hilbert space (RKHS) whose dimensionality is independent of 0). Within this RKHS the optimal large-margin classifier is representable finite
have conducted series of studies on multi-agent communication in distributed environments [17, 25, 26, 27, 28), and the present paper can be regarded as one of them.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

CON
annotate-
Supervisor
train
Classifler
test O
Task
Channel
CNN
Supervision
pretrein
question
Student
init
Agent
Inswer-
test
Unlabelled
Data
c)
wWW
CON
Tanelled
pretrein
Transformers (Pg..GPT-4)
promot
Kernel Machine
init
Task
Start (t-o)
examples
test
Terminate (t-7)
Data
The communication model of our approach. A teacher tells students possibly taking on muliple tasks by addressing difference questions The different agents communicates over channels such as Internet and local networks. a) Typical machine learning b) Teacher-student model c) Support vector generation (ours)
Figure 1:
weighted sum of kernel evaluations, suggesting that explicit, compact, and interpretable classifiers can be extracted from PLMs without fine-tuning.
We introduce SVG, training-free free, two-stage procedure (Fig, Ic). Stage 1 employs a Metropolis- Hastings sampler queried against the PLM to synthesise sentences near the decision surface. Stage solves the dual support-vecto-macee those with non-zero dual weights: the surviving sentences form explicit support set. At test time classification reduces to a weighted vote over at most supports, and the dual coefficients provide exact influence scores yielding faithful sentence-level rationales.
problem restricted to these candidates, retaining only
Contributions. (i) We formalise the PLM-induced language kernel and prove that SVG minimises empirical hinge loss within the span of its supports, obtaining a generalisation bound that scales with the support count rather than with [0]. (ii) We devise an efficient CPU implementation whose entire training phase finishes in under three minutes per GLUE task without any gradient updates. (iii) seven zero-shot tasks from GLUE benchmark, SVG matches or surpasses strong prompting baselines while providing exact post-hoc explanations. (iv) We release anonymised code and scripts to facilitate independent verification.
Collectively, these results demonstrate that interpretable and resource-efficiee classifiers can be distilled from frozen language models, paving the way for PL.M-level accuracy in compute-constrained settings.
2
Related Work
Before the deep learning era, kernels (i.e. "similarity") were widely employed measure the distance between a pair of inputs, especially sequences of information such as web documents or protein sequences, which can be of variable length. The representative one is the cosine similarity between TF- IDF vectors, which give good results for information retrieval [21], with a probabilistic interpretation given in [12]. String kernels [15, 33], which compare the number substrings in common, have been used for amino acid sequences, The string kernel is known as scalable Mercer's 's kernel, which can be computed in linear time for length of string using suffix trees [18, 34, 38]. One of the string kernels which consider strings of a fixed length 1: is known as the k-spectrum kernel, and has been used to classify proteins into SCOP superfamilies [18]. The string kernel has been generalized to compare trees [9], which is useful for parse trees and evolutionary trees. In computer vision, pyramid match kernel [14] is used to compare two images of feature vectors obtained from SIFT [20]. One of draw wwwbacks of these kemelss is that they cannot capture the recurring structure of strings, because they merely assume the document as a hag-of-words (or -eg-orrfeatures), and only care about the frequeney of the words.

---

The paradox between the number of parameters of neural network and its generalisation performance has been one of the unsolved problems the field of deep learning: traditional learning theories state that if the parameters of model exceed the number of samples, the model is over-trained and cannot adapt sufficiently to unknown samples. One hypothesis for this that neural networks acquire the ability to "interpolate" between any two samples after having fully memorised all samples, which also has experimental evidence in the form of the double descent phenomena [24]. Nevertheless, another problem with this paradox is the derivation of explicit optimal solutions, i.e. how to regularise model with degrees of freedom that go beyond the train data. [41] map the MNIST samples into a dual space and optimise the hinge loss with "kernel trick" to obtain regularised explicit representation consisting of the weights of each sample and the Gram matrix, which a similarity representation between each sample. This solution also showed higher performance than the expected value of the solution optimised simply by the stochastic gradient descent, suggesting a strong relationship between neural networks and kernels.
Self-attention [37] is a powerful building block that has enabled the development of large language models (LLMs) such as BERTs and GPTs, which can quickly capture the in-context relationships between tokens in a text sample. LLMs have been used for various tasks including text generation, question answering, and dialogue generation [6, 30, 31, 42]. Recent research has focused on controlled text generation, generating text that adheres to a set of constraints while being fluent and relevant to the given context [19, 44]. Zero-shot learning, a more challenging task, uses transfer learning or data augmentation to overcome data scarcity and computational constraints [13, 22]. Despite its computational advantage, self-attention requires the use of GPUs to accommodate the large number of parameters (l0] ~ 1010) needed to memorize all examples. This can be computationall expensive and may not be feasible for certain applications.
3 Language Kernels
Suppose we are a data scientist company who has been assigned task of sentiment analysis, checking whether the given text represents a negative or positive review for each film manufactured by the client. In a zero-shot learning scenario. instead of requesting disclosure of training data from the client, we can derive an inner product "(x)" o(w,) between an input æ E X' and a label wy E X for y E [+1], where X' is a set of strings, and Φ represents text embeddings, and we can directly insert desired labels such as Φ(x)" [[ (0(" ("positivee") - o( (""negative"")] Though it seems too straightforward, one can find out that the accuracy for SST-2 is 0.83?. The inner product k(2, wyy) = <(x)" To 6(w,) measuring the text similarity of the sentence pair k X² + E is called a kernel.
Here we use a simple string of "positive"""nncgaivv"" as an example, but there are countless such task description expressions, each with slightly different vector representations (e.g.. "good"/"bbd"", Therefore, we generalize the above method into a set of n synonyms per label (#1,..., .2n) = {wy) ..., " (((=1) with decision function fa X' - I as follows:
fa(z)= = Σ a,k(2,
(1)
where a; 2 0 represents the sample-wise attention of each description.
The problem is how to find the solution of the system with 2n degrees of freedom without auxiliary oracles such as real labeled samples. Surprisingly, the optimal solution is found by solving the dual objective of SVMs [10] as follows:
2n
2n
2n
2n.
max J(a) = =(a)=
a. 2
ΣE
20,,k(, i=1
s.t. Σ -ay=0 0 Osasc (2)
where C >>> is a regularization parameter that controls the trade-off between maximizing the margin and minimizing the training error. The number of support vectors nişv < 2n is bounded by the VC-dimension [4, 36), which measures the complexity of a binary classification problem with the maximum number of samples the classifier shatter. For any positive definite function re(-.)) Rddd + IR and text embedding Φ(-) : X - Rd, we define a language kernel as ks #22 := ((((2)))((2)) ((x2)).
2 Contirmed through text-embedding ada-002. The chance rate is 0.50 and the sfateoof-hhe-an is 0.95 (fully- RoBERTa-larg-FFF -T, iee. supervised learning with GPUs) [13].
3

---

Algorithm 1 The Support Vector Generation (SVG)
Require: 20, 00, 0 T, 92 {C,
1: for t = to T do
2: It+1 (0(-12))
3: M++1 + 4 fa fa(
4:
a' +=trrin(De++114
5:
if (1,, T(+1) < random(0, 1) then
6:
a+a
7: else
8:
(z Vo+1, , +- (xt, Vc. 0)
9:
end if
10:
end for
11: return ;" (r) EH
4
Support Vector Generation
The zero-shot decision function in Eq. (1) is represented as follows:
Sa (Inew) = Ex.y<= [k(Twew x>yl y),
(3)
where the æ is probability distribution with countable spikes X' {+1], each spike having an appropriate ordered index i. Here, if we denote the ordered set by D, π can be expressed as follows:
a3/l|alla T T(. v) - 0,
if (a. y) (rr, V:)
if (I, v) $
where ||a||] =; as is the I norm of Note that even if (ar, v) E DD, if x does not support the decision boundary Le., a; = 0, then the case is equivalent to (I. y) $ D. Thus, when we denote the set of support vectors, i.e., the vectors with 0; > 0, Dsv, the D is identified with Dsy (a.c.). As the optimal a is found by minimizing Eq. (2), it is sufficient to find the Dşv (neither D nor 0!) to represent the problem. This trick binds the optimal hyperplane to a VC-dimension (ss (Dsv)) which is finite and smaller than dim 0, especially for deep neural nets, not to mention that (D), and then makes the problem far easier than fine-tuning and training data generation.
The idea of SVG is to sample from the prior pa(x, y) and optimize the decision boundary with the kernel machines. However, directly sampling from P4 is not casy due to intractability, in many cases, we only know a scaled and/or conditional go 3. To address this, we employ Markov chain Monte Carlo (MCMC), particularly Betropolis-Hasting (MH) sampling [8], assuming 'D is an ergodic process 1 - D2
... whose empirical distribution p(z, y/ ((((() converges to π (I., v) (L + xc, a.c.). MH aims to achieve the detailed balance on the state transition at the fixed point holding (1x, y1 ( the normalization of both sides are canceled. Given a sample x, at each step t, the sampling step of MH proposes new sample from a distribution q0 - Ineww qe (Znow (x)). ). and decides whether to update T441 with Fnew or De based on the following acceptance probability [8]
T(I++1, Yt+1)qe (Iz [=++1), which is tractable because
77 (IInww Mnew) ) go go (Few)) A(ze, knew) min 1,
(5)
x (3r, yt) qo (10neew (T+))
We estimate the backward using go(=e(=mew) the concatenation. As π is unknown, we approximate it with scaled #t41 (Fnew; Ynew
go (TrTn x. /// go (Trnww) where [Knew x.e is
'Though completion models provide the transition pe (Zeompletion (gemmps) = II pe (wy [rpemmpss Wes). ). the stable probability pa (Xcomplassca) is intractable: fxx pa (Zcooplettonn [ppamge (2prompp) This problem, concerning the "initial token" (o)), is related to symbol grounding and multi-modality.
'Intuitively, the weight go (2 too low. For example, in text generation, "lions are + mammals" correct, but "mammals are - lions" are not always correct or depends on contexts, so qs ("mammals") |I "lions") (oons")//gr /gge "'lions""mmamal"" should be low. We can confirm that A = 0 if f(( is misclassified or A = if the process satisfies the detailed balance for backward/forwwrd sampling go (wze) = qo (x (.++1))
1x)//0(zz - x++1) penalizes when the backward path x++1 Ire

---

Table Complexity analysis of SVG and PLMs. n: the number of samples, m: the maximum length of texts in token, d: the dimension of the embeddings, nsv: the number of support vectors (S n))
Train
Predict
Kernel machines (ours) Transformers [37]
O .d) On, .mm
(ngy : d) O d)
Table 2: Results from zero-shot learning of GLUE benchmark with CPUs and without any GPUs or GPU memories. The values of i are from [13]. The experiment was repeated three times, and the average and standard deviation are listed. Bold indicates the best score. The rightmost column shows the average elapsed time for a single experiment.
Single Sentence
Sentence- pair
SST-2
CoLA
QQP
MRPC
RTE
QNLI
MNLI
(Acc.)
(Mat.)
(FI)
(FI)
(Acc.)
(Acc.)
(3-Acc.)
(sec)
Chance rate
50.0
50.0
50.0
50.0
50.0
33.3
SVG (ours)
91.70.9
9.13.2
72.91.a
63.712.1
57.92.8
64.0s.2
51.81.2
48.1
without MCMC
88.91.5
4.3
65.72.1
58.1s.o
55.11.2
55.94.a
44.91.2
1.9
Prompting' 1
83.60.0
2.0g.0
49.7p.a
61.90.a
51.30.e
50.8o.a
51.70.0
max(0, (,, 4/4 1 Nnew Jo (Frew)), where a' is the dual coefficient learnt assuming (I++1; Mt+1) (Fnow, sgn ag (Fnew)) ))- As by definition, Vnew takes either binary value, we can write Vnew sgnfa' (Inow) a.c. without loss generality. With Re+1, the above equation is approximated as,
À (Iit, Unew) min 1, 2'41 9o (T Zt (xz4)
of go (are; knew]) qo (Frew)
The proposed SVG algorithm outlined in Algorithm 1. This method generates additional support vectors that help shape decision boundaries achieved by sampling from the prior distribution and subsequently updating the parameters of the kernel machine. The advantage of this method that it allows artificial expansion of the training data, which is particularly useful in scenarios where available data scarce. By encapsulating the data distribution more effectively, the SVG ensures improved performance of zero-shoo shot learning models, even under computational constraints. Table 1 illustrates the complexity analysis. Even though the QP solver for Eq. (2) takes O(n3) times, we assume the practical algorithm for SVMs such Sequential Minimal Optimization (SMO) [29], which takes (n²) times. In sew- learning, SVG faster than the Transformers because n & m?.
Numerical Experiment
To evaluate the effectiveness of our proposed approach, conducted experiments the General I Language Understanding Evaluation (GLUE) benchmark [39]. The GLUE benchmark comprises a set of sentence sentence-pair language understanding tasks, providing a comprehensive evaluation of the performance of language models in various natural language understanding scenarios. We have compared the performance of proposed method, SVG baseline methodology: 'Prompting' [6]. In this comparison, we employed a conventional prompting technique, zero-shot learning approach that adopts set of manually-constructer prompts an exemplar for the labels. For context, we used a non- non-fine-tun fen---inttueee tuned PLM [13] comprising our baseline.
The experimental configuration used two CPU-only virtual machines computational environment and an OpenAl puy-as-you-go account as the trained language model. Three executions per task were carried out in multi-process manner, with one CPU (not GPU) of GHz and GB memory are assigned to each process. The training was completed in three minutes, which is far faster and more economical than networks with GPUs.
public cloud the
5.1 Zero-shot Learning
The experimental results are shown in Table We report the accuracy and score for each task, comparing the performance of SVG with the baseline methods. The results show that SVG outper-
5

---

SST-2 91.7%
0.3
The performences were stallar and 50 believble." (Ks. +
The actors' perforances were incredible (Sx. +11
A powertul and inspiring story of ...rgg and thumph (kc. +1]
1.4
laughed and cried throughout increciele expertence kepe
movie (xx..
never torget (Pe).
0.8
The humor heartwarming film that ha ars the and (Nee. +1)
whole time
0.1
was
This movie gave me such warm feeling Oaas +1
X.yi
-0.8
locked any trace emotioe or dipth (Sxx.
story was
forgetablc IX4.
-16
The
rether
predictable
This Tim did not provide the that espected (ris.
--24
Nothing soout this movie was enjoyable Ohy
found the ending to be overly disappinting (60. -1
-0.2
-32
The special effects were homendou.... over-the-top 1nt.
the story difficult to follow was monsensical (Xoa.
engeg tory movic. Nothing redecmable
(toe
0.2
0.1
0.0
6.1 0.2
0.00
d)
Figure 2: Left: Generated samples and the decision boundary, with the points in circles representing support vectors and contours with (x) +1, 0, -1 from the left. SST-2 (Accuracy: 0.917), text- curic-001. Right: The spikes - high-performed support vectors generated from the SVG algorithm. The vectors are visualized from chain out of five parallel MCMC search.
forms the baseline methods in terms of both accuracy and Fl score, demonstrating the effectiveness of SVG in improving the performance of zero-shot learning tasks, even in resource-constraing environments. Fig. 2 shows the generated samples and the decision boundary.
The superior performance of SVG can be attributed to the combination of the generative capabilities of PLMs and the computational efficiency of kernel machines. By generating support vectors from the PLMs, SVG is able to augment the training data for zero-shot learning tasks without the need for fine-tuning or additional computational resources. This allows SVG to achieve comparable or even superior performance to the baseline methods, while using only CPU resources.
The experimental results overall corroborate the effectiveness of SVG in zero-shot learning, specifi- cally ssouure-coossrriine contexts. Merging PLMs with kernel machines introduces new opportu- nities for elaborated language processing assignments. Such combination facilitates the creation of precise and efficient natural language understanding systems.
Performance of kernel machines highly depends on the hyperparameter C, the upper bound of each entry of a, and is on a trade-off between over- and underfitting [11], I]. If C' is too large, the model yields overfitting, and vice versa. One of the common search algorithms is the combination of grid search and cross-validation [35]; divide 'D into K discrete batches randomly and test the each with K
90
Accuracy
80
1 others, and choose the best candidate maximizing the metrics, as shown in Fig. 3. We optimized C' = Co/n for the number of samples n and select the best candidate Co from discrete log space {10-2,.. ..... 1010} at every to 10 step.
70
60
K 5, no = 100
K 3, no 10
10a
10a
10a
10a
A proposal de as close as π improves the acceptance ratio of MH. Other than completion API as we have employed in this paper, there are several heuristics of data augmentation such back translation and text attack [5, 23).
C
Figure 3: Results of performance by fixed C on SST-2.
5.2 Few-shot Learning
Although we can simply initialize D with the given labeled samples from the training set and optimize a with Eq. (2), SVG also retains the ability to generate additional support vectors that resemble these labeled samples. This is primarily because these generated support vectors are expected to be located around the decision boundary, indicating that they should closely mirror the given labeled samples. Where this approach deviates from data augmentation is in its ability to generate samples

---

90
6
generative
gon-earrrae
500
generative non generative
80
400
Accuracy
Entropy
4
Dimension
300
3
60
2
generative non-generative
0
100
10
10?
10a
10a
10a
Few-shot samples
Few-shot samples
Few-shot samples
Figure 4: The result of few-shot learning of SVG through SST-2 in comparison to the conventional kernel machines, such as non-generative SVMs. The accuracy shows that even for the scarce dataset, SVG can complement the lack of data points. Entropy H[ == log ||a=]] E [ass/|a/|]) log og shows that SVG successfully obtains the complexity for the scarce dataset. The right figure shows the number of support vectors acquired by each method.
which more semantically similar to the given labeled samples than the samples generated by data augmentation. This can be attributed to the fact that SVG is based on a kernel machine, method that effectively captures the similarity between samples by transforming the input data into high-dimensional space, enabling a more nuanced similarity measure.
Fig. 4 shows the results of few-shot learning. In machine learning theory, including few-shot learning, that accuracy increases with the number of data, Interestingly, the case of SVG, accuracy reaches a maximum when the number of data is two (i.e. one positive example and one negative example each), then drops and steadily increases again when the number of train data sufficient. One possible explanation for this result is that while the generalisation performance is improved by incorporating different distributions as long as the number of 'external' data to be added is small, the quality of the external data is inferior to the quality of the train data generated by the SVG itself, which in turn hurts the accuracy of the model. This phenomenon is referred to as 'few-shot double descent' in SVG, though the reasons for this not analysed further in this paper.
6 Discussion
6.1
Applications
While we have mainly discussed the case of single sentences with two classes, in tasks of natural language understanding, we often have to deal with more than two classes and more than one sentence. GLUE benchmark has sentence-pair classification tasks such paraphrase identification (QQP, MRPC) and inference (RTE, QNLI), and multi-class tasks (MNLI) where the model has to predict one of three classes (entailment, neutral, contradiction) for sentence pair. Each of the tasks also has a training data which has not been covered yet in this paper, though application to few-shot learning is also possible. In this section, discuss how extend the proposed method to such tasks.
6.1.1 Sentence-pairs
For sentence-pale pair classification, we can adopt the same methodology as the case of single-sentence classification, albeit with few alterations. Initially, task is to represent the features of the sentence-pait We can accomplish this by utilizing a language kernel his X + R, which accepts two sentences as input and is defined as follows
kig [[=1; 22), [2;; æ4]) = x(6(02) - ((x)), ((xs) - ((I3)),
where o X Rd denotes text embedding and k : 2d + R represents a positive definite kernel.
1mh i.e. pairings sentences constitute language just as individual sentences do, hence hs (2 + E also constitutes a language kernel. Following this assumption, the language kernel can be alternatively written as hop : X + IR. This topic is slated for discussion in future work.
We have assumed recurring topology X²

---

Table 3: The model-agnostic multi-class task descriptions 21...., .......... EM which yield high performance in SVG. The placeholder of quote, labels and sample are obtained from the GLUE original paper [39], which can also be scraped at ht nttps: //tensorff low org/ perg/datasee datasettsss ca og/gi ue. Instead of text-curie-00 for MCMC, the initial samples will be inferred once via larger completion models such as text- davinci-003.
(a) Template
<quot>>
<labeel>>, <labela?,..................>>>,. M: <labelm>
The possible ten examples of the <sample> of "i: <label,>" are:
(b) SST-2 (single sentence, 2 classes)
"The Stanford Sentiment Treebank consists of sentences from movie reviews and human annotations of their sentiment. The task is to predict the sentiment of a given sentence. We use the two-way (positive/negative) class split, and only sentence-level labels."
I: positive, 2: negative
The possible ten examples of the sentence of "2: negative"
First, we need to compute the acceptance probability Å (Ii, Knew) for a sentence-pair. To do this, use the same approach as for single-sentence classification, but with few modifications. Second, we need to compute the probability go (e(new//zz) for the sentence- pair. To do this, we can use language model qe X²2 + R that takes sentences as input defined as qe (0([=1;02))
1 exp
log go (z+) where Z is a normalization constant and g0(2;; is language model for Σ Di=
a single sentence. Finally, we can compute the acceptance probability A ([x) [shew xnow)) for sentence-pair using qe.
6.1.2 Multi-class
In multi-class classification tasks, we have to deal with more than two classes. Typically, we can use the os-rest (OVR) or one-vs-one (OVO) approach. The former constructs a binary classifier for each class, and the class with the highest score is chosen as the predicted class, whereas the latter constructs a binary classifier for each pair of classes, and the class with the highest number of wins chosen the predicted class.
For the OVR approach, we can use the same algorithm as for the binary classification case, with the only difference being that the labels yi are now one-hot vectors instead of the binaries {=1]. For OVO, we can use the same algorithm, but with the labels y, being a vector of length MM - 1)/2. where M is the number of classes. To calculate the acceptance ratio in the MCMC step, we need to compute the backward probability 4o (zy neew). For OVR, we can simply use the same formula as for the binary classification case. On the other hand, for OVO we need to compute the backward probability for each pair of classes.
6.2 Tuning Technique
As approximate the target probability π with 7. there is risk of exponential amplification of the model's 's approximation error as the generative progresses, especially if the dynamics T म. + Th non-contractive in the measurable space X. To mitigate this drawback, we employ a "cross- validating" approach for posterior estimation in MCMC [16]. We run K independent chains in parallel, and use the posterior approximation # from the previous chain as an approximation to m for the current chain, instead of using # directly. Additionally, we have introduced the following heuristics into the implementation:
burn-in period fo. assuming that the variance of 7; stabilizes after to iterations. We update the reference model गt every to steps, instead of updating it at every step, to reduce the sensitivity of the initial samples and improve the overall stability of the approximation.
We incorporate probabilistic SVM [40] and multiply the probability (new))) क, This helps to refine the estimation of the posterior by incorporating the predictive power of the SVM.

---

If the newly generated samples Nnew duplicates any of the previous ones, or if is one of the predefined stop words egg, " <BOS>, or <LF'> we set a4+1 0 and generate new token.
To address the issue of sensitivity to initial samples, we employ transfer learning for initial sampling. We directly draw 2no seeds from the task description in the original paper of the target dataset [39]. We carefully select task descriptions that are model-agnostic and unbiased, enabling us to train our model zero-shot manner. While we use mid-size completion model text-curie-001 during the process, we utilize text-davinci-003 for the initial sampling. We have observed that repeating this initial sampling process also yields high performance, but increases the overall computational cost compared to MCMC. Therefore, using this heavy sampling approach once at / = 0 is the most practical option. In Table 3, we provide examples of the model-agnostic task descriptions used for transfer learning. These examples demonstrate the effectiveness of our approach in achieving high performance across different tasks. An exhaustive list of models and hyperparameters used in the experiments of this paper is presented in Table 4.
6.3 Limitation
While our proposed approach has yielded promising results, it presents certain limitations. First, SVG's effec- tiveness hinges on the quality of the representative samples that the PL.Ms generate. The interpretability of these generative models often lacks clarity, and they may inadvertently perpetu- ate undesirable biases inherent in the training data [43].
Table Models and hyperparameters used for GLUE bench- mark, shared by all the tasks after tuned with SST-2.
Description
Value
Text embedding
text sett-embedding ada-002*
Language model
iext-davinci-0031 (seeds)
text-curie-001 (MCMC) libsvm!
fx
Kernel machine model Examples per class Dimension of the embeds
100+1000 (seeds/MCMC)
d
1536
Second, in spite of superior memory efficiency compared to contemporary PLMs, the computational efficiency of kernel methods still tends to de- teriorate significantly when dealing with high- nigh-dimenssonal embeddings many-shot problems. As for compu- tational speed (nn)), kernel methods frequently lag behind PLMs O(nm²) of n >> m
m
Length in tokens
2048
Co
Upper bound of the duals on
10
10 10
R
Non- -on-liie
RBE Linear
Scaler of the RBF kernels
auto
K
Chains in parallel.
5
to
Burn-in period
Multi-class settings
OVR, OVO 5
Cross-validating bins
Temperature
.5 .05
Prompt
* as _?
Stop words
<BOS> | <LF>|.
We anticipate addressing this limita- tion in our future work. One proposed direction of research involves the development of more informed and reliable strategies for generating representative samples. This could mean leveraging insights from active learning or using novel architectures that encourage diversity in the generated examples. We also aim to explore efficient kernel methods and applicable approximations, such random Fourier features [32], to tackle large data scenarios. Pursuing these avenues, we believe, will bring us closer to creating efficient and effective zero-shot learning models with wider range of applicability.
GGPT3 [6] '17)
7 Conclusion
We presented Support Vector Generation (SVG), training-free method that converts a frozen language model into an explicit, interpretable kernel classifier by (i) sampling candidate sentences near the decision boundary and (ii) solving support-vector machine in the PLM-induced language kernel. SVG minimises empirical hinge loss within the span of its supports, completes training on single CPU in under three minutes, andon seven zero-shot GLUE tasksmatches or slightly exceeds strong prompting baselines while providing faithful sentence-level rationales. These results demonstrate that large language models can be distilled into compact, explainable, and compute- efficient predictors, opening a path toward wider deployment of shoo NL.P in resturce-constrained settings.

---

References
[1] Sajjad Abnar and Willem Zuidema. Quantifying attention flow in transformers. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020), pages 4190-4197, 2020. URL http::tttt//haaeeeeeeee /aw.antoooooooyoooooooao cttpy//0gg/2000.00. acl-main. 385.
[2] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems 31 (NeurIPS 2018), pages 9505-9515, 2018. URL https: htt.tpss//ppoceeddggg.s.. neurins cc/paper/200 8/ file/ 222/2/224282222 htt.aaaeeee..aa222e22eeeeeceaaaa Pap.ee.apppppp.p.e pdÉ.
[3] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. Palm 2 technical report. arXiv preprini arXiv:2305. 10403, 2023.
[4] A Blumer, A Ehrenfeucht, D Haussler, and M Warmuth. Learnability and the Vapnik- Chervonenkis dimension. of the ACM, 36(4):929 90999965, 1989.
[5] Brislin. mack-translation for cross-cultural research. J. of Cross-cultural Psychology, 1(3): 185-216, 1970.
[6] T Brown, B Mann, N Ryder, M Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, Sastry, A Askell, al. Language models are few-shot learners. NeurlPS, 2020.
[7] Chang and C Lin. LIBSVM: a library for support vector machines. ACM Trans, on Intelligent Systems and Technology (TIST), 203):1-27, 2011.
[8] S Chib and E Greenberg. Understanding the metropoliishhstings algorithm. The American Statistician, 49(4):327-335, 1995.
[9] M Collins. Discriminative reranking for natural language parsing. In ACL, 2002.
[10] C Cortes and V Vapnik. Support-vector networks. Machine Learning, 20:273-297, 1995.
[11] K Duan, S Keerthi, and A Poo. Evaluation of simple performance measures for tuning SVM hyperparameters. Neurocomputing. 51:41-59, 2003.
[12] Elkan. Logistic regression, naive Bayes, and the importance of proper feature scaling. KDD, 2005.
[13] T Gao, A Fisch, and D Chen. Making pre-trained language models better few-shot learners. arXiv:2012.15723 2020.
[14] Grauman and T Darrell. The pyramid match kernel: Discriminative classification with of image features. IEEE Trans. on Pattern Analysis and Machine Intelligence. 29(6):1455-1430, 2007.
[15] Hastie, R Tibshirani, and Friedman. The elements inference, and prediction. Springer, 2009.
of statistical learning: data mining,
[16] Held, B Schrödle, and H Rue. Posterior and cross-validatory predictive checks: a comparison of meme and inla. Statistical modelling and regression structures: Festschrift in honour of ludwig fahrmeir, pages 91-110, 2010.
[17] Ito, Ohsawa, and H Tanaka. Information diffusion enhanced by multi-takk peer prediction. In Proceedings of the 20th International Conference Information Integration and Web-Based Applications & Services, 2018.
[18] Leslie, Eskin, Weston, and W Noble. The spectrum kernel: a string kernel for SVM protein classification. Pacific Symposium Biocomputing, pages 579-590, 2003.
[19] Y Liu, Z Yang, T Zhao, D Xiong, X Wang, and B Liu. Generating long and coherent paragraphs with topic-aware neural networks. arXiv: 1903. 03051, 2019.
[20] Lowe. Object recognition from local scale-invariant features. IEFE Truns. on Pattern Analysis and Machine Intelligence, 21(6))810-822 1999.
10

---

[21] C "Manning, P Raghavan, and H Schütze. Introduction to information retrieval. 2008.
[22] Meng, Huang, Y Zhang, and Han. Generating training data with language models: Towards zero-shot language understanding. NeurIPS, 2022.
[23] J Morris, Lifland, Yoo, J Grigsby, D Jin, and Y Qi. TextAttack: A framework for adversarial attacks, data augmentation, and adversarial training in NLP. In EMNLP, 2020.
[24] Nakkiran, G Kaplun, Y Bansal, T Yang, B Barak, and Sutskever. Deep double descent: Where bigger models and more data hurt. J. of Statistical Mechanics: Theory and Experiment, 2021(12):124003, 2021.
[25] S Ohsawa. Truthful self-play. In ICLR, 2023.
[26] S Ohsawa, Y Obara, and T Osogami. Gated probabilistic matrix factorization: learning users' attention from missing values. In IJCAI, 2016.
[27] S Ohsawa, K Akuzawa, T Matsushima, G Bezerra, Y Iwasawa, H Kajino, S Takenaka, and Matsuo. Neuron as an agent. In ICLR Workshop, 2018.
[28] H Okamoto, M Suzuki, Higuchi, S Ohsawa, and Y Matsuo. Dual space learning with variational autoencoders. In ICLR Workshop, 2019.
[29] J Platt. Sequential minimal optimization: fast algorithm for training support vector machines. 1998.
[30] A Radford, K Narasimhan, T Salimans, I Sutskever, and J Martens. Improving language understanding by generative pre-training. OpenAl Blog, 2018.
[31] A Radford, Wu, R Child, D Luan, D Amodei, and Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8):9, 2019.
[32] A Rahimi and Recht. Random features for large-scale kernel machines. In NIPS, 2008
[33] C Rasmussen and C Williams. Gaussian processes for machine learning. Adaptive Computation and Machine Learning, pages 100-109, 2006.
[34] J Shawe-Taylor and N Cristianini. Kernel methods for remote homology detection. In KDD, 2004.
[35] I Syarif, A Prugel-Bennett, and Wills. SVM parameter optimization using grid search and genetic algorithm to improve classification performance. TËI.KONIKA (Telecommunncattion Computing Electronics and Control), 14(4):500-1150, 2016.
[36] V Vapnik and A Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and its Applications, 16(2):264, 1971.
[37] A Vaswani, N Shazeer, N Parmar, Uszkoreit, Jones, A Gomez, L. Kaiser, and Polosukhin. Attention is all you need. NeurIPS, 2017.
[38] S Vishwanathan, A Smola, and B Schölkopf. Fast kernel machines. NIPS, 2003.
[39] A Wang, A Singh, J Michael, F Hill, O Levy, and S Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019.
[40] T Wu, C Lin, and R Weng. Probability estimates for multi-class classification by pairwise coupling. NIPS, 2003.
[41] C Zhhngg S Bengio, M Hardt, B Recht, and O Vinyals. Understanding deep learning requires rethinking generalization. In /CLR, 2016.
[42] 7. Zhang, E Dinan, Urbanek, A Sordoni, and A Trischler. Dialogpt: Large-scale generative pre-training for conversational response generation. In NAACL, 2019.
[43] J Zhao, T Wang, M Yatskar, V Ordonez, and K Chang. The limitations of deep learning in adversarial settings. ACL, 2018
[44] X Zhu, J Dai, M Ye, 7. Zhu, Liu, and Song. Generative adversarial learning towards fast weakly supervised detection. In CVPR, 2019.

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and Sec. I state that SVG provides an interpretable, training- free kernel classifier and match the theoretical (Sec. 3) and empirical (Sec. findings without over-claiming generalisation beyond the GLUE tasks considered.
Justification:
2. Limitations
Question:
Does the paper discuss limitations of the work performed by the authors?
Answer: [Yes]
Justification: Sec. 6.3 Limitation details reliance on LLM quality, scalability issues of kernel methods, and potential bias amplification.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Assumptions for Thms. 3.13.2 are stated in Sec. 3; complete proof's appear in Appendix A.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Sec. 5, Table 4, and the supplement list all Hyper-parameters, data splits, and evaluation protocols; an anonymised GitHub repo with scripts is linked in the supplementary material.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Public GLUE datasets are cited; anonymised code (including inference scripts and instructions) provided in the supplemental ZIP and will be made public upon accep- tance.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Sec. 5 and Table 4 enumerate chain counts, burn-in, C-grid, kernel type, proposal temperature, and dataset splits.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Table reports mean + standard deviation over three independent runs with different random seeds; Sec. 5.1 explains the variability source.
12

---

8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Sec. 5.1 describes use of 3 GHz 1-core CPUs with 1 GB RAM on public cloud; Table (right column) reports per-task wall-clock time.
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:
Answer: [Yes]
Justification: The work follows the code; bias risks are acknowledged (Sec. 6.3) and no personal data is processed.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Sec. 6.3 discusses positive impacts (energy-efficient inference, interpretability) and negative ones (possible biased rationales, misuse of generated text).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No new high-risk model or dataset is released; SVG code only calls the OpenAI API under their existing usage policies.
12. Lieenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: GLUE (MIT license) and LIBSVM (BSD-style) are cited; OpenAl GPT-3 API terms are referenced in Sec. 4.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new datasets or pretrained modelsonly code.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No huwan- subject or crowdsourcing study was conducted.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
13

---

Answer: [NA]
Justification: No human-subject research is involved.
16. Declaration LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research?
Answer: [Yes]
Justification: Sec. 4 details how frozen OpenAl completion models (text-davinci-003, text- curie-00l) are used for MetropolisHaatings proposals and initial seed generation, which is central to SVG.
14


Reviews:
[{"review_id": "RpIoptFwpg", "reviewer": "Reviewer_EHmm", "summary": "This paper introduces Support Vector Generation (SVG), a method that converts a frozen LM into an interpretable, training-free classifier for zero- and few-shot NLP tasks. SVG combines Metropolis–Hastings sampling and support vector machine optimisation in the RKHS induced by the language model’s embedding. The method identifies a small set of natural language sentences as support vectors, which act as both classifiers and rationales. The authors show that SVG matches or outperforms strong prompting baselines on seven GLUE tasks while running on CPUs in under three minutes per task. Theoretical guarantees are also provided, with the generalisation bound depending on the number of supports rather than model size.", "questions": "- Could the authors clarify whether SVG’s performance degrades if the PLM used to induce the kernel is weaker? In other words, how sensitive is SVG to the choice of embedding model?\n- The interpretability claim is interesting. Could the authors include concrete examples of support sentences and their influence scores?\n- Have the authors considered applying SVG to more complex tasks beyond GLUE?\n- What are the limits of scalability for SVG?", "limitations": "yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- The paper is technically sound, with a clear algorithmic description and proofs of theoretical claims.\n- The paper is generally well written\n- Ability to runn inference and training on CPUs in minutes is a meaningful contribution \n- The idea of distilling a PLM into a support-vector-like classifier with interpretable supports is novel according to my knowledge.\n\n\nWeaknesses:\n- Experimental analysis is limited to GLUE tasks and relatively small models (OpenAI APIs), and it is not clear how SVG scales to harder tasks or different domains\n- Some sections (notably the MCMC derivation) are dense and hard to follow, particularly for readers unfamiliar with kernel methods or MCMC (but this can be just my lack of expertise in this domain)\n- The connection between the language kernel and interpretability could be better explained with examples.", "was_revised": true, "final_justification": "I am keeping my score unchanged. It is a positive accept score, but two concerns prevent me from increasing it further: (1) evaluation only on the GLUE benchmark, and (2) reliance on closed-source GPT models while framing the work as more reproducible.", "rebuttal": "We thank you for the constructive and thoughtful feedback. Below, we address each of the raised points.\n\n> Experimental analysis is limited to GLUE tasks and relatively small models (OpenAI APIs), and it is not clear how SVG scales to harder tasks or different domains.\n\nWe chose GLUE as a representative and widely accepted benchmark to evaluate zero-shot NLP performance across diverse classification tasks. While we agree that broader evaluation would be beneficial, the current results already demonstrate the method's generality. We view scaling SVG to more complex or domain-specific tasks as a valuable direction for future work.\n\n> Some sections (notably the MCMC derivation) are dense and hard to follow, particularly for readers unfamiliar with kernel methods or MCMC (but this can be just my lack of expertise in this domain).\n\nWe appreciate this feedback. To keep the method transparent and reproducible, we included the full derivation. However, we acknowledge that the exposition could be streamlined, especially for readers less familiar with these techniques. We will consider clearer presentation in a future revision.\n\n> The connection between the language kernel and interpretability could be better explained with examples.\n\nThank you. SVG produces a small number of support vectors, each corresponding to a natural language sentence with an associated influence score (dual weight). These vectors act as both the classifier and its rationale. Figure 2 (right) shows concrete examples of such supports, and we agree that including more examples in supplementary material would be beneficial.\n\n> Could the authors clarify whether SVG’s performance degrades if the PLM used to induce the kernel is weaker? In other words, how sensitive is SVG to the choice of embedding model?\n\nYes, the performance of SVG may be influenced by the quality of the underlying language model, as the kernel is constructed from its embedding space. That said, since SVG relies on relative similarity rather than absolute scores, it retains a degree of robustness. A comprehensive study of this sensitivity remains future work.\n\n> The interpretability claim is interesting. Could the authors include concrete examples of support sentences and their influence scores?\n\nAs noted in Figure 2 (right), SVG outputs support vectors with non-zero dual coefficients, which directly serve as influence scores. These vectors can be inspected post hoc as faithful explanations. We agree that showcasing additional examples could help clarify this point and would be open to including them upon acceptance.\n\n> Have the authors considered applying SVG to more complex tasks beyond GLUE?\n\nWe appreciate this suggestion. We focused on GLUE due to its task variety and relevance for zero-shot classification. We are interested in extending SVG to more complex settings, such as multi-label or generative tasks, in future work.\n\n> What are the limits of scalability for SVG?\n\nSVG is designed for compactness and efficiency: the classification decision is made using at most 32 support vectors. This enables practical CPU-only deployment. However, as discussed in Section 6.3, scalability may be limited in scenarios with very large support sets, and efficient kernel approximations (e.g., random features) may be necessary in those settings."}, {"review_id": "lCp0ej3fPO", "reviewer": "Reviewer_vTmc", "summary": "This paper introduces Support Vector Generation (SVG), a kernel-based framework that converts a frozen language model into an interpretable, training-free classifier for zero- and few-shot learning. SVG operates by combining a sampling method with support vector machine optimization to reproduce kernel Hilbert space induced by the language model’s embedding. Each classification decision is based on a weighted combination of at most 32 natural-language sentences, which serve as explicit support vectors and provide faithful rationales. The theoretical analysis shows that SVG minimizes the empirical hinge loss over the span of the supports and admits a generalization bound independent of the language model size. Experiments on the GLUE benchmark show that SVG matches or surpasses prompting-based zero-shot baselines in accuracy across multiple tasks—without any fine-tuning or GPU acceleration. Notably, their CPU-only implementation completes training in under three minutes per task and maintains competitive inference speed. These results suggest that SVG offers a viable path toward efficient, interpretable NLP systems under compute constraints.", "questions": "I have put up a set of questions in the weakness part and hope to get the insight from the authors on positions and contributions in relevance to the newer studies in LLMs.", "limitations": "The discussion of the limitations should have more discussions on the state-of-the-art of research in LLM in the last several years.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n\n1. The motivation of using Support Vector Generation (SVG), that converts a frozen language model into an interpretable, training-free classifier for zero- and few-shot learning is interesting.\n\n2. SVG combines a sampling method with support vector machine optimization to reproduce kernel Hilbert space induced by the language model’s embedding and then the classification decision can be based on a weighted combination of a small number of natural-language sentences, serving as explicit support vectors and providing faithful rationales. \n\n3. Experiments on the GLUE benchmark show that SVG matches or surpasses prompting-based zero-shot baselines in accuracy across multiple tasks—without any fine-tuning or GPU acceleration. \n\n4. CPU-only implementation of training suggests that SVG offers a viable path toward efficient, interpretable NLP systems under compute constraints.\n\nWeakness:\n\n1. The comparison with prompting was on a benchmark done in 2020, years before GPT or real LLMs came into being.  I am not sure whether such a comparison will be convincing.  Why not comparing with more recent LLM to demonstrate your pontential?\n\n2. Even when you review the related work, it seems most of them are not on recent year's work on LLMs, such as GPT, Llama, Qwen, Search-R1.   Why do you want to give a state-of-the-art overview and comparison of the most recent work?\n\n3. For prompting, there are also a lot of recent work on prompting, such as chain-of-thought, tree-of-thought, and many newer ones.  I am wondering why the paper does not do an overview and compare with the recent work to give readers a good reasoning and the value of your study in comparison with the new ste-of-the-art?", "was_revised": true, "final_justification": "Based on the authors' rebuttal and the other reviews, I have decided to raise the review score to 4 (borderline accept).  Thanks.", "rebuttal": "Thank you for your review and sincere suggestions.\n\n> The comparison with prompting was on a benchmark done in 2020, years before GPT or real LLMs came into being. I am not sure whether such a comparison will be convincing. Why not comparing with more recent LLM to demonstrate your potential?\n\nOur goal is to isolate the contribution of Support Vector Generation (SVG) under CPU‑only, zero‑shot constraints.\n\nFirstly, regarding the benchmark choice, GLUE remains the only public suite that (i) supplies classification labels for all tasks and (ii) is broadly reused by recent kernel/distillation work, enabling a like‑for‑like comparison.\n\nSecondly, sota LLMs (GPT‑4, Gemini 1.5, etc.) require GPU inference and longer prompts; this violates the edge‑device / privacy‑sensitive deployment scenario we target.\n\nThirdly, as for the fairness to baselines, SVG’s decision function uses <32 kernel evaluations; therefore we pair it with token‑balanced zero‑shot prompts rather than multi‑hundred‑token chain‑of‑thought variants to avoid giving SVG an artificial deficit.\n\n> Even when you review the related work, it seems most of them are not on recent year's work on LLMs, such as GPT, Llama, Qwen, Search‑R1. Why do you want to give a state‑of‑the‑art overview and comparison of the most recent work?\n\nWhile this paper prioritizes universality and broad applicability, software ecosystems evolve so quickly that continually switching to each freshly released model as you proposed would deliver only marginal benefit. Nevertheless, we can accommodate your concerns by removing model‑specific names from the Introduction part.\n\n> For prompting, there are also a lot of recent work on prompting, such as chain‑of‑thought, tree‑of‑thought, and many newer ones. I am wondering why the paper does not do an overview and compare with the recent work to give readers a good reasoning and the value of your study in comparison with the new state‑of‑the‑art?\n\nCoT/ToT are complementary to SVG rather than competitive, and thus these prompting methods would improve the performance and *vice versa*.\nThat said, this combination has trade-offs that typically inflate token budgets by ×10–×50, erasing the CPU cost advantage that motivates SVG. \n\nOur study focuses on classification latency under severe compute limits rather than generative potency.  We deliberately benchmark against token‑minimal prompts to provide a conservative baseline, and still found out that SVG yields +8 – 12 F1 on GLUE tasks while using only CPU inference (Table 2).\n\n> The discussion of the limitations should have more discussions on the state‑of‑the‑art of research in LLM in the last several years.\n\nDue to space limitation, we constrained Section 6.3 (Limitation) to acknowledge two key gaps vis‑à‑vis modern LLM trends:  \n1. Dependence on embedding quality: SVG can directly adopt newer open‑source embeddings (e.g., text‑embedding‑3‑large, E5, GTR‑XL); no code changes are required.  \n2. Quadratic kernel cost: We outline random‑feature approximations and active‑learning proposals as concrete next steps.  \n\nWe believe that these points situate SVG within ongoing LLM progress without overselling future work.\n\nWe appreciate your detailed feedback. The clarifications above address the concerns while respecting page limits and the CPU‑only, zero‑shot scope of our submission."}, {"review_id": "uQOx8HWCHf", "reviewer": "Reviewer_dtMy", "summary": "This paper introduces Support Vector Generation (SVG), a kernel-based framework that converts a frozen language model into an interpretable, training-free classifier for zero- and few-shot learning. It matches / exceeds the performance of prompting-based baselines on several GLUE benchmark tasks using only a CPU, with training <3mins per task.", "questions": "I've highlighted some actionable suggestions in the weaknesses section.", "limitations": "-", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1.\tResults are impressive without expensive hardware\n2.\tInterpretable model - each classification is the result of a weighted vote from a small set of up to 32 actual sentences, which serve as explicit support vectors\nWeakness:\n1.\tFundamental dependence on the quality and biases of the frozen pretrained model. Probably worth auditing the final set of support vectors.\n2.\tCurrent implementation relies on proprietary models – worth conducting supplementary experiments on open source models to improve accessibility of the method and extended research \n3.\tA deeper discussion on \"few-shot double descent\" could be helpful", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for their detailed and encouraging review. We respond to the specific concerns and suggestions below.\n\n> Fundamental dependence on the quality and biases of the frozen pretrained model. Probably worth auditing the final set of support vectors.\n\nWe agree that SVG inherits characteristics from the underlying PLM, including any biases it may encode. As each decision in SVG is based on a small, human-readable set of support sentences, we believe it offers a unique opportunity for manual auditing and inspection, which may help identify and mitigate such issues in practice.\n\n> Current implementation relies on proprietary models – worth conducting supplementary experiments on open source models to improve accessibility of the method and extended research.\n\nThis is an excellent point. We used OpenAI APIs to simplify reproducibility and avoid confounding implementation factors, but we agree that open-source alternatives (e.g., LLaMA or sentence-transformers) are natural candidates for future evaluation and community adoption. We appreciate the suggestion.\n\n> A deeper discussion on \"few-shot double descent\" could be helpful.\n\nThank you. We briefly mention this phenomenon in Section 5.2 and Figure 4, where accuracy initially dips and later recovers as more labeled examples are introduced. We agree this deserves a more detailed theoretical and empirical treatment, which we consider a promising direction for future work.\n\n> I've highlighted some actionable suggestions in the weaknesses section.\n\nWe sincerely appreciate the constructive suggestions and your overall positive assessment of the work."}, {"review_id": "KeMKKCc9nr", "reviewer": "Reviewer_LEM1", "summary": "This paper proposes the method named Support Vector Generation, or SVG for short, that utilizes kernel methods to convert a frozen pretrained language model to an interpretable classifier. It first uses Metropolis-Hastings to sample some natural language sentences (support vectors), which are then used to vote for the final decision through a weighted decision function. The weights of this function can be learned through the optimization methods akin to support vector machines. The authors experiment with their methods on the GLUE datasets and show that SVG can greatly improve the zero-shot performance of large language models (LLMs) with very constrained CPU-only compute resources.", "questions": "1. Did you ever try running this algorithm using open-source sentence embedding models, such as the checkpoints provided by sentence-transformers? This might reduce the cost and latency. Some smaller models can still be run on CPUs.\n2. Typo in eq (4), L115. It should be \"if $(x,y)$.\"", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Besides what's shown in the summary section, the paper has the following additional strengths:\n\n1. The paper is well-written, providing enough background knowledge and good explanation of the method. It also provides detailed discussion on the choice of the hyperparameters.\n2. SVG can be valuable in scenarios of limited access to LLMs or limited computational resources, as it can improve the performance of language models with CPU-only machines.\n3. The paper overcomes the challenges in sampling from intractable distributions (i.e., autoregressive language models) by sampling with the Metropolis-Hastings algorithm and some approximations.\n\nIt has the following weaknesses:\n\n\n1. The application of SVG is very limited. It can only be applied to a predefined classification problem. Though the authors discussed its extension to multi-class and sentence-pair examples, these skills are achieved with some tricks like one-over-rest or a different form of kernel functions. In this sense, the \"zero-shot\" ability of SVG is very constrained compared to a generative model.\n2. SVG relies on access to the embeddings of a pretrained language model, which can be costly because of its iterative access to the endpoint, and many endpoints do not provide embeddings for sentences. The actual cost of SVG is far more than just CPU machines.\n3. This method is significantly slow due to the nature of the MH sampling process and the latency of API calls.", "was_revised": true, "final_justification": "I think the author's reply echos with my comments and we are now on the same page. If the paper is accepted, authors may use more ink on the limitations.\n\nI will still lean towards acceptance and keep my score unchanged. This paper is unique among all LLM papers.", "rebuttal": "We thank you for the thorough and encouraging review. Below we address the concerns and questions raised.\n\n> The application of SVG is very limited. It can only be applied to a predefined classification problem. Though the authors discussed its extension to multi-class and sentence-pair examples, these skills are achieved with some tricks like one-over-rest or a different form of kernel functions. In this sense, the \"zero-shot\" ability of SVG is very constrained compared to a generative model.\n\nWe agree that SVG is currently tailored to classification tasks, which includes single-sentence, sentence-pair, binary, and multi-class classification as detailed in Section 6. While this differs from the broader generative capabilities of large language models, our aim is to provide interpretable and efficient alternatives specifically for classification under compute constraints.\n\n> SVG relies on access to the embeddings of a pretrained language model, which can be costly because of its iterative access to the endpoint, and many endpoints do not provide embeddings for sentences. The actual cost of SVG is far more than just CPU machines.\n\nWe acknowledge this point. While SVG is CPU-only during training and inference, it assumes access to an embedding API (e.g., OpenAI embeddings). In practice, the cost remains manageable due to the small number of queries (hundreds per task). Moreover, SVG can be adapted to work with local open-source embeddings, which we view as promising future work.\n\n> This method is significantly slow due to the nature of the MH sampling process and the latency of API calls.\n\nWe agree that MH sampling introduces latency, especially with API-based models. To mitigate this, we restrict the number of MCMC chains and propose efficient initial seeding strategies (Sec. 6.2). We also note that even with sampling, training completes in under three minutes per task (Table 2), making SVG practical in many scenarios.\n\n> Did you ever try running this algorithm using open-source sentence embedding models, such as the checkpoints provided by sentence-transformers? This might reduce the cost and latency. Some smaller models can still be run on CPUs.\n\nThis is a valuable suggestion. While our current implementation uses OpenAI embeddings to ensure compatibility and reproducibility, adapting SVG to open-source models is straightforward and aligns with our broader goal of democratizing access to interpretable zero-shot classifiers.\n\n> Typo in eq (4), L115. It should be \"...\n\nThank you for catching this. We will correct the typo."}]

Author rebuttal:

Meta-review: This paper introduces Support Vector Generation (SVG), a kernel-based framework that builds a sequence classifier based on a frozen language model for zero- and few-shot learning. The method optimizes support vectors (e.g., in the form of natural-language sentences) in the reproducing kernel Hilbert space (RKHS) induced by the language model's embedding. The SVG method is training-free and lightweight, thus contributing to efficient and interpretable NLP systems under compute constraints.

The reviewers generally appreciated the method to be novel and the empirical evaluation to convincingly demonstrate the advantage of SVG. However, they also pointed out several weaknesses, such as the exclusive use of old benchmarks (GLUE) in the evaluation and the lack of comparisons to more recent SOTA (open-source) models, which indicated that the advantage of SVG might only exist on a narrow set of tasks (NLU style). 

On balance, the reviewers still think the paper is worthy of acceptance at NeurIPS due to its novelty and promising results (albeit the current evaluation mainly focuses on narrow/outdated NLU tasks). The AC encourages the authors to discuss how the method can be generalized to a broader range of tasks (e.g., open-ended generation).
Decision: Accept (poster)