Title: Support  Vector  Generation: Kernelizing Large Language Models for Efficient Zero‑Shot NLP
Venue: NeurIPS 2025
Accepted: yes
Keywords: Zero-shot Learning

Kernel Machines

Large Language Models

Support Vector Generation

Resource-constrained NLP
Abstract: We introduce Support Vector Generation (SVG), a kernel-based framework that converts a frozen language model into an interpretable, training-free classifier for zero- and few-shot learning. SVG operates by combining Metropolis–Hastings sampling with support vector machine optimization in the reproducing kernel Hilbert space (RKHS) induced by the language model's embedding. Each classification decision is based on a weighted combination of at most 32 natural-language sentences, which serve as explicit support vectors and provide faithful rationales. Our theoretical analysis proves that SVG minimizes the empirical hinge loss over the span of the supports and admits a generalization bound independent of the language model size. Experiments on the GLUE benchmark show that SVG matches or surpasses prompting-based zero-shot baselines in accuracy across multiple tasks—without any fine-tuning or GPU acceleration. Notably, our CPU-only implementation completes training in under three minutes per task, and maintains competitive inference speed. These results suggest that SVG offers a viable path toward efficient, interpretable NLP systems under compute constraints.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the method named Support Vector Generation, or SVG for short, that utilizes kernel methods to convert a frozen pretrained language model to an interpretable classifier. It first uses Metropolis-Hastings to sample some natural language sentences (support vectors), which are then used to vote for the final decision through a weighted decision function. The weights of this function can be learned through the optimization methods akin to support vector machines. The authors experiment with their methods on the GLUE datasets and show that SVG can greatly improve the zero-shot performance of large language models (LLMs) with very constrained CPU-only compute resources.\n\nSTRENGTHS AND WEAKNESSES:\nBesides what's shown in the summary section, the paper has the following additional strengths:\n\n1. The paper is well-written, providing enough background knowledge and good explanation of the method. It also provides detailed discussion on the choice of the hyperparameters.\n2. SVG can be valuable in scenarios of limited access to LLMs or limited computational resources, as it can improve the performance of language models with CPU-only machines.\n3. The paper overcomes the challenges in sampling from intractable distributions (i.e., autoregressive language models) by sampling with the Metropolis-Hastings algorithm and some approximations.\n\nIt has the following weaknesses:\n\n\n1. The application of SVG is very limited. It can only be applied to a predefined classification problem. Though the authors discussed its extension to multi-class and sentence-pair examples, these skills are achieved with some tricks like one-over-rest or a different form of kernel functions. In this sense, the \"zero-shot\" ability of SVG is very constrained compared to a generative model.\n2. SVG relies on access to the embeddings of a pretrained language model, which can be costly because of its iterative access to the endpoint, and many endpoints do not provide embeddings for sentences. The actual cost of SVG is far more than just CPU machines.\n3. This method is significantly slow due to the nature of the MH sampling process and the latency of API calls.\n\nQUESTIONS:\n1. Did you ever try running this algorithm using open-source sentence embedding models, such as the checkpoints provided by sentence-transformers? This might reduce the cost and latency. Some smaller models can still be run on CPUs.\n2. Typo in eq (4), L115. It should be \"if $(x,y)$.\"\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-19"}

comments: SUMMARY:
This paper introduces Support Vector Generation (SVG), a kernel-based framework that converts a frozen language model into an interpretable, training-free classifier for zero- and few-shot learning. It matches / exceeds the performance of prompting-based baselines on several GLUE benchmark tasks using only a CPU, with training <3mins per task.

STRENGTHS AND WEAKNESSES:
Strengths:
1.	Results are impressive without expensive hardware
2.	Interpretable model - each classification is the result of a weighted vote from a small set of up to 32 actual sentences, which serve as explicit support vectors
Weakness:
1.	Fundamental dependence on the quality and biases of the frozen pretrained model. Probably worth auditing the final set of support vectors.
2.	Current implementation relies on proprietary models – worth conducting supplementary experiments on open source models to improve accessibility of the method and extended research 
3.	A deeper discussion on "few-shot double descent" could be helpful

QUESTIONS:
I've highlighted some actionable suggestions in the weaknesses section.

LIMITATIONS:
-
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Support Vector Generation (SVG), a kernel-based framework that converts a frozen language model into an interpretable, training-free classifier for zero- and few-shot learning. SVG operates by combining a sampling method with support vector machine optimization to reproduce kernel Hilbert space induced by the language model’s embedding. Each classification decision is based on a weighted combination of at most 32 natural-language sentences, which serve as explicit support vectors and provide faithful rationales. The theoretical analysis shows that SVG minimizes the empirical hinge loss over the span of the supports and admits a generalization bound independent of the language model size. Experiments on the GLUE benchmark show that SVG matches or surpasses prompting-based zero-shot baselines in accuracy across multiple tasks—without any fine-tuning or GPU acceleration. Notably, their CPU-only implementation completes training in under three minutes per task and maintains competitive inference speed. These results suggest that SVG offers a viable path toward efficient, interpretable NLP systems under compute constraints.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\n1. The motivation of using Support Vector Generation (SVG), that converts a frozen language model into an interpretable, training-free classifier for zero- and few-shot learning is interesting.\n\n2. SVG combines a sampling method with support vector machine optimization to reproduce kernel Hilbert space induced by the language model’s embedding and then the classification decision can be based on a weighted combination of a small number of natural-language sentences, serving as explicit support vectors and providing faithful rationales. \n\n3. Experiments on the GLUE benchmark show that SVG matches or surpasses prompting-based zero-shot baselines in accuracy across multiple tasks—without any fine-tuning or GPU acceleration. \n\n4. CPU-only implementation of training suggests that SVG offers a viable path toward efficient, interpretable NLP systems under compute constraints.\n\nWeakness:\n\n1. The comparison with prompting was on a benchmark done in 2020, years before GPT or real LLMs came into being.  I am not sure whether such a comparison will be convincing.  Why not comparing with more recent LLM to demonstrate your pontential?\n\n2. Even when you review the related work, it seems most of them are not on recent year's work on LLMs, such as GPT, Llama, Qwen, Search-R1.   Why do you want to give a state-of-the-art overview and comparison of the most recent work?\n\n3. For prompting, there are also a lot of recent work on prompting, such as chain-of-thought, tree-of-thought, and many newer ones.  I am wondering why the paper does not do an overview and compare with the recent work to give readers a good reasoning and the value of your study in comparison with the new ste-of-the-art?\n\nQUESTIONS:\nI have put up a set of questions in the weakness part and hope to get the insight from the authors on positions and contributions in relevance to the newer studies in LLMs.\n\nLIMITATIONS:\nThe discussion of the limitations should have more discussions on the state-of-the-art of research in LLM in the last several years.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Support Vector Generation (SVG), a method that converts a frozen LM into an interpretable, training-free classifier for zero- and few-shot NLP tasks. SVG combines Metropolis–Hastings sampling and support vector machine optimisation in the RKHS induced by the language model’s embedding. The method identifies a small set of natural language sentences as support vectors, which act as both classifiers and rationales. The authors show that SVG matches or outperforms strong prompting baselines on seven GLUE tasks while running on CPUs in under three minutes per task. Theoretical guarantees are also provided, with the generalisation bound depending on the number of supports rather than model size.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is technically sound, with a clear algorithmic description and proofs of theoretical claims.\n- The paper is generally well written\n- Ability to runn inference and training on CPUs in minutes is a meaningful contribution \n- The idea of distilling a PLM into a support-vector-like classifier with interpretable supports is novel according to my knowledge.\n\n\nWeaknesses:\n- Experimental analysis is limited to GLUE tasks and relatively small models (OpenAI APIs), and it is not clear how SVG scales to harder tasks or different domains\n- Some sections (notably the MCMC derivation) are dense and hard to follow, particularly for readers unfamiliar with kernel methods or MCMC (but this can be just my lack of expertise in this domain)\n- The connection between the language kernel and interpretability could be better explained with examples.\n\nQUESTIONS:\n- Could the authors clarify whether SVG’s performance degrades if the PLM used to induce the kernel is weaker? In other words, how sensitive is SVG to the choice of embedding model?\n- The interpretability claim is interesting. Could the authors include concrete examples of support sentences and their influence scores?\n- Have the authors considered applying SVG to more complex tasks beyond GLUE?\n- What are the limits of scalability for SVG?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-06"}