Transductive Learning for Textual Few-Shot Classification in API-based Embedding Models
Authors: Pierre Colombo
Victor Pellegrain
Malik Boudiaf
Myriam Tami
Victor Storchan
Ismail Ben Ayed
Pablo Piantanida
Venue: EMNLP 2023 Main (2023)
Keywords: NLP
few shot
text classifiers

Abstract: Proprietary and closed APIs are becoming increasingly common to process natural language, and are impacting the practical applications of natural language processing, including few-shot classification. Few-shot classification involves training a model to perform a new classification task with a handful of labeled data. This paper presents three contributions. First, we introduce a scenario where the embedding of a pre-trained model is served through a gated API with compute-cost and data-privacy constraints. Second, we propose a transductive inference, a learning paradigm that has been overlooked by the NLP community. Transductive inference, unlike traditional inductive learning, leverages the statistics of unlabelled data. We also introduce a new parameter-free transductive regularizer based on the Fisher-Rao loss, which can be used on top of the gated API embeddings. This method fully utilizes unlabelled data, does not share any label with the third-party API provider and could serve as a baseline for future research. Third, we propose an improved experimental setting and compile a benchmark of eight datasets involving multiclass classification in four different languages, with up to 151 classes. We evaluate our methods using eight backbone models, along with an episodic evaluation over 1,000 episodes, which demonstrate the superiority of transductive inference over the standard inductive setting.

Transductive Learning for Textual Few-Shot Classification in API-based Embedding Models
Pierre Colombo 2,, Victor Pellegrain *5.6
Malik Boudiaf Victor Storchan', Myriam Tami Ismail Ben Ayed'. Celine Hudelot Pablo Piantanida" ETS Montreal, LIVIA, ILLS, Canada Equall, Paris, France
4 ILLS, MILA, CNRS, CentraleSupélec, Canada IRT SystemX Saclay, France, 6 MICS, CentraleSupelec, Universite Paris-Saclay, France
Mozilla.ai, Paris, France
Abstract
Brown al., 2020), GPT-4 (OpenAI, 2023), GPT- (Brown et al., 2020), T5 (Raffel et al., 2020), and BERT (Devlin et al., 2018), which have achieved unprecedented performance in text classification (Liu et al., 2019b), language modeling, machine translation (Fan et al., 2021), and coding tasks (Chen et al., 2021a).
Proprietary and closed APIs are becoming in- creasingly common to process natural language. and are impacting the practical applications of natural language processing, including few- shot classification. Few-shot classification in- volves training a model to perform a new clas- sification task with handful of labeled data. This paper presents three contributions. First, we introduce scenario where the embedding of a pre-trained model is served through gated API with compute-cost and data-privacy con- straints. Second, we propose transductive in- ference, a learning paradigm that has been over- looked by the NLP community. Transductive inference, unlike traditional inductive learning, leverages the statistics of unlabeled data. We also introduce a new parameter- free transduc- tive regularizer based the Fisher-Ro Rao loss, which can be used on top of the gated API embeddings. This method fully utilizes unla- beled data, does not share any label with the third-party API provider and could serve as a baseline for future research. Third, we propose improved experimental setting and compile a benchmark of eight datasets involving multi- class classification in four different languages, with up to 151 classes. We evaluate our meth- ods using eight backbone models, along with an episodic evaluation over 1,000 episodes, which demonstrate the superiority of transductive in- ference over the standard inductive setting.
Despite the success of the scaling paradigm, sig- nificant challenges still exist especially when the many practical constraints of real-world scenarios have to be met: labeled data be severely lim- ited (i.e., few-shot scenario (Song et al., 2022; Ye et al., 2021)), data privacy is critical for many in- dustries and has become the subject of increasingly many regulatory pieces (Commission, 2020, 2016), compute costs need to be optimized (Strubell et al., 2019). Furthermore, these challenges are made even more complex stronger foundation models are now available only through APIs (e.g., Ope- nAl's GPT-3, GPT-4 or ChatGPT, Anthropic's Claude or Google's s PaLM (Chowdhery et al.. 2022)) which has led to some of their parame- ters being concealed, presenting new challenges for model adaptation (Solaiman, 2023). This pa- per is centered on the fundamental task of few- shot text classification, specifically focusing on cloud-based/APP access. Specifically, we formulate three requirements for API-based few-shot learning (FSL) (see Fig. 1):
1 Introduction
(R1) Black-box scenario. We focus on learning from models that are opaquely deployed in produc- tion to the end-user, who only has access to the end-point of the encoder, ie., the resulting text em- bedding produced by the final layer of the network. (R2) Low resources / computation time. Al sys- tems often required to make rapid predictions at high frequencies in various real-world applications. Therefore, any few-shot classifier used in such sce- narios should have low training and inference time, as well as require minimal computational re- sources.
Recent advances in Natural Language Processing (NLP) have been largely driven by the scaling paradigm (Kaplan et al., 2020; Rosenfeld et al., 2019), where larger models with increased param- eters have been shown to achieve state-of-the-art results in various NLP tasks (Touvron et al., 2023; Radford et al., 2019). This approach has led to the development of foundation models such as Chat- GPT (Lehman al., 2023; Kocoń et al., 2023;
'These authors contributed equally to this work

---

(R3) Limited Data Sharing. When utilizing API models, data sharing becomes major concern. In the current landscape, providers are increasingly of- fering less transparent procedures for training their networks. As a result, users prefer sharing as little information as possible, such as labeling schemn and annotated data, to safeguard their data privacy. Shorteomings of Existing Works. While numer- previous studies have addressed the popular few-shot classification setting, to our knowledge no existing line of work adequately satisfies the three API requirements described above. In particular, prompt-base based FSL (Schick and Schütze, 2020a) and parameter-efficient fine-tuning FSL (Houlsby et al., 2019) both require access to the model's gradients, while in-context learning scales poorly with the task's 's size (e.g number of shots, number of classes) (Chen et al., 2021b; Min et al., 2021, 2022: Brown al., 2020) and requires full data sharing. Instead, we focus on methods that can operate within API-based constraints.
larizer can serve baseline for future research.
Contributions
In this paper, we make several contributions to the field of textual FSL. Precisely, our contributions threefold:
A new textual few-shot scenario: We present a new scenario for FSL using textual API-based mod- els that accurately capture real-world constraints. Our new scenario opens up new research avenues and opportunities to address the challenges associ- ated with FSL using API-based models, paving the way for improved performance in practical applica- tions.
A novel transductive baseline. Our paper pro- poses transductive FSL algorithm that utilizes a novel parameter-free Fisher-Rao-based loss. By leveraging only the network's 's embedding (R1), our approach enables fast and efficient predictions (R2) without the need to share the labeling schemn or the labels of few-shot examples making it compli- ant with (R3). This innovative method marks a significant step forward in the field of FSL. A truly improved experimental setting. Previous studies on textual few-shot classification (Schick and Schütze, 2022, 2020b; Mahabadi et al., 2022; Tam et al., 2021; Gao et al., 2020) have predomi- nantly assessed their algorithms on classification tasks with restricted number of labels (typically less than five). We take step forward and create benchmark that is more representative of real- world scenarios. Our benchmark relies on total of eight datasets, covering multiclass classification tasks with up to 151 classes, across four different languages. Moreover, we further enhanced the eval- uation process by not only considering 10 classi- fiers trained with 10 different seeds (Logan IV et al., 2021; Mahabadi et al., 2022), but also by relying on episodic evaluation on 1,000 episodes (Hospedales et al., 2021). Our results clearly demonstrate the superiority of transductive methods.
Under R1, R2, and R3 requirements, the stan- dard inductive learning (Liu et al., 2022) may be quite limiting, To mitigate the labeled data scarcity while retaining API compliance, we revisit trans- duction (Vapnik, 1999) in the context of textual few-shot classification. Specifically, in the con- text of FSL. transductive FSL (Liu et al., 2019a) advocates leveraging unlabeled test samples of a task as an additional source of information on the underlying task's data distribution in order to bet- ter define decision boundaries. Such additional source essentially comes for free in many offline applications, including sentiment analysis for cus- tomer feedback, legal document classification, or text-based medical diagnosis.
Our findings corroborate recent findings in com- puter vision (Liu al., 2019a; Ziko et al., 2020; Lichtenstein al., 2020; Boudiaf et al., 2020; Hu et al., 2021b), that substantial gains can be obtained from using transduction over induction, opening new avenue of research for the NLP community. However, the transductive gain comes at the cost of introducing additional hyperparameters, and care- fully tuning them. Motivated by Occam's razor principle, propose novel hyperparameter- free transductive regularizer based on Fisher-Rao dis- tances and demonstrate the strongest predictive per- formances across various benchmarks and models while keeping hyperparameter tuning minimal. We believe that this parameter-fin transductive regu-
2 Related Work
2.1 Few-shot learning in NLP
Numerous studies have tackled the task of FSL in Natural Language Processing (NLP) by utiliz- ing pre-trained language models (Devlin al., 2018: Liu al., 2019b; Radford et al., 2019; Yang et al., 2019). These methods can be classified into three major categories: prompt-based based, parameter- efficient tuning, and in-context learning.

---

Slack the encoder
No training possible
Local training
X
Y
fe
EΦ
Useroriy
Xs
Zs
9 5
Accessible
Ys United Data Sharing: lobeling sahema and labels not shared
Y'
Api Col-provider-orly uccesa
Figure 1: API- based FSL scenario. The black-box API provides embeddings from the pretrained encoder fa. The black-box scenario discards existing inductive approaches and in-context learning methods due to the inaccessible of the model's parameters ((R)) and privacy concerns ((R3). This scenario, allows tuning classification head 9o (using induction or transduction) at low computational cost (R2) while retaining all support labels locally.
Prompt-based FSL: Prompt-based FSL involves the use of natural language prompts or templates to guide the model perform specific task (Ding al., 2021: Liu al., 2023). For example, the sem- inal work (Schick and Schütze, 2020a) proposed a model called PET. which uses a pre-defined set of prompts to perform various NLP tasks as text classification. They also impose a choice of a ver- balizer which highly impacts the classification per- formances (Cui et al., 2022; Hu et al., 2021a). How- ever, recent studies have questioned the benefits of prompt-based learning due the high variability in performance caused by the choice of prompt (Liu al., 2022). To address this issue, researchers have proposed prompt tuning which involves few learn- able parameters addition to the prompt (Lester et al., 2021). Nevertheless, these approaches face limitations when learning from API: (i) encoder access for gradient computation is infeasible (as in R1), (ii) prompting requires to send data and label which raises privacy concerns (as in R3), and (iii) labeling new points is consuminn (see in R3) and expensive due to the need to send shots for each input token'.
pute gradients of the encoder (as per R1) and the requirement to send both the labeling schema and the labels, which violates R3.
In Context Learning (ICL). In-context learning models are models that utilize imputtoo-uupuu train- ing examples as prompts to make predictions, with- out any parameter updates (Wei et al., 2022). These models, such as text text-davinci, rely solely on the provided examples to generate predictions, with- out any additional training. However, significant drawback of this approach is that the user must sup- ply the input, label examples, and task description, which becomes prohibitively expensive when the number of classes or shots increases, is slow (Liu et al., 2022) (R2) and raises data privacy concerns (as highlighted R3). Additionally, the inabil- ity to reuse text embeddings for new tasks or with new labels without querying the model's 's API limits practicality and scalability, making reusable encod- ing unfeasible for in-context learning models?. Meta-learning. Meta-learning approaches have for quite long stood as the de-facto paradigm for FSL ((Snell et al., 2017: Rusu et al., 2019; Sung et al., 2018b; Lee ct al., 2019; Raghu et al., 2019; Sun et al., 2019a)). In meta-learning. the objective is to provide the model with the intrinsic ability to learn in data-efficient manner. For instance, MAML ((Finn et al., 2017a; Antoniou et al., 2018)), arguably the most popular meta-leariin method, tries train a model such that can be fine-tuned end-to-end using only few supervised samples while retaining high generalization ability. Un- like the three previous lines work, meta-learning methods operate by modifying the pre-training pro-
Parameter-efficier fine-tuning. These methods, such as adapters (Houlsby et al., 2019; Pfeiffer et al., 2020), keep most of the model's parameters fixed during training and only update small feed- forward networks that are inserted within the larger model architecture. A recent example is T-FEW (Liu et al., 2022), which adds learned vectors that rescale the network's internal activations. Addition- ally, requires a set of manually created prompts for each dataset making it hard to use in practice. Relying on srrameter-efficiiee fine-tuning methods with an API is not possible due to the need to com-
2 Furthermore, as the number of considered classes in- creases. the fixed size of the transformer limits the number of possible shots that can be fed to the model. Previous stud- ies have often neglected this limitation by focusing on few numbers of lahels.
'The cost of API queries is determined by the number of input tokens that are transmitted.

---

cedure and therefore assume access both the training data and the model, which wholly breaks both R1 and R3.
that has a set of unseen classes Viest. Each task involves a few labeled examples from K differ- ent classes chosen at random among Veest. These labeled examples constitute the support set $ = {rt, yi) kETs> with a size of |S] = Ng x K. Ad- ditionally, each task has an unlabeled query set
2.2 Inductive vs transductive learning
Learning an inductive classifier on embeddings generated by an API-based model, as proposed by (Snell et al., 2017), is a common baseline for performing FSL. This approach is prevalent in NLP. where parametric model is trained on data to infer general rules that are applied to label new, unseen data (known inductive learning (Vapnik, 1999)). However, in FSL scenarios with limited labeled data, this approach can be highly ambiguous and lead to poor generalization.
Q = (ai) composed of |Q] = No x K un- seen examples from each of the K classes. Is and IQ represent the drawn indices during the sampling process for support set and query set, respectively. Pre-trained models use few-shot techniques and the labeled support sets to adapt to the tasks at hand and are evaluated based on their performances on the unlabeled query sets.
Remark Setting the values of N and K in textual FSL is not standardized, as discussed in Sec. 3. 1. Therefore, in all of our experiments, we have relied on setting (N, K) E {5, 10)².
Transduction offers an attractive alternative to inductive learning (Sain, 1996). Unlike inductive learning, which infers general rules from training data, transduction involves finding rules that work specifically for the unlabeled test data. By utilizing more data, such as unlabeled test instances, and aiming for a more localized rule rather than gen- eral one, transductive learning has shown promise and practical benefits in computer vision (Boudiaf et al., 2020, 2021; Ziko et al., 2020). Transduc- tive methods yield substantially better performance than their inductive counterparts by leveraging the statistics of the query set (Dhillon et al., 2019). However, this approach has not yet been explored in the context of textual data.
3.2 Proposed Transductive Method
NLP few-shot classifiess rely only on inductive inference, while computer vision has shown signifi- cant performance improvements using transductive inference for FSL. Transductive inference succeeds in FSL because it jointly classifies all unlabeled query samples of single task, leading to more efficient and accurate classification compared to inductive methods that classify one sample at a time. Let begin by introducing some basic no- tation and definitions before introducing our new transductive loss based on the Fisher-Rao distance. In the API-based few-shot classification set- ting, our goal is to train a classification head Io 2 + K that maps the feature representa- tions to the posterior distribution space for mak- ing predictions. To simplify the equations for the rest of the paper, we use the following no- tations for the posterior predictions of each E Is U IQ and for the class marginals within Q: Pa: - go( = P(Y - k|X - x 0 Φ) and Σ Eicoo Pik P(Ya = k; 0, Φ) where
3 API-based Few-shot Learning
3.1 Problem Statement
Let be the considered vocabulary, we denote I2* its Kleene closure. The Kleene closure corresponds to sequences of arbitrary size written with tokens in 1 Given an input space X' with X C 22 and a latent space Z, we consider a pre- trained backbone model ʃe X' + 2 Rd, where A E θ represents the parameters of the encoder and d is the embedding dimension size. the bas-ba setting, we assume that we are unable to access the exact structure of fo as mentioned in R1. However, we do have access to the last encoder embedding which is available for our use (see R1).
Ω, Le.. Ω- U
Dr = 인 X and are the r.v.s associated with the raw fea- tures and labels, respectively. and where YQ means restriction of the r.v. Y to set Q.
For training the classification head in the trans- ductive setting, prior rescarch aims at finding ø such that @ = arg min CE - À x RQ², with CE:- [sT Eiers Σk Vik log(pæ) being the cross-entropy supervision on the support set (in which yeu is the :tth coordinate of the one-hot en-
The objective of few-shot classification is to learn a classifier from limited labeled data and gen- eralize to new, unseen tasks or classes. To accom- plish this, randomly sampled few-shot tasks are Nestt created from a test dataset D Diest := { y:) Migtt
' is set to in all the experiments.

---

coded label vector associated to sample i) and RQ being a transductive loss on the query set Q.
the Fisher-Rao distance between soft-predictions
q == (gi..... ... 9K) and p == (Pi,... (PI...., ,K), which is given by (Picot et al.. 2023):
Note that this transductive regularization has been proposed in the literature based on the Info-
Max principle (Cardoso, 1997; Linsker, 1988), and the inductive loss can be found by setting À = 0. In what follows, we review the regularizers introduced in previous work.
(q, p) == 2 arrcos
Σ
Vqk x Pk
(3)
The proposed transductive regularizer denoted by RER, for each single few-shot task, can be de- scribed as measuring the Fisher-Ras distance be- tween pairs of query samples:
Entropic Minimization (HI) An effective regu- larizer for transductive FSL can be derived from the field of semi-supervised learning, drawing inspira- tion from the approach introduced in (Grandvalet and Bengio, 2004). This regularizer, proposed in (Dhillon et al., 2019), utilizes the conditional Shan- non Entropy (Cover, 1999) of forecast results from query samples during testing to enhance model generalization. Formally:
RQ
log
Pik X Pyk (4)
sEIQ
j jEIQ
diR (pi Pj)
loog
Cos
(5)
Q
ICL
jelq
K Pak log(P:))
R
(1)
where dFR (P; Pj) is the Fisher-Rao distance be- tween pairs of soft-prediction (pi, Pý). Further- more, it is shown that expression (4) yields sur- rogate of the Mutual Information as shown by the following proposition. This result the best of our knowledge is new, as far as we can tell.
Q
SEL
Mutual Information Maximization (I) A promis- ing alternative entropic minimization for address- ing the challenges of transductive FSL to adopt the Info-max principle. (Boudiaf et al., 2020) ex- tended this idea, introduced (Hu et al., 2017), and propose as regularizer a surrogate of the mutual- information Ra)(a)
Theorem 1. (Fisher-Ree Rao as surrogate to maxi- mize Mutual Information) Let (qi) be a collec- tion of soft predictions corresponding the query samples. Then, it holds that V S a S.
K
RQ + log IQ1 RQ(11 S RQ(a),
(6)
Σ Pu log P& + a [Q]
Σ iETQ 1=1
Σ Pa log(p:)) (2)
Proof: Further details are relegated to Ap. A. Advantage of R over R²(aa): Similarly R (a). RER can be exploited to maximize the Mu- tual Information. However, RFR is parameter-free and thus, does not require tuning a.
Limitation of existing strategies: Despite its effectiveness, the previous method has a few lim- itations that should be taken into account. One of these limitations is the need to fine-tune the weight of different entropies using the hyperpa- rameter This parameter-tuning process can be time-consuming and may require extensive experi- mentation achieve optimal results. Additionally, recent studies have shown that relying solely on the first Entropic term, which corresponds to the Entropic minimization scenario in Equation 1, can lead to suboptimal performance in FSL.
3.4 Additional Few-shot Inductive Baseline
In addition to the transductive methods of Sec. 3.2, will explore three additional inductive methods for few-shot classification: prototypical networks, linear probing, and semi-supervised classifier.
Prototypical Networks (PT) PT learn a met- ric space where the distance between two points corresponds to their degree of similarity. During inference, the distance between the query example and each class prototype is computed, and the pre- dicted label is the class with the closest prototype. PT has been widely used in NL. and is considered as a strong baseline (Snell et al., 2017; Sun et al., 201 9b; Gao et al., 2019).
3.3 A Fisher- Rao Based Regularizer
In the FSL scenario, minimizing parameter tuning is crucial. Motivated by this, in this section, we introduce new parameter-fre transductive regu- larizer that fits into the InfoMax framework. Addi- tionally, our loss inherits the attractive properties of

---

Table I: Datasets Statistics.
pretrained using the closed task (Taylor, 1953). We consider two different sizes of the RoBERTa model, namely RoBERTa (B) with 124M parameters and RoBERTa (L) with 355M parameters and Distil- RoBERTa, a lighter version of RoBERTa trained through distillation process (Hinton et al., 2015); for a total of 82M parameters.
Dataset
Classes (K)
Tweet
20
Emotion
25
Amazon
30
B77
77
Clinc
151
Linear Probing (CE) Fine-tuning a linear head on top of a pretrained model is a popular approach to learn a classifier for classification tasks and was originally proposed in (Devlin et al., 2018).
2. Three sentence- centence-rrnsformmess encoder (Reimers and Gurevych, 2019). Following (Muennighoff et al., 2022), we consider MPNET-base (Song et al., 2020), MiniLM (Wang et al., 2020), and Albert Small V2 (Lan et al., 2019).
Semi-supervised Baselines (SSL). We addition- ally propose two seme-super baselines follow- ing steps. In the first step, classifier is trained using the support S and used to label Q. In the second step, the final classifier is trained on both S and Q with the pseudo label obtained from the first step.
3. Multilingual models. To address realistic multi- lingual scenarios, we rely on three sizes of XLM- RoBERTa (Conneau et al., 2020, 2019): base (B), large (L) and XL (XL).
4. text daxinna model: to mimic the typical set- ting of API-based models, we also conduct experi- ments on text-davinci, only accessible through OpenAT's API.
4
An Enhanced Experimental Setting
4.1 Datasets
Benchmarking the performance of FSL methods on diverse sets of datasets is critical to evaluate their generalization capabilities in a robust manner as well as their potential for real-world applica- tions. Previous work on FSL (Karimi Mahabadi et al., 2022; Perez et al., 2021) mainly focuses on datasets with a reduced number of classes (i.e.. K < 3). Motivated by practical considerations we choose to build datasets with larger number of classes. Specif- ically, we choose Go Emotion (Demszky et al., 2020), Tweet Eval (Barbicri et al., 2020), Clinc (Larson et al., 2019), Banking (Casanueva et al., 2020) and the Multilingual Amazon Reviews Cor- pus (Keung et al., 2020). These datasets cover a wide range of text classification scenarios and are of various difficulty*. A summary of the datasets used can be found in Tab. 1.
4.3 Evaluation Framework
Prior research in textual FSL typically involves sampling low number of tasks, typically less than 10, of each dataset. In contrast, utilize an episodic learning framework that generates a large number of N-shots K-ways tasks. This frame- work has gained popularity through inductive meta- learning approaches, such as those proposed by (Finn et al., 2017b; Snell et al., 2017; Vinyals et al., 2016; Sung et al., 201 8a; Mishra et al., 2017; Rusu et al., 2019; Oreshkin et al., 2018), it mimics the few-shot environment during evaluation and improves model robustness and generalization. In this context, episodic training implies that differ- ent model is initialized for each generated few-shot task, and all tasks are compiled independently in parallel. This approach allows to the computation of more reliable performance statistics by evaluat- ing the generalization capabilities of each method more diverse set of tasks. To account for the model's generalization ability, average the sults for each dataset over 1000 episodes, with the N considered classes varying in every episode. For each experiment, we consider the F1-Score.
benchmark composed of
4.2 Model Choice
The selection of an appropriate backbone model is a critical factor in achieving high performance in few-shot NLP tasks. To ensure the validity and robustness of our findings, we have included a di- verse range of transformer-based backbone models in our study, including
5 Experiments
1. Three different sizes of RoBERTa based models (Liu et al., 2019b). Similar to BERT, RoBERTa is
5.1 Case Study of text-davinci
In this experiment, we investigate the performance of text-davinci in both its language model and
Datascts are available in Dataset (Lhoest et al., 2021)

---

Table 2: Aggregated performance over K, N, the dif- ferent datasets for text-davinci |r| stands for the averaged imput length.
Table 3: Aggregated performance over K,N, the differ- datasets and considered backbone.
N-shots
10
5
N-shots
10
5
x
K- K-ways
10
5
10
5
K-ways
10
10
5
FR
69.83
77.46
66.70
75.03
14.2
FR
52.09
61.99
48.71
56.55
Н
10.00
20.00
10.01
20.04
14.2
50.07
59.17
46.42
55.74
68.38
75.82
65.15
73.06
14.2
H
15.07
27.39
15.33
25.84
CE
68.21
75.47
64.92
72.70
14.2
CE
48.31
56.87
45.27
53.94
67.95
75.41
64.60
72.50
14.2
SSL
50.39
58.78
47.33
55.85
SSL
68.27
75.55
64.99
72.75
14.2
ICL
68.9
76.24
65.2
74.3
900
PT
47.29
56.05
44.32
53.20
embedding- model forms. We assess its clas- sification capabilities using the aforementioned baseline and explore the language model's s perfor- mance when applied in an in-context learning (ICL) setup with prompting.
Cline
08.92.3
25
30
35
40
45
50
55
60
65
70
75
80
85
90
95
Banking
70.
77.2
Takeaways. From Tab. 2, we observe that SSL performs comparably CE, which simpler to and will be considered as the baseline in the next part of our study. Although ICL slightly out- performs CE, its implementation comes at a sig- nificant cost. In ICL, each class requires N shots, forcing the user to send long input query with additional instructions. This query length becomes prohibitive the number of classes increases, and average, it is 58 times longer than using the em- bedding base API in our benchmark. The lengthy input and ICL approach make time-consuming for generation (violating R1), require the user to provide labels (violating R2), and prevent the reuse embeddings for future use (e.g., retrieval, cluster- ing). Additionally, ICL is 60 times more expensive than CE. Thus, we will discard ICL for the subse- quent part of this study.
Emotions Tweet
24.9
13
20
25
30
35
Figure 2: Performance on the monolingual datasets.
focus the mono-lingual datasets. Fig. 2 reveals that the global trends observed Tab. remain consistent across datasets of varying difficulty lev- els. Notably, we observed consistent improvements achieved by transductive regularizers (such as or FR) over CE. However, the relative improvement is highly dependent on the specific dataset being eval- uated. Specifically, FR achieves +6.5% Fl-score Banking, but only shy +1.5% on Tweet. A strong baseline generally suggests highly discrimi- native features for the task, and therefore a strong upside leveraging additional unlabeled features, and vice versa. Therefore, we hypothesize that the potential gains to be obtained through transduction correlate with the baseline's performance."
5.2 Overall Results
Global results: To evaluate the effectiveness of various few-shot methods, we conducted a com- prehensive analysis of their classification perfor- mance across all datasets, all backbones, and all considered N-shots/K -ways scenarios. Results reported in Tab. 3. An interesting observation is that transductive approaches and FR outperform their inductive counterparts (CE and PT). Notably, we found that vanilla entropy minimization, which solely relies on H, consistently underperforms in all considered scenarios. Our analysis revealed that FR surpasses traditional fine-tuning based on cross-entropy by margin of 3.7%.
5.3 Study Under Different Data-Regime
In this experiment, we investigated the performance of different loss functions under varying condi- tions of 'ways' and 'shots'. As shown in Fig. 3, we observed that increasing the number of classes (ways') ) led to decrease in Fl while increasing the number of examples per class ('shots') led an improvement in Fl. This can be explained by
Mono-lingual experiment: In order to thor- oughly analyze the performance of each method, we conducted dataset study, beginning with a
Additional multilingual results (Lee, on es, de, fr) can he found on Sec. B.3. They exhibit the same behavior.

---

Temet
8
21
26
1
8
N100100
N/G
Mroki0
NS-433
N5
N10-55
MIC-100
NS NE-010
N10
NG-B
NIC-010
NTD-XS
NS-66
N0-610
WIONG
NGHS
MI0-10
NS-61
N10
NG-66
Task --Shot- K-Way
Task- N-Shol- -ayay
Figure 3: The effect of ways and shots on test performance on monolingual (left) and multilingual (right) datasets.
Table 4: Training time for episode on MI-CPU.
05
Loss
CPU Time
77 0
CE
0.45s
67
FR
0.83s
62
H
0.75s
8 69
522
I
0.83s
PT
0.01s
Figure 4: Impact of model size.
SSL
0.80s
the fact that having more data enables the classifier to better discern the characteristics of each class. Interestingly, the relationship between the num- ber of shots and classification Fl may not be the same for all classes or all loss functions. Fig. shows that different loss functions (e.g. FR on banking) henefited greatly from adding a few shots, while others did not show as much improvement. However, this variability is dependent on the spe- cific dataset and language being used, as different classes may have different levels of complexity and variability, and some may be inherently easier or harder to classify than others.
5.5 Practical Considerations
In this experiment, we adopt practical standpoint and aim to evaluate the effectiveness of an API model, specifically text-davinci. Tab. 4, we report the training speed of one episode on a MAC with CPU. Overall, we observed that the transduc- tive loss is slower as it necessitates the computation of the loss the query set, whereas PT is faster it does not involve any optimization. Furthermore, note that FR is comparable in speed to I. To provide a better understanding of these results, we can compare our method with existing approaches (in the light of R2). For instance, PET (Schick and Schütze, 2020a) entails training time of 20 min- utes on A 100, while ADAPET (Tam et al., 2021) necessitates 10 minutes on the same hardware.
5.4 Ablation Study On Backbones
In this experiment, we examined how different loss functions perform when increasing the number of parameters in various models. The results, pre- sented in Fig. 4. show the average performance across the experiments and are organized by the loss function. We observed inverse scaling law for both the RoBERTa and XLM-RoBERTa fam-- ily of models, where increasing the number pa- rameters led to a decrease in performance for the losses tested. However, within the same family, we observe that the superiority of FR remains consis- tent. An interesting finding from Fig. 4 is that the transductive regularization technique using FR out- performs other methods on text text-davinci. This highlights the effectiveness of FR in improving the performance of the model and suggests that transductive regularization may be a promising ap- proach for optimizing language models.
6 Conclusions
This paper presents a novel FSL framework that utilizes API models while meeting critical con- straints of real-world applications (i.c., RI, R2, R3). This approach is particularly appealing as it shifts the computational requirements (R2), elimi- nating the need for heavy computations for the user and reducing the cost of embedding. To provide better understanding, embedding over 400k se- quences cost as low as 7 dollars. In this scenario, our research highlights the potential of transductive losses, which have previously been disregarded by the NLP community. A candidate loss is the Fisher- Rao distance which is parameter-free and could serve simple bascline in the future.

---

7 Limitations
References
We are optimistic that our research will have a pos- itive impact on society, Nonetheless, it is essential to acknowledge the limitations of API-based few- shot classification models despite their promising results in various tasks, Firstly, the performance of the introduced methods is heavily dependent on the quality of available API models. the API mod- els do not provide sufficient information or lack diversity, the introduced methods may struggle to accurately classify input texts. Secondly, the black- box nature of the backbone limits the interpretabil- ity of API-based few-shot classification methods, which may hinder their adoption. Ultimately, the aim of this work is to establish baseline for future research transductive inference. As result, all existing transductive methods are compared in
Antreas Antoniou, Harrison Edwards, and Amos Storkey, 2018. How to train your maml. arXiv preprint arxi::1810.09502.
Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa-Anke, and Leonardo Neves. 2020. TweetE- val:Unified Benchmark and Comparative Evaluation Tweet Classification. Proceedings of Findings of EMNLP.
Malik Boudiaf, Hoel Kervadec, Ziko Imtiaz Masud. Pablo Piantanida, Ismail Ben Ayed, and Jose Dolz. 2021. Few-shot segmentation without meta-learning: A good transductive inference is all you need? In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13979-13988.
Malik Boudiaf, Imtiaz Ziko, Jérôme Rony, José Dolz. Pablo Piantanida, and Ismail Ben Ayed. 2020. In- formation maximization for few-shot learning. Ad- vances in Neural Information Processing Systems, 33:2445-2453 2457.
this study.
Acknowledgements
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances neural information processing systems, 13:1877-1901.
This work was performed using HPC resources from GENCI-IDRIS (Grants 2022- AD01101838,
2023-103256 and
J-F Cardoso. 1997. Infomax and maximum likelihood for blind source separation. JEFF Signal processing letters, 4(4):112-114
lñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. In Pro- ceedings of the 2nd Workshop on Natural Language Processing for Conversational Al.
Emile Chapuis, Pierre Colombo, Matteo Manica. Matthieu Labeau, and Chloé Clavel. 2020. Hier- archical pre-training for sequence labelling spoken dialog. In Findings of the Association for Computa- tional Linguistics: EMNLP 2020, pages 2636-2648, Online. Association for Computational Linguistics.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021a. Evaluating large lan- guage models trained on code. arXiv preprint arXiv:2107.0337. 07.377744
Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. 2021b. Meta-learning via language model in-context tuning. arXiv preprint 11x0v.2110.07814.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi. Sasha Tsvyashchenko, Joshua Maynez, Abhishek

---

Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengeheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivami Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz. Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways.
11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 6539-6550, Online. Association for Computational Linguistics.
Pierre Colombo, Guillaume Staerman, Chloé Clavel, and Pablo Piantanida. 202 Automatic text eval- uation through the lens of Wasserstein barycenters. In Proceedings of the 2021 Conference Empiri- cal Methods in Natural Language Processing, pages 19450-104660 Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Pierre Colombo, Guillaume Staerman, Nathan Noiry, and Pablo Piantanida. 2022d. Learning disentangled textual representations via statistical measures of sim- ilarity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Vol- ume 1: Long Papers), pages 2614-2630, Dublin, Ireland, Association for Computational Linguisties.
Pierre Colombo, Emile Chapuis, Matthieu Labeau, and Chloé Clavel. 2021a. Code-switched inspired losses for spoken dialog representations. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 8320-8337 Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Pierre Colombo. Wojeiech Witon, Ashutosh Modi. James Kennedy, and Mubbasir Kapadia. 2019. Affect-driven dialog generation. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3734-377 9734-3743, Minneapolis, Min- nesota. Association for Computational Linguistics.
Pierre Colombo, Emile Chapuis, Matthieu Labeau, and Chloé Clavel. 2021b. Improving multimodal fusion via mutual dependency maximisation. In Proceed- ings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 231-245, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
European Commission. 2016. Regulation (eu) 2016/679 of the european parliament and of the council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing directive 95/46/ec (general data protection regulation).
Pierre Colombo, Chloé Clavel, Chouchang Yack. and Giovanna Varni. 2021c. Beam search with bidirec- tional strategies for neural response generation. In Proceedings of the 4th International Conference on Natural Language and Speech Processing (ICNLSP 2021), pages 139-146, Trento, Italy. Association for Computational Linguistics.
European Commission. 2020. Proposal for a regula- tion of the european parliament and of the council european data governance (data governance act). COM((020) 767 final.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint
Pierre Colombo, Eduardo Dadalto, Guillaume Staerman, Nathan Noiry, and Pablo Piantanida. 2022a. Beyond mahalanobis distance for textual ood detection. Ad- vances in Neural Information Processing Systems, 35:17744-1777 17759.
Pierre Colombo, Nathan Noiry, Ekhine Irurozki, and Stéphan Clémençon. 2022b. What are the best sys- tems? new perspectives on nlp benchmarking. Ad- vances in Neural Information Processing Systems. 35:26915 22.222...6...
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross- cross-lingus lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics.
Pierre Colombo, Maxime Peyrard, Nathan Noiry, Robert West, and Pablo Piantanida. 2022c. The glass ceiling of automatic evaluation in natural language generation. arXiv preprint arXiv:2208. 14585.
Thomas M Cover. 1999. Elements of information theory. John Wiley & Sons.
Pierre Colombo, Pablo Piantanida, and Chloé Clavel. 202 ld. A novel estimator of mutual information for learning disentangle textual representations. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the
Ganqu Cui, Shengding Hu, Ning Ding, Longtao Huang, and Zhiyuan Liu, 2022. Prototypical verbalizer for prompt- few-shot tuning. arXiv preprint arXiv:2203.09770

---

Maxime Darrin, Pablo Piantanida, and Pierre Colombo. 2022. Rainproof: An umbrella to shield text gener- ators from ouf-o--distribution data. arXiv preprint arXiv:2212.09171.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. ArXiv.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. 2021. Meta-learning in neural networks: A survey. IEEE transactions patlern analysis and machine intelligence, 44(9):514-5519)
Maxime Darrin, Guillaume Staerman, Eduardo Dadalto Câmara Gomes, Jackie CK Cheung, Pablo Piantanida, and Pierre Colombo. 2023. Unsuper- vised layer-wise score aggregation for textual ood detection. arXiv preprint arXiv:2302.09852.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficien transfer learning for nip. In- ternational Conference on Machine Learning, pages 2790-2799, PMLR.
Dorottya Demszky, Dana Movshovitz-Attice Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. GoEmotions: A Dataset of Fine- Grained Emo- tions. In 58th Annual Meeting of the Association for Computational Linguistics (ACL).
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint 1810::0800000055
Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Juanzi Li, and Maosong Sun. 2021a. Knowl- edgeable prompt-tuning: Incorporating knowledge into prompt verbalizer for text classification. arXiv preprint SkKiv:2108.0033.
Guneet S Dhillon, Pratik Chaudhari, Avinash Ravichan- dran, and Stefano Soatto. 2019. A baseline for few-shot image classification. arXiv preprint arXiv: 790..029.9..
Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Mat- sumoto, and Masashi Sugiyama. 2017. Learning discrete representations via information maximizing self-augmented training. In Proceedings of the 34th International Conference on Machine Learning Vol- ume 70.
Ning Ding. Shengding Hu, Weilin Zhao, Yulin Chen, Zhiyuan Liu, Hai-Tao Zheng, and Maosong Sun. 2021. Openprompt: An open-source framework for prompt-learning. arXiv preprint 1rXiv:2111.09900
Yuqing Hu, Vincent Gripon, and Stéphane Pateux. 2021b. Leveraging the feature distribution in transfer- based few-shot learning. In Artificial Neural Net- works and Machine Learning-ICAN 2021: 30th International Conference on Artificial Neural Net- works, Bratislava, Slovakia, September 14-17, 2021, Proceedings, Part II 30, pages 487-499. Springer.
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma. Ahmed EI- Ei-Kish Siddharth Goyal, Mandeep Baines, Onur Celebi. Guillaume Wenzek, Vishrav Chaudhary, et al. 2021. Beyond english-centric multi- lingual machine translation. The Journal of Machine Learning Research, 22(1) (11:))339944))))
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford. Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.0830
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017a. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on ma- chine learning, pages 1126-113 -135. PMLR.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017b. Model-agnostic meta-learning for fast adaptation of deep networks. Proceedings of the 34th Interna- lional Conference on Machine Learning.
Rabech Karimi Mahabadi, Luke Zettlemoyer, James Henderson, Marzieh Saeidi, Lambert Mathias, Veselin Stoyano, and Majid Yazdani. 2022. Perfect: Prompt- freet--feeedeee and efficient few-shot learning with lan- guage models. In Annual Meeting of the Association for Computational Linguistics.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2020. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.1572.
Phillip Keung, Yichao Lu, György Szarvas, and Noah A. Smith. 2020. The multilingual Amazon reviews cor- pus. In Proceedings the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Tianyu Gao. Xu Han, Zhiyuan Liu, and Maosong Sun. 2019. Hybrid iieention-bssed prototypical networks for noisy seot relation classification. In Proceed- ings of the AAAI conference on artificial intelligence. volume 33, pages 6407-6414.
Yves Grandvalet and Yoshua Bengio. 2004. Semi- supervised learning by entropy minimization. Ad- vances in neural information processing systems, 17.
Jan Kocoñ, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydio, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz. Anna Kocoñ. Barttomiej Koptyra, Wik- toria Mieleszezenko BaadeszzzeenkooKowszzewzzz Piotr Mitkowski. Marcin Oleksy, Maciej Piasecki, Fukasz Radliáski, Konrad Wojtasik, Stanislaw Wožniak, and Prze- myslaw Kazienko. 2023. Chatgpt: Jack of all trades. master of none.
Anas Himmi, Ekhine Irurozki, Nathan Noiry, Stephan Clemencon, and Pierre Colombo. 2023. To- wards more robust nip system evaluation: Han- dling missing scores benchmarks. arXiv preprint arXiv:2305. 10284.

---

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learn- ing of language representations. arXiv preprint arXiv:/909.11 17942.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint
Stefan Larson, Anish Mahendran, Joseph J. Peper. Christopher Clarke. Andrew Lee. Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of- scope prediction. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Rohert L Logan IV, Ivana Balažević, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2021. Cutting down prompts and parameters: Simple few-shot learning with language models. arXiv preprint 106. 13353.
Rabech Karimi Mahabadi, Luke Zeullemoye, James Henderson, Marzieh Saeidi, Lambert Mathias, Veselin Stoyanov, and Majid Yazdani. 2022. Per- feet: Prompt- free and efficient few-shot learning with language models. arXiv preprint vrxiv:2204.0117
K Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatt.. 2019. Meta-learning with dif- ferentiable convex optimization. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10657-10665.
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Han- nanch Hajishirzi. 2021. Metaicl: Leamning to learn in context. arXiv preprint arXiv:21 15943..
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstra- tions: What makes in-context learning work? arXiv preprint arXiv:2202. 12837.
Eric Lehman, Evan Hernandez, Diwakar Mahajan, Jonas Wulff, Micah J. Smith. Zachary Ziegler, Daniel Nadler, Peter Szolovits, Alistair Johnson, and Emily Alsentzer. 2023. Do we still need clinical language models?
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and P. Abbecl. 2017. A simple neural attentive meta- learner. In International Conference on Learning Representations.
Brian Lester, Rami A1-Rfou, and Noah Constant. 2021. The power of scale for parameter-effiiien prompt tuning. arXiv preprint arXiv:2104.0869/
Quentin Lhoest, Albert Villanova del Moral, Yacine Jemite,, Abhishek Thakur, Patrick von Platen, Suraj Patil. Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, et al. 2021. Datasets: commu- nity library for natural language processing. arXiv preprint Xiv: 200.21100000 02846..
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2022. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316
OpenAI. 2023. Gpt-4 technical report.
Boris N. Oreshkin, Pau Rodriguez, and Alexandre La- coste. 2018 Tadam:
Moshe Lichtenstein, Prasanna Sattigeri, Rogerio Feris, Raja Giryes, and Leonid Karlinsky. 2020. Tafssl: Task-adaptive feature sub-space learning for few- shot classification. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VII, pages 522-539. Springer.
Task dependent adaptive metric for improved few-shot learning. In Proceedings of the 32nd International Conference on Neural Infor- mation Processing Systems, page 719-729.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models. Ad- vances in neural information processing systems. 10000
Ralph Linsker. 1988. Self-organization in perceptual network. Computer, 133
Jonas Pfeifferr Andreas Rücklé Clifton Poth. Aishwarya Kamath, Ivan Vulić, Sehastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020. Adapterhub: A framework for adapting transformers. arXiv preprint arXiv:2007.07779.
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mo- hta, Tenghao Huang. Mohit Bansal, and Colin Raffel. 2022. Few-shot parametor-effiiient fine-tuming is better and cheaper than in-context learning. arXiv preprint arXiv:2255.05:.
Georg Pichler, Pierre Jean A Colombo, Malik Boudiaf, Günther Koliander, and Pablo Piantanida. 2022. A differential entropy estimator for training neural net- works. In International Conference on Machine Learning, pages 17691-17715. PMLR.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods natural language processing. ACM Computing Surveys, 1-35).1-35.
Marine Picot, Francisco Messina, Malik Boudiaf, Fab- rice Labeau, Ismail Ben Ayed, and Pablo Piantanida. 2023. Adversarial robustness via fisher-rao regular- ization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):2698-21 10.
Yanhin Liu, Juho Lee, Minseop Park, Sachoon Kim, Eunho Yang, Sung Ju Hwang, and Yi Yang. 2019a. Learning to propagate labels: Transductive propaga- tion network for few-shot learning. ICLR.

---

Marine Picot, Nathan Noiry, Pablo Piantanida, and Pierre Colombo. 2022a. Adversarial attack detec- tion under realistic constraints.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. 2020. Mpnet: Masked and permuted pre- training for language understanding. Advances in Neural Information Processing Systems, 33:16857- 16867.
Marine Picot, Guillaume Staerman, Federica Granese, Nathan Noiry, Francisco Messina, Pablo Piantanida, and Pierre Colombo. 2022b. simple unsupervised data depth- based method to detect adversarial im- ages.
Yisheng Song. Ting Wang, Puyu Cai, Subrota K Mondal, and Jyoti Prakash Sahoo. 2022. A comprehensive survey of few-shot learning: Evolution, applications. challenges, and opportunities. ACM Computing Sur- veys.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Emma Strubell, Ananya Ganesh, and Andrew McCal- lum. 2019. Energy and policy considerations for deep learning in nip.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-555 5551.
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. 2019a. Meta-transfer learning for few-shot learning. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 403 412.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. 2019. Rapid learning or feature reuse? towards understanding the effectiveness of maml. arXiv preprint 1rXiv:1909009177
Shengli Sun, Qingfeng Sun, Kevin Zhou, and Tengchao Lv. 2019b. Hierarchical attention prototypical net- works for few-shot text classification. In Proceed- ings of the 2019 conference on empirical methods in natural language processing and the 9th interna- tional joint conference on natural language process- ing (EMNI.P-ICNNLP), pages 476-485.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Process- ing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLP).
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip Torr, and Timothy Hospedales. 2018a. Learning to compare:
Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Be- linkow, and Nir Shavit. 2019. A constructive predic- tion of the generalization error across scales. arXiv preprint arXiv: 19t::::999.222...
Relation network for few-shot learning.
Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip HS Torr. and Timothy M Hospedales. 2018b. Learning compare: Relation network for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1199-1208.
Andrei Rusu, Dushyant Rao. Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. 2019. Meta-learning with latent embedding optimization.
Derek Tam, Rakesh R Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. 2021. Improving and simplifying pattern exploiting training. arXiv preprint arXiv:2 11955.
Stephan R Sain. 1996. The nature of statistical learning theory.
Timo Schick and Hinrich Schütze. 2020a. Exploit- ing cloze questions for few shot text classification and natural language inference. arXiv preprint arXiv:2001.0767..
Wilson L Taylor. 1953, "cloze procedure": A new tool for measuring readability. Journalism quarterly, 30(4):415
Timo Schick and Hinrich Schützee 2020b. It's not just size that matters: Small language models are also few-shot learners. arXiv preprint arXiv:2009.071B 18.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint 10X.
Timo Schick and Hinrich Schütze. 2022. True few-
shot learning with prompis-a real-world perspective. Transactions of the Association for Computational Linguistics, 10:716-731.
Vladimir N Vapnik. 1999. An overview of statistical learning theory. IEEE transactions on neural net- works, 10(5))988-99)) 9999.
Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. Ad- vances neural information processing systems, 30.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, ko- ray kavukcuoglu, and Daan Wierstra. 2016. Match- ing networks one shot learning. In Advances Neural Information Processing Systems.
Irene Solaiman. 2023. The gradient of generative ai
release: Methods and considerations. arXiv preprint wrXiiv2302.04844.

---

Wenhui Wang, Furu Wei, Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural In- formation Processing Systems, 33:5776-5788.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten
Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022.
Chain of thought prompting elicits reasoning large language models. arXiv preprint arXiv:220..11903
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for lan- guage understanding. Advances in neural informa- tion processing systems, 32.
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. 2021. Crossfit: A few-shot learning challenge for cross-task generalization in nlp. arXiv preprint arXiv:2104.08835.
Imtiaz Ziko, Jose Dolz, Eric Granger, and Ismail Ben Ayed, 2020. Laplacian regularized few-shot learn- ing. In International conference on machine learning, pages 11660-11670 PMLR.

---

A Proof of Proposition
In this Appendix, we prove the inequality (Eq. 6) provided in Proposition 1. The right-hand side of (Eq. 6) follows straightforwardly from the definition of (a(a) and the non-negativity the Shannon entropy. order to prove the first inequality, we need to introduce the following intermediate result.
For any arbitrary random variable (r.v) X and countable r.v Y, and any real number /3, let
X
8
(x Y) = Ex log Ex
P(Y
where the r.v X* follows the same distribution than X. Notice that it is obvious that IxX; = I(X;))
where I (X; Y) is Shannon Mutual Information.
Lemma 1. For any arbitrary r.v. X and countable E.V. Y, we have
I(X;Y) 2 Ix(X;Y), for S 1 1.
Proof of the lemma: We must show that the different of (X; Is(X; is nonnegative. To this end, we write this difference as:
I(X:Y) (
(7)
pl 9 (XXXX)
E log
(8)
Ex (Y
p XY|X P(Y
2 log x
(9)
Ex (X
Pl 10-88/ (x)(x)))()
log
Σ
Ex. P(ylX")
(10)
Ex (x
yEy
log
(11)
ExP Pd (1) X)
yEy
log
Σ yEy
Ex (
(12)
0,
(13)
where the first inequality follows by applying Jensen's inequality to the function t + - log(t).
Proof of Proposition 1: From Lemma using Jensen's inequality, we have
X
I(X;Y)- E log Ex
(14)
X
X
8
2 x Ex
(15)
X(Y
P(Y
18
2 - Ex log ExExx
(16)
P(
Exx log Ex
(17)
Σ vEY
(18)
where inequality (15) follows by applying Lemma 1 and inequality (16) follows by exploiting the convexity of the function ( - log(() for any 0 < 3 < 1. Finally, it not difficult to check from the definition of the Fisher-Rao distance given by expression (3) that
drR(P (P(y) X P() -()1)=x'1)) x11)
cos
(19)
2
P(y| = x)P(2(X = =x*).
(20)
DEY

---

Table 5: Preliminary experiment results. Accuracy of the different backbone.
Model
Params
Emotion
Twitter
Clinic
Banking
Amazon
en
en
en
en
es
Albert Small V2 (XS)
11M
25.2
18.3
67.0
88,1
33,5
X
X
X
MiniLM (S)
33M
30.2
19.3
67.1
92.3
39.5
X
X
MPNET-base
109M
30.2
22.5
67.4
94.3
41.3
X
DistilRoBERTa (S)
82M
23.3
26.0
68.5
90.9
40.0
X
X
X
RoBERTa (B)
124M
21.0
25.5
66.7
91.4
39.2
X
X
RoBERTa (I.)
355M
15.0
23.0
64.5
90.0
38.1
X
XLM-RoBLKTa (B)
278M
21.0
22.1
66.5
87.0
40.1
19.2
17.5
18.3
XLM-RoBLKIa (L)
559M
14.0
18.0
64.5
86.2
38.2
17.5
15.6
18.1
XLM-RoBERTa (XL)
3,48B
25.4
19.0
68.9
95.0
41.0
18.9
17.9
22.0
text-davinci
175B
38.9
35.3
70.4
98.7
48.4
30.4
34.0
33.5
Using the identity given by (19) in expression (18), and setting , = 1/2, we obtain the following lower bound on T(X; Y):
dFR (FR(P(w/X),
Ex- log Ex cos
2
The incquality (6) immediately follows by replacing the distribution of the r.v. X with the empirical distribution on the query and P(ylar) with the soft-prediction corresponding to the feature I, which concludes the proof of the proposition.
Additional Experimental Results
B.1 Preliminary Classification Results
Preliminary Experiment. In our experiments, the backbone models are of utmost importance. Our objective in this preliminary experiment is to assess the efficacy of these models when fine-tuning only the model head across variety of datasets. Through this evaluation, generalization abilities and any dataset-specific factors that may influence their performance. This information can be utilized analyze the performance of different models the few-shot scenario, described in Sec. 5. We present the results of this experiment in Tab. 5, noting that all classes were considered, which differs from the episodic training approach detailed in Sec. 5.
aim gain insight into their
B.2 A Dive Into text-davinci results
text-davinci appears to be the backbone providing the most informative a priori embeddings in Tab. 5 and could be considered as the prime model for API-based FSL, showcasing the current requirements in this area. It is thus a typical candidate for application uses that must meet the following criteria (R1)- (R3). Therefore,
put special emphasis its related results.
Fig. 6 (top) details the text-davinci results of the experiments conducted on the mono-lingual datasets. These plots highlight the consistency of the tendencies that emerged in Tab. 5, Tab. and Fig. 2, namely: the superiority of transductive approaches (F R and I) over inductive ones (CE and PT), the underperformance of the conveyed by text-davinci learned embeddings over other backbones, resulting in higher F1 scores on all datasets.
strategy (H), and the higher amount of information
These phenomena still occur in the multi-lingual setting, as illustrated in Fig. (bottom), stressing the superiority of transductive (and especially FR) over other approaches for presumably universal tasks, beyond English-centered ones, and without the need for using language-specific engineering as for prompting-based strategies.
Note that for both of these settings, the entropie-minimizztiin-bb strategy (H) seems be capped at a 15% Fl score, thus with no improvement over other backbones embeddings, and independently of the dataset difficulty.

---

1.0
CE
0.8
PT
H
Accuracy
0.6
FR
0.4
0.2
0.0
Tweet
Banking
Emotions
Clinc
Task
Figure The different losses when training on text-davinci embeddings.
6
5.0,
(53.9)
18
23
43
48
26
28.5
13
18
23
28
33
23
28
33
Figure Losses Performance on multilingual datasets.
B.3 Multilingual Experiment
To provide an exhaustive analysis, we report the same experiment that is made in Sec. 5.2, for multi-lingual models on the Amazon dataset. While both Latin languages (French and Spanish) share close results, with an Fl gain of 2.8% for FR over CE, the results in the German and English language exhibit an F1 increased by almost 4%.
B.4 Importance of Model Backbones on Monolingual Experiment
In this section, we report the results our experiment aggregated per backbone. The goal is to understand how the different losses behave on the different backbones. The results are presented in Fig. 10. While the trends observed in the previous charts are retrieved for the majority of backbones, some of these models are exceptions. For example, while transductive methods perform generally better than inductive methods, the CE-based method seems to perform slightly better than for XLM-RoBERTaxxl. Additionally, while FR is the most effective method for the majority of backbones, is surpassed by for the all- distilroberta-vl model. Furthermore, the inverse-scaling-law details are found for the RoBERTa(B/L) and XI.M-RoBERTa (B/L.) models per dataset. In general, it interesting to note that although model performance is constrained by dataset difficulty, the performance order of each method is consistent across all 4 datasets for each considered backbone.

---

0.0
Acturity
02
Tount
Ganking
Emostons
Towel
Baniinng
Emasion
Cater
Towet
Gendiin
Emnsirns
Task
Task
Task
1.0
0.6
SSL
02
Toome
Banking
Endiant
Gioe
Tames
Baesiing
Emeckone
Gioe
Tower
Baniiing
Emistone
Oiter
Task
Task
Task
1.0
0.8
Aocurao
0.8
0.2
0.0
Toom
Banking
Emotane
Cloce
Twmet
Basking
Emnrione
Gloe
Toret
Basking
Envatoon
Clipa
Task
Task
Task
Figure 10: Performance on monolingual datasets.
B.4.1 Results Per Language
In this experiment, we report the performance of different losses on the Amazon dataset by averaging the results over the number of shots, ways, and model backbones. The results are presented in Tab. 6. Our observations indicate that the transductive regularization improves the results for all languages over the inductive baseline (i.e., CE), with substantially higher gain for the German language. Additionally, we note that the observed improvements for FR more consistent. This further demonstrates that the transductive loss can be useful in few-shot NLP. In the future, we would like to explore the application transductive inference to other NLP tasks such as sequence generation (Pichler et al., 2022: Colombo et al., 2019, 202 b0) and classification tasks (Chapuis et al., 2020; Colombo et al., 2022d,b; Himmi et al., 2023) as well as NLG evaluation (Colombo et al., 2021e, 2022c, 2021c,a,b) and Safe Al (Colombo et al., 2022a; Picot et al., 2022a,b; Darrin et al., 2022, 2023).
fr
de
en
es
FR
29.36
33.98
53.89
28.47
27.74
31.41
51.75
26.79
H
15.04
15.13
15.04
15.04
CE
27.15
30.24
50.89
26.21
PT 26.37
29.16
50.34
25.44
SSL
27.20
30.29
50.94
26.26
Table 6: Global Results for multilingual Amazon


Reviews:
[{"review_id": "taJRNIi5cJ", "reviewer": "Reviewer_RYzB", "summary": "", "questions": "Why is it enough to show the Theorem 1 only at one direction - R^{FR} <= R^{I}(\\alpha}? for example I think we can say that 0 <= R^{I}(\\alpha} because of non-negativity of Shanon entropy - but we cant use 0 as regularizer. What do I miss? \n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer rYZB for their careful reading of the manuscript. **We are glad they acknowledge that our contributions may have a strong positive impact on the research field and that he does not see any reason not to accept the paper.**\n\nBelow we answer their question on the Theorem 1: \n\nAs stated by Theorem 1, we have: $R^{FR}_{Q} + \\log |Q| \\leq R_{Q}(1) = I(X_{Q},Y_{Q}) \\leq R_{Q}(\\alpha)$\nThus,  maximizing  $R^{FR}_{Q}$ could be viewed as a proxy for maximizing the mutual information (as we maximize a lower bound on the mutual information).\n\nThe confusion of Reviewer rYZB  may arise from the fact that R_{Q}^I(1) = I(X_{Q},Y_{Q}) and not \\( R_{Q}^I(1) = -I(X_{Q}, Y_{Q}). To enhance readability, we will add a line of clarification in Eq. 2:\n \\hat{H}(Y_{Q}) - \\alpha \\hat{H}(Y_{Q} | X_{Q}) = I_{\\alpha}(Y_{Q},X_{Q}) \nThus,  -R^{FR} \\geq -I_{\\alpha}(Y_{Q},X_{Q}) , and by minimizing -R^{FR}, we minimize an upper bound on the mutual information.\n\n**We thank the reviewer for their improvement suggestions that we will make sure to incorporate in order to enhance the paper's quality. We hope our answers address all concerns of reviewer rYZB, and we hope they would be keen to consider raising their score.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "All the three paper contributions may have a strong positive impact on the research field and should be shared with the NLP community:\n1. The model defined at the paper meets the needs of real life scenarios where strong pre-trained models are executed by third parties and are available only by API and there are strong constraints of sharing private date with that third parties. \n2. The ransductive inference paradigm that was previously overlooked by the NLP community looks like a powerful tool and I believe it can be used not only at framework of this paper.    \n3. the benchmark and the FSL approach presented at this paper can be used to evaluate future works at this field by using the presented FSL approach as a baseline.  ", "reasons_to_reject": "I don't see any reason not to accept this paper", "questions_to_authors": "Why is it enough to show the Theorem 1 only at one direction - R^{FR} <= R^{I}(\\alpha}? for example I think we can say that 0 <= R^{I}(\\alpha} because of non-negativity of Shanon entropy - but we cant use 0 as regularizer. What do I miss? \n", "paper_topic_and_main_contributions": "The paper solves the few-shot classification problem. Its main contributions are:\n1. they defined a model where the embedding of a pre-trained model is calculated as a black-box, via API, without accessing the models internals, and in addition without providing the labels (privacy constraint). \n2. they extended the transductive inference paradigm that was previously used at computer vision field to the NLP.\n3. in addition to the theoretical proof of the feasibility of their approach the team evaluated the new method by running a wide set of experiments and comparing its performance to broad set of alternative methods, on a multi-language benchmark of eight datasets involving multi-class classification with up to 151 classes. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "_", "Justification_for_Ethical_Concerns": "_"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Page 3: can the colors tones be changed in a way that the figure 1 will be readable at grayscale also? it is hard to distinguish between colors at BW printed version.\nPage 4: the usage of footnote at line 346 looks a little bit confusing - since on the first look it looks like a R_{Q} in a third degree.\nPage 7: it will be better to use the same order at tables 2 and 3 \nPage 8: the figure 4 i very hard to read - maybe it worth to move the label to outside of the chart\nPage 14: The line 1125 is broken - the formula overlaps the text of the second column \nPage 15: to change the order of table 6 to be aligned with 2 and 3 \nPage 16: at figure 8 the 'FR' should be in lowercase"}}, {"review_id": "ssmWLnUZkO", "reviewer": "Reviewer_d1BR", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to Reviewer d1BR for their meticulous evaluation of the manuscript. **We appreciate their recognition of the timeliness and relevance of the chosen problem for the community. Additionally, we are pleased that they concur with the novelty of our technical contribution regarding the Fisher Rao method.**\n\n\nBelow we clarify our positioning: \n\nIn our scenario, we work with a limited quantity of locally labeled data (referred as the support set in section 3.1 in the problem statement. The distinguishing feature of our approach lies in its treatment of the target set. Unlike the conventional NLP setting, which predominantly relies on inductive inference, overlooking the distributional information embedded in predictions, our regularizers leverage the distributionnal information within the query set. This allows our method to operate in a transductive manner, making effective use of the (unlabeled) query set.\n\nIt is essential to highlight that (as Reviewer d1BR suggests) our method remains flexible enough to adapt to scenarios where a larger pool of labeled data is accessible. However, this particular aspect goes beyond the immediate scope of this paper and is not studied in the paper which rely on N-shots (5 or 10; please see l318) per class. This is particularly relevant when using APIs. If a substantial volume of labeled training data is available, training a model locally might be a more suitable approach. We will add this suggestion in the future research directions of the paper. \n\n**We hope our answers address all concerns of reviewer d1BR, and we hope they would be keen to consider raising their score.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It is good work tackling a timely problem, with solid evaluation conducted in real-world settings. The proposed transductive learning regularizer using the Fisher-Rao distance is new and has good performance.", "reasons_to_reject": "In the proposed scenario, one has to submit every piece of text to the foundation model in order to obtain the embedding. I don't see why this is related to the Few-Shot Learning setting, because whether there is a label or not, the label is not shared with the foundation model. So if one has a lot of labeled data locally, one can simply perform the normal supervised learning, but using the text embeddings through the foundation model API. Why is the proposed scenario related to Few-Shot Learning?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper formulates a new Few-Shot Learning problem for text classification: In the proposed scenario, one can only access a deployed foundation model through APIs, without seeing the internal states of the model; one submits text sequences to the foundation model and receives embeddings of the text. Then, Few-Shot Learning methods are applied to the text embeddings to perform a text classification task.\n\nThe paper discusses several requirements of the considered scenario, and argues that the proposed solution meets the requirements most in the real world. Further, the proposed solution emphasizes a new transductive learning regularizer using the Fisher-Rao distance.\n\nEvaluation is conducted on real-world applications, as well as in controlled studies. The proposed transductive learning regularizer has exhibited its usefulness in the evaluation.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6M8zLlE7N0", "reviewer": "Reviewer_CwfZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to Reviewer CwfZ for their reading of the manuscript. **We are glad that they find the paper clear and find the novel problem and experimental results promising.**\n\nBelow we answer the question on the Fisher Rao regularizer:\n\n**The intuition behind using the pairwise Fisher-Rao distance as a  transductive regularizer**\n\nWe really appreciate this comment and we totally agree that giving additional intuitive arguments as to why the transductive Fisher-Rao (FR) loss in Eq. (4) works is important for readers. We assumed that Theorem 1, which shows that the FR loss is a proxy for maximizing the mutual information in Eq. (2), is enough. However, we realize that we did not provide intuition as to why maximizing the mutual information in Eq. (2) is relevant (assuming that this is known in computer vision, but we agree with the reviewer that it needs clarification for an NLP audience). In the following, we provide (1) intuitive arguments for maximizing the mutual information in Eq. (2); and (2) direct intuitive arguments, beyond Theroem 1, based on the pairwise distances in the FR loss in Eq. (4), and how it encourages prediction confidence and class balance (similar to maximizing the mutual information). This will also help clarify the advantage of FR over the mutual information (i.e., the adaptive trade-off between prediction confidence and class balance in FR).  We will add these explanations to the paper:\n\n**1- Intuitive arguments for maximizing the mutual information:** Minimizing the conditional entropy (second term in the mutual info in Eq. 2)  encourages the model to output confident predictions close to the vertices of the simplex (i.e. close to one-hot encoding vectors). This conditional-entropy loss is a clustering objective as it models the cluster assumption: The classifier’s boundaries should not occur at dense regions of the unlabeled features . The marginal-entropy loss (the first term in the mutual information in Eq. 2) encourages the distribution of labels to be uniform, thereby avoiding degenerate solutions assigning all samples to one single class. \n\n**2-  Direct intuition behind minimizing the pairwise FR distances in Eq. 4 (without Th1 linking to the mutual info):** From Eq. (4), one could see that maximizing FR minimizes the pairwise dot products between  $\\sqrt{{\\mathbf p}_i}$ and $\\sqrt{{\\mathbf p}_j}$ for probability simplex predictions ${\\mathbf p}_j$ and ${\\mathbf p}_j$. This dot product is the Bhattcharryya coefficient between ${\\mathbf p}_i$ and ${\\mathbf p}_j$, and it takes values in $[0, 1]$: 1 when ${\\mathbf p}_j$ = ${\\mathbf p}_j$  and 0 when simplex vectors ${\\mathbf p}_j$ and ${\\mathbf p}_j$ are orthogonal. Therefore, maximizing FR encourages sparsity of probability predictions ${\\mathbf p}_j$ and hence confidence, pushing them towards the vertices of the simplex (similarly to the conditional entropy term $R_Q^H$ in the mutual info in Eq. 2). Intuitively, unconfident predictions in the middle of the simplex (i.e. close to the uniform distribution) lead to high pairwise dot products (close to 1), which decrease the FR loss in Eq. (4) and hence correspond to undesirable solutions. However, unlike conditional entropy $R_Q^H$, the FR loss also penalizes degenerate, extremely imbalanced solutions that assign all points to a single class. Indeed, even with confident predictions, assigning all points to a single class yields high pairwise dot products and, therefore, decreases the FR loss. In summary, the FR loss has it all, both prediction confidence (cluster assumption) and class balance in one single loss, with the trade-off between them being adaptive (more explanation on this below). This is an important advantage over maximizing the mutual information with fixed $\\alpha$.                              \n\n\n\n**Why use $R_Q^FR$ instead of $R_Q^I(1)$, which is a non-parametric, tighter bound?**\n\nWe tried to set $\\alpha=1$ in our experiment, i.e., using the Mutual Information $R_Q^I(1)$, but it yields suboptimal results, which are lower than those we obtained with the $R_Q^I(0.1)$ loss for example (similar results are reported in Boudiaf et al. 2020). Indeed, fixing the confidence/class balance tradeoff parameter $\\lambda$ during gradient-based optimization of the mutual info may yield degenerate solutions (e.g. extremely imbalanced solutions). Assume, for instance, that the current solution is confident but extremely imbalanced. Then, if $\\lambda$ is not small enough, the conditional entropy term (second term in the mutual info in Eq. 2) may act as a strong barrier, preventing moving away from such a degenerate solution. On the contrary, in FR in Eq. (4), when the predictions are close to the vertices of the simplex (i.e. confident predictions close to one-hot encoding vectors), one could see that FR in Eq. (4) is approximately the marginal entropy term (first term in the mutual info in Eq. 1), up to a constant. Intuitively, this is akin to adaptively choosing a very small $\\lambda$ in the mutual information to penalize the trivial one-class solution. This adaptive confidence/class-balance tradeoff is the main advantage of FR over the mutual information in Eq. (2). \n\nFollowing the reviewer's comments, we will make sure to include more details in the future version of the manuscript.\n\n**We hope our answers address all concerns of reviewer CwfZ, and we hope they would be keen to consider raising their score.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is generally written clearly.\n\nThe proposed setting is reasonable and the experiment results seem promising.", "reasons_to_reject": "The technique novelty of the paper concentrates in Section 3.3, i.e., the Fisher-Rao regularizer, which is insufficiently explained. Especially, what is the intuition that we have to compute the Fisher-Rao distance between each pair of the query instance and how it can work as a regularizer? I think these are missing points that are important for readers to trust the experimental results.\n\nIn the theorem, both the proposed $R_Q^{FR}$ and $R_Q^{I}(1)$ can lower-bounds $R_Q^{I}(\\alpha)$. And, it seems from (6) that the $R_Q^{I}(1)$ is a tighter lower-bound. Then, why we want to use $R_Q^{FR}$ instead of $R_Q^{I}(\\alpha)$, which is also non-parametric?", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper propose an API based setting for few-shot learning, where the encoder is concealed in a black-box API and a classifier is trained on the API output embeddings. Further, it considers the transductive setting that assumes the query data being available during training.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper discusses the problem of few-shot classification for NLP by using black-box APIs. The setting is very relevant to the current state, where many foundational models are served by companies only through APIs without sharing their internals representations or model architecture details. The authors proposes to use a Transductive Learning approach through a Fisher-Rao regularizer.
The reviewers agree that the paper is timely with respect to the current state of the art. The reviewers agree that the proposed approach is interesting, the experiments are showing the efficacy of the appraoch and that in general  the paper can provide a positive impact to the NLP community.
Decision: Accept-Main