Revisiting Logistic-softmax Likelihood in Bayesian Meta-Learning for Few-Shot Classification
Authors: Tianjun Ke
Haoqun Cao
Zenan Ling
Feng Zhou
Venue: NeurIPS 2023 poster (2023)
Keywords: Few-shot learning
Gaussian processes
Conditional conjugate

Abstract: Meta-learning has demonstrated promising results in few-shot classification (FSC) by learning to solve new problems using prior knowledge. Bayesian methods are effective at characterizing uncertainty in FSC, which is crucial in high-risk fields. In this context, the logistic-softmax likelihood is often employed as an alternative to the softmax likelihood in multi-class Gaussian process classification due to its conditional conjugacy property. However, the theoretical property of logistic-softmax is not clear and previous research indicated that the inherent uncertainty of logistic-softmax leads to suboptimal performance. To mitigate these issues, we revisit and redesign the logistic-softmax likelihood, which enables control of the \textit{a priori} confidence level through a temperature parameter. Furthermore, we theoretically and empirically show that softmax can be viewed as a special case of logistic-softmax and logistic-softmax induces a larger family of data distribution than softmax. Utilizing modified logistic-softmax, we integrate the data augmentation technique into the deep kernel based Gaussian process meta-learning framework, and derive an analytical mean-field approximation for task-specific updates. Our approach yields well-calibrated uncertainty estimates and achieves comparable or superior results on standard benchmark datasets. Code is publicly available at \url{https://github.com/keanson/revisit-logistic-softmax}.

Revisiting Logistic-softmax Likelihood in Bayesian Meta-learning for Few-shot Classification
Tianjun Ke Haoqun C Zenan Ling', Feng Zhou
*Center for Applied Statistics and School of Statistics, Renmin University of China
SScoool of EIC, Huazhong University of Science and Technology (keanson, cachaoqun2007, Leng.zhou)&rur edu. en, lingzenanChust. edu, cn
Abstract
Meta-learning has demonstrated promising results in few-shot classification (FSC) by learning to solve new problems using prior knowledge. Bayesian methods are effective at characterizing uncertainty in FSC, which is crucial in high-risk fields. In this context, the logistic-softmal likelihood is often employed as an alternative the softmax likelihood in multi-class Gaussian process classification due its conditional conjugacy property. However, the theoretical property of logistic- softmax is clear and previous research indicated that the inherent uncertainty of logistic-softmax leads to suboptimal performance. To mitigate these issues, we revisit and redesign the logistic-softmax likelihood, which enables control of the priori confidence level through a temperature parameter. Furthermore, we theoretically and empirically show that softmax can be viewed of logistic-softmax and logistic-softmax induces a larger family of data distri- bution than softmax. Utilizing modified logistic-softmas, we integrate the data augmentation technique into the deep kernel based Gaussian process leea-le framework, and derive an analytical mean-field approximation for task-specific updates. Our approach yields well-calibrated uncertainty estimates and achieves comparable or superior results on standard benchmark datasets. Code publicly available at https: /// /g tituh...... com/ cwm/ wwweannonnreviiieee wereit-loggiiiiiiisseee tmax.
special case
1 Introduction
Meta-learning refers to the ability to quickly learn a new task given set of training tasks that share common structure [13; 15; 30). This concept is critical for achieving human-like performance computationally with little data, and recent algorithms have shown success in few-shot classification and regression problems [15: 18]. When dealing with limited data, it is essential analyze robust. meta-learning methods that properly deal with uncertainty which is inevitable in the context [1]. Some existing works have suggested that the Bayesian inference mechanism provides principled way address this issue, moving towards robust meta-learning with uncertainty quantification [7; 27; 38],
The Bayesian framework combines prior distribution and likelihood function that models the data distribution. By performing posterior inference on parameters, it provides a natural framework for capturing inherent model uncertainty [8; 23]. Gaussian process (GP) is Bayesian nonparametric method that places prior distribution on functions rather than parameters, ensuring better expressive- ness [3]. The GP prior is defined by covariance function which can be parameterized by deep neural networks, commonly referred to as a deep kernel [35]. GP with a deep kernel has demonstrated state-of-ther the- performance in few-shot regression problems [20], where posterior inference admits closed-form solution due to the conjugacy between the Gaussian likelihood and the GP prior.
"Equal contributions.
CCoresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

However, there are several challenges in GP-based meta-learning with deep kernels for classification tasks. To begin with, unlike the regression scenario, the widely-used softmax likelihood does not lead to conjugacy for GPs, making posterior inference intractable. To overcome this issue, several approximate inference methods have been proposed, such as label regression with Gaussian likelihood [20], One-vs-Each (OVE) approximation of softmax [27], and the logistic-softmax likelihood [9]. The logistic-softmax likelihood replaces the exponential function in the softmax likelihood with a logistic function, ensuring conditional conjugacy after data augmentation [9]. However, previous research suggests that the logistic-softmal function tends to exhibit inherent lack of confidence, leading to suboptimal performance in few-shot classification tasks [27]. Additionally, most GP-based meta-learning models employ Gibbs sampling for posterior inference, which can be computationally demanding for achieving convergence [26]. Furthermore. the coordination of task-level posterior inference and meta-level optimization for deep kernel methods remains an open question.
In this paper, we bridge these gaps by redesigning the logistic-softmax likelihood and deriving mean- field approximation for posterior inference. contrastive learning, adding a temperature parameter to rescale the logits in softmax is prevalent approach [4: 12; 33]. Motivated by this concept, we introduce the temperature parameter to the logistie-softmax likelihood to control its inherent prior confidence. Moreover, we discover that softmax can be viewed as special case of logistic-softmax and logistic-softmax induces a larger family of data distribution than softmax. To the best of our knowledge, this theoretical property has not been covered in the literature. Furthermore, we apply the modified logistic-sofie softmax likelihood to GP based meta- meta-learning for few-shot classification. Since the ongistic-sfftmat likelihood enables conditional conjugate GP model, we naturally derive an analytical mean-field approximation for task-specific updates. Compared to the existing literature [27], this variational inference method is more efficient than Gibbs sampling. We empirically show that our mean-field approximation achieves comparable results to the Gibbs sampling in practice. We also contribute the coordination problem of bi-level optimization in Bayesian meta-learning methods.
Specifically, our contributions are as follows: (1) we introduce the logistic-softmax with temperature and prove its unique limiting behavior, which may have broad applicability across diverse machine learning domains; (2) we theoretically and empirically show that softmax can be viewed as a particular case of logistic-softmal and logistic-softmal induces larger family of data distribution than softmax; (3) we derive an analytical mean- field approximation for task-level posterior inference with redesigned logistic-softmax through data augmentation; (4) we demonstrate the effectiveness of the redesigned logistic-softman through few-shot classification accuracy and uncertainty calibration on several benchmark datasets; (5) we contribute the coordination of task-level inference and meta-level optimization that appears in Bayesian meta-learning, which may provide insights for future research.
2 Preliminaries
Meta-learning involves learning from set of tasks in order to acquire knowledge and generalize to new tasks [5; 20J. In this work, the t-th task comprises a dataset Dr consisting of both support and query data, denoted as S, and Q, respectively. The support data S, consists of limited number of samples (N-shot), represented as pairs of input-output (xn, Bn.) while the query set Qt typically includes larger number of samples. During training, models exposed to various tasks sampled from the dataset D to learn and extract valuable information across different contexts. Subsequently, when presented with unseen task t", the model aims to leverage the acquired knowledge from both training data and the support set S,- to make accurate predictions on the query set Q.-.
Gaussian Process Classification is a probabilistic framework used for solving classification prob- lems. At its core, a GP is a probability distribution over functions, where the values of the function fxx evaluated at arbitrary set of inputs have a joint Gaussian distribution with a mean vector and covariance matrix determined by kernel function. In the context of a C-class classification problem, separate GP latent functions [ are employed to model the logits for each class. These latent functions represent the underlying mapping from the input space to the logit of each class, capturing the uncertainty associated with the classification decision. By modeling the class logits with GPs, it enables principled probabilistic inference and prediction.
Deep Kernel is proposed by Wilson et al. [35] which combines kernel methods and neural networks, harnessing the expressive power from both sides. This deep kernel extends traditional covariance kernels by integrating deep architecture into the base kernel formulation. The deep kernel is defined
2

---

as k(x,'' 10,w) ( 0), where k' represents the base kernel with parameters O. To enhance flexibility, the inputs x and x' are transformed by a deep neural network g with weights w. The deep kernel parameters consist of A and w. A notable advantage of deep kernels is their capability to learn metrics through data-driven optimization of input space transformation, contrast to traditional kernels that often rely on Euclidean latance--bssel metrics.
3 Logistic-softmax with Temperature
In this section, we introduce novel logistic-softmar func- tion that incorporates temperature parameter. We first emphasize two unique features of this function as the temperature approaches zero. We then demonstrate that logistic-softmax surpasses softmax as a versatile categor- ical likelihood theoretically and empirically. Specifically, we show that softmax can be viewed as a special case of logistic-softmax and logistic-softmax induces a larger family of data distributions than softmax. Fig. I briefly previews the relationship between the two likelihoods.
Logistic-softmax
One-hot
Softmax
Pottem
Uniform
Pattem 2
3.1 Definition of Logistic-softmax with Temperature
Figure 1: Diagram representing the fea- tures and relationship of logistic-softmax and softmax.
We define the logistic-softmax function with temperature as:
o(fh/T)
ply f.)) E (S2/T)
(3.1)
where we assume C classes, f. (x,)) f. temperature parameter and or(-) is the logistic function as opposed to the exponential in the traditional softmax. While there are other possible choices for incorporating the idea of temperature, we find this particular form is the most straightforward one while preserving the desired conditional conjugate property of logistic-softmax as shown in later sections.
f. k (L... T is the
Although reminiscent of the softmax likelihood with temperature, the logistic-softmax likelihood displays distinct limiting behavior, illustrated by the following theorem.
Theorem 3.1. Denote the logistic-softmax function with temperature LS(f,, Define / { fa > 0} C [C), we have
C
max fn < 0 and c* = argmax f <E|C]
cE(C]
lim LS(fn, T
Σ CET
Eos
S > 0 c
max
[T|
where e, E C is the one-hot vector with 1 in its e-th coordinate.
The proof is provided in Appendix I. On the one hand, the Dogistic-softmar likelihood converges to a one-hot vector when fü < for all E [C]. On the other hand, the likelihood transforms into uniform distribution over the classes corresponding to the indices of the positive elements if multiple elements of f, are greater than 0.
We interpret the theoretical result from two perspectives. First, the original logistic-softmax likelihood has been criticized for its lack of confidence [27]. However, with the introduction of temperature, the modified logistic-sfftmax likelihood can become excessively confident as the temperature approaches zero. This implies that can effectively control the confidence of the logistic-softmal softmax likelihood by adjusting the temperature, thus resolving existing issues. Second, when multiple components of the logits are positive, this likelihood exhibits remarkable fairness all positive classes. This characteristic is profound and distinct since it enables adaptation to multi-label classification scenarios, where assigning an equal probability to each correct class is essential. Notably, to our knowledge, no existing likelihood can identify multiple correct classes, which limits the tools available for multi-label classification. Within this domain, the prevalent approach to treat each class as a binary classification problem separately. Our discovery paves the way for new paradigm in this domain, as
3

---

the logistic-softmax function allows for the identification of several positive labels simultaneously. However. in this work, we primarily concentrate on the application of logistic-softmaat in Bayesian meta-learning and postpone the investigation of multi-label classification to future research.
3.2 Comparison of Logistic-softmax and Softmax with Temperature
We present several results demonstrating that logistic-softmax surpasses softmax as a versatile categorical likelihood function theoretically and empirically, We begin by giving a theorem that shows the logistic-softmax function converges to the softmax function pointwise.
Theorem 3.2. For all f. E ERC, E R {0} and Cu E R, we have
lim LS(f. ,) S(f,, -) S(fn C),
where S(f,. r) denotes the softmax function with temperature.
The proof is provided in Appendix II. This theorem shows that softmax is translational invariant function that can be approximated by logistic-softmax with a sufficient negative shift of logits. We also note that logistic- softmax is not translational invariant. Empirically. we find that C" = 5 provides an accurate upproximation if f, is near zero. Now we present a stronger result indicating that logistic-softmax is more versatile than softmax with regards to modeling the data distribution.
Theorem 3.3. Assume the logits f ~ Gi(la, k"), where a is the mean function and ke is the
kernel function for each class c E [C]. Denote y = [yi..... yN] IT as the random label vector of N given points. Suppose E s and k2 E X, where s/ and X are two function classes. Define I (E a, xX as the family of the marginal distribution p(y) X, a, k9) induced by a E a and ke E H on given points X E Nxp xp with a likelihood function E. Under mild condition on s we have
F
x)
Furthermore, we have
(( C
The proof is provided in Appendix III. We elaborate on this theoretical result below. The first equation indicates that data distribution induced by softmax is translational invariant with regards to a, while the one by logistic-softmax softmax is not. Recall that Theorem 3.2 implies softmax is translational invariance w.r.t. its input variable. Intuitively, as the location parameter a is integrated due to translational invariance, the family of the marginal distribution of y corresponding to softmax is only decided by the kernel function k, while the family for logistie-softmax has an additional parameter a. With this intuition, it should be rather straightforward to understand our second result. Informally, the second result states that logistic-sofimax has larger family than softmax. In other words, logistic-softmax possesses a stronger capability in modeling the data distribution for each class.
Next, we delve deeper into the relationship between logistic-softmax and softmax. For comparison, we present toy classification task with three classes (C 3) and one sample belonging to the first class (y- = 1). We place a standard normal prior on f1 and fa and clamp fs at -100 for ease of visualization. We plot the likelihoods in Fig. 2 where we observe distinct pattern between softmax and logistic-softmax. Although both likelihoods become more confident (likelihood output changes at a faster rate) as the temperature parameter decreases, logistic-softmax exhibits unique probability patterns when both fi and 1/ are greater than O, while softmax gives the same sets of regardless of location. We present the zoom-in plots to further explain the difference in Fig. 2. These plots demonstrate that when both fi and fo are a bit less than O, bogistic-sfftmax accurately approximates softmax in every temperature. Indeed, the pattern of softmax in the whole plane is the same as the pattern that appeared in quadrant 3 of logistic-softmall This observation supports the idea behind Theorem 3.2 and Theorem 3.3, which shows that logistic- softmax is capable of modeling any data distribution that can be modeled by softmax, as long as we adjust the logits to the negative domain or put sufficiently small negative mean on f. Furthermore, as Fig. 2 shows, the softmax likelihood is unable to model the data distribution of logistic-softmax with positive logits. Therefore, we conclude that logistic-softmax is more powerful than softmax for its expressiveness in modeling categorical distribution.
borderline parallel to fi - f2

---

Softmax Likelihood
Logistic-Softmax Likelihood
r=
r=1
t=2
10
t=
t=1
high
Original
0.50
8.53
low
-3
10
10
10
10-10
10-20
10-10
10-10
10
high
Zoom-in
93
5,50
3.73
-12
low
More confident
Less confident
Figure Plot p(y 1ff) where fs clamped 1000. We provide separate zoom-in plots of softmax and logistic-softmax in the 2nd row. In the upper-right area (where all fi and f2 are greater than 0), the logistic- softtmx function exhibits unique probability patterns that softmax cannot model. In the bottom-left area (where all fi and f2 are smaller than 0), logistic- accurately approximates softmax for every temperature and every location all at once.
3.3 Further Discussions of Potential Applications
As the logistic-softmax function possesses more expressiveness than the softmax function, we include a short discussion of its potential applications in various domains. To begin with, it can applied to Gaussian process classification tasks, including class-imbalanced scenarios [37], active learning [40], and time-series data analysis [6]. In this case, Dogistic-softmal brings desired conditional conjugacy to make inference tractable and provides additional flexibility in data modeling than softmax as indicated in our paper. Moreover, the sofistic-sfttmax function can be a great choice in modern Bayesian methods, such Bayesian neural networks and neural network Gaussian processes although further adaptation needed. Furthermore, logistic-softman might be capable of replacing softmax beyond the Bayesian domain since we prove its flexibility over the softmax function. For example, as the logistic-softmax function captures positive signals for multiple classes, it may have prospective advantages in scenarios like multi-label classification [14] and multi-label contrastive learning [39]. ushering in new paradigms thanks to its unique properties.
4 Logistic-softmax with Temperature in Bayesian Meta-learning
Logistic-softmar was initially proposed to address the conjugation issue that arises in multi-class Gaussian process classification. As the Bayesian framework is advantageous in uncertainty calibration, some researchers focus adapting the multi-class Gaussian process to few-shot classification tasks, where Bayesian meta-learning one the prevalent paradigms. specific, Snell Zemel [27] attempts to apply original logistic-softmax to Bayesian meta-learning but fails to provide efficient inference and optimal result. follow this research line and validate the advantages of logistic- softmax with temperature, we utilize the logistic-softmax likelihood in the GP-based meta-learning framework. Additionally, we use the data augmentation technique to derive a fully analytical mean- field inference method for this model.
4.1 Framework of Bayesian Meta-learning
We review the Bayesian meta-learning framework used in the research line of MAML [7], DKT [20], and OVE [27]. To begin with, Bayesian meta-learning commonly utilizes a hierarchical structure that includes an inner loop and an outer loop. Conventionally, task-specific parameters are updated in the inner loop and task-common parameters are updated in the outer loop [7]. For efficiency, DKT proposed to replace the inner loop with a marginal likelihood computation that integrates out task-specific GP function for each task [22]. Then, the task-common hyperparameter O of the deep kernel can be learned by maximizing the marginal likelihood across all tasks in the outer loop. More specifically, denote the input support and query data of task t D the target data DV. Da and
5

---

D are the collections of these datasets over all tasks. The marginal likelihood takes the form
DD D ( II p(D) p))pp: O)dd..
(4.1)
where Φ; is the task-specific parameters of task t. To obtain an analytic integral Eq. (4.1), DKT used GP prior for p(Ø; @). The integral can be computed analytically in regression cases since the model likelihood is Gaussian as opposed to classification cases where the conjugacy is broken since categorical. However, obtain a GP model that is conditional conjugate after data augmentation. Thus, an efficient inference method is needed for the augmented model. We describe task-level inference in next section.
the likelihood is Bernoulli
the logistic-softmax likelihood to
4.2 Task-level Bayesian Inference
In this subsection, we introduce how an efficient task-level Bayesian inference method is developed. For specific task, we denote the (support and query) input dataset as X [xi..... ; XN] T with label dataset 182.....384 uN] where Vn [C], N and C are the number of observations and classes respectively. The multi-class GP classification model includes latent GP functions for each class f [= ,..,,,..'" fC]T where fe GP(a k"), a is mean function and kc is kernel for c-th class.
We utilize the logistic-softmax function with temperature Eq. (3.1) to model the likelihood for the multi-class GP classification. Inspired by Galy-Fajou et al. [9]. three sets of auxiliary latent variables are augmented to expand the logistic-softmax likelihood to obtain a conditional conjugate model for each task, including Gamma variables A, Poisson variables M, and Pólya-Gamma variables Ω. Given the GP priors on fe, we obtain the augmented joint density (proof provided in Appendix IV);
MY,A,
2 2 (9 +m))
exp
Vn
m
fc
wh 2
PG m. uh. 0)
A exp( (m)))
C II N((f K*),))
(4.2)
m
where -== [ -f. f&] is the e-th column F. A, is the augmented Gamma variable, m, the augmented Poisson variable, wf is the augmented Pólya-Gamma the kernel matrix for c-th class with N samples. The original model has been now transformed into a conditional conjugate one, which possesses excellent mathematical properties.
Gamma variable, a is the mean and K is
Mean-field Approximation Based on Eq. (4.2), we can obtain the closed-form conditional densi- ties for all variables, which constitutes Gibbs sampler if we iteratively draw sample from each conditional distribution. However, the Gibbs sampler is not efficient because the sampling operation time-consuming. In order improve efficiency, we derive mean-field variational inference that provides an approximate posterior but with better efficiency.
In the mean-field algorithm, we need to approximate the true posterior p(A, M, S Y) by variational distribution which is assumed to factorize over some partition of latent variables. Here, we assume the variational distribution gA,, M, Ω. F) qı (M, 22 (A, F). Following the standard derivation provided Appendix V, we obtain the optimal density for each factor:
N.C II n.e=1
N
@ M)
PG
m. + sh Fi), (4.3a)
(A) = II
Ga(An
On. C),,
(4.3c)
==1
N.C II m,cmm
C
ai (M
Po(mn 1n));
(4.3b)
(F)
W(f
Ø Σ",
(4.3d)
where

---

1
3n =V [ V 1% + 0
(4.4a)
Ee = (diag( / )²) + K )-, (4.4d)
(xp( - 12T
Sy -1y) + ''KK a, (4.4e)
2/ 2C cosh(ff/22
(4.4b)
Ir 2T
In + ynn tanh
On Σ 2 1,
(4.4c)
S Elw
2/n
2
(4.4f)
where (-) is the digamma function. Update the posterior of Ωt M, À and F iteratively by Eq. (4.3),
we obtain an efficient mean-field algorithm to provide the approximate posterior for each task.
4.3 Meta-level Optimization
We learn set of hyperparameters of deep kernels in the outer loop that maximizes the marginal likelihood across all tasks, which is also called the empirical Bayes [17]:
T
T
Σ logp(Y, ()) - Σ log / M, O ddA 1=1
(4.5)
t=1
where the subscript indicates the (-th task. Unfortunately, the integral in Eq, (4.5) intractable, so we utilize an approximate approach: given the variational parameters of each task, we try to maximize the evidence lower bound (ELBO) L as a function of θ [16]. Thanks to data augmentation, we have analytical expression of the ELBO which is provided in Appendix VI, and the gradient VeL can be computed by the automatic differentiation [2]. addition, Snell & Zemel [27] has found that the predictive likelihood (PL) can also serve a loss of O. We follow their derivation and use the approximate gradient estimator:
M
1
VeLPL
Σ m=1
Vologp(y. = x., X, Y, θ),
M
where M denotes the number of samples drawn from Eq. (4.6b).
Alternating between the optimization steps w.r.t the variational parameters of each task in the inner loop and the hyperparameters of deep kernels in the outer loop, obtain an efficient Bayesian meta-learning method for classification. However, the coordination the gradient flow between the inner loop and the outer loop remains open question. Previous research detaches the task-level variables the inner loop and updates the hyperparameters of the deep kernel using gradients produced by the outer loop [27]. Nevertheless, we have discovered that the task- variables in the inner loop should not be detached when using mean-field approximation.
4.4 Prediction
Given a test task with support dataset containing the input X {xi] and label Y from which the model can learn about the new classes, and an unlabeled data point x, in the query dataset, the predictive probability of test label y. l is:
{yr)
-f /PLL. k (.. II olf X, Y, O)df.
ply. k
X., X, -
(4.6a)
afS X) = (PS2111))( (f))((1) Y.Y, N 22,
(4.6b)
where & (1....,,. C}, θ the learned kernel hyperparameter in the training procedure, p(y. k: f.) is the logistic-softmax likelihood, f, = [f (x........,f _x q(p X is Eq. (4.3d). n = KG μıe and o he_- - Ki /t kf. + Ki "'K The integral in Eq. (4.6a) intractable, so we resort to Monte Carlo to compute and classify the test point by the highest pre- dictive probability. The complete training and test procedure is described in Alg. 1 of Appendix VII.
7

---

5
Experiments
In this section, we present the results for few-shot classification tasks including accuracy and uncertainty quantification. We consider two challenging standard benchmark datasets: the Caltech- UCSD Birds [32] and mini-ImageNet [24].
5.1 Few-shot Classification and Domain Transfer
In this subsection, we describe the experimental setup and report the results of our few-shot classi- fication tasks. While we notice that there are various settings for few-shot classification (e.g.. the distribution calibration technique in Yang et al. [36]). we adopt the vanilla setting in Bayesian meta- learning to better compare the likelihoods. Following the procedure of prior work [20]. we employ a standard Conv4 architecture [31] as the backbone and assess our models under six different settings, including 1-shot and 5-shot scenarios for both in in-domain and cross domaiin tasks. We benchmark our models against various baselines and state-of -fetthrrrt models, including Feature Transfer [5], Baseline++ [5], Matching [31], ProtoNet [28], RelationNet [29], MAML [7], DKT [20], Bayesian MAML [38], A ABML [23]. LS [9], and OVE [27]. As for LS, we utilized the Gibbs sampling version implemented by Snell & Zemel [27], which is more computationally demanding than our mean-fiell field method. We note that DKT, L.S, and OVE are similar to our proposed method as they are all GP-based models but use different likelihood functions and inference methods. To ensure a fair comparison, we also apply the cosine kernel as in Patacchiola et al. [20] and Snell & Zemel [27].
We train and evaluate our models (denoted as CDKT) with the ELBO loss (denoted as ML) using the default number of epochs from Patacchiola
[20], and with the predictive likelihood loss (denoted as PL) using 800 epochs. However, we find that the ML (T < 1) domain- transser experiment requires 800 epochs avoid underfitting. We set the temperature parameter to T 0.5 for the 5-shot PL experiment of mini-ImageNet and domain transfer, and 0.2 for all other experiments. Drawing inspiration from Section 3, we observe that placing negative mean prior improves our method by 0.5% to 1% across all experiments. To ensure numerical stability, we train our models with zero mean prior and use constant negative mean of 5 at test time for all ML (T < 1) experiments. We also use this technique for the PL (r < 1) experiment of the domain-transfe that the training of PL loss is generally less efficient than the ML loss, as it requires Monte Carlo sampling. Our mean field approximation converges at extremely fast rate and we only need 2 steps for lask-level updates. More experimental details are provided in Appendix VIII.
task. We also note
We report the average accuracy and standard deviation of our models evaluated on 5 batches of 600 episodes with different random seeds in Table 1. Our model achieves the highest accuracy both 1- shot (65.2 21 ()) and 5-shot 19%) experiments on the CUB dataset and achieves comparable results on the mini-ImageNNt dataset. As for the domain-transfer task, we also achieve the highest result for the 1-shot (40.43%) scenario and a near-optimal result for the 5-shot (56. 18%) scenario. Overall, while our mean-field approximation method works well for the ELBO loss (ML), the predictive likelihood loss (PL) generally does not match the performance demonstrated by the Gibbs sampling version LS proposed by Snell & Zemel [27]. However, we find the PL loss effective for the domain-tranne scenario, indicating its potential at dealing with this type of task. Moreover, observe a significant improvement (3% - 5%) in accuracy by adding a small temperature scaling (T 0.2 or 0.5) to the default logistic-softmax function (r = 1) in all scenarios. This finding suggests that our approach to controlling the priori confidence of logistic-softmax is highly effective.
5.2 Uncertainty Quantification
Uncertainty quantification is important aspect of few-shot learning because classifiers trained on limited data may have high variance and uncertainty in their predictions. This uncertainty can arise from a variety of sources, such as the small amount of training data, the complexity of the model, and the variability of the input data. A robust meta-learning method should be able to properly deal with such uncertainty, especially in high-risk areas including medical diagnosis, judicial adjudication, and autonomous driving.
We use 2 widely-applied metrics for uncertainty quantification, namely expected calibration error (ECF) [11] and maximum calibration error (MCF) [19]. ECE measures the average difference between confidence (probability outputs) and accuracy within each bin. MCE is similar to ECE but it
8

---

Table 1: Average I-shot and 5-shot accuracy and standard deviation on 5-way few-shot classification. Baseline results are from Patacchiola et [20] and Snell & Zemel [27]. Results are evaluated over batches of 600 episodes with different random seeds. We highlight the best results in bold.
CUB
mini-ImageNNt
mini-ImageNNet I-shot
CUB
Method
I-shot
Sishol
I-sho
5-shol
5-shot
Feature Transfer
46.19 1 0,64
68.40
1 0.79
39.51
0.23
60.51 1 0.55
32.77 1 0.35
50.34
0.27
Baseline++
61.75 = 0.95
78.51
+ 0.59
47.15
0.49
66.18 - 0.18
39.19
019::011
57.31
= 0.11
NatchingNet
60.19 = 1,02
75.11
0.33
18.25=0.00
62.71
+ 0.44
36.98 0.00
50.72 = 0.36
ProtoNet
52.52 1.90
75.93
1.30 48.76
64.07
33.27 +
RelationNet
62.52 0.34
78.22
1 0.07
0.17
64,20
1 0.28
17.13.10.200
51.76 1.48
MAMI,
56.11
74.84
0.49
61.58
34.01
4883
DKT + Cusine
63,37 1 0.19
77.73
1 0.26
48.64
1 0.45
62.85
1 0.37
40.22 0.54
55.65
0.05
Bayesian MAML
72.87
0.30
62,60
0.336
51.35
Bayesian MAML (Chaser)
53.93 = 0.72
71.16 + 0.32
43.74 = 0.46
59.23 + 0.34
36.22 0.50
51.53 = 0.43
ABML
56.08 1:
0.266
LS (Gibbs)+ + Cosine (ML)
60.23 0.54
74.58 -
46.75 = 0.20
59.93 + 0,31
36.41 0.18
50.33 = 0.13
IS (Gibhs) Cosine (PI.)
60.07 0.29
0.007
47.05
0.20
66.01 + 0.25
0.26
56.70 0.31
OVE PG GP+ Cosine (ML)
63.98 0.43
77,44 0.188
0.35 50.02
64.58 1 0.31
39.66 1: 0.18
55.71
0.31
OVE PG GP Cosine (PL)
60.11 0.26
79.07
1: 0.05
48.00
0.24
67.14
1 0.23
37.49.1.00
57.23
0.31
CDKT +
Cosine (ML) (r < 1)
65.21 = 0.45
79.10
+: 0.33
47.54
0.21
63.79 +: 0.15
40.43 0.43
55.72 = 0.45
CDKT Cosine (ML)
60.85
75.98
+ 0.33
43.50
0
59.69 +
0.
52.42 0.50
CDKT Cosine (PL. ()
59.49 - 0.35
76.95
+ 0.28
44.97
+ 0.25
60.87
+ 0.24
39. 18 0.3.4
56.18
0.28
CDKT Cosine (T 1)
52.91 0.29
73.34
0.40
40.29
0.14
60.23 1 0.16
37.62
0.32
54.32 0.19
Table Expected calibration error (ECE) and maximum calibration error (MCE) for 5-shot 5-way tasks on CUB, mini- Mini-ImageNet. and domain-transser. Baseline results are from Snell & Zemel [27]. All metries are computed on 3,000 random tasks from the test set.
CUB
mini- maag-ema ECE
Piini-mmaeNNet->CEE
Method
ECE
MCE
MCE
ECE
MCE
Feature Transfer
0.187
0.250
0.368
0.641
0.275
0,646
Baseline++
0.421
0.502
0.395
0.598
0.315
0.537
MatchingNet
0.023
0.031
0.019
0.043
0.030
0.079
ProtoNet
0.034
0.059
0.035
0.050
0.009
0.025
RelationNet
0,438
0.593
0.330
0.596
0.234
0.554
DKT Cosine
0.187
0.250
0.287
0.446
0.236
0.426
Bayesian MAML
0,018
0.047
0.027
0,049
0.048
0.077
Bayesian MAML (Chaser)
0.047
0.104
0.010
0.071
0.066
0.260
LS (Gibbs) Cosine (ML)
0.371
0.478
0.277
0.490
0.220
0.513
LS (Gibbs) + Cosine (PL)
0.024
0.038
0.026
0.041
0.022
0.042
OVE PG GP + Cosine (ML)
0.026
0.043
0.026
0.039
0.049
0.066
OVE PG GP + Cosine (PL)
0.005
0.023
0.008
0.016
0.020
0.032
CDKT Cosine (ML)
0.005
0.036
0.009
0.015
0.007
0.020
CDKT Cosine (PL)
0.018
0.223
0.025
0.140
0.010
0.029
measures the maximum difference. Following the protocol of Patacchiola et al. [20] for evaluation, we first tune the temperature parameter on the validation set for alignment and then compute the ECE and MCE on the test set. The results of 5-shot experiments are summarized in Table 2. We observe that our introduction of temperature significantly improves the calibration of logistic-softmaat compared to the one implemented by Snell & Zemel [27]. Specifically, we obtain the lowest ECE on CUB (0.005) and domaín transfer (0.007) and the lowest MCE on mini-ImageNet (0.015) and domain transfer (0.020). Our model performs marginally worse than state-of-the-an in MCE on CUB and ECE on -mageem Overall, this result indicates that our model is highly reliable on uncertainty calibration, demonstrating promising robustness in sho-shot scenarios.
Figure 3 shows the reliability diagrams of our models. The confidence barplot should match the diagonal line for a robust uncertainty quantification model. We observe that the models trained with the ELBO loss (ML) generally perform better than those trained with the predictive likelihood (PL). Specifically, ML models closely match the diagonal line while PL models tend to underestimate the accuracy of low-confidence outputs. conclusion, our findings indicate that ML models are better at handling uncertainty in the context of mean-field approximation inference. This is an interesting result as contrasts the findings in Snell & Zemel [27], where PI. models with Gibbs sampling generally perform better. Nonetheless, we leave the analysis of this phenomenon to future work.

---

5.00
52 64 64
83 2-6
94 64
K2
24
(a) CUB (ML)
(b) Mini (ML)
(c) DT (ML)
(d) CUB (PL)
(c) Mini (PL)
(f) DT (PL)
Figure 3: Reliability diagrams on 5-shot classification with expected calibration error (ECE) and maximum calibration error (MCE) metrics. Mini denotes the mini- Mini-ImaeeNet dataset, and DT denotes
the domain transfer task of CUB -> mini-ImageNet. Results are computed on 3,000 test tasks.
6 Related Work
GP classification has been extensively studied, and several approaches have been proposed to address its challenges, including label regression and Laplace approximation [25; 34). Recently, Polson et al. [21] introduced Polya-Gamma augmentation, and Galy-Fajou et al. [9]; Snell & Zemel [27] utilized different likelihood functions and data augmentation to approximate the softmax function. These methods have advanced the field by providing intuitive frameworks and improving modeling capabili- ties for GP classifiers. Our work builds upon these foundations by focusing on more generalized likelihood formulation with enhanced flexibility and modeling capabilities for GP classifiers.
Bayesian meta-learning has been explored through various approaches leverage prior knowledge and adapt to new tasks. Finn et al. [8] introduced Bayesian hierarchical modeling perspective, capturing uncertainty at different levels. Grant al. [10] recast meta-learning
inference in GP. Yoon et al. [38] introduced Bayesian MAML on the basis of Finn et al. [7]. Patacchiola et al. [20]: Snell & Zemel [27] utilized GPs with deep kernels for task-specific inference. These works made contributions to Bayesian meta-learning by addressing parameter updates, uncertainty modeling, and prior distributions. Our work contributes by providing an effective alternative for task-level updates and further provides insights into the coordination problem of bi-level optimization.
7 Limitations
While the logistic-softmax with temperature has proven to be theoretically superior to softmax in data modeling, its performance and suitability are only verified Bayesian meta-learning, leading to certain limitations. Further research is necessary to explore the logistic-softmax function's 's performance and adaptability various domains and problem settings.
8 Conclusions
In this paper, we introduce the logistic-softmax function with temperature which simple yet highly effective in improving classification accuracy and uncertainty calibration. Furthermore, we delve into the theoretical property of the redesigned logistic-softmax function including both limiting behavior and data modeling capability. Moreover, we apply mean-field approximation for deep kernel based GP meta-learning for the first time. We also shed some light on the coordination problem between the inner loop and the outer loop that appeared in li-le optimization. In the future, it is an interesting track to apply redesigned Pogistic-softman to other domains such as multi-label classification.
Acknowledgments and Disclosure of Funding
This work was supported by NSFC Project (No. 62106121), the MOE Project of Key Research Institute of Humanities and Social Sciences (22JJD1 10001), the fund for building world-class universities (disciplines) of Renmin University of China (No. KYGJC2023012), and the Public Computing Cloud, Renmin University of China.
10

---

References
[1] Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth,
P., Cao, X., Khosravi, A., Acharya, U. R., et al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 76:243-297 2944, 2021.
[2] Baydin, A. G., Pearlmutter, B. A., Radul, A. A., and Siskind, J. M. Automatic differentiation in machine learning: survey. Journal of machine learning research, 18, 2018.
[3] Bishop, C. M. Pattern Recognition and Machine Learning. springer, 2006.
[4] Chen, T., Korbblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
[5] Chen, W., Liu, Y., Kira, Z.., Wang, Y. F., and Huang. J. A closer look few-shot classification. In International Conference on Learning Representations. OpenReview.ner, 2019.
[6] Constantin, A., Fauvel, M., and Girard, S. Mixture of multivariate gaussian processes for classification irregularly sampled satellite image time-series. Statistics and Computing, 32 (5):79, 2022.
[7] Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pp. 1126-1135. PMLR, 2017.
[8] Finn, C., Xu, K., and Levine, S. Probabilistic model-agnostic meta-learning, Advances in neural information processing systems, 31, 2018.
[9] Galy-Fajou, T., Wenzel, F., Donner, C., and Opper, M. Multi-class gaussian process classifi- cation made conjugate: Efficient inference via data augmentation. In Uncertainty in Artificial Intelligence, pp. 755-765, PMLR, 2020.
[10] Grant, E., Finn, C., Levine, S., Darrell, T., and Griffiths, T. Recasting gradient-based meta- learning as hierarchical bayes. arXiv preprint arXiv:1801.0881 2018.
[11] Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330. PMLR, 2017.
[12] He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9729-9738, 9738, 2020.
[13] Hospedales. T., Antoniou, A., Micaelli, P., and Storkey, A. Meta-learning in neural networks: survey. IEEE transactions on pattern analysis and machine intelligence, ((19):54495114 2021.
[14] Lanchantin, J., Wang. T., Ordonez, V., and Qi, Y. General multi-label image classification with transformers. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 14478-16488 2021.
[15] Li, X., Sun, Z., Xue, J.-H., and Ma, Z. A concise review of recent few-shot meta- meta-learning methods. Neurocomputinng 456:463-468, 2021.
[16] Mandt, S., Hoffman, M., and Blei, D. A variational analysis of stochastic gradient algorithms. In International conference on machine learning, pp. 354-363. PMLR, 2016.
[17] Maritz, J. S. and Lwin, T. Empirical bayes methods. Chapman and Hal/CCRC, 2018.
[18] Nichol, A., Achiam, J., and Schulman, J. On first-order meta-learning algorithms. arXiv preprint arXiv: 1809902999, 2018.
[19] Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J., Lakshminarayanan, B., and Snoek, Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.

---

[20] Patacchiola, M., Turnrr, J., Crowley, E. J., O' O'Boyle, M. F. P., and Storkey, A. J. Bayesian meta-learning for the few-shot setting via deep kernels. In Advances Neural Information Processing Systems, 2020.
[21] Polson, G., Scott, G., and Windle, J. Bayesian inference for logistic models using Pólya- Gamma latent variables. Journal of the American statistical Association, 108(504):333--144 2013.
[22] Rasmussen, C. E. Gaussian processes in machine learning. In Summer School Machine Learning, pp. 63-71. Springer, 2003.
[23] Ravi, S. and Beatson, A. Amortized bayesian meta-learning. In International Conference on Learning Representations, 2019.
[24] Ravi, S. and Larochelle, H. Optimization as model for few-shot learning. In International conference on learning representations, 2017.
[25] Rifkin, R. and Klautau, A. In defense of one-vs-all classification. The Journal of Machine Learning Research, 5:101-141, 2004.
[26] Ruiz, F. and Titsias, M. A contrastive divergence for combining variational inference and meme. In International Conference on Machine Learning, pp. 5537-5545. PMLR, 2019.
[27] Snell, J. and Zemel, R. S. Bayesian few-shot classification with one-vs-cach polya-gamma gamma augmented gaussian processes. In International Conference on Learning Representations, ICLR 2021. OvenReview.net, 2021.
[28] Snell, J., Swersky, K., and Zemel, R. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
[29] Sung, F., Yang, Y., Zhang, L., Xiang, T., Torr, P. H., and Hospedales, T. M. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 119--1208, 2018.
[30] Vilalta, R. and Drissi, Y. A perspective view and survey of meta-learning, Artificial intelligence review, 18:77-95, 2002.
[31] Vinyals, O., Blundell, C., Lillicrap. T., Wierstra, D., et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
[32] Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The caltech-uesd birds-200-2011 dataset. 2011.
[33] Wang, F, and Liu, H. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2095-4995555 2504, 2021.
[34] Williams, C. K. and Barber, D. Bayesian classification with gaussian processes. IEEE Transac- tions on pattern analysis and machine intelligence, 20(12):1342-133 1351, 1998.
[35] Wilson, A. G., Hu, Z., Salakhutdinov, R., and Xing, E. P. Deep kernel learning. In Artificial intelligence and statistics, pp. 370-378. PMLR, 2016.
[36] Yang, S., Liu, L., and Xu, M. Free lunch for few-shot learning: Distribution calibration. arXiv preprint arXiv:2101.06395, 2021.
[37] Ye, C.. Barnes, N., Petersson, L., and Tsuchida, Efficient gaussian process model on class- imbalanced datasets for generalized zero-shot learning. In 2022 26th International Conference on Pattern Recognition (ICPR), pp. 2078-2085 IEEE, 2022.
[38] Yoon, J., Kim, T., Dia, O., Kim, S., Bengio, Y., and Ahn, S. Bayesian model-agnostic meta- learning. Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 7343-7353, 2018
12

---

[39] Zhang, S., Xu, R.. Xiong, C., and Ramaiah, C. Use all the labels: A hierarchical multi-label contrastive learning framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 16660-16669, 2022.
[40] Zhao, G., Dougherty, E., Yoon, B.-J., Alexander, F., and Qian, X. Efficient active learning for gaussian process classification by error reduction. Advances in Neural Information Processing Systems, 34:9734-977 9774, 2021.
13


Reviews:
[{"review_id": "6hCOcFcPpG", "reviewer": "Reviewer_iEEK", "summary": "In this work, the logistic-softmax likelihood is redesigned, allowing control of the a priori confidence level through a temperature parameter. The modified logistic-softmax is shown to encompass softmax as a special case and induces a larger family of data distributions. By integrating this modified likelihood into a deep kernel-based Gaussian process meta-learning framework with data augmentation, well-calibrated uncertainty estimates are achieved in experiments, and competitive results are obtained on standard benchmark datasets.\n\nPost rebuttal: I have read the authors' rebuttal and I appreciate the authors' effort in addressing my concerns.", "questions": "Q1: Have you considered other forms of the temperature in the modified logistic-softmax likelihood, for example having it as the power of the functions?\nQ2: Can you give a bit more detail of how the modified logistic-softmax function in multi-label classification?", "limitations": "The limitations have been discussed in the paper: the performance of the proposed modified logistic-softmax is only evaluated in Bayesian meta-learning.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+The logistic-softmax function with temperature is a nice idea and has potential to be used in multiple domains.\n+The theoretical analysis of the logistic-softmax likelihood is solid.\n+Some promising results are presented.\n+The paper is well written and easy to follow.", "weaknesses": "-The proposed modified logistic-softmax function with temperature can be fundamental for different machine learning problems, it is not clear why it is specifically used for Bayesian meta-learning for few-shot classification. It is suggested to be tested on simpler tasks first before applying it to tasks like Bayesian meta-learning.\n-The proposed method performs marginally worse than state-of-the-art in maximum calibration error on a few benchmark datasets. It is not clear why how it happens. Theoretically, an adaptive temperature is expected to give better performance for all the tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for reviewing our paper. We answer your questions below.\n\n> Q1. The proposed modified logistic-softmax function with temperature can be fundamental for different machine learning problems, it is not clear why it is specifically used for Bayesian meta-learning for few-shot classification.It is suggested to be tested on simpler tasks first before applying it to tasks like Bayesian meta-learning.\n\nThank you so much for pointing this out. Logistic-softmax was initially proposed to address the conjugation issue that arises in multi-class Gaussian process classification. As the Bayesian framework is advantageous in uncertainty calibration, some researchers focus on adapting the multi-class Gaussian process to few-shot classification tasks, where Bayesian meta-learning is one of the prevalent paradigms. In specific, OVE attempts to apply it to Bayesian meta-learning but fails to achieve optimal results. Therefore, we specify Bayesian meta-learning because it is one of the initial applications for the original logistic-softmax likelihood in literature and is consistent with our research line. Nonetheless, our theoretical analysis has broad applicability across various domains beyond its motivation from the Bayesian framework.\n\nMeanwhile, we do realize that the presentation of our paper could be adjusted for better clarity and we thank you for pointing this out. We will elaborate on why we apply the logistic-softmax function to Bayesian meta-learning and explain our motivation more clearly in the revised paper. In addition, we will add more discussions on other potential applications of logistic-softmax. \n\n\n\n> Q2. The proposed method performs marginally worse than state-of-the-art in maximum calibration error on a few benchmark datasets. It is not clear why how it happens. Theoretically, an adaptive temperature is expected to give better performance for all the tasks.\n\nThank you for raising this concern. We notice that our result on maximum calibration error is worse in only one dataset. One possible explanation for this phenomenon is that the adaptive temperature is tuned on the validation set (line 295). It is highly possible that in the test dataset, there are some outliers that this particular temperature is unable to effectively handle. To provide some context, the MCE result of 0.036 is from a bin with a small number of samples.\n\n\n\n> Q3. Have you considered other forms of the temperature in the modified logistic-softmax likelihood, for example having it as the power of the functions?\n\nThank you for reminding us, we will add a section discussing possible forms of temperature in the revised paper. We consider the current form because we are motivated by temperature in contrastive learning, where the most popular choice is to scale the logit directly. In addition, we also believe that it is the simplest way to maintain the conditional conjugate structure, while using the power of the function may lead to an intractable integral unless further adapted.\n\n\n\n> Q4. Can you give a bit more detail of how the modified logistic-softmax function in multi-label classification?\n\nThank you for raising this question. In general, multi-class classification is often treated as a combination of binary classification problems. For example, [1-3] use independent classifiers for each label, and design a sum of binary cross-entropy(BCE) loss as the training objective. From a higher level, this phenomenon is due to the lack of an appropriate likelihood function that identifies multiple positive labels at the same time, and researchers have to rely on the One-vs-Rest scheme. Specifically, if we use the softmax function (at a low temperature) to process the logits, the output fails to depict the probability of multiple labels, as softmax converges to a one-hot vector (for example, the positive label includes cat and dog, logits are 5 and 4.9, softmax (at a low temperature) outputs 0.99 and 0.01). With our modified logistic-softmax function, it can leverage the upside of a low temperature while preserving an appropriate probability output (logits are 5 and 4.9, logistic-softmax (at a low temperature) outputs 0.51 and 0.49). Although we acknowledge the potential barriers in the application to multi-class classification (e.g., designing novel loss function paired with logistic-softmax), we hold a strong belief in its potential to motivate new paradigms in multi-label classification.\n\n[1] Lanchantin, Jack, et al. \"General multi-label image classification with transformers.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n[2] Wang, Haoran, et al. \"Can multi-label classification networks know what they don’t know?.\" Advances in Neural Information Processing Systems 34 (2021)\n\n[3] Panos, Aristeidis, Petros Dellaportas, and Michalis K. Titsias. \"Large scale multi-label learning using Gaussian processes.\" Machine Learning 110 (2021)\n\n"}, {"review_id": "KwAKaDDFu8", "reviewer": "Reviewer_L2f9", "summary": "This paper proposes to modify the logistic Softmax likelihood by including a temperature coefficient in GP-based meta learning for few-shot classification. This is motivated by the observation that prediction made by logistic Softmax likelihood do not have confidence.\nTheoretically, they demonstrate that using the temperature, one could control for the confidence in logistic softmax. Furthermore, it is proved that softmax likelihood is a special case of logistic softmax. They also propose to use mean-field variational inference to approximate the posterior which is computationally more efficient than the typically adapted Gibbs sampler. It is demonstrated that their method achieves superior performance in uncertainty quantification. However, the performance improvements measured by accuracy are not as impressive as the uncertainty quantification.\n", "questions": "Q1. I understand that the typical benchmark for few-shot classification uses 5 classes. Have you seen improvements with your method when there are more than 5 classes? Like 10 or 20? (Answering this question won’t impact my decision negatively.)\n\nQ2. Can you also include https://arxiv.org/pdf/2101.06395.pdf method in your benchmark?\n\nQ3. I understand that the temperature has been tuned for your experiments, but it would be nice to see the variation in the accuracy and uncertainty estimation as the temperature changes. Can you demonstrate this relationship for a subset of your experiments?\n", "limitations": "NA", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The problem is well motivated, and all the required theoretical derivations have been included.\nThere are new theoretical findings presented.", "weaknesses": "The idea of including temperature is not novel but its application to a new domain is novel.\nIncluding one more recent method for few-shot classification, referenced in my questions, can improve the benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your advice to this paper. We answer your questions below:\n\n> Q1. I understand that the typical benchmark for few-shot classification uses 5 classes. Have you seen improvements with your method when there are more than 5 classes? Like 10 or 20? (Answering this question won’t impact my decision negatively.)\n\nWe understand your concern about our method's effectiveness on 10 or 20-way few-shot problems. We acknowledge that we have not run experiments on those settings because most benchmarks in Bayesian meta-learning do not include results for 10 or 20-way few-shot problems. In future research, we will investigate the potential impact brought by the choice of class numbers, but we believe more classes will not fundamentally change the effect of our current method.\n\n\n\n> Q2. Can you also include https://arxiv.org/pdf/2101.06395.pdf method in your benchmark?\n\nThe referred paper proposes a novel idea for few-shot learning by calibrating data distribution. Although the method brings promising results on similar benchmarks, we suppose its framework is a bit different from the Bayesian meta-learning methods considered in this paper. In fact, the calibration method is compatible with our proposed method since we essentially introduce a Bayesian framework to train classifiers, and the calibration method works with arbitrary classifiers. However, we would love to add some discussion about the referred paper to give a more comprehensive view of typical few-shot classification methods.\n\n\n\n> Q3. I understand that the temperature has been tuned for your experiments, but it would be nice to see the variation in the accuracy and uncertainty estimation as the temperature changes. Can you demonstrate this relationship for a subset of your experiments?\n\nThank you for the question. We have added some results on the CUB dataset with different temperature parameters.\nWe hope the additional empirical evidence resolves your concern.\n\n| Temperature | 0.2              | 0.5              | 0.75             | 1                | 1.5              |\n| ----------- | ---------------- | ---------------- | ---------------- | ---------------- | ---------------- |\n| CUB 1 shot  | 65.76 $\\pm$ 0.40 | 65.16 $\\pm$ 0.28 | 64.02 $\\pm$ 0.29 | 60.85 $\\pm$ 0.38 | 59.43 $\\pm$ 0.25 |\n| CUB 5 shot  | 79.10 $\\pm$ 0.33 | 78.48 $\\pm$ 0.18 | 77.20 $\\pm$ 0.13 | 75.98 $\\pm$ 0.33 | 72.13 $\\pm$ 0.20 |\n\n"}, {"review_id": "ZfOGL6v1OD", "reviewer": "Reviewer_JdA9", "summary": "The paper revisits the design of logistic-softmax function in the context of classification in Bayesian machine learning. In particular, the paper shows that a logistic-softmax function with a temporature could be more expressive than the conventional softmax function. However, due to the intrinsic nature of the logistic-softmax function, the inference might be non-conjugate, resulting in an intractable solution. To mitigate such an issue, the paper adopts the data augmentation technique proposed in the original logistic-softmax paper and propose a mean-field variational inference as an approximation to make the inference more efficient. The paper demonstrates the capability of the logistic-softmax with temporature in the context of few-shot meta-learning and shows that the approaches that integrate such method could achieve comparable performance in the two classificatio benchmarks: CUB-200-2011 and mini-ImageNet.", "questions": "Could the authors discuss further some potential applications using the logistic-softmax function of interest (beside meta-learning mentioned in the paper)?", "limitations": "- This is an increamental improvement from the original paper of the logistic-softmax function.\n- As mentioned in section 7, the current analysis is carried out in the context of Bayesian meta-learning. It might worth to investigate in other settings.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper has carried out a thorough investigation of the properties of the logisitic-softmax function with a temporature parameter. In particular, the paper shows that the logistic-softmax function with a temporature can easily model a one-hot vector or a uniform one (Theorem 3.1) or it can converge to a softmax function under certain conditions (Theorem 3.2). In addition, the logistic-softmax function of interest could model a richer family of distribution functions (Theorem 3.3).\n- The explanation of the paper is clear with easy-to-follow intuitition. The formulations are also clear to increase its clarity.", "weaknesses": "The weakness of the paper might be at the applications of the logistic-softmax function. Currently, the paper targets to meta-learning, but to me, the paper is about the logistic-softmax function and meta-learning is just a mere application to demonstrate. Since the authors specify explicitly meta-learning, I cannot ask for other exploration. However, the paper might be strengthen more if it could include other applications using such a function for classification with GP.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your advice on this paper. We answer your questions below:\n\n> Q1. Currently, the paper targets to meta-learning, but to me, the paper is about the logistic-softmax function and meta-learning is just a mere application to demonstrate. However, the paper might be strengthen more if it could include other applications using such a function for classification with GP.\n\nWe agree that logistic-softmax is the main focus of this paper. Despite meta-learning being one of its typical applications, we acknowledge that logistic-softmax should be employed in other domains. In future research, we are committed to exploring additional applications of logistic-softmax within and beyond GP classification tasks.\n\n\n\n> Q2. Could the authors discuss further some potential applications using the logistic-softmax function of interest (besides meta-learning mentioned in the paper)?\n\nThank you for raising this question. We believe our logistic-softmax function has the potential to replace the softmax function in various domains. To begin with, our function can be applied to Gaussian process classification tasks, including class-imbalanced scenarios [1], active learning [2], and time-series data analysis [3]. In this case, logistic-softmax brings desired conditional conjugacy to make inference tractable and provides additional flexibility in data modeling than softmax as indicated in our paper. Moreover, the logistic-softmax function can be a great choice in modern Bayesian methods, such as Bayesian neural networks and neural network Gaussian processes although further adaptation is needed. Furthermore, our logistic-softmax function might be capable of replacing softmax beyond the Bayesian domain since we prove its flexibility over the softmax function. For example, as the logistic-softmax function captures positive signals for multiple classes, it may have prospective advantages in scenarios like multi-label classification [4] and multi-label contrastive learning [5], ushering in new paradigms thanks to its unique property.\n\n[1] Ye, Changkun, et al. \"Efficient Gaussian Process Model on Class-Imbalanced Datasets for Generalized Zero-Shot Learning.\" 2022 26th International Conference on Pattern Recognition (ICPR). IEEE, 2022.\n\n[2] Zhao, Guang, et al. \"Efficient active learning for Gaussian process classification by error reduction.\" Advances in Neural Information Processing Systems 34 (2021)\n\n[3] Constantin, Alexandre, Mathieu Fauvel, and Stéphane Girard. \"Mixture of multivariate gaussian processes for classification of irregularly sampled satellite image time-series.\" Statistics and Computing 32.5 (2022)\n\n[4] Lanchantin, Jack, et al. \"General multi-label image classification with transformers.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n[5] Zhang, Shu, et al. \"Use all the labels: A hierarchical multi-label contrastive learning framework.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n\n\n> Q3. This is an incremental improvement from the original paper of the logistic-softmax function.\n\nWe acknowledge that our work is built upon the original paper on logistic-softmax, but we would like to emphasize that the main theoretical findings of our research have not been explored in the existing literature to the best of our knowledge. Furthermore, the application of logistic-softmax in Bayesian meta-learning with mean-field posterior inference is a novel contribution of our work. Through empirical evaluation, we have achieved state-of-the-art performance on several benchmarks, providing empirical validation for our theoretical analysis. While we admit the modification is simple, we believe our theoretical and methodological findings support the significance and relevance of our research. \n\n"}, {"review_id": "DSa6FyLcrj", "reviewer": "Reviewer_1tNK", "summary": "This paper tackles the problem that classification tasks in Bayesian meta-learning meet mathematical problems with the softmax function. The previously proposed logistic-softmax function as an alternative can be optimized, but tends to exhibit inherent lack of confidence in prediction. To solve this problem, the authors propose to add a temperature scaling parameter in the function. This simple change not only solves the confidence problem when setting the temperature to less than 1, but also enjoys both theoretical and empirical advantages, as verified in the paper. For efficient and intractable optimization, the authors additionally use the data augmentation technique to derive a fully analytical mean-field inference method for the Bayesian meta-learning model.", "questions": "- In theorem 3.3, $a$ is said to be the mean function with no restriction, so the mean of different sample points can be different. However, in line 444 in the appendix, all different samples have the same mean, can the author explain this?\n- How does your method compare with non-Gaussian processes like neural process [1]? It seems that neural process is more flexible and efficient than Gaussian process.\n\n[1] Garnelo et al. Neural processes.", "limitations": "The authors have discussed the limitations which I found reasonable. I suggest the authors think more about the role of Gaussian process in bayesian meta-learning, as well as the role of bayesian meta-learning in few-shot classification. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ While being simple, the revised logistic-softmax function indeed enjoys better theoretical and empirical advantages, and these are all verified in the paper with proofs or experiments.\n+ The proposed logistic-softmax function with temperature has the potential to be applied to broader research areas such as multi-class classification.\n+ The writing is clear, making the paper easy to follow.", "weaknesses": "- When we just focus on few-shot classification, I have to say that all bayesian meta-learning algorithms are not comparable to even very simple algorithms like ProtoNet or CE Baseline (I mean, if well-tuned, not the performance in Table 1) with the same architecture Conv-4. Also, as standard algorithms now use ResNet-12 as the backbone, all bayesian meta-learning algorithms still use Convnet, making the performance not comparable to those in most recent reported papers. This leads to doubt that do these bayesian meta-learning algorithms still perform well if we use ResNet-12 as the backbone? If the reason not to use ResNet-12 is the computation burden, then the biggest problem may be how to make bayesian inference more tractable. Also, another problem of interest is to figure out why bayesian meta-learning with complicated training steps cannot perform well on few-shot learning, even if equipped with nice theoretical properties. Since this paper does not answer these questions, I think the value of this paper is restricted.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable advice. As an overview, our paper mainly focuses on two themes: 1. theoretical analysis of \nthe logistic-softmax function. 2. application of the logistic-softmax function to Bayesian meta-learning. We admit the second section mainly follows the research line of the Bayesian meta-learning community while paying less attention to the larger context of few-shot classification. Although we do not offer a comprehensive remedy for the problems faced by the Bayesian method in FSC, we believe our theoretical analysis on logistic-softmax and the derivation of the mean-field approximation provides valuable insight and tools for the strive of the Bayesian methods community. In the revised version, we will add more discussions with regard to the role of Gaussian process in Bayesian meta-learning, as well as the role of Bayesian meta-learning in FSC in the Limitations section. Now we answer your questions below.\n\n> Q1. When we just focus on few-shot classification, I have to say that all bayesian meta-learning algorithms are not comparable to even very simple algorithms like ProtoNet or CE Baseline (I mean, if well-tuned, not the performance in Table 1) with the same architecture Conv-4.\n\nThank you for the comment. In Table 1, the result for ProtoNet is state-of-the-art performance under the same training protocol in the existing literature. Therefore, we may not fully understand the meaning of 'well-tuned' in your concern and would love to engage in further discussion.\n\nIn addition, though some Bayesian methods may not be competitive in terms of accuracy, we would like to point out that the major advantage of Bayesian methods is the capability of uncertainty calibration in a probabilistic framework. This is often desirable in fields relying on risk measurement, especially in the context of few-shot classification. \n\n\n\n> Q2. Also, as standard algorithms now use ResNet-12 as the backbone, all Bayesian meta-learning algorithms still use Convnet, making the performance not comparable to those in most recent reported papers. This leads to doubt that do these bayesian meta-learning algorithms still perform well if we use ResNet-12 as the backbone?\n\nThank you for the question. We would like to emphasize that our work mainly focuses on the theoretical property of modified logistic softmax. Its application to Bayesian meta-learning is largely an attempt to verify our theoretical analysis with empirical results. Therefore, we use the most common backbone structure and training protocol inherited from Bayesian meta-learning. Since OVE and DKT (and other Bayesian meta-learning methods) differ from our method only in their likelihood functions in essence, we use the same Convnet setup as them for better comparison of the logistic-softmax function and others.\n\nBesides, we refer to the paper of DKT [1] which uses both ResNet-10 and Convnet as the backbone and both reaches competitive results. As a similar Bayesian meta-learning method, we believe our method can extend to different neural network structures as well. \n\n[1] Patacchiola, Massimiliano, et al. \"Bayesian meta-learning for the few-shot setting via deep kernels.\" Advances in Neural Information Processing Systems 33 (2020)\n\n\n\n> Q3. If the reason not to use ResNet-12 is the computation burden, then the biggest problem may be how to make Bayesian inference more tractable.\n\nThank you for raising this question. We agree that the biggest problem is to make Bayesian inference more tractable. In the context of Bayesian meta-learning, the main challenge is to compute the integral for posterior inference. Our derivation of mean-field approximation targets to resolve this problem, as we provide a closed-form expression for variational parameters. In practice, our inference method is more efficient than the Gibbs sampling one while reaching close classification accuracy. \n\n\n\n> Q4. Also, another problem of interest is to figure out why bayesian meta-learning with complicated training steps cannot perform well on few-shot learning, even if equipped with nice theoretical properties. Since this paper does not answer these questions, I think the value of this paper is restricted.\n\nWe acknowledge that the theoretical explanation for sub-optimal results for Bayesian methods in meta-learning is not explored in our paper. However, we would like to emphasize that the main focus of this paper is the modified logistic-softmax likelihood, whose good theoretical property invites potential applicability in various domains beyond few-shot learning. Nonetheless, we agree that this is a thrilling and urgent topic and we would love to do some future work for exploration. \n\n\n\n> Q5. In theorem 3.3, $a$ is said to be the mean function with no restriction, so the mean of different sample points can be different. However, in line 444 in the appendix, all different samples have the same mean, can the author explain this?\n\nWe really appreciate the reviewer for pointing out this typo and thank the reviewer for the chance for us to clarify it here. The mean function does not have to be constant w.r.t. different sample points. We have proofread the proof of Thm3.3, and our analysis is still correct when the mean function is not constant.\n\n\n\n> Q6. How does your method compare with non-Gaussian processes like neural process [1]? It seems that neural process is more flexible and efficient than Gaussian process.\n\nThank you for reminding us. The neural process is a promising and efficient method, and we will add some discussion on the extension to non-Gaussian processes such as the neural process in the revised paper."}]

Author rebuttal:
We extend our sincere appreciation to all reviewers for their time, effort, and insightful feedback. We are encouraged by their recognition of the significance of our work in introducing an effective modification to control the confidence of logistic-softmax, uncovering novel theoretical properties and broad applicability, deriving an efficient mean-field inference method, conducting comprehensive numerical experiments, and maintaining clear and concise writing.

In the following, we respond meticulously to each of the reviewers' comments. Our aim is to ensure that we address all the concerns and offer clarity and reassurance where needed. Should any additional questions arise, we invite reviewers to engage in further discussion. Once again, we express our gratitude for your time and dedication in reviewing our work.

Meta-review: The submission revisits the logistic-softmax likelihood in the context of Bayesian meta-learning for few-shot classification. It proposes a modification which incorporates a temperature coefficient to address the lack of confidence inherent to the approach. It also sidesteps intractability issues using a mean-field variational inference approximation to make inference more efficient. Test episode query accuracies are presented for CUB, mini-ImageNet, and mini-ImageNet → CUB for 1-shot and 5-shot classification, along with expected calibration error (ECE) and maximum calibration error (MCE).

Reviewers note that the submission is clear and well-written (iEEK, JdA9) and presents a broadly-applicable idea (iEEK) showing promising results (iEEK) and backed up by a solid theoretical analysis (iEEK, L2f9, JdA9).

The most salient reviewers concerns are:

* Reviewers iEEK and JdA9 would like to see the idea applied in settings beyond Bayesian meta-learning for few-shot classification. The authors explain that they chose this setting as it is "one of the initial applications for the original logistic-softmax likelihood in the literature" and that they will clarify this point in the submission. This addresses Reviewer JdA9's concern.
* Reviewer iEEK notes that the proposed approach does not consistently outperform competing approaches in terms of MCE. The authors acknowledge that this is the case for one dataset, which could be the result of a mismatch between the validation set and outliers in the test set. Their response sounds reasonable to me: I think what's more important here is the overall pattern of strong results.
* Reviewer L2f9 would like to see a comparison against Distribution Calibration, and Reviewer 1tNK points out that Bayesian meta-learning approaches still standardize around the four-layer convnet architecture, whereas other few-shot classification works have moved on to ResNet-12 and other higher-capacity architectures. The authors explain that Bayesian methods are not necessarily competitive in terms of accuracy, but they have uncertainty calibration capabilities which other few-shot classification approaches do not. To me, this is a reasonable justification for the scope of competing approaches to compare against.
* Reviewer 1tNK is also concerned that the original Prototypical Networks paper presents stronger numbers than what is reported in the submission. The authors clarify that they report results that are presented in the OVE paper, whose first author is also the first author of the Prototypical Networks paper.

Overall all reviewers are in favor of acceptance (albeit weakly), and the authors' response addresses the reviewers' concerns adequately from my vantage point.
Decision: Accept (poster)