SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Revisiting Logistic-softmax Likelihood in Bayesian Meta-learning for Few-shot Classification
Tianjun Ke†; Haoqun Cao†; Zenan Ling†, Feng Zhou†§
†Center for Applied Statistics and School of Statistics, Renmin University of China
$^{\ddagger}$ School of EIC, Huazhong University of Science and Technology
{keanson, caohaoqun2007, feng.zhou}@ruc.edu.cn, lingzenan@hust.edu.cn
Meta-learning has demonstrated promising results in few-shot classification (FSC) by learning to solve new problems using prior knowledge. Bayesian methods are effective at characterizing uncertainty in FSC, which is crucial in high-risk fields. In this context, the logistic-softmax likelihood is often employed as an alternative to the softmax likelihood in multi-class Gaussian process classification due to its conditional conjugacy property. However, the theoretical property of logistic-softmax is not clear and previous research indicated that the inherent uncertainty of logistic-softmax leads to suboptimal performance. To mitigate these issues, we revisit and redesign the logistic-softmax likelihood, which enables control of the a priori confidence level through a temperature parameter. Furthermore, we theoretically and empirically show that softmax can be viewed as a special case of logistic-softmax and logistic-softmax induces a larger family of data distribution than softmax. Utilizing modified logistic-softmax, we integrate the data augmentation technique into the deep kernel based Gaussian process meta-learning framework, and derive an analytical mean-field approximation for task-specific updates. Our approach yields well-calibrated uncertainty estimates and achieves comparable or superior results on standard benchmark datasets. Code is publicly available at https://github.com/keanson/revisit-logistic-softmax.
Meta-learning refers to the ability to quickly learn a new task given a set of training tasks that share a common structure [13; 15; 30]. This concept is critical for achieving human-like performance computationally with little data, and recent algorithms have shown success in few-shot classification and regression problems [15; 18]. When dealing with limited data, it is essential to analyze robust meta-learning methods that properly deal with uncertainty which is inevitable in the context [1]. Some existing works have suggested that the Bayesian inference mechanism provides a principled way to address this issue, moving towards robust meta-learning with uncertainty quantification [7; 27; 38].
The Bayesian framework combines a prior distribution and a likelihood function that models the data distribution. By performing posterior inference on parameters, it provides a natural framework for capturing inherent model uncertainty [8; 23]. Gaussian process (GP) is a Bayesian nonparametric method that places a prior distribution on functions rather than parameters, ensuring better expressiveness [3]. The GP prior is defined by a covariance function which can be parameterized by deep neural networks, commonly referred to as a deep kernel [35]. GP with a deep kernel has demonstrated state-of-the-art performance in few-shot regression problems [20], where posterior inference admits a closed-form solution due to the conjugacy between the Gaussian likelihood and the GP prior.
$\S$  Corresponding author.
However, there are several challenges in GP-based meta-learning with deep kernels for classification tasks. To begin with, unlike the regression scenario, the widely-used softmax likelihood does not lead to conjugacy for GPs, making posterior inference intractable. To overcome this issue, several approximate inference methods have been proposed, such as label regression with Gaussian likelihood [20], One-vs-Each (OVE) approximation of softmax [27], and the logistic-softmax likelihood [9]. The logistic-softmax likelihood replaces the exponential function in the softmax likelihood with a logistic function, ensuring conditional conjugacy after data augmentation [9]. However, previous research suggests that the logistic-softmax function tends to exhibit inherent lack of confidence, leading to suboptimal performance in few-shot classification tasks [27]. Additionally, most GP-based meta-learning models employ Gibbs sampling for posterior inference, which can be computationally demanding for achieving convergence [26]. Furthermore, the coordination of task-level posterior inference and meta-level optimization for deep kernel methods remains an open question.
In this paper, we bridge these gaps by redesigning the logistic-softmax likelihood and deriving a mean-field approximation for posterior inference. In contrastive learning, adding a temperature parameter to rescale the logits in softmax is a prevalent approach [4; 12; 33]. Motivated by this concept, we introduce the temperature parameter to the logistic-softmax likelihood to control its inherent prior confidence. Moreover, we discover that softmax can be viewed as a special case of logistic-softmax and logistic-softmax induces a larger family of data distribution than softmax. To the best of our knowledge, this theoretical property has not been covered in the literature. Furthermore, we apply the modified logistic-softmax likelihood to GP based meta-learning for few-shot classification. Since the logistic-softmax likelihood enables a conditional conjugate GP model, we naturally derive an analytical mean-field approximation for task-specific updates. Compared to the existing literature [27], this variational inference method is more efficient than Gibbs sampling. We empirically show that our mean-field approximation achieves comparable results to the Gibbs sampling in practice. We also contribute to the coordination problem of bi-level optimization in Bayesian meta-learning methods.
Specifically, our contributions are as follows: (1) we introduce the logistic-softmax with temperature and prove its unique limiting behavior, which may have broad applicability across diverse machine learning domains; (2) we theoretically and empirically show that softmax can be viewed as a particular case of logistic-softmax and logistic-softmax induces a larger family of data distribution than softmax; (3) we derive an analytical mean-field approximation for task-level posterior inference with redesigned logistic-softmax through data augmentation; (4) we demonstrate the effectiveness of the redesigned logistic-softmax through few-shot classification accuracy and uncertainty calibration on several benchmark datasets; (5) we contribute to the coordination of task-level inference and meta-level optimization that appears in Bayesian meta-learning, which may provide insights for future research.
Meta-learning involves learning from a set of tasks in order to acquire knowledge and generalize to new tasks [5; 20]. In this work, the  $t$ -th task comprises a dataset  $D_{t}$  consisting of both support and query data, denoted as  $S_{t}$  and  $\mathcal{Q}_t$  respectively. The support data  $S_{t}$  consists of a limited number of samples ( $N$ -shot), represented as pairs of input-output  $(x_{n},y_{n})$ , while the query set  $\mathcal{Q}_t$  typically includes a larger number of samples. During training, models are exposed to various tasks sampled from the dataset  $D$  to learn and extract valuable information across different contexts. Subsequently, when presented with an unseen task  $t^*$ , the model aims to leverage the acquired knowledge from both training data and the support set  $S_{t^*}$  to make accurate predictions on the query set  $\mathcal{Q}_{t^*}$ .
Gaussian Process Classification is a probabilistic framework used for solving classification problems. At its core, a GP is a probability distribution over functions, where the values of the function  $f(x)$  evaluated at an arbitrary set of inputs have a joint Gaussian distribution with a mean vector and a covariance matrix determined by a kernel function. In the context of a  $C$ -class classification problem, separate GP latent functions  $\{f^c\}_{c=1}^C$  are employed to model the logits for each class. These latent functions represent the underlying mapping from the input space to the logit of each class, capturing the uncertainty associated with the classification decision. By modeling the class logits with GPs, it enables principled probabilistic inference and prediction.
Deep Kernel is proposed by Wilson et al. [35] which combines kernel methods and neural networks, harnessing the expressive power from both sides. This deep kernel extends traditional covariance kernels by integrating a deep architecture into the base kernel formulation. The deep kernel is defined
as  $k(\mathbf{x}, \mathbf{x}' \mid \boldsymbol{\theta}, \mathbf{w}) = k'(g_{\mathbf{w}}(\mathbf{x}), g_{\mathbf{w}}(\mathbf{x}') \mid \boldsymbol{\theta})$ , where  $k'$  represents the base kernel with parameters  $\boldsymbol{\theta}$ . To enhance flexibility, the inputs  $\mathbf{x}$  and  $\mathbf{x}'$  are transformed by a deep neural network  $g$  with weights  $\mathbf{w}$ . The deep kernel parameters consist of  $\boldsymbol{\theta}$  and  $\mathbf{w}$ . A notable advantage of deep kernels is their capability to learn metrics through data-driven optimization of input space transformation, in contrast to traditional kernels that often rely on Euclidean distance-based metrics.
3 Logistic-softmax with Temperature
In this section, we introduce a novel logistic-softmax function that incorporates a temperature parameter. We first emphasize two unique features of this function as the temperature approaches zero. We then demonstrate that logistic-softmax surpasses softmax as a versatile categorical likelihood theoretically and empirically. Specifically, we show that softmax can be viewed as a special case of logistic-softmax and logistic-softmax induces a larger family of data distributions than softmax. Fig. 1 briefly previews the relationship between the two likelihoods.
Figure 1: Diagram representing the features and relationship of logistic-softmax and softmax.
3.1 Definition of Logistic-softmax with Temperature
We define the logistic-softmax function with temperature as:
$$
p (y = k \mid \mathbf {f} _ {n}) = \frac {\sigma \left(f _ {n} ^ {k} / \tau\right)}{\sum_ {c = 1} ^ {C} \sigma \left(f _ {n} ^ {c} / \tau\right)}, \tag {3.1}
$$
where we assume  $C$  classes,  $f_{n}^{c} = f^{c}(\mathbf{x}_{n})$ ,  $\mathbf{f}_n = [f_n^1,\dots ,f_n^C ]^\top$ ,  $k\in [C]\coloneqq \{1,\ldots ,C\}$ ,  $\tau$  is the temperature parameter and  $\sigma (\cdot)$  is the logistic function as opposed to the exponential in the traditional softmax. While there are other possible choices for incorporating the idea of temperature, we find this particular form is the most straightforward one while preserving the desired conditional conjugate property of logistic-softmax as shown in later sections.
Although reminiscent of the softmax likelihood with temperature, the logistic-softmax likelihood displays distinct limiting behavior, illustrated by the following theorem.
Theorem 3.1. Denote the logistic-softmax function with temperature as  $\mathrm{LS}(\mathbf{f}_n,\tau)$ . Define  $I\coloneqq \{i:f_n^i >0\} \subset [C]$ , we have
$$
\lim  _ {\tau \to 0 ^ {+}} \operatorname {L S} (\mathbf {f} _ {n}, \tau) = \left\{ \begin{array}{l l} \boldsymbol {e} _ {c ^ {*}}, & \text {i f} \max  _ {c \in [ C ]} f _ {n} ^ {c} <   0 \text {a n d} c ^ {*} = \underset {c \in [ C ]} {\arg \max } f _ {n} ^ {c} \\ \frac {1}{| I |} \sum_ {c \in I} \boldsymbol {e} _ {c}, & \text {i f} \underset {c \in [ C ]} {\max } f _ {n} ^ {c} > 0 \end{array} \right.
$$
where  $\pmb{e}_c \in \mathbb{R}^C$  is the one-hot vector with a 1 in its  $c$ -th coordinate.
The proof is provided in Appendix I. On the one hand, the logistic-softmax likelihood converges to a one-hot vector when  $f_{n}^{c} < 0$  for all  $c \in [C]$ . On the other hand, the likelihood transforms into a uniform distribution over the classes corresponding to the indices of the positive elements if multiple elements of  $\mathbf{f}_n$  are greater than 0.
We interpret the theoretical result from two perspectives. First, the original logistic-softmax likelihood has been criticized for its lack of confidence [27]. However, with the introduction of temperature, the modified logistic-softmax likelihood can become excessively confident as the temperature approaches zero. This implies that we can effectively control the confidence of the logistic-softmax likelihood by adjusting the temperature, thus resolving existing issues. Second, when multiple components of the logits are positive, this likelihood exhibits remarkable fairness to all positive classes. This characteristic is profound and distinct since it enables adaptation to multi-label classification scenarios, where assigning an equal probability to each correct class is essential. Notably, to our knowledge, no existing likelihood can identify multiple correct classes, which limits the tools available for multi-label classification. Within this domain, the prevalent approach is to treat each class as a binary classification problem separately. Our discovery paves the way for a new paradigm in this domain, as
the logistic-softmax function allows for the identification of several positive labels simultaneously. However, in this work, we primarily concentrate on the application of logistic-softmax in Bayesian meta-learning and postpone the investigation of multi-label classification to future research.
3.2 Comparison of Logistic-softmax and Softmax with Temperature
We present several results demonstrating that logistic-softmax surpasses softmax as a versatile categorical likelihood function theoretically and empirically. We begin by giving a theorem that shows the logistic-softmax function converges to the softmax function pointwise.
Theorem 3.2. For all  $\mathbf{f}_n\in \mathbb{R}^C$ $\tau \in \mathbb{R}\setminus \{0\}$  and  $C_0\in \mathbb{R}$ , we have
$$
\lim_{C^{\prime}\to +\infty}\operatorname {LS}(\mathbf{f}_{n} - C^{\prime},\tau) = \operatorname {S}(\mathbf{f}_{n},\tau) = \operatorname {S}(\mathbf{f}_{n} - C_{0},\tau),
$$
where  $\mathrm{S}(\mathbf{f}_n,\tau)$  denotes the softmax function with temperature.
The proof is provided in Appendix II. This theorem shows that softmax is a translational invariant function that can be approximated by logistic-softmax with a sufficient negative shift of logits. We also note that logistic-softmax is not translational invariant. Empirically, we find that  $C' = 5$  provides an accurate approximation if  $\mathbf{f}_n$  is near zero. Now we present a stronger result indicating that logistic-softmax is more versatile than softmax with regards to modeling the data distribution.
Theorem 3.3. Assume the logits  $f^{c} \sim \mathcal{GP}(a, k^{c})$ , where  $a$  is the mean function and  $k^{c}$  is the kernel function for each class  $c \in [C]$ . Denote  $\mathbf{y} = [y_{1}, \ldots, y_{N}]^{\top}$  as the random label vector of  $N$  given points. Suppose  $a \in \mathcal{A}$  and  $k^{c} \in \mathcal{K}$ , where  $\mathcal{A}$  and  $\mathcal{K}$  are two function classes. Define  $\mathcal{F}(\ell \mid \mathcal{A}, \mathcal{K})$  as the family of the marginal distribution  $p(\mathbf{y} | \mathbf{X}, a, k^{c})$  induced by  $a \in \mathcal{A}$  and  $k^{c} \in \mathcal{K}$  on given points  $\mathbf{X} \in \mathbb{R}^{N \times p}$  with a likelihood function  $\ell$ . Under mild condition on  $\mathcal{A}$ , we have
$$
\mathcal {F} (\mathrm {S} \mid \mathcal {A}, \mathcal {K}) = \mathcal {F} (\mathrm {S} \mid \mathcal {K}).
$$
$$
\mathscr {F} (\mathrm {S} \mid \mathscr {A}, \mathscr {K}) \subset \mathscr {F} (\mathrm {L S} \mid \mathscr {K}).
$$
The proof is provided in Appendix III. We elaborate on this theoretical result below. The first equation indicates that data distribution induced by softmax is translational invariant with regards to  $a$ , while the one by logistic-softmax is not. Recall that Theorem 3.2 implies softmax is translational invariance w.r.t. its input variable. Intuitively, as the location parameter  $a$  is integrated due to translational invariance, the family of the marginal distribution of  $y$  corresponding to softmax is only decided by the kernel function  $k$ , while the family for logistic-softmax has an additional parameter  $a$ . With this intuition, it should be rather straightforward to understand our second result. Informally, the second result states that logistic-softmax has a larger family than softmax. In other words, logistic-softmax possesses a stronger capability in modeling the data distribution for each class.
Next, we delve deeper into the relationship between logistic-softmax and softmax. For comparison, we present a toy classification task with three classes ( $C = 3$ ) and one sample belonging to the first class ( $y = 1$ ). We place a standard normal prior on  $f_{1}$  and  $f_{2}$  and clamp  $f_{3}$  at  $-100$  for ease of visualization. We plot the likelihoods in Fig. 2 where we observe a distinct pattern between softmax and logistic-softmax. Although both likelihoods become more confident (likelihood output changes at a faster rate) as the temperature parameter decreases, logistic-softmax exhibits unique probability patterns when both  $f_{1}$  and  $f_{2}$  are greater than 0, while softmax gives the same sets of borderline parallel to  $f_{1} - f_{2} = 0$  regardless of location. We present the zoom-in plots to further explain the difference in Fig. 2. These plots demonstrate that when both  $f_{1}$  and  $f_{2}$  are a bit less than 0, logistic-softmax accurately approximates softmax in every temperature. Indeed, the pattern of softmax in the whole plane is the same as the pattern that appeared in quadrant 3 of logistic-softmax. This observation supports the idea behind Theorem 3.2 and Theorem 3.3, which shows that logistic-softmax is capable of modeling any data distribution that can be modeled by softmax, as long as we adjust the logits to the negative domain or put a sufficiently small negative mean on  $f$ . Furthermore, as Fig. 2 shows, the softmax likelihood is unable to model the data distribution of logistic-softmax with positive logits. Therefore, we conclude that logistic-softmax is more powerful than softmax for its expressiveness in modeling categorical distribution.
Figure 2: Plot of  $p(y = 1|\mathbf{f})$  where  $f_{3}$  clamped to -100. We provide separate zoom-in plots of softmax and logistic-softmax in the 2nd row. In the upper-right area (where all  $f_{1}$  and  $f_{2}$  are greater than 0), the logistic-softmax function exhibits unique probability patterns that softmax cannot model. In the bottom-left area (where all  $f_{1}$  and  $f_{2}$  are smaller than 0), logistic-softmax accurately approximates softmax for every temperature and every location all at once.
3.3 Further Discussions of Potential Applications
As the logistic-softmax function possesses more expressiveness than the softmax function, we include a short discussion of its potential applications in various domains. To begin with, it can be applied to Gaussian process classification tasks, including class-imbalanced scenarios [37], active learning [40], and time-series data analysis [6]. In this case, logistic-softmax brings desired conditional conjugacy to make inference tractable and provides additional flexibility in data modeling than softmax as indicated in our paper. Moreover, the logistic-softmax function can be a great choice in modern Bayesian methods, such as Bayesian neural networks and neural network Gaussian processes although further adaptation is needed. Furthermore, logistic-softmax might be capable of replacing softmax beyond the Bayesian domain since we prove its flexibility over the softmax function. For example, as the logistic-softmax function captures positive signals for multiple classes, it may have prospective advantages in scenarios like multi-label classification [14] and multi-label contrastive learning [39], ushering in new paradigms thanks to its unique properties.
4 Logistic-softmax with Temperature in Bayesian Meta-learning
Logistic-softmax was initially proposed to address the conjugation issue that arises in multi-class Gaussian process classification. As the Bayesian framework is advantageous in uncertainty calibration, some researchers focus on adapting the multi-class Gaussian process to few-shot classification tasks, where Bayesian meta-learning is one of the prevalent paradigms. In specific, Snell & Zemel [27] attempts to apply original logistic-softmax to Bayesian meta-learning but fails to provide efficient inference and optimal result. To follow this research line and validate the advantages of logistic-softmax with temperature, we utilize the logistic-softmax likelihood in the GP-based meta-learning framework. Additionally, we use the data augmentation technique to derive a fully analytical mean-field inference method for this model.
4.1 Framework of Bayesian Meta-learning
We review the Bayesian meta-learning framework used in the research line of MAML [7], DKT [20], and OVE [27]. To begin with, Bayesian meta-learning commonly utilizes a hierarchical structure that includes an inner loop and an outer loop. Conventionally, task-specific parameters are updated in the inner loop and task-common parameters are updated in the outer loop [7]. For efficiency, DKT proposed to replace the inner loop with a marginal likelihood computation that integrates out task-specific GP function for each task [22]. Then, the task-common hyperparameter  $\Theta$  of the deep kernel can be learned by maximizing the marginal likelihood across all tasks in the outer loop. More specifically, denote the input support and query data of task  $t$  as  $D_t^x$ , the target data as  $D_t^y$ .  $D^x$  and
$D^{y}$  are the collections of these datasets over all tasks. The marginal likelihood takes the form
$$
p \left(D ^ {y} \mid D ^ {x}, \boldsymbol {\Theta}\right) = \prod_ {t} \int p \left(D _ {t} ^ {y} \mid D _ {t} ^ {x}, \phi_ {t}\right) p \left(\phi_ {t} \mid \boldsymbol {\Theta}\right) d \phi_ {t}, \tag {4.1}
$$
where  $\phi_t$  is the task-specific parameters of task  $t$ . To obtain an analytic integral of Eq. (4.1), DKT used a GP prior for  $p(\phi_t \mid \Theta)$ . The integral can be computed analytically in regression cases since the model likelihood is Gaussian as opposed to classification cases where the conjugacy is broken since the likelihood is Bernoulli or categorical. However, we can use the logistic-softmax likelihood to obtain a GP model that is conditional conjugate after data augmentation. Thus, an efficient inference method is needed for the augmented model. We describe our task-level inference in next section.
4.2 Task-level Bayesian Inference
In this subsection, we introduce how an efficient task-level Bayesian inference method is developed. For a specific task, we denote the (support and query) input dataset as  $\mathbf{X} = [\mathbf{x}_1,\dots ,\mathbf{x}_N]^\top$  with label dataset  $\mathbf{y} = [y_{1},\ldots ,y_{N}]^{\top}$  where  $y_{n}\in [C]$ ,  $N$  and  $C$  are the number of observations and classes respectively. The multi-class GP classification model includes latent GP functions for each class  $\mathbf{f} = [f^{1},\dots ,f^{C}]^{\top}$  where  $f^{c}\sim \mathcal{GP}(a^{c},k^{c})$ ,  $a^c$  is mean function and  $k^c$  is kernel for  $c$ -th class.
We utilize the logistic-softmax function with temperature in Eq. (3.1) to model the likelihood for the multi-class GP classification. Inspired by Galy-Fajou et al. [9], three sets of auxiliary latent variables are augmented to expand the logistic-softmax likelihood to obtain a conditional conjugate model for each task, including Gamma variables  $\lambda$ , Poisson variables  $\mathbf{M}$ , and Pólya-Gamma variables  $\Omega$ . Given the GP priors on  $f^c$ , we obtain the augmented joint density (proof provided in Appendix IV):