Maximum Entropy Information Bottleneck for Confidence-aware Stochastic Embedding
Authors: Sungtae An
Nataraj Jammalamadaka
Eunji Chong
Venue: Submitted to ICLR 2023 (2023)
Keywords: Deep learning
Computer vision
Stochastic embedding

Abstract: Stochastic embedding has several advantages over deterministic embedding, such as the capability of associating uncertainty with the resulting embedding and robustness to noisy data. This is especially useful when the input data has ambiguity (e.g., blurriness or corruption) which often happens with in-the-wild settings. Many existing methods for stochastic embedding are limited by the assumption that the embedding follows a standard normal distribution under the variational information bottleneck principle. We present a different variational approach to stochastic embedding in which maximum entropy acts as the bottleneck, which we call "Maximum Entropy Information Bottleneck" or MEIB. We show that models trained with the MEIB objective outperform existing methods in terms of regularization, perturbation robustness, probabilistic contrastive learning, and risk-controlled recognition performance.

Under review as a conference paper at ICLR 2023
MAXIMUM ENTROPY INFORMATION BOTTLENECK FOR CONFIDENCE- STOCHASTIC EMBEDDING
Anonymous authors Paper under double-blind review
ABSTRACT
Stochastic embedding has several advantages over deterministic embedding, such as the capability of associating uncertainty with the resulting embedding and ro- bustness to noisy data. This is especially useful when the input data has ambigu- ity (e.g., blurriness or corruption) which often happens with in-the-will wild settings. Many existing methods for stochastic embedding are limited by the assumption that the embedding follows a standard normal distribution under the variational information bottleneck principle. We present a different variational approach stochastic embedding in which maximum entropy acts as the bottleneck, which we call "Maximum Entropy Information Bottleneck" or MEIB. We show that models trained with the MEIB objective outperform existing methods in terms regularization, perturbation robustness, probabilistic contrastive learning, and risk-controlled recognition performance.
INTRODUCTION
Mean Branch f/
11
Backbone Encoder f
Task
Decoder fo Loss
Covariance Branch f
Σ
Figure
Stochastic embedding framework.
Stochastic embedding mapping of an input the mapped regions of similar inputs are placed nearby. Unlike deterministic embedding, where = f() is a point in R.,, stochastic embedding can represent the input uncertainty, such as data corruption or ambiguity, by controlling the spread of probability density over manifold Oh et al. (019))
random variable p(2(x) R" in which
Figure depicts typical stochastic embedding framework with the neural networks parameterized by 0. Input æ is mapped to Gaussian distribution (2; H4, Σ) by a stochastic encoder that con- sists of backbone feature extractor f followed by two separate branches f/ and fg each of which predicts the 11 and Σ While the covariance matrix Σ, in prior work well as in this paper, is assumed to be diagonal where f outputs a D-dimensional vector, would straightforward to ex- tend it full covariance matrix, for instance, using Cholesky decomposition Dorta et al.) (2018). Embeddings sampled from this Gaussian are then consumed by a decoder fo for the downstream task, e.g., classification.
Majority of leading methods for stochastic embedding Oh et al. 2019); Chang et al. (2000);
Sun et al. (2020); Chun et al. (202T): [i al. (202Tb) built upon the variational informa- tion bottleneck (VIB) principle Alemi et al. (2017) where the stochastic encoder p(z[z) is regu- larized by Kullback-Leible (KL) divergence, <L(p<>|>)]lr
where p(2(z) (2; M,, Σ) and
r(2) = (2; 0, I) in general. This effectively impels the embeddings to be close to standard normal distribution, which an explicit assumption that may not always hold true.
We the terms fh (x) and (z) interchangeably with f (f8 (z:) and fo (x)) respectively.

---

Under review as a conference paper at ICLR 2023
(a) Deterministic
(b) Stochastic, KLD
(c) Stochastic, ME
Embedding space characteristies. Each color represents a class of data. The color-filled shapes refer to the deterministic or the mean point of stochastic embeddings. The ellipses around the shapes depict the standard deviation of stochastic embeddings. The circles and the diamonds represent training and testing data, respectively. The solid lines are the decision boundaries learned.
Figure 2:
Furthermore, Bütepage et al. (2021) showed that the standard variational autoencoder (VAE) trained with KL divergence from the standard normal prior Kingma & Welling (204) fails correlate the latent variance with the input uncertainty; the variance decreases with the distance to the latent means of training data, which contrary expectation. Since VAE special case of an unsu- pervised variant of VIB, this phenomenon also holds for VIB; our experiments show VIB assigns smaller variance to more uncertain inputs (see the supplemental Section A). Motivated by this find- ing, we explicitly use the variance (entropy) as confidence indicator rather than measure of input uncertainties and encourage the model to assign larger variance to more certain inputs.
In this paper, we propose Maximum Entropy Information Bottleneck (MEIB) to lift such constraints of using a fixed prior and instead use the conditional entropy of the embedding HZ as the only grincipleJaynes]
regularization. Based the maximum entropy principle Jaynes (1957), postulate that stochastic uncertainty is best represented by the probability distribution with the largest entropy. By maxi- mizing H (Z))), ), the embedding distribution is promoted be more random, pushing for broader coverage in the embedding space, with a trade-off on the expressiveness of Z. about target Y. The resulting distribution is also the one that makes the fewest assumptions about the true distribution of data Shore & Johnson 1980).
Figure 2 depicts our intuition; (a) deterministic encoders would learn embeddings "just enough" to classify the training samples unless any regularization technique, such as a margin loss, is con- sidered. It would be vulnerable to small changes test inputs.
(b) the embedding distribution by typical stochastic encoders (e.g.. VIB) trained with the KL divergence regularization will tend to cover fixed prior. Note that it is generally difficult to pick true prior distribution. Also, unnecessary restrict the embedding distribution be within specific bound. (c) with MEIB, on the other hand, by maximizing the conditional entropy of the stochastic embeddings, we would have a better regularization effect it makes the area secured by the embedding distribution for the given input as broad as possible.
The key contributions of MEIB to the previous stochastic embedding methods are summarized as follows:
While it provides a comparable regularization in handwritten digit classification, MEIB outperforms existing approaches in the challenging person re-identification task with three popular datasets.
MEIB shows significantly better perturbation robustness compared to VIB in handwritten digit classification.
MEIB performs better than VIB when used a probabilistic contrastive learning frame- work.
Providing reliable confidence measurements, MEIB shows an outstanding risk controondo recognition performance in digit classification and person re-identification tasks.
2

---

Under review as a conference paper at ICLR 2023
2 RELATED WORK
Research on stochastic embeddings has gained popularity in recent years. Oh et al. 2019) proposed similarity-basee method based the pairwise soft contrastive loss and the VIB principle, which has been later applied to human pose embedding [unn et al.] (2020), cross-modal retrieval Chun al. (2021), and ordinal embedding Li al. (2021b). Chang et al. (2020) proposed stochastic embedding for face recognition using softmax loss and KL divergence regularization without relying on similarity. All of these methods assume embedding distribution be unit Gaussian. Unlike these approaches, Probabilistic Face Embedding (PFE) Shi & Jain (2019)
Stochastic Embeddings
turns deterministic embeddings into Gaussians with fixed mean by training post hoc network to maximize the mutual likelihood of same-class embeddings. Follow-up research on PFE includes its extension to triplets Warburg et al. (202T) and for spherical space [i et al. (2021a). However, this line of work is limited to fixed embedding mean. The work most closely related to ours is DistributionNet Yu et al. (019) which introduced entropy-based regularization and inspired Yang al. (2021) for their uncertainty-awee aware loss. However, these methods put a margin to bound the total entropy rather than maximizing it as we do.
Maximum Entropy Maximum entropy is a general principle that has already been widely adopted in designing machine learning models, including supervised and reinforcement learning Zheng et al.) (2017); Ahmed al. (2019). Pereyra al. (2017) used the negative entropy of the class prediction distribution, - -H (po ]), as regularization term in the loss function to prevent over-confident predictions. In reinforcement learning, the maximum entropy framework encourages diverse explo- rations in both on-policy and off-policy settings Mei et al.) (2019); Haarnoja et al. (2018); Han & Sung (202T). However, most previous work, the entropy regularization has been applied at the decision levels, the distribution of class or action predictions. In this work, on the other hand, we focus on the entropy of the stochastic embedding of inputs.
MAXIMUM ENTROPY INFORMATION BOTTLENECK
MEIB Objective The maximum entropy principle Jaynes 1957) states that the current state of knowledge about the given system is best represented by the probability distribution with the largest entropy (Shore & Johnson (1980). By combining this with our hypothesis, the goal is learn an encoding Z that is maximally expressive about Y while maximizing the expected amount of infor- mation (entropy) about Z X:
max 1(2, Y; s.t. H(Z)X:0) 2 He.
9
Introducing a Lagrangian multiplier 3, we have the maximization objective:
JMERR [[2, Y: + (Z
(2)
where 3 controls the trade-off between the predictiveness and the spread of Z given X. Using the lower bound suggested by Alemi et al. (2017) for the first term of the objective I [Z2 Y), we have
f dx dy de p(x))(())))))))))) log ((vl=)
- BB dx dz p(x) p(z)2) log logp(2/2)) L.
(3)
This lower bound L can be computed by approximating the joint distribution p(z, v) p(x) p(yla))
using the empirical data distribution p(æ, y) = N/ Σ 82,, (x) 6y (v) (Alemi et al. 2017
V d= log ((u.l2) p(z(xx)) (ogke))oogp(((())) n=1
LR
Σ
(4)
Consequently, the loss function to minimized is:
LMEBB
Σ
Ea-p( logg(un/))) (2H(
(5)
3

---

Under review as a conference paper at ICLR 2023
where use the typical reparameterizatioo trick Kingma & Welling (2014) backpropagate gra- dients through the sampling & ~ p(z(zn). We use a single sample of 2 by default unless it is specified.
Relationship to VIB
The minimization loss function of VIB (Alemi et al. 2017 is:
N
CVIB
Ex-p(e)
(6)
N
log ((2n ) Ba-p(=/
+ 3 (-3H(p( r(1)
(7)
N n=1
N
Ba-p(s/a,) (on 13 (2 -MEB
(8)
N
where H)((Z]),)) r (2)) is the cross-entropy of r(2) relative to the distribution pl p(((zn)
N(ZIf (x, )), which is given by
D + d=1
H(p(Z(an),r(2)) = (1 (ln(2 +
where 1*g = 16 (3).) and E. Di) Pf a diagonal covariance, respectively detailed derivation can be found in the supplemental material Section Therefore, the VIB loss function is an upper bound of the MEIB loss function with a positive value B.
Confidence Measure of MEIB MEIB encourages obvious inputs that be easily classified to take broader embedding areas by assigning larger entropy. On the contrary, the inputs closer to other classes would have smaller entropy reduce the chance of misclassification according the loss function. Therefore, we adopt the conditional entropy H (Z])z) as our confidence measure for the input x, which is given by Cover Thomas (2006):
1 (n(2 Ee
D 2 Ee (1 (1+In22))
H H
(9)
Specifically, we use the dimension-win wise average conditional entropy H(Z[x)/D to achieve a damensionnaagootiic confidence measure.
EXPERIMENTAL RESULTS AND DISCUSSION
In this section, we show experimental results for various tasks to demonstrate the effectiveness of MEIB in terms of the regularization, perturbation robustness, and confidence measure. All compu- lational experiments were implemented using PyTorch Paszke et al. (2019) 1.9 with Python 3.7 on workstation equipped with NVIDIA * RTX 2080 Ti graphic card. Please refer the supplemental material Section H for the implementation and training details for each experimental task.
4.1 DIGIT CLASSIFICATION
First, we evaluate MEIB on handwritten digit classification as the simplest form benchmark task. We use the QMNIST dataset Yadav & Bottou (2019) to utilize its larger set of 60,000 testing data compared to 10,000 of those in the original MNIST dataset. while has almost identical training data. We adopt the same architecture employed in Alemi et al. 2017); the backbone encoder f is multilayer perceptron (MLP) with two fully-connected (FC) layers of 1024 hidden units with ReLU activations. Both f/ and f/ was applied after f We found that applying batch normalization (BN) layer at the end of f improves the performance MEIB with a noticeable gap (see the supplementary material Section F The decoder 1 is an FC layer with the sofumax function that outputs p(y (z) ten classes.
FC layer of D hidden units where the exponential (exp) function

---

Under review as a conference paper at ICLR 2023
Table 1: QMNIST test set error rate (%)
Method
D
D 256
Deterministic
4.64 0.43
1.71 0.04
Dropout
4.08 + 0.29
1.62 + 0.02
VIB
3.29 0.32
1.75 + 0.03
MEIB
3.95 0.41
1.76 1 0.05
VIB (12 MC samples)
3.21 + 0.32
1.45 + 0.02
MEIB (12 MC samples)
3.31 + 0.35
1.48 + 0.04
Using the same architecture, we compare the following embedding approaches with MEIB: deter- ministic baseline, dropout (Srivastava al. (2014). and VIB. The deterministic baseline represents the typical usage of neural network models without stochasticity. By omitting the variance estima- tor module/ f (z), the embedding of input is deterministically given by
f (z)). Dropout one of the most popular regularization methods for neural networks; thus, it is considered the first benchmark regularization method to compare Alemi et al. (2017); Ghiasi et al. (2018). We use the same deterministic model, but dropout is applied with the probability of 0.5 during the training time. Unlike MEIB and VIB, both deterministic models were trained only with the cross-entropy loss. We set 3 a/D D where a is equal 0.01 for VIB, 0.1 for MEIB with D D = 256. Please refer to the supplementary material Section F for the hyperparameter study. All models trained with five different random seeds, and report the mean and standard deviation for each performance metric.
and 1 for MEIB with
Regularization Effect Table I shows the classification results by each method the QMNIST test set. Specifically, we compared the methods with two different embedding sizes, D = 2 and 256. With D = 2, the stochastic methods performed better than the deterministic ones regardless of the usage of dropout. Using the larger embedding size of D = 256, on the other hand, the deterministic model trained with dropout performed the best while the others have very similar performance considering the standard deviation. However, using 12 Monte Carlo (MC) samples of 2, the stochastic methods outperformed the deterministic ones with larger gap than the single sample case. MEIB provides a reasonable amount of regularization comparable to VIB.
Perturbation Robustness The test time noise/perturbation robustness of deep neural network model is an important aspect, especially when the model is considered to be deployed for a real- world application. We evaluate the robustness of models toward adversarial examples an alterna- tive form of perturbation robustness evaluation because models that are weak to adversarial examples might be vulnerable to not only intended attacks but also unexpected noise in test time Gilmer et al. (2019). Since the primary purpose of MEIB and the other compared methods is not defense against strong adversarial attacks, we use the Fast Gradient Sign Method (FGSM) Goodfellow et al. (2015):
I = x + c sign( V [0, (0,,a,)))))))))
(10)
where 7 is the crafted adversarial example, e is a scale factor of the perturbations, and J is the target loss function, e.g., cross-entropy loss. The FGSM is often regarded as a weak adversary but is still widely used as a first benchmark adversary method due its simplicity. We report evaluation results with stronger adversaries in the supplemental Section G for interested readers. We crafted the adversarial examples from the QMNIST test set using the FGSM with e E [0.0,0.5] with the step size of 0.5 on all models (D = 256) trained with different random seeds for each method. We used 12 MC samples of for the stochastic methods, MEIB and VIB.
Figure shows the misclassification rate of each method toward the different strengths of the FGSM perturbations. MEIB is more robust than the other methods with significant gaps. Furthermore, the error rate of MEIB increases very slowly with the increasing strength of perturbations until about e == On the other hand, VIB is more vulnerable than both deterministic and dropout baselines, typically with more severe perturbations. It might be because chose the target model with the best performance with the clean dataset, which is a rational choice, for all methods. Thus, a VIB model trained with different 33 might yield better robustness, but it still would be difficult to close
5

---

Under review as a conference paper at ICLR 2023
MEIB
VIB
Dropout
Deterministic
1:75
1.50
Error Rate (%)
Eraor Rate (%)
1.25
60
1.30
0.75
0.50
0.25
0.30
0.0
0.1
0.2
0.3
0.4
0.5
10
20
30
40
50
60
70
80
90
Rejection Rate
(a) Perturbation Robustness
(b) Risk-controlled Classification
Figure 3: Performance comparisons the QMNIST dataset.
the gap with MEIB. Please refer to the supplemental Section C for the perturbation robustness with different a values.
Risk-controlled Classification In many real-world application scenarios, it would be favorable refuse any decision instead of making false prediction when the model is not confident about the input. For doing this. we need a way to correctly estimate the confidence, or the uncertainty. of inputs to MI. models. Using the estimated confidence, we may reject the inputs with insufficient con- fidence; it is called risk-controlled recognition (Shi & Jain (2019). We evaluated the risk-controlled classification performance on the QMNIST test set by rejecting the inputs by the confidence esti- mated by each method. Similarly the confidence measured by MEIB (Section 3). we empirically found that the mean of variance vector from fø (z) of VIB is proportional to the confidence. For the deterministic and dropout baselines, as a proxy measure of confidence Raajann et al. (2017). We set the rejection rate from 0% to 95% with a step size of 5%. A single sample of risk-controlled classification performance by all methods with D = 256. MEIB outperformed all other methods across the most range of the input rejection rate. Specifically, the error of MEIB dropped more than half of the initial value after rejecting 10% of uncertain inputs and reached about 0.1% when half of the inputs were rejected.
the L2 Ly-norm of the embedding vector for each input
used for MEIB and VIB. Figure 3b shows the
Embedding Distribution Figure f depicts the embedding space learned by VIB and MEIB with D = 2. It shows that each embedding distribution by MEIB takes as much area possible depend-
25
P
-2
-5.0
-75
(a) VIB
(b) MEIB
Figure 4: 2D embedding space learned for the QMNIST dataset. The ellipses represent the standard deviation of the stochastic embeddings for subset of training data.

---

Under review as a conference paper at ICLR 2023
Table 2: Performance (AP) comparison of HIB with different main framework methods
N = 2
N 3
D
Test Data
VIB
MEIB
VIB
MEIB
Clean
0.955 + 0.004
0.959 0.004
0.950 + 0.003
0.954 0.003
Corrupted
0.840 + 0.004
0.836 + 0.010
0.844 = 0.007
0.842 + 0.005
Clean
0.980 + 0.001
0.982 + 0.002
0.980 = 0.003
0.984 + 0.002
3
Corrupted
0.861 + 0.003
0.864 + 0.004
0.900 = 0.006
0.898 + 0.003
Clean
0.991 + 0.001
0.993 + 0.001
0.990 = 0.002
0.991 + 0.001
Corrupted
0.888 = 0.003
0.894 + 0.010
0.912 = 0.004
0.913 + 0.003
Clean
1.000 0.000
1.000 + 0.000
1.000 = 0.000
1.000 0.000
100
Corrupted
0.931 = 0.005
0.936 + 0.004
0.952 = 0.004
0.968 = 0.002
ing on its location from the decision boundaries, which is consistent with our hypothesis. On the other hand, every embedding by VIB has small standard deviation and thus covers much smaller area in both dimensions (axes) than those of MEIB. It would be reasonable to consider increasing a of the prior distribution .V (0, 2² I) used in VIB modeling to achieve a similar effect of entropy maximization by MEIB. However, even VIB with very large variance priors still performs much worse than MEIB for both aspects of perturbation robustness and risk costccontollled classification (see the supplemental Section D). It implies that MEIB increases of inputs in an adaptive way while VIB tries to match the given prior distribution.
4.2 HEDGED INSTANCE EMBEDDING
Oh et al. 2019) proposed the hedged instance embedding (HIB), metric learning method that explicitly models the input uncertainty in the stochastic embedding space. HIB utilizes the VIB objective with the probabilistic contrastive learning framework. The authors also proposed new dataset, called N-digit MNIST, basically images of N adjacent MNIST digits. We examined the HIB framework with the MEIB objective instead of VIB simply by replacing the KL divergence term in the original HIB loss function with the negative conditional entropy of embeddings, keeping all the other aspects of the neural network model same, including the hyperparameters. Please refer to the supplementary material for the architectural details. Table 2 original HIB with the VIB objective and the MEIB-variant by the average precision (AP) on the test set. MEIB-variant outperformed the original HIB in every case with the clean test set, except the very high-dimensional embedding of D 100, where both performed well. For the corrupted test data, on the other hand, the original HIB performed slightly better in low-dimensional embedding D = scenarios, and both performed comparably with D 3. However, MEIB outperformed with a digens-dime embedding D = 4 and D = 100 in both N = 2 and 3 cases. In most applications, it is very uncommon to use such a small size of embeddings with two or three dimensions, except for direct visualization of data relationships An et al. (2020). Consequently, it suggests that the MEIB objective would better fit the contrastive learning framework of HIB than the VIB objective upon using reasonable size of embedding dimensions.
reports the performance of the
4.3 PERSON RE-IDENTIFICATION
Person re-identification (ReID) is an important computer vision task that is utilized in various ap- plications, including intelligent security and surveillance systems Gong et al. (20IT). Unlike typical image classification tasks, the objective of the person RelD task to find the ranked matches of pedestrian images captured across multiple non-overlapping cameras (Cen et al. (202T). Person ReID is challenging due to image-level corruptions and appearance changes, including occlusions Ye et al. (2021). Therefore, it is critical to have embedding method robust to noises and ca- pable of confidence measuring for potential cosk-contrlllabiiity. For evaluating those aspects, we used three datasets popularly employed in person ReID literature: Market- 150 Zheng et al. (2015), MSMTI7We. et al. (2018), and LPW [Song et al. (0188). For LPW, we used a quarter subset of by selecting every fourth frame of each identity due to the limited computational resource.
7

---

Under review as a conference paper at ICLR 2023
Table 3: Performance on the person RelD datasets
Dataset
Method
mAP (%)
Rank- (%)
Rank-5 (%)
Rank-10 (%)
Baseline
75.65 0.21
90.48 + 0.45
96.47 0.06
97.66 + 0.15
Market- 1500
DistNet
74.61 + 0.06
90.10 + 0.29
96.25 + 0.23
97.61 + 0.26
PFE
76.49 0.23
90.48 + 0.51
96.53 0.11
97.75 = 0.20
DUL
77.12 0.19
90.09 - 0.17
95.84 = 0.15
97.29 0.18 0.18
MEIB
79.67 + 0.15
92.14 0.16
96.79 0.16
97.83 + 0.09
Baseline
39.69 0.16
69.31
0.31
82.66 + 0.17
86.73 + 0.06
MSMT17
DistNet
38.16
68.79 0.21
82.44 + 0.32
86.67 + 0.28
PFE
0.14
70.62 0.24
83.68 + 0.17
$7.50 0.20
DUL
38.70
0.22
80.98 0.50
85.38 = 0.25
MEIB
44.77 + 0.39
85.34 0.25
88.80 + 0.24
Baseline
34.82 0.33
52.76 0.80
67.06 + 0.94
73.32 0.87
LPW
DistNet
#2.37=0.22
50.25 + 0.20
65.36
72.02 = 0.89
PFE
$3.35:0.33
50.36 0.41
64.99
71.69 = 0.76
DUL
33.41 0.65
49.41
63.90
70.13 0.83
MEIB
37.64 + 0.41
54.85 0.52
69.05 0.58
75.20 0.49
For all methods compared in RelD experiments, we used ResNet50) He al. (2016) as the backbone encoder f8. The mean estimator f of an FC layer with 512 hidden units followed by BN layer used for all methods. For the variance estimator fø used an MLP of FC512-BN- ReLU-FC5 exp for MEIB and followed the original architecture proposed by the authors for each method, except that used 512 hidden units for their FC layers. We used 1 for MEIB and the recommended hyperparameter values by the authors for each method.
We compare the performance of MEIB with the following baseline methods not only from person ReID literature but also from previous work on face recognition, which shares similar task charac- teristies: DistributionNet (DistNet) Yu al. (2019), Probabilistic Face Embedding (PFE) Shi & Jain (211)) and Data Uncertainty Learning (DUL, equivalent to VIB) et al. (2020). In addition, we compare the deterministic baseline of the same architecture without the variance estimator fø- We also tried HIB for the Rell) task, but we could not achieve meaningful results. All models were trained for the classification task with a softmax classifier of fe predict the true identity labels. In testing time, the Euclidean distance between every pair of gallery and query images is calculated using the deterministic embeddings or the mean of stochastic embeddings f (x) of them to rank the pairs For PFE, we use the negative mutual likelihood score (MLS) used in the original work of PFE as the distance metric.
We first evaluated the methods without considering the input confidence and risk control. Table summarizes the results where we report the mean average precision (mAP) Schüttze et al. (2008) and cumulative matching characteristics (CMC) curve Gray al. (2007) at rank-1, rank-5. and rank- 10. MEIB outperforms all the others by from 2 up to 6.6 percentage points of mAP throughout the datasets considered. It implies that maximizing the conditional entropy of embeddings in MEIB has better regularization effects than the other methods.
The importance of risk-controlled recognition is more emphasized in the person ReID task due to the cost of misidentifying a person; can have a serious societal impact, such as falsely tagging someone as criminal. Considering a real-world application of risk-controlled ReID model de- ployment, a realistic situation is that we can prepare a well-curated clean set of gallery images in advance and expect that arbitrary query/probe images with potential corruptions will be given after model deployment. Thus, we kept the original test gallery images for each dataset and evaluated the methods with varying amounts of rejected test query images. The query images are sorted using the confidence estimated by each method: MEIB and the deterministic model use the same approaches described in Section 4.1 DistNet and DUL use the mean of variance vector from (x), while PFE the mean reciprocals of the variance vector elements their confidence measures. Then, the first R% of low- con-coiide images are removed from evaluation. For each remaining query image, the entire gallery images ranked by the distance, and the evaluation metrics calculated.
8

---

Under review as a conference paper at ICLR 2023
MEIB
DUL
-w.. PFE
DistNet
Baseline
10
60
(86)
mA/ (%0
60
40
90
75
(a) Market-1500
(b) MSMT17
(c) LPW
Figure 5: Risk-controlled person RelD performance.
Figure 5 shows the risk-controlled identification performance of the methods for each dataset. While MEIB starts with the best performance among all methods, shown previously, the amount of performance improvement at lower rejection rates is more significant than the other methods in most cases. Furthermore, the gap between MEIB and the other methods is kept the most range of the rejection rate. It confirms that MEIB provides an effective confidence measure and risk-control capability. F shows the example query images for each dataset (except MSMT17 prohibited by the license) with the associated confidence (conditional entropy) value. There are conspicuous trends: (1) the low-confident images are mostly blurry while the high-confident images are relatively clearer, (2) the persons in the low-confident images are often occluded by another object or person, and (3) the most persons in the low-confident images wear clothes with an achromatic color without patterns while those in the high-confident images wear clothes with vivid colors or patterns such as stripes. This would another evidence that MEIB provides reasonable confidence measurements.
5 CONCLUSION
In this work, we presented MEIB, novel framework that produces stochastic embeddings dis- tributed with the maximum entropy. MEIB provides a regularization effect and robustness toward adversarial noise by securing the maximum area in the embedding space, leading better classi- fication performance. Moreover, the experimental results in digit classification and person RelD tasks showed that MEIB enables an effective risk-controlled recognition by providing reliable con- fidence measurements. While the experiments in this work utilized somewhat elementary neural network backbone encoders, would be straightforward to combine MEIB with more sophisticated architectures to yield further classification performance improvement.
3.271 3.493
3.571
3.574 3.586 3.593 6.757 6.918 6.932 6.993 7.044 7.398
(a) Market- 1501
3.025
3.317
3.319
3.413
3.416
3.445 6.294
6.317
6.440
6.613
6.704
6.896
(b) LPW
Figure 6: Example query images that MEIB is least confident (left six) and most confident (right six) from each ReID dataset. The number above each image is the confidence measured by MEIB the dimension-we wise average entropy.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sravanti Addepalli, Vivek BS, Arya Baburaj, Gaurang Sriramanan, and R Venkatesh Babu. Towards achieving adversarial robustness by enforcing feature consistency across bit planes. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1020-1029, 2020.
Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. In International conference on machine learning, Pp- 151-160. PMLR, 2019.
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information bottleneck. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Sungtae An, Shenda Hong, and Jimeng Sun. Viva: sumi-super visualization via variational autoencoders. In 2020 IEEE International Conference on Data Mining (ICDM), pp. 22-31. IEBE, 2020.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of se- curity: Circumventing defenses adversarial examples. In International conference on machine learning, pp. 274-283. PMLR, 2018.
Judith Bütepage, Lucas Maystre, and Mounia Lalmas. Gaussian process encoders: Vaes with reliable lapent uncertainty, In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 84-99. Springer, 2021.
Jie Chang.
Zhonghao Lan, Changmao Cheng, and Yichen Wei. Data uncertainty learning in face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 5710-57111 17100-77,7777 2020.
Minghui Chen, Zhiqiang Wang, and Feng Zheng. Benchmarks for corruption invariant person identification. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus. Probabilistic embeddings for cross-modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8415-8424, 2021.
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory (Wiley Series in Telecom- munications and Signal Processing). Wiley-Interscienee USA, 2006. ISBN 0471241954.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter- free attacks. In International conference on machine learning, pp. 2206- 2216. PMLR, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Ccomputer Vision and Pattern Recognition, pp. 248 248.255. IEEE, 2009.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4690-4699, 2019.
Garoe Dorta, Sara Vicente, Lourdes Agapito, Neill DF Campbell, and Ivor Simpson. Structured uncertainty prediction networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5477-5485, 2018
Mengyue Geng, Yaowei Wang. Tao Xiang, and Yonghong Tian. Deep transfer learning for person re-identification. arXiv preprint 161 1.05244, 2016.
10

---

Under review as a conference paper at ICLR 2023
Golnaz Ghiasi, Tsung-Y Yi Lin, and Quoc V Le. Dropblock: A regularization method for convolu- tional networks. Advances in neural information processing systems, 31, 2018.
Justin Gilmer, Nicolas Ford, Nicholas Carlini, and Ekin Cubuk. Adversarial examples are a natural consequence of test enror in noise. In International Conference on Machine Learning, pp. 2280- 2289. PMLR, 2019.
Shaogang Gong, Chen Change Loy, and Tao Xiang. Security and surveillance. In Visual analysis of humans, pp. 455-472. Springer, 2011.
lan J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Douglas Gray, Shane Brennan, and Hai Tao. Evaluating appearance models for recognition, reac- quisition, and tracking. In Proc. IEEE international workshop on performance evaluation for tracking and surveillance (PETS), volume 3, pp. 1-7. Citeseer, 2007.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning. pp. 1861-1870. PMLR, 2018.
Seungyul Han and Youngchul Sung. A max-min entropy framework for reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Edwin T Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620, 1957.
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. In 3rd Interna- tional Conference on Learning Representations (ICLR), 2015.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the International Conference on Learning Representations, 2014.
Shen Li, Jianqing Xu, Xiaqing Xu, Pengcheng Shen, Shaoxin Li, and Bryan Hooi. Spherical con- fidence learning for face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 15629-15637, 2021a.
Wanhua Li, Xiaoke Huang, Jiwen Lu, Jianjiang Feng. and Jie Zhou. Learning probabilistic ordinal embeddings for uncertainty-awert regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13896-13905, 2021b.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL forum? d
Jincheng Mei, Chenjun Xiao, Ruitong Huang. Dale Schuurmans, and Martin Müller. On princi- pled entropy exploration in policy optimization. Proceedings of the 28th International Joint Conference on Arlificial Intelligence, pp. 3130-3136, 2019.
Maria-Irina Nicolae, Mathieu Sinn, Minh Ngoc Tran, Beat Buesser, Ambrish Rawat, Martin Wis- tuba, Valentina Zantedeschi, Nathalie Baracaldo, Bryant Chen, Heiko Ludwig, Ian Molloy, and Ben Edwards. Adversarial robustness toolbox v1.2.0. CoRR, 1807.01069, 2018. URL fttps://arxe
wotg/pdf//B// 01069
Scong Joon Oh, Kevin P. Murphy, Jiyan Pan, Joseph Roth, Florian Schroff, and Andrew C. Gal- lagher. Modeling uncertainty with hedged instance embeddings. In 7th International Conference Learning Representations, (ICLR), 2019.

---

Under review as a conference paper at ICLR 2023
Adam Paszke, Sam Gross, Francisco Massa, Adam Lever, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. Regulariz- ing neural networks by penalizing confident output distributions. In Workshop Track Proceedings of International Conference on Learning Representations, 2017.
Rajeev Ranjan, Carlos D Castillo, and Rama Chellappa. L2-constrained softmax loss for discrimi- native face verification. arXiv preprini arXiv: urXiv:/703.095000 09507, 2017.
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence adam and beyond. In 6th International Conference on Learning Representations, ICLR, Conference Track Proceedings, 2018.
Hinrich Schütze, Christopher D Manning, and Prabhakar Raghavan. Introduction to information retrieval, volume 39. Cambridge University Press Cambridge, 2008.
Yichun Shi and Anil K Jain. Probabilistic face embeddings. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2002-6911,2015
John Shore and Rodney Johnson. Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy. IEEE Transactions information theory, 26(1)266(11 1980.
Guanglu Song, Biao Leng, Yu Liu, Congrui Hetang, and Shaofan Cai. Region-based quality estima- tion network for large-scale person re-identification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199 2014.
Jennifer Sun, Jiaping Zhao, Liang-Chite Chich Chen, Florian Schroff, Hartwig Adam, and Ting Liu. View-invariant prohabilistic embedding for human pose. In European Conference on Computer Vision, pp. 53-70. Springer, 2020.
Feng Wang, Xiang Xiang, Jian Cheng, and Alan Loddon Yuille. Normface: 1.2 hypersphere embed- ding for face verification. In Proceedings of the 25th ACM international conference on Multime- dia, pp. 1041-1049, 2017.
Frederik Warburg, Martin Jørgensen, Javier Civera, and Søren Hauberg. Bayesian triplet loss: Un- certainty quantification in image retrieval. In Proceedings of the IEEE/CVF International Con- ference Computer Vision, pp. 12158-12166 2021.
Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian. Person transfer gan to bridge domain gap for person re-identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
Chhavi Yaday and Leon Bottou. Cold case: The lost mnist digits. In Advances in Neural Information Processing Systems, volume 32, 2019.
Gengcong Yang, Jingyi Zhang, Yong Zhang, Baoyuan Wu, and Yujiu Yang. Probabilistic modeling of semantic ambiguity for scene graph generation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12527-12536, 2021.
Mang Ye, Jianbing Shen, Gaojie Lin, Tao Xiang, Ling Shao, and Steven CH Hoi. Deep learning for person re-identification: A survey and outlook. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Tianyuan Yu, Da Li, Yongxin Yang, Timothy M Hospedales. and Tao Xiang. Robust person identification by modelling feature uncertainty. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 552-561, 2019.
12

---

Under review as a conference paper at ICLR 2023
Guanhua Zheng, Jitao Sang, and Changsheng Xu. Understanding deep learning generalization by maximum entropy. arXiv preprint arXiv:1711.07788, 2017.
Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang. Jingdong Wang. and Qi Tian. Scalable person re-identification: A benchmark. In Proceedings of the IEEE international conference on computer vision, pp. 11 1116-118 124,, 2015.
Kaiyang Zhou and Tao Xiang. Torchreid: A library for deep learning person re-identification in pytorch. arXiv preprint 1910: 10093, 2019.
A UNCERTAINTY ESTIMATION OF VIB
Figure [ shows the QMNIST test set samples with the lowest and highest variance averaged over fea- ture dimension, i.e.. 1o Ed= og estimated by VIB. It shows that VIB assigns relatively smaller variances for uncertain inputs and higher variances for clear inputs. Moreover, Figure S shows the risk-controlled classification (see Section 4.1 in the main paper) performance of VIB with the as- cending and descending order of the average variance. In the ascending order, the samples with small variances are rejected first. Similarly, the high variance samples are rejected first in the descending order case. If we assume that the variance proportionally represents the 'uncertainty' associated with the inputs, we will reject the samples with the higher variance first. In this case of the descending order rejection, the risk-controlled classification did not work as expected; the more samples were rejected, the worse performance was achieved. On the contrary, the risk-controlled classification worked correctly using the ascending order, regardless of its performance. Therefore, we can infer from these results that it is more rational to utilize the variance oß in VIB to represent a 'confident' area/interval rather than the 'uncertainty".
0.584 0.613
0.625 0.629
0.632
0.634
0.636
0.639
0.640
0.642
0.643
0.648
4
d
9
9
4
1
1
(a) Low average variance
1.046
1.024
1.009
1.006
1.005
1.003
0.994
0.993
0.991
0.989
0.988
0.987
5
3
&
4
7
5
6
3
7
8
O
2
(b) High average variance
Figure 7: Sample QMNIST images with the uncertainty estimated by VIB.
10
Ascending
Desrending
Error
2
10
20
30
40
50
60
70
80
90
Rejection Rate (%)
Figure 8: Risk-controlled classification performance of VIB with the ascending and descending order of the average variance.
13

---

Under review as a conference paper at ICLR 2023
B CROSS-ENTROPY (ppo(z
Let two D-dimensional multivariate Gaussian distributions po(2|>mn) = A2 Mo, Σe) and r(2) = N (2; Mo, Στ) where 14g (2x,), Ee = للوه = f (2,), diagonal covariance, Ho 0, zero mean vector, and Στ = I, an identity covariance matrix, respectively. Then, H (p(2((n), r(z)) the cross-entropy of r(2) relative to p(z(an), is given by
H(p(2(am), - / p(z)zn) (e((=))(((()))) Inr(z) dz
(11)
1 / M22 Mo, Ex) (1n((2)) (
2
In (z -o0 2T (2 - (00) dz
(12)
1 + (2- T) 0)) dz
Σa) (l
(13)
2
1
(((:) En) (ll((2)+ + + dz
(DInn(2))
(14)
1 (ln((2T) / N(=)Ho,
( Eo) Ez) + (1 Ho [x)(=T=) 2)
(15)
2
1
(D In(2T) + Ez-po(HI [2]2])
(16)
2
1
Dln(2x) + Esnpo(sten)
Σ
(17)
2
2d
d==
1 Dln(27)
E Ex-po(s(za) [
(18)
2
Dln(2 +
Σ
E Ex~3o (x(an) [Rd] Var Varg=po(elaan) [2d] 12a))
(19)
2
d=1
1 (n
+
Σ
(
(20)
2
d=1
where each of 2d, Ma and oh a is the d-th dimension of the corresponding vector. Note again that Ee - = (xx), diagonal covariance, i.c., each dimension of : is independent to each other.
C PERTURBATION ROBUSTNESS BY DIFFERENT &
Figure shows the misclassification rates against the FGSM perturbations for MEIB trained with different values. It shows more consistent trend that larger provides more robustness for the given perturbation strength, unlike the VIB case that shows saddle-shaped curves in Figure 9bb It is reasonable because larger (thus, a larger (3) value encourages the model assign larger entropy and thus secure a larger area for the distribution of given input z, which leads to larger margins from the decision boundary.
D VIB WITH LARGE-VARIANCE PRIORS
We trained and evaluated VIB models with different values of a in their prior distribution. Figure TO shows the embeddings of VIBs trained with 2-dimensional bottleneck and different values of a. By visual inspection, it may seem getting similar to those of MEIB shown in Figure 4b as or increases. On the other hand, Figure [] controlled classification performance by VIB trained with the different values a while keeping the other configurations same in Section 4 including the 256-dimensional bottleneck. It shows that the adversarial robustness is improved with increasing o, until or = 10, but becomes worse again with much larger 100, while all of them were still worse than MEIB with significant
presents the perturbation robustness (toward FGSM attacks) and risk-
14

---

Under review as a conference paper at ICLR 2023
0.3
0.4
100
80
Error Rate (%)
60
40
10-6
10-4
10-1
10-1
20
(a) MEIB
(b) VIB
Figure Adversarial robustness with different a values.
gap. Moreover, the rist-controlled classification performance has little difference with different values.
On the other hand, the magnitude of as estimated by MEIB for QMNIST dataset are in range below than those by VIB with o 5, shown in Table It provides an insight that MEIB increase of inputs in a more effective way.
Table 4: || o ||a estimated by each method
Method
Mean
SD
Min
Max
VIB (o =)
0.38
0.08
0.18
0.53
VIB (a = 5)
1.80
0.39
0.56
4.03
VIB (o 10)
3.22
0.84
0.70
9.33
VIB (a 100)
18.77
8.01
2.04
81.42
MEIB
1.10
0.28
0.45
1.95
E RANK-1 ACCURACY FOR RISK-CONTROLLED PERSON REID
Figure 12 shows the risk-controlled CMC rank-1 accuracy for each method on each ReID dataset.
The results were obtained from the same experiments described in Section 4.3 in addition to the risk-controlled mAP in Figure 5 of the main paper. It confirms again that MEIB provides the most effective confidence measure and risk-control capability compared the other methods over the most range of the rejection rate.
F HYPERPARAMETER STUDY
Figure 13 shows the performance of VIB and MEIB for different values of a and the usage of the BN layer the end of the 1 branch in the QMNIST digit classification task with D = 256. For VIB, the BN layer is beneficial only for the small values of a (0 ~ 10-4), and the best result was common
achieved without using the BN layer at a 0.01. For MEIB, on the other hand, phenomenon that using the BN layer at the end of f noticeably improves the performance across all values, while the best performance was obtained with a = 1 using the BN layer. Furthermore, Figure shows that using the BN layer at the end of the f/ branch also significantly improves the performance of MEIB on all datasets considered in the person ReID task, commonly with the best performance
concrete study about the effect of BN in MEIB, which we leave as our future work, it can be assumed that a = with BN layer at the end f/ is a reasonable default configuration for these tasks.
1. While need
15

---

Under review as a conference paper at ICLR 2023
BG
100
-10
-20
-10
10
(a)
(b)
5
300
30
200
10
00g
-100
20
30
-20
8
-40
-30
-20
-10
10
20
30
-700
-100
100
200
(c) 07
10
(d) o 100
Figure 10: 2D embedding space learned for the QMNIST dataset by VIB with different of the prior distributions. The ellipses represent the standard deviation of the stochastic embeddings for subset of training data.
100
MEIB
VIB (c- 31
VID (+r
B0
(ar
1.50
VIB (d 5)
.....VIB (or 101
Error Rate (%)
VIB (a 100)
1.25
100)
1.00
0.50
20
0.25
0.1
0.2
0.4
0.5
0
20
40
60
80
Rejection Race (%6
(a) Perturbation Robustness
(b) Risk-controlled Classification
Figure 11: Performance comparisons on the QMNIST dataset.
G STRONGER ADVERSARIAL ATTACKS EXPERIMENTS
While the FGSM evaluated in Section 4 is popular first-choice baseline method, it is often inef- fective compared to multi-step attack methods. Although the main contribution this paper is not an adversarial defensive method, evaluated MEIB and other baseline methods the following stronger adversaries for a more concrete adversarial robustness analysis: the multi-step Projected Gradient Descent (PGD) Madry al. 2018), two variants of Auto-PGI (APGD), one based on the
16

---

Under review as a conference paper at ICLR 2023
MEIB
DUL
AFE
DistNet
Ease line
97.5
90
70
Rank-1 (%)
Rank-1 (%)
95.0
Pank:1
80
60
92.5
90.0
70
87.5
20
30
50
60
80
10
30 40
60
70
80
90
Rejection Rate
Rejection Rate
Rejection
(a) Market-1501
(b) MSMT17
(c) LPW
Figure 12: CMC rank-1 accuracy for risk-controlled person RelD)
NoBN
UseBN
1.75
1.70
12MC Error Rate (5)
1.65
12MC Error Rate
2.0
1.60
1.55
1,50
1.45
'10
10"
101
10-1 10-1 10-1 10-2
10- 10-4
10-3
10-1
10-2
10-1
10a
10
a
a
(a) VIB
(b) MEIB
Figure 13: Hyperparameter comparison for QMNIST (lower is better).
cross-entropy loss (APGDcE) and one with the difference of logits ratio (DLR) loss (APGDplR). and AutoAttack (AA), which is an ensemble of those two APGDs Croce & Hein (020). We used the projection on the -ball radius for all attacks where we set 0.3. PGD, we used 1000 steps (PGD-1000) with the step size Estep = 0.01 and 100 random restarts. For APGDs and AA, Expectation over Transformation (EOT) Athalye et al. (0188) was used with an average over 20 times of the gradient computations for more effective attacks the stochastic models, MEIB and VIB. We used Adversarial Robustness Toolbox (ART) Nicolae et al. (2018) to leverage its PGD implementation and the AA implementation provided by the authors including APGDs.
Table 5) shows the evaluation results on class-balanced 10, 000 samples of the QMNIST test set. MEIB outperformed all the other baseline methods across all different types of adversarial attacks. Although MEIB was also effectively fooled by these multi-step and adaptive PGD attacks, it per- forms better than other non-adversarially trained models such Mixup evaluated with the PGD attacks on the original MNIST Latase(Addepa) et al. (2020). It would be possible to achieve better robustness once MEIB trained with adversarial training methods.
H IMPLEMENTATION AND TRAINING DETAILS
QMNIST Digit Classification All models were trained for 200 epochs with batch size of 100. Adam optimizer Kingma & Ba (015) was used with 31 0.5, 92 = 0.999, and the learning rate of 0.0001, which is decayed by the factor of 0.97 every other epoch. The exponential moving average of the model parameters with decay factor of 0,999 was tracked during the training, and the final
17

---

Under review as a conference paper at ICLR 2023
NoBN
UseBN
80.0
42.5
75.0
40.0
MAP
30
72.5
37.5
35.0
25
67.5
10
10
10"
10"4
10
10
10
(a) Market-1500
(b) MSMT17
(c) LPW
Figure 14: MEIB hyperparameter comparison for person RelD (higher is better).
Table Classification Accuracy (%) with different adversarial attacks on the QMNIST test samples
Method
Clean
PGD-1000
APGDcE
APGDDLR
AA
Deterministic
98.39 0.11
1.13 0.04
0.00
0.00 0.00
0.00 0.00
Dropout
98.45 0.08
1.09
0.01 1.011
0.00 0.00
0.00
VIB
98.34 0.14
0.07
0.89
0.10
0.02
MEIB
98.38 + 0.15
2.42
6.79 = 1.40
1.60 1
averaged parameters were used at test time. We trained and evaluated all methods with five different random seed values. All pixel values of input digit images were rescaled [-1, 1]. The embedding
size of D 256 used by default for all experiments unless was specified. For the stochastic methods, MEIB and VIB, we used 12 MC samples of 2 for the adversarial robustness experiment and single sample for the risk-controlled classification experiment.
Hedged Instance Embedding We used the same architecture in the original HIB work Oh et al. 2011)) except for some parts mentioned by the authors. The backbone encoder fo consists of two convolutional layers, 32 and 64 filters of 5 x 5 kernels, each followed by ReLU activation and a max-pooling with 2 x 2 kernels. Each f and f an FC layer of hidden units, and BN layer is attached at the end of f Both MEIB and VIB variants of the HIB models were trained over 500,000 iterations with a batch size of 128. We followed the same batching strategy used Oh et al. (2019) ensure had enough number of both positive and negative pairs of images. Adam optimizer Kingma& & Ba 2015) was used with 31 = 0.5, B2 = 0.999 and the learning rate of 0.0001.
imple- ment the methods and conduct the experiments. Except for PFE and DistNet, all other models were initialized with the ResNet50 parameters pre-trained on the ImageNet dataset Deng al. 2009 and trained for 60 epochs with a batch size of 32. Specifically, we adopted the two-stepped transfer learn- ing strategy Geng al. (2016). where the backbone encoders f were frozen for the first 5 epochs. AMSGrad optimizer Reddi et al. (2018) was used with the learning rate of 0.0003, which was re- duced by the factor of every 20 epochs. PFE and DistNet were initialized from the fully-trained deterministic baseline model and fine-tuned over another 60 epochs only for the parts specified in their original work. While the same optimizer was used, the initial learning rate was set as 0.0001, which was also reduced by the factor of 10 every 20 epochs. For all methods, each batch consists of 4 random identities and 8 random images for each identity. All input images were rescaled to 128 x 256, and random horizontal flipping with the probability of 0.5 was applied. We reported the mean and standard deviation of the performance for all models trained with five different random seeds.
Person Re-identification We utilized the Torchreid framework Zhou & Xiang (019)
18

---

Under review as a conference paper at ICLR 2023
I MISCLASSIHIED QMNIST SAMPLES
Figure 15 shows a subset QMNIST test images misclassified by MEIB after filter out 80% of the test set in the risk-controlled classification experiment in Section 4.1 Although MEIB was overconfident, i.c., assigned inappropriately high entropy values, for these uncertain images, each image has a plausible shape for the predicted class also.
+ 1 8 +
9
8.764
8.810
8.817
8.827
8.861
8.917
8.980
Figure 15: Sample QMNIST images misclassified by MEIB. Two digits above each image represent Label -Prediction and the number below each image is the conditional entropy estimated.
J L2 NORMALIZATION FOR MEIB AND VIB
L2 normalization feature embedding Ranjan et al. (2017) is widely used with the softmax loss Wang et al. (2017), angular margin- Banggin-based loss Deng et al. (2019). and contrastive loss Chen et al. (2020). We also trained and tested MEIB and VIB with normaazel samples to see its benefits on stochastic embeddings. Figure T6 shows the perturbation robustness and risk-controlled classi- fication performance, similar to Figure 3 of the MEIB and VIB models with and without L2 nor- malization of embeddings. MEIB performs much worse with the L2 normalization in terms of both perturbation robustness and risk-controlled classification. While VIB shows slightly improved ro- bustness with L2 normalization, its cist-controlled classification performance become worse than without the normalization shortly after about 30% of rejection rate. It could be difficult for stochas- tic embeddings, especially MEIB, to utilize their advantage in angular space since their magnitudes are not considered anymore. For example, the main intuition of MEIB is spreading out each input embedding as wide as possible to take more space within a decision region. However, after applying L2 normalization, only the angle of each feature vector contributes to the decision regardless of their magnitude in the embedding space.
MEIB
(L2
VIB
(L2 Norm.)
100
1.75
80
1.50
Error Rate (%)
60
1.00
40
0.75
0.50
20
0.25
0.00
0.0
0.1
0.2
0.4
0.5
20
40
60
80
Rejection Rate (%)
(a) Perturbation Robustness
(b) Risk-controlled Classification
Figure 16: Comparison of Ly noromalle MEIB and VIB the QMNIST dataset.
19


Reviews:
[{"review_id": "CIv7Ql_qLRX", "reviewer": "Reviewer_yBZK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript presents a new stochastic embedding method named Maximum Entropy Information Bottleneck (MEIB). Authors show that the well-known VIB is an upper bound of MEIB. Experimental results show that MEIB performs more robustly against adversarial attacks or other types of perturbation than VIB. The proposed method also outperforms the other four algorithms in the ReID missions on all three datasets. ", "summary_of_the_review": "A simple but effective method for stochastic embedding. Some results are not well justified. ", "strength_and_weaknesses": "Strength:\n1. A simple method but works well. It is also good that authors evaluated on a real ReID task. \n2. The manuscript is easy to follow. \n\nWeaknesses:\n1. In fact, it is easy to verify that the basic IB objective is also an upper bound of the so-called MEIB: \n$L_{IB}=I(Z,Y)-\\beta* I(Z,X)=I(Z,Y)-\\beta* [H(Z)-H(Z|X)]=[I(Z,Y) + \\beta* H(Z|X)] - \\beta* H(Z)= L_{MEIB}-\\beta* H(Z)>=L_{MEIB}$\nThe difference is just the absence of negative entropy of $Z$. \n\nAuthors have empirically demonstrated the effectiveness of the MEIB and its better performance of robustness as well as accuracy in specified scenarios. It would be more convincing if the authors present some theoretical explanations on why MEIB performs better than IB or VIB by just removing the regularization of H(Z) or the cross-entropy term. \n\nI can expect MEIB provide more accurate estimation on input uncertainty. However, it is hard for me to understand why MEIB is also robust to adversarial pertubations. Figure 3 is confusing to me, especially the VIB performs worst in adversarial robustness. This observation actually conflicts with previous literature (such as nonlinear IB or HSIC-bottleneck or even the classic VIB) which suggests that one can improve adversarial robustness by a regularization on H(X;Z) or even H(Z). \n\n2. The name (i.e., maximum entropy information bottleneck) of new method is also a bit confusing to me. I am not sure if the new method can be interpreted as a bottleneck or not. Usually, the IB or VIB involves a trade-off by maximizing a term, while also constraining another term. However, the new objective actually maximizes both I(Z,Y) and H(Z|X), i.e., there seems no bottleneck effect.\n\n3. In Section 4.2, can you validate that \"MEIB or its variant should perform better with much higher dimensional embeddings\"? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9L4Kpc4Kva", "reviewer": "Reviewer_HaaB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Stochastic embeddings are explored to benefit its capability of associating uncertainty for robustness to noisy data. Instead of using a standard normal distribution prior for variational information bottleneck principle (that typically use KL to learn), this work uses maximum entropy as bottleneck objective to learn the embedding. The proposed approach outperforms prior based approach in terms of robustness.", "summary_of_the_review": "This work is self-contained and well written with sufficient discussion to support the claim, yet the novelty is limited and motivation is not very clear. Before the questions are answered, I would suggest a borderline reject.\n\nAfter discussion and reviewing authors feedback, the feedback answers most of my questions, but did not address the weakness part especially did not convince me of the sufficient novelty. Thus I'll clear update my recommendation as reject. This is a good paper, but not good enough.", "strength_and_weaknesses": "*Strength*\n- It is good to see the relationship between MEIB and VIB mathematically to understand the proposed approach better.\n- MEIB significantly improve the robustness of the embeddings compared to VIB, dropout and deterministic\n- MEIB performs much better than peer approaches in person re-id datasets\n\n*Weaknesses*\n- The proposed approach is incrementally novel from VIB. Please clarify the highlight of the motivation. Even mathematically, MEIB and VIB look similar, the motivation matters to justify the novelty.\n- Classification performance wise, MEIB is on-par with VIB in MNIST, though the robustness is much better, the trade-off is not discussed. Instead it seems a binary decision instead of a dynamic trade-off between error rate and robustness by tuning any hyperparameter. A suggestion is It might be nice to re-consider the equation (8) to add a (\\lambda * prior) item for an unmature example to mitigate this gap\n- More to see clarity section", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2xSs9sSrR6", "reviewer": "Reviewer_PNBE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel cost function for representation learning/stochastic embedding of data. Specifically, they adapt the information bottleneck functional, which aims at maximizing $I(Y;Z)-\\beta I(X;Z)$ by removing the term corresponding to the entropy of $Z$. Essentially, the authors result at maximizing $I(Y;Z)+\\beta H(Z|X)$, where the former term ensures that the representation $Z$ is useful for the downstream task, and where the latter maximizes the stochasticity of the representation given the input. The authors show that their approach results in a higher adversarial robustness and risk-controlled recognition performance.", "summary_of_the_review": "The paper is well-written and proposes an interesting adaption of VIB with small, but non-neglibible novelty. The experimental evidence is good, but not fully convincing because some of the experimental setups are not clear or well-justified. Regarding the theoretical argumentations, I would appreciate a more in-depth discussion of the effects of $H(Z)$, $H(Z|X)$, and $I(X;Z)$ as regularization terms, and how MEIB builds on them.\n\n*EDIT*: After discussion with other reviewers, I think that the paper is not sufficiently novel to merit publication. I lowered my score accordingly.", "strength_and_weaknesses": "The paper is generally well-written and the math is easy to follow. The cost function is well motivated and its connection with existing methods (such as VIB) is explained. The paper is heavy on the experimental side, which I greatly appreciate. The experimental evidence is good (but see also below), and the supplementary material complements the paper nicely.\n\nRegarding the main weaknesses, I have to admit that the experimental setup and choices are not always entirely clear and that thus it is difficult to judge the significance of the experimental results. Further, some connections to the VIB and related methods are not fully clear and need to be expanded upon.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-UNRIwTfrt", "reviewer": "Reviewer_YBtS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a Maximum Entropy Information Bottleneck (MEIB), which is a different take on the information bottleneck compared to the well-known variational information bottleneck.\n\n\n", "summary_of_the_review": "A high quality paper that may spur new interest in information bottleneck techniques for deep networks. ", "strength_and_weaknesses": "I liked very much the idea, both the simplicity and the motivation behind it. I also found very intriguing the property of MEIB being a lower bound on VIB. The experiments are extensive, combined with deep ablation studies; they convincingly show that models trained with the MEIB objective outperform existing methods in terms of regularization, perturbation robustness, probabilistic contrastive learning, and risk-controlled recognition performance.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject