Mixture of Balanced Information Bottlenecks for Long-Tailed Visual Recognition
Authors: Yifan Lan
Cai xin
Jun Cheng
Shan Tan
Venue: Accepted by TMLR ()

Abstract: Deep neural networks (DNNs) have achieved significant success in various applications with large-scale and balanced data. However, data in real-world visual recognition are usually long-tailed, bringing challenges to efficient training and deployment of DNNs. Information bottleneck (IB) is an elegant approach for representation learning. In this paper, we propose a balanced information bottleneck (BIB) approach, in which loss function re-balancing and self-distillation techniques are integrated into the original IB network. BIB is thus capable of learning a sufficient representation with essential label-related information fully preserved for long-tailed visual recognition. To further enhance the representation learning capability, we also propose a novel structure of mixture of multiple balanced information bottlenecks (MBIB), where different BIBs are responsible for combining knowledge from different network layers. MBIB facilitates an end-to-end learning strategy that trains representation and classification simultaneously from an information theory perspective. We conduct experiments on commonly used long-tailed datasets, including CIFAR100-LT, ImageNet-LT, and iNaturalist 2018. Both BIB and MBIB reach state-of-the-art performance for long-tailed visual recognition.

Jun Cheng Published in Transactions on Machine Learning Research (06/2025)
Mixture of Balanced Information Bottlenecks for Long- Tailed Visual Recognition
Yifan Lan Huazhong University of Science and Technology
yifenlan@hust.du..o edu.n
Xin Cai
rincoi1998 daw.ail..mmmm.....mmmmmmmmmmmmom
Huazhong University of Science and Technology
jcheng24 @hust. edu.cn
Huazhong University of Science and Technology
Shan Tan
shantan Ghaust edu.cn
Huazhong University of Science and Technology
Reviewed on OpenReview: https: opwneeviee.........mm net/ forum? id- 9esALSuZGA
Abstract
Deep neural networks (DNNs) have achieved significant success in various applications with large-scale and balanced data. However, data real-world visual recognition are usually long-tailed, bringing challenges to efficient training and deployment of DNNs. Information bottleneck (IB) is clegant approach for representation learning. In this paper, we propose balanced information bottleneck: (BIB) approach, in which loss function re- re-balancing and self- dittliisillation techniques are integrated into the original IB network. BIB is thus capable of learning a sufficient representation with essential label-related information fully preserved for long-tailed visual recognition. To further enhance the representation learning capability, we also propose a novel structure of mirlure of multiple balanced information bottlenecks (MBIB), where different BIBs are responsible for combining knowledge from different net- work layers. MBIB facilitates end-to-end learning strategy that trains representation and classification simultaneously from an information theory perspective. We conduct experi- ments commonly used long-tailed datasets, including CIFAR100-LTT, ImageNet-L.T, and iNaturalist 2018. Both BIB and MBIB reach state-of-the-ard performance for long-tailed visual recognition.
1 Introduction
With the emergence of ImageNet, COCO, and other datasets, deep neural networks (DNNs) have achieved great success in various computer vision tasks such as image classification, object detection, and image segmentation. The success of deep learning largely due to the large-scale and balanced data in these tasks. However, rcal-world data are usually long-tailed (Buda al. 2018), which means that a few classes (head classes) occupy most. of the instances in the data, while most classes (lail classes) occupy only few instances. When using standard training methods (using cross entropy the loss function and instance balanced sampling) to train model on long-tailed training sct, the model usually performs well for the head classes but poorly for the tail classes, which brings a great challenge to the training and deployment of DNNs [Bda et al. 2018)
Recently, many methods have been proposed solve the problem of imbalanced data distribution. Among these methods, re-sampling and loss function re-balancing are the two most commonly used techniques. Re-sampling makes imbalanced dataset balanced by over-sampling or under-sampling (Buda et al. 2018

---

Published in Transactions on Machine Learning Research (06/2025)
Re-balancing methods include re-weighting (Buda et al. 2018 and logits adjustment [een et al. 2020)_ Kang et al. (2019) lound that although the classification performance of model trained by standard methods is worse than that of re-balancing methods, the obtained feature space is better. They proposed to decouple the training process into two stages: learning representation and learning classifier.
Ando & Huang 2017:
The decoupling training method leads to good results (Peileng el. al. 2023: Zhou et al., 2023 Zhang et al.
2021). Zhoong et al. (2021) showed that partial network parameters obtained the learning representation stage, such as parameters of Batch Normalization (BN), are not suitable for the second stage of learning classifier directly, which suggests that learning representation and learning classifier should not be regarded as two completely independent stages. ceaturee-ofFFxperrt (MoE) techniques involve the training of multiple neural networks, each specializing in handling distinct segments of a long-tailed dataset. BBN (Wang et al. 2020), RIDE (Zhou et. al. 2020) and SADE (Zhang et al. 2022) serve as representation attioss of MoE mot. However, the enhanced capabilities of MoE come at the cost of increased computational loads. Recently, Laurent et al. (2022) highlighted that, for long-tailed visual rocognition, the key is not just the classification rule but the ability to learn and identify correct fealures.
The Information Bottleneck (IB) is elegant approach for representation learning. It originates from the rate-distortion theory (Tishby 1999 Tishby & Zaslavsky 2015), and has made extraordinary progress in many tasks, such as image classification (Alemi et al. 2017). image segmentation (Luo et al., 2019), multi- view learning (Wang et al. 2021b), reinforcement learning (Goyal et al.) 2019) and so on. IB rethinks what a "good" representation is: for a given task, the best representation should contain sufficient and minimal amount of information. In the IB theory, sufficiency is achieved by maximizing mutual information between the representation z and the label y, and minimality is achieved by minimizing mutual information bet the input and the representation 2. By introducing Lagrange multiplier B, the IB method can be optimized by minimizing
I + (1;;2), [a; = (xz(P(a.)))) & Po).
(1)
where (a;; b) represents mutual information between and b, which measures the mutual dependence between a and b. DKL is the Heilbackk-eebbee divergence, and P. @ P& is the outer product distribution which assigns probability (a(a)- Pb( to each (a, b).
IB provides a new learning paradigm that naturally avoids over-fitting and enhances model robustness. To optimize the IB objective in problems involving high-dimensional variables, variants such as VIB (Alemi et al. 2017), Drop-IB (Kim et al., 2021), and Nonlinear-IB (Kolchinsky et al. 2019) are proposed. VIB is the first to indirectly optimize the variational upper bound of IB, and has become one of the most widely used IB variants for its simplicity and effectiveness.
Although IB has made lot of progress on balanced datasets, (2) is still affected by label distribution when applicd to long-tailed datasots. This limitation arises from the scarcity of tail class samples, making it difficult for DNNs to learn a sufficient representation z. Besides, mutual information is difficult to estimate in DNNs To solve these problems, we propose a novel balanced information bottleneck (BIB) method this study. We use the loss function re-balancing technique to alleviate challenges posed by label distribution. Concurrently, we implicitly optimize the information bottleneck objective by self diff-iitillation to reserve much information related to labels as possible in the process of information flow. To further enhance the representation learning ability, we introduce a novel framework of mixture of multiple balanced information bottlenecks (MBIB). MBIB is the first network to leverage multiple balanced information bottlenecks, each responsible for extracting knowledge from different network layers. By optimizing multiple IB objectives simultancously, MBIB ensures a comprehensive and effective representation learning process, leading to improved performance. We conduct experiments on benchmark datasets, including CIFAR100-LT, ImageNet- LT, and iNaturalist 2018.
The contributions of this study are summarized as follows:
1) Firstly, we propose a novel balanced information bottleneck (BIB) method to handle long-tailed data for real-word visual recognition.
2) Secondly, to the best of our knowledge, we propose the first network with a mixture of multiple bal- anced information bottleneck (MBIB), optimizing diverse IB objectives simultancously effectively learn representation and classifier in an end-to-end fashion.
2

---

Published in Transactions on Machine Learning Research (06/2025)
3)
Finally, methods achieve state-of-the-art performance among singlo-expert methods and compet- itive performance with MoF. methods while maintaining higher efficiency, as evidenced by the average classification accuracy across multiple benchmark datasets.
2 Related Work
2.1 Long-Tailed Tailed Visual Recognition
Re-sampling and loss function re-balancing are the most widely studied approaches in imbalanced classifica- tion. Re-sampling aims to construct balanced training set, including pver-sampling and under-sampling. Due to the low diversity of tail classes, over-sampligg often results in models over-fitting tail classes. Under- sampling discards part of samples in head classes, which damages the diversity of head classes and decreases model generalization performance. Kang et. al. (2019) proposed progressive sampling algorithm combin- ing over- and uuder-sampling to transition the distribution from an imbalanced distribution to balanced distribution. However, the problem of re-sampling still cxists.
Loss function re-balancing gives different weights to different classes or instances to re- re-balance the model at the level of the loss function. This strategy mainly includes re- weighting (Cui al 2019 Byrd & Lipton
2019) and logits adjustment (Ren al. 2020 Menon et al.] 2021). Cui et al. (2019) proposed re-weighting the loss function with the number of effective samples. Lin et al. (2020) suggested that the model should pay
more attention to hard samples by giving hard samples a large weight. On the other hand, Ren et al. (0020)
and Menon et al. 2021 adjusted logits to make gradients more balanced during training. Kang et al. (0019)
found that although re-balancing and re-sampling can improve the performance of imbalanced classification.
they result in worse representation. Kang et al. (0199) proposed to decouple the training process into two stages. The first stage uses standard training methods to learn a good representation. In the second stage, strategies such as re-sampling, re-weighting and so on are used fine-tune the classifier to learn a good classifier. The decoupling training approach provides
training paradigm for long-tailed recognition.
Many studies have focused how to obtain better representation (Kang et al. 2021 Liu et al. 2021 Zhong ct al. 2022) or learn a better classifier (Wang et al., 2021d Zhong et al. 2021).. MoE methods also have achieved a great success by effectively combining the knowle leowledge from multiple experts. BBN (Zhou et al. 2020) introduces a two-branches network to address long-tailed recognition. RIDE (Wang et al., 2020) trains multiple experts with the softmax loss respectively and enforces a KI-divergence basod loss enhance the diversity among various experts. SADE (Zhang et al., (202)) pioneers novel apectrum-spannte multi-expert framework and introduces innovative expert training scheme. Distillation strategy like DIVE (He et al. 2021 and contrastive learnig like PaCo (Cui et al. 2021] have also achieved successes.
2.2
Information Bottleneck
IB divides the model into two parts: encoder and decoder. The encoder codes random variable into random variable z, and the decoder decodes the random variable z into random variable y. IB assumes that variables 2, follow a Markov chain ++ x + 2, and & is the bottleneck of the information flow. IB expects z to retain information from x to y as much as possible while forgetting information unrelated to y. This means that for a given task, z is the best representation in the perspective of rate-distortion theory. However, despite the beauty of the TB theory, the calculation of mutual information IB is complicated, especially in deep learning (Slonim 2006). To address this problem, the Variational Information Bottleneck (VIB) (Alemi et al. 2017) optimizes the IB objective by using its upper bound. According to VIB, -I(2 and ((x; 2) are bounded as
I(2,y) S Ep(z,y) - ( = Lo v),
[x;;)) < Ep(r,s) logp(=)>) Ep(x) r(z)
E Epania (z) [K(g(s]=))]((()
(2)
Here, Epte.) and Epiz,)) is the expectation over the joint distribution p(z, y) and p(x, 2). Similarly, Epanu is the expectation over the data distribution (a)) q(g(2), q(2(z), r(2) are variational approximations p(yl=), p(z[c), p(z), respectively. These variational distributions are introduced to make the optimization
3

---

Published in Transactions on Machine Learning Research (06/2025)
computationally simpler by approximating the intractable true distributions. VIB is widely used for its simplicity and effectiveness. However, IB is essentially compromise wetween representation and classifica- tion, and obtaining an optimal compromise point is difficult. To avoid this compromise, [Tian et al. (2021) proposed a method to implicitly optimize the information bottlencck objective through solf-distilation.
3 Method
In this section, we first formalize the problem setup and the motivation for our approach from the perspective of information theory Section 3.1 We then introduce the proposed Balanced Information Bottleneck (BIB) framework in Scction 3.2 followed by its extension, the Mixture of Balancod Information Bottlenocks (MBIB) in Section 3.3 which leverages optimization of multiple IB and improves the representation further. We also explain the effectiveness of MBIB from the perspectives from information theory and self-distillation.
3.1 Preliminaries
Let D {( y: be a training set, where y/ is the label for data x; and K is the number of classes. Without loss of generality, let n1 n2
nk, where n, is the number of training samples for class i, hence the total number of training samples is N E Ra- Unlike the long-tailed distribution of the training set, the test set follows uniform distribution, ensuring an equal number of samples across all classes for a balanced evaluation of the model's performance in each class.
The objective is learn sufficient and minimal representation = that preserves information relevant to the label y, while discarding irrelevant and redundant information in æ. From the Information Bottleneck (IB) perspective, this corresponds to optimizing the objective RIB in Eq.
3.2 Balanced Information Bottleneck (BIB)
The information bottlenock theory aims obtain a sufficient and minimal representation of the input, the sense of effectively characterizing the output. Let v be an observation of the input extracted from an encoder such CNN, and be representation encoded from by fully connected layer, as shown in Figure [ Considering CNNs' powerful feature extraction capability, introduce the following assumption to enable tractable optimization of the IB objective (Tian et al. 2021
Assumption 3.1 (Sufficiency of Observation v). The observation extracted by the encoder is assumed to retain all label-relevant information from the input i.e., [(u; = 1(r;))
Therefore, our problem is translated into how to find a representation a that preserves the sufficient and minimal information of the observation v to the label y.
Theorem 3.2 (Three Sub-optimization Objectives). Under Assumption 3.1 the Information Bottleneck (IB) objective can be decomposed into the following three sub-objectives:
Marimize I(0;)) - encourage to relain label-relevant informalion;
Marimize [(2; y) encourage to be predictive of the label;
Minimizing [1(;; y) (2;w))
encourage to contain less redundant information.
We need to optimize these three objectives together to get a minimal and sufficient representation z.
Proof of Theorem 1. We decompose T(x; z) to two terms:
[w); = ((: v) (w; z/p),
(3)
where the first term represents the information in 2 that is related the label, and the sccond term represents information that is related the label. To optimize the IB objective, we should maximize T(2;y) while minimizing 7(u; zly). Minimizing 2(,;) is equivalent to minimizing [7(v;) T(2:y) Tian

---

Published in Transactions on Machine Learning Research (06/2025)
CNN
PC
g6:0)
DSCLox
GAP
16:00)
VSDLos
DSCLon
Figure The network structure of BIB. FC means the Fully Connected layer and GAP means Global Average Pooling, f(;; 0) and g(2; 0) are classifiers. BSCLoss means the balancod softmax cross entropy loss (Loss) and Loss2), and VSDloss means variational self-distillation loss (Lossz). The output is the mean of the outputs from f(e; 0) and 0)
et al.
2021 Therefore, we can decompose the objective into three sub-optimization objectives: maximizing I(u: v) maximizing [(2;g) y) and forcing (2: y) approximate [v:;)).
To handle long-tailed data and optimize the IB objective, we propose a balanced information bottleneck (BIB), which integrates re-balancing techniques and self-distillation into the original network architecture. This approach mitigates data imbalance while jointly optimizing the three sub-objectives defined in Theorem 3.2 to improve the quality of representation 2. We now formulate loss function to address each of the three objectives from Theorem 3.2
We start with maximizing (v; y) and (2; According to VIB AAemi et al. 2017). -I(e; y) and - y2; are bounded as Ep(r.) - log ((yle) and (p)())) log ((ylz). For long-tailed datasct, we apply a re-balancing technique over [v:y)):
(u(li)) (x(lo) neeh(0:0)
(4)
K
Σ n
where the subscript 8 represents the training set distribution, f(;;; 0) represents the classifier following The proof can be found in Appendix A The loss corresponding to maximizing (1; becomes:
Loss Ep(e.)) log gs (y)v))
(5)
Similarly, to maximize T(2;8), we get the loss as:
Loss = Ep(z,) - log (v)z).
(6)
entropy losses. We introduce classes weighting factor inversely proportional to the label frequency to strengthen the learning of the minority class and re-balance the losses better (Du ct al. 2023). The weighting factor is:
Loss1 and Loss2
K (1/d,)"
Wc
K (1/d)))
(7)
where d, is the i-th class frequency the training dataset, and m is a hyperparameter. Therefore, Loss and Loss2 are regarded as balanced softmax cross entropy losses (BSCLoss).
Then, to force yz approximate (v; v), we only need to ensure that (ylv) approximates (y)z)) Tian
et al. (2021) proved that making q(ylv) approximate 2(y)z), i.e., minimizing the KL-divergence between ( and (()le, can effectively make Hyle)) approximate H (y)=). Therefore, our third loss is to minimize the Drl
Lossy Eg(((z) [
Eg(((2) (=)()) ( (qlulo))lo (()lo) - log q(vl=)]]l
Eg((lz) [lg( a(vlv) log d[][=)]]
(8)

---

Published in Transactions on Machine Learning Research (06/2025)
Note that this is like the method of self-distillation. To stabilize the optimization, we don't optimize q(ylv); instead, we detach it from the backward propagation process. We call Loss3 the variational self-distilaation loss (VSDLoss). Furthermore, to mitigate the effect of the long-tailed distribution, we use clees-dependent self-distillation temperatures: ((sl))) S and g(vrl2) eslls where T, = ("mmr ,", Y is a
E efttorr
hyperparameter, ((v;0) represents the classifier following v.
The overall loss is given by
LOssBIB(r.s) Loss1 + Loss2 + B Loss3;
(9)
where is a hyperparameter. To sum up, Loss] and Loss are balanced cross entropy losses (BSCLoss) to maximize [(v;)) and y) Loss3 is the variational self-distillation loss (VSDLoss) to force T(z; to
approximate vv; Therefore, we can optimize the IB objective implicitly by minimizing Loss
The code of BIB loss is presented in Appendix J
3.3 Mixture of Balanced Information Bottleneck (MBIB)
From the perspective of information theory, BIB assumees that the CNN-derived observation v of the input 2 satisfies [(2; y) = [x;y]] y), as stated in Assumption 3.1 implying that v preserves all mutual information between x and the label y. However, this assumption encounters limitations due to the inherent data processing inequality (Cover & Thomas 1991 in the information processing chain.
Specifically, consider a CNN network composed of three consecutive parts, denoted CNN1, CNN2, and CNN3, as illustrated in Figure 2 Let vi, 122 and bt denote the intermediate representations extracted from
these partial networks, where viy corresponds to the full observation e used in BIB (Section 3.2 We formally state the following result:
Theorem 3.3 (Data Processing Inequality). Let on, 02, and va be intermediate representations obtained from partial networks CNN,, CNN2, and CNNs, respectively. Then the following inequality holds:
[(v3:)) < [(v2: v) S (v11g)).
(10)
This theorem reflects the fundamental principle that, as data is progressively transformed and compressed through the network, the mutual information between the representation and the target label inevitably diminishes. As a result, Assumption 3.1 made in BIB and prior works (Tian et al.) 2021) does not always hold in practice; instead, we generally have [(v: I [[ y) due to inevitable information loss along the processing chain. Consequently, the final representation (V3 Figure 2 is not a sufficient observation containing all mutual information between and y, and solely applying BIB between v and may fail to fully exploit the label-relevant information.
One promising model enhancement strategy is leveraging the label-related information retained in v and 12, given that they contain more mutual information with y than Us ,as indicated in Theorem 3.3 By applying BIB between vi and = (BI,(v1,))). as well as between 12 and (BI =)), we are able to conserve a greater amount of label-relevant information in 81 and U2. This enables feature z a sufficient and comprehensive representation of U1, 12 and vy simultancously and containing more label-related information, ultimately improving the model's ability to capture long-tailed label information.
Hence, we propose novel network structure, which consists of the mirture of multiple balanced information bottleneck (i.c., MBIB), as shown in Figure 2 MBIB is a network capable of simultancously optimizing diverse information bottleneck. The overall loss function is:
LossMBER LossBIB(e).a) + b- Loss + Loss BIB(vs.))
(11)
where a and b are hyperparameters adjust the proportions of different BIBs. Specifically, when a 0 and b 0, MBIB degrades to the BIB proposed in Section 3.2 As the values of a and b increase, the model emphasizes information from U1 and 02 more. Each LOSsBIB is composed of three losses as below:
Loss BSCLoss + BSCLoss2 B- V.DLoss,
(12)
6

---

Published in Transactions on Machine Learning Research (06/2025)
CNN
CNN
CNN,
V/3 PC
Z 96:0)
GAP
logits
VSDLos
VSDLoss
VSDLoss
logits;
logits;
logits,
BSCLLoss
BBSLLos
LSCCoss
Figure 2: The network structure of MBIB. CNN1, CNN2 and CNN; are different parts of the CNN network. FC means the Fully Connected layer, and GAP means Global Average Pooling. h(v1; u(v2; f(vs; and g(2; 0) are classifiers. The output is the mean of the outputs from f 0) and g(2; 0).
where there are two balanced cross entropy losses and one variational self-distillation loss.
From the perspective of self-distillation, we can further explore the effectiveness of MBIB. VSDLosses in Eqs. equation 9 and equation 12 facilitate self-distillation processes among layers at different depths and the output layer. For example, the VSDLoss in (T1 is optimized in the following form:
min V SDT.ssSs..sssBm..a & go (Prr
(13)
To optimize this objective, minimize the KI.-divergence between g(vl=) and ((vlv1), thereby performing distillation between and z. This process effectively integrates knowledge from and 2.
Previous studics (Jin et al. 2023 have underscored that shallow parts of the deep model are able to perform better on certain tail classes, and layers of different depths excel at recognizing different classes in long-tailed data. This suggests that fusing knowledge from shallow and deep layers can better fit the long-tailed data, and VSDLosses achieve this through self-distillation.
Therefore, MBIB enables more comprehensive output that integrates the knowledge of different shallow layers. At the same time, this approach allows shallow layers to learn from deeper ones. Consequently, this strategy can enhance the representation learning capability of the network by integrating knowledge form different depths.
4 Experiments
4.1 Datasets and Setup
Long-Tailed CIFAR-100. CIFAR-100 includes 60K images, of which 50K images are for training and 10K for verification. There are 100 classes in total. We the same long-tailed version of the CIFAR-100 dataset in Cao et al. (2019) for fair comparison. The imbalance degree of the dataset is controlled by the Imbalanced Factor (IF- Nmax/Nmin: where Nmax represents the highest frequency and Nm represents the lowest frequency). We conduct experiments on CIFAR-100-LT with IF of 100, 50, and 10.
ImageNet-LT, ImageNet-LT is a subset of long-tailed distribution sampled from ImageNet through Pareto distribution. The ImageNet-LLT training set contains 115.8K images, and a total of 1000 classes. The highest frequency 1280, and the lowest frequency 5. There are 20 images of each class in the validation set, and 50 images in each class of the test set.
iNaturalist 2018. iNaturalist 2018 is large-scale, long-tailed fino-grained dataset. iNaturalist 2018 includes 437.5K images and 8142 classes in total, with 1000 samples for the class with the highest frequency and 2 samples for the class with the lowest frequency.
According to the setting in Liu al. 2019). we divide the datasct into three subsets according to the number of samples: Many shot (more than 100 samples), Medium shot (between 20 and 100 samples), and Few shot (less than 20 samples).
7

---

Published in Transactions on Machine Learning Research (06/2025)
Table 1: Top-1 accuracy(%) of ResNet32 on CIFAR-100-LI with 200 epochs. Data bold is the overall accuracy of our methods, while underlined data indicates overall performances superior to ours. This for- matting is consistent in other tables.
Imbalanced Factor
Method
100
50
10
One-Stage
CE
38.3
43.9
55.7
Focal Loss
38.4
44.3
55.8
LDAM-DRW
42.0
46.6
58.7
BBN
42.6
47.0
59.1
CDT
44.3
58.9
BSCE
42.7
47.2
58.5
BIB(Ours)
44.9
49.8
60.4
MBIB(Ours)
47.5
51.2
60.9
Two-Stage
cRT
41.2
46.8
57.9
T-norm
41.1
46.7
57.1
KCL
42.8
46.3
57.6
TSC
43.8
47.4
59.0
SSP
43.4
47.1
58.9
MoE
BBN
42.6
47.0
59.1
RIDE(2F)
47.0
RIDE
48.0
SADE
49.8
53.9
63.6
Others
DiVE
45.4
51.1
62.0
4.2 Implementation Details
Training details on CIFAR-100-LT. For CIFAR-100-LT, we process samples the same way as in Cao et al. (2019) We use ResNet.32 as the backbone network. To keep consistent with the previous settings Cao et al. 2019 we use the SGD optimizer with momentum of 0.9 and weight decay of 0.0003. We train 200 epochs for cach model. The initial lcarning rate is 0.1, and the first five epochs use the lincar warm-up. The learning rate decays by 0.01 at the 16 and the 180th epoch. The batch size of all experiments is 128.
Training details on ImageNet-LT. For ImageNet-LTT, we report the results of two backbone networks: 10 and Xt50. We train 90 epochs for all models, using the SGD optimizer with a momentum of 0.9 and weight decay 0.0005. For ResNet10, we use a cosine learning rate schedule decaying from 0.05 with batch size of 128. For ResNeXt50, we use cosine learning rate schedule decaying from 0.025 to with batch size of 64.
ResNet50 as the backbone network. The model trained 90 or 200 epochs using the SGD optimizer with momentum of 0.9 and weight decay of 0.0001. The batch size is 64, and we use a cosine learning rate schedule decaying from 0.025 to 0.
Training details
iNaturalist 2018. For the iNaturalist 2018,
For the setting hyperparameters, we take 3 in {0, 1,,1, 4, 5) according to different datasets. For all of the datasots, we use a 0.1,
and m 0.1. For CIFAR100-LT, we use 7 = 0, and for ImageNet-I.T and iNaturalist 2018, we use 7 0.5. To make the results more robust, we the mean of fv; and (); as the final result of the test sample.
8

---

Published in Transactions on Machine Learning Research (06/2025)
Table 2: Top-1 accuracy(P) (%) of ResNet10 and ResNeXtt50 on ImageNet-LT with 90 epochs. A $ or indicates training extended to 180 or 200 epochs.
ResNet11/ ResNet10/ReeNeext55 ReesNxt55
Method
Many
Medium
Few
All
One-Stage CE
57.0/65.9
25.7/37.5
3.5/7.7
34.8/44.4
Focal Loss
36.4/64.3
29.9/37.1
16.0/8.2
30.5/43.7
LDAM-DRS
-/63.7
-/47.6
-/30.0
36.0/51.4
LADE
-/62.3
-/49.3
-/31.2
-/51.9
BSCE
53.4/62.2
38.5/48.8
17.0/29.7
41.3/51.4
weight balancingt
-/62.0
-/49.7
-/41.0
-/53.3
RBLt
-/64.8
-/49.6
-/34.2
-/53.3
BIB(Ours)
54.7/64.7
40.0/51.2
21.7/32.7
43.2/53.9
MBIB(Ourg
56.4/67.0
41.8/52.8
23.2/33.5
44.9/55.7
Two-Stage
cRT
-/61.8
-/46.2
-/27.4
41.8/49.6
T-norm
-/59.1
-/46.9
-/30.7
40.6/49.4
DisAlign
-/61.5
-/50.7
-/33.1
-/52.6
WCDAS
53.8/-
41.7/-
25.3/-
44.1/-
CC-SAM
-/63.1
-/53.4
-/41.4
-/55.4
SReprt
-/-
-/-
-/-
-/54.6
MoE
BBN
-/0..0
-/43.3
-/40.8
-/41.2
RIDE(3E)
/66.9
-/52.3
-/4..5
44.3/55.5
SADE
-/65.3
-/55.2
-/42.0
-/57.3
Others
DIVE
-64..1
-/50.4
-/31.5
-/53.1
PaCo
-/59.7
-/51.7
-/36.6
-/52.7
4.3
Main Results
Baseline. We compared four mainstream approaches, including one-stage, two-stage, MoE and other ap- proaches such distillation and contrastive learning. The one-stage approach includes Focal loss (Lin et al.
2020 LDAM (Cao et al. 2019), BSCE (Ren et al. 2020 weight balancing Alshammari ct al. 2022)
RBL Peifeng et al. 2023), etc. The two-stage approach includes cRT (Kang al. 2019). KCL (Kang el al.
2021),
TSC (Li et al. 2021), SSP (Yang & Xu (2200). WCDAS 2023). CC-SAM et al.) 20223]
etc.
The MoE approach includes BBN ZZoou et al.
2020
RIDE Wang al.) 2020), SADE[Zhang et al.
2022
The distillation approach is DiVE et al.
2021 The contrastive learning method is PaCodCui
et al.
2021 Especially, if only the results of the are concerned, BIB degenerates to BSCE.
CIFAR-100-LT. Table compares BIB and MBIB with baseline methods on CIFAR-100-LT.. As the results show, BIB achieves improvements on imbalanced factors. MBIB higher than most of the methods and even outperforms some MoE. models. Although some MoE methods outperform MBIB, they incur much larger computational costs than our methods.
ImageNet-LT. Table 2 compares BIB and MBIB with baseline methods ImageNet-LT. We conduct experiments on two backbones networks: ResNet10 and ResNeXt50. The results of RIDE, SADE and PaCo are from Zhang et al. (202bb). The experimental results show that BIB and MBIB can achieve consistent performance improvement on both small and large neural networks. The overall accuracy of MBIB is higher than all of the baseline methods except SADE.
iNaturalist 2018. Table 3 compares BIB with bascline methods on iNaturalist 2018. Notably, MBIB achieves the best. performance for 200 training epochs among all baseline methods including MoE models.
9

---

Published in Transactions on Machine Learning Research (06/2025)
Table 3: Top-1 accuracy(X) of ResNet50 on iNaturalist 2018.
Method
Epoch
Many
Medium
Few
All
One-Stage
CE
90/200
72.2/75.7
63.0/66.9 69.9
57.2/61.7
61.7/65.8
ResLT
Cui
2022
200
68.5
70.4
70.2
LADE
Hong et al
2021
200
70.0
BSCE
RRn et al.
2020)
90/200
67.2/69.6
66.5/69.8
67.4/69.7
66.9/69.8
weight balancing Alshammaari et al.
2022
200
71.0
70.3
69.4
70.0
BlB(Ours) MBIB(Ours:
90/200
70.9/73.9
69.9/72.9
69.6/72.1
69.9 69.9/72.7
90/200
70.7/72.6
70.6/73.6
70.2/73.1
70.4/73.3
Two-Stage
cRT
(Kang et al.
2019
90/200++10
69.0/73.2
66.0/68.8
63.2/66.1
65.2/68.2
7-norm
(Kng ot al.
2019
90/200++10
65.6/71.1
65.3/68.9
65.9/69.3
65.6/69.3 6/.3
KCL (Kang al. 2021
200++30
68.6
TSC
Li et al.
2021
400++30
72.6
70.6
67.8
69.7
DisAlign
Zhang et al. 2021
90/200++30
64.1/69.0
68.5/71.1
67.9/70.2
67.8/70.0
WCDAS
(Han 2023
200++30
75.5
72.3
69.8
71.8
CC-SAM
Zhou ct a..
2023
200+-30
65.4
70.9
72.2
70.9
SRepr
Nam et al.
2023
200++20
70.8
MoF
BBN Zhou et al. 2020
90/180
49.4/-
70.8/-
65.3/-
66.3/69.6
RIDE( Waang et al. 2020
100/200
70.9/70.5 74.5
72.4/73.7 72.5
73.1/73.3
72.6/73.2
SADE
Zhang et al. 2022
200+-5
73.0
72.9
Others
DIVE He et al.
2021
90/200
-/-
-/-
-/-
69.1/71.7 71.6
PaCo
Cui et al.
2021
200
68.5
72.0
71.8
4.4 Ablation Study
How the value of 8 affects our methods? 3 is hyperparameter in the loss function, which affects the degree of information compression by the network. Figure 3 shows the impact of different on the overall accuracy of the CIFAR100-LT datasct. The results show that the optimal 33 may be different for different imbalanced factors and methods, and we can make more fine adjustments if necessary. However, we think this may not be necessary, because simple search in (0, 1, 4,5) can already obtain satisfactory results.
How the the value of and b affects methods? and b are hyperparameters that adjust the proportions of different BIB components within the MBIB framework. Figure 4 shows the heatmap of accuracy on the CIFAR100-LT dataset with respect to different and b. The results show that the values of and significantly influence the overall accuracy. Similar to the 3, we can fine-tune and to get the models having better performances.
How the the quantity of obsorvation affects our methods? We utilize three observation v our methods. Ablations on and b show that when b zero, 3-MBIB (three-observation MBIB) degenerates to 2-MBIB and accuracy often lowers. We also investigated 4-MBIB, 5-MBIB and 6-MBIB with results shown in Table 4 We found the quantity of observation affects MBIB's performance, improving with more However, once exceeds 3, the performance improvement diminishes and even starts to decline. We altribute this to two key factors. Firstly, 3-BBIB has fully utilized the useful information in intermediate observations, thereby limiting the additional information gained by further increasing the number of e. Secondly, introducing more BIB objectives complicates the optimization process, making it harder for the model to achieve effective optimization, which ultimately leads to performance decrease for 5-MBIB and 6-BBIB.
10

---

Published in Transactions on Machine Learning Research (06/2025)
60.0
600
Overall Accuracy (%6)
57.5
52.5
501
525
50.5
51.1
512
505
50.0
465
45.0
415
10-100
10-30
10-10
42.5
The value of B
Figure 3: The impact of different 8 on the overall accuracy of CIFAR-100-LT (we fixed 0.1 and b = 0.3). The solid lines show the results of MBIB and the dashed lines correspond to BIB.
Table Accuracy of MBIB with different quantitics of observation u. We set in 2-MBIB and 0.3, b 0.3 in the others. 4-MBI((x) denotes the introduction of BIB(I, 2) to 3-MBIB.
Quantity of v
2-MBIB
M-BI
4-BBIB(x)
4-MBIB
5-MBIB
6-BBIB
Accuracy
45.2
46.6
46.8
46.9
46.1
45.4
Due to the page limit, the impact of different parts of BIB, as well as comparisons between BIB and BSCE are provided in Appendix B and C
4.5 Analysis of the Posterior Probability Distribution
Ideally, the mean positive posterior probability per class should be equal to 1:
[(u(2z) Σ {(ullzz) = 1.
(14)
This means that the closer ((y])r)) is to 1, the better. The experiment results reveal BIB's superiority in most classes compared to BSCE and MBIB behaves better than BIB in the tail classes. The detailed results and analysis are shown in Appendix D
4.6 Analysis of the Learned Representation
A good representation should have the following characteristics: the representations of the same class are very close, and the representations between different classes are far away (Wang et al. 2021c). We can evaluate the quality of the representation by the mean of the average intra-class distance (Dintra); the mean
Table 5: The value of p of different methods on the CIFAR-100-LT (IF=100) lesting set.
Metric
Methods
Many
Medium
Few
All
BSCE
1.26
1.30
1.48
1.34
BSCE MLP
0.85
0.98
1.29
0.99
BIB v
0.94
1.06
1.37
1.06
BIB
0.80
0.91
1.15
0.91
MBIB
1.01
1.04
1.31
1.09
MBIB z
0.78
0.86
1.02
0.86
11

---

Published in Transactions on Machine Learning Research (06/2025)
89
44.74444
45.3
14.34488-4
84.844.4444
4..1
44.4
46.5
46.64..
44.
67
64.744.7
45.7454445
44.2
43.8
45.0
44.3
46.0
15..6.1145.745
45.64.1444.7
455
0.5444 44.7
46.046.0.440
0.0046.2440
46.0
45.4
43.445.2440
45.044444.
44.2
0.2-5.2445.7
46.
46.0
46.
46.0
45.
62-455444 45.4458
44.0
41.45.345.54455 45.5
26.2
46.5
45.0
44,9
45.5
45
44.88
-43
60
64.4
45.4
16.5
46.3
66.1
16
44.244.8
9.6
0.
02
03
94
0.5
06
6.7
0.5
0.9
Figure 4: The impact of different a and b on the overall MBIB accuracy of CIFAR-100-LT (we fixed B = 5).
inter-class distance (Dinter), and the ratio (p) between them. Dintros Dinter and are calculated as follows:
Dintra
Σ Tj.
[r
Tull2
(15)
K
R2
1=1
Dinter
Σ Σ
|
cgll
(16)
K(K 1) j-i
Dintra
p = Dinter
(17)
where R2 is the representation set of class i, and G the class center of class i, i.e. c; R.
Σ Ty
Tj. The
better the representation, the smaller the value of p.
Table compares p obtained by different methods on the testing set. The value of p obtained by BIB and MBIB smaller than that of BSCE and BSCE MLP (BSCE MLP indicates that the network structure is the same branch in BIB, and the loss function is BSCE). In addition, We visualize the representation of the test set obtained by different models using t-SNE (van der Maaten & Hinton 2008 The visualization and its analysis are shown in Appendix
Table 6: Efficiency comparisons (test size 3 640 x 640)
Method
MBIB
SADE
RIDE 4E)
Params(k)
486.02
783.86
1018.00
FL.OPs(G)
27.93
40.69
50.98
4.7
The efficiency compared with MoE
MoE methods are often the state-of-the-art approaches in long-tailed recognition. Although they can achieve higher accuracy on some datascts than our methods, they always require larger computational resources. Therefore, MoE methods are less practical than our methods scenarios with limited computational re- sources. We compare the efficiency of MBIB with MoE methods (RIDE and SADE) in Table
5 Discussion
We have cmpirically observed a significant cnhancement network performance upon the introduction of BIB. Additionally, undertook an exploration of two alternative mixture of BIB structures, as de- picted in Figure One structure applied sequential BIB connections between layer pairs (e.g., BIB((1, v2).
12

---

Published in Transactions on Machine Learning Research (06/2025)
1013
VL
(a)
V.
(b)
Figure 5: Two alternative multi-BIB structures.
(1, va), and (13 z)), as shown in Figure a),, while the other integrated BIB connections among all feature representations as shown in Figure (b... We named them respectively SE-MBIB and ALL-MBIB. However, experiments on the CIFAR100-T I. dataset have revealed that both SE-MBIB and ALL-MBIB often exhibit worse performance compared to our MBIB method, as shown in Appendix F The original intention to introduce sequential BIB connections was to propagate the optimal representation throughout the network in a cascading manner, where 122 is considered a sufficient representation U1, V' is a sufficient representation of 12 and z is a sufficient representation of eg. This would ultimately result in a serving as sufficient representation of Ui, consequently increasing mutual information [(z;y). However, this notion is inherently less rigorous, even if 12 is sufficient representation of and 82 is a sufficient representa- tion of 22, V3 does not necessarily constitute sufficient representation of 01. While each individual BIB operation guarantees the output as optimal and sufficient representation of the input, the sequential BIB connections do not inherently ensure that the global output is an optimal and sufficient representation of the initial input. In other words, the representation may deviate from the initial inputs and fall into local optima instead of global optima during the sequential propagation. Thus, SE-MBIB with the sequential BIB connections appears be less justifiable. The second structure ALL-MBIB, which introduces the improper sequential BIB connections on top of MBIB, is found to be detrimental to performance.
While our proposed methods demonstrate performance that falls slightly short of some state-of-the-as mixture of experts (MoE) models for long-tailed datasets, this presents a rich avenue for future research in BIB application. MoF methods involve the integration of multiple expert networks, each specialized in the recognition of distinct data segments. Our MBIB method could potentially be integrated into MoE frameworks. For example, MBIB may be incorporated into each expert network, enhancing the feature learning effectiveness of each expert, and thereby improving the overall performance. Inspired by two-stage methods, our network could explore staged training strategies, such decoupling the training of the feature extractor and classifier, to optimize both components and ultimately achieve superior performance.
6 Conclusion
This paper proposes end-to-end learning methods named BIB and MBIB for long-tailed visual recognition based information bottleneck theory. BIB uses self-distillation to optimize the objective and re- re-balance the classes, improving tail class performance without damaging head class performance. MBIB optimizes various information bottleneck within a single network simultaneously to utilize more information related to labels. Moreover, MBIB can fuse knowledge from different depths of the network to better fit the long-tailcs data, further improving the model performance. Our experiments show that the quality of the feature spaces Iearned by BIB and MBIB is better than that of the re-balancing method like BSCE. SSCE.Experiments on datascts like CIFAR100-LT, Le,,e, and iNaturalist. 2018 show that both BIB and MBIB perform well, even better than some recently proposed two-stage methods and MoE methods.
References
Alexander A. Alemi, Ian S. Fischer, Joshua V. Dillon, and Kevin P. Murphy. Deep variational information bottleneck. ArXiu, abs/1611.00410, 2017.
13

---

244, October Published in Transactions on Machine Learning Research (06/2025)
Shaden Alshammari, Yuxiong Wang, Deva Ramanan, and Shu Kong. Long-tailed recognition via weight balancing. In CVPR, 2022.
Shin Ando and Chun Yuan Huang. Deep over- sampling framework for classifying imbalanced data. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp- 770-785, 2017.
Mateusz Buda, Atsuto Maki, and Maciej A. Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural networks the official journal of the International Neural Network Society, 106:249-259, 2018.
Jonathon Byrd and Zachary Chasc Lipton. What is the effect of importance weighting in deep learning? In ICML, 2019.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Aréchiga, and Tengyu Ma. Learning imbalanced datasets with aabel-listribttion-awwae margin loss. In NewrIPS, 2019.
Thomas M. Cover and Joy A. Thomas. Elements of information theory. 1991.
Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 715-724, 2021.
Jiequan Cui, Shu Liu, Zhuotao Tian, Zhisheng Zhong, and Jiaya Jia. Reslt: Residual learning for long-tailed recognition. IEEE Transactions on Pottern Analysis and Machine Intelligence, 2022.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge J. Belongie. Class balanced loss based on effective number of samples. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9260 9269, 2019.
Fci Du, Peng Yang, Qi Jia, Fengtao Nan, Xiaoting Chen, and Yun Yang. Global and local mixture consistency cumulative learning for long-tailed visual recognitions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15814-15823, 2023.
Anirudh Goyal, Riashat Islam, Daniel Strouse, Zafarali Ahmed, Matthew M. Botvinick, H. Larochelle, Sergey Levine, and Yoshua Bengio. Infobot: Transfer and exploration via the information bottleneck. ArXiu, abs/1901.10902, 2019.
Boran Han. Wrapped cauchy distributed angular softmax for long-tailed visual recognition. In Proceedings of the 40th International Conference on Machine Learning, ICML'22 '2, 2023.
Yin- Yin Hc, Jianxin Wu, and Xiu-Shen Wei. Distilling virtual examples for long-tailos recognition. In Pro- ceedings of the IEEE/CVF International Conference on Compuler Vision (ICCV), pp. 235 235-244, 2021.
Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang, Disentangling label distribution for long-tailed visual recognition. 2021 IEEE/CVF Conference on Computer Vision and Pallern Recognition (CVPR), pp. 6622 6632, 2021.
Yan Jin, Mengke Li, Yang Lu, Yiu-ming Cheung, and Hanzi Wang. Long-tailed visual recognition via self- heterogeneous integration with knowledge excavation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23695 23695-23704 23704, 2023.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalan- tidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations, 2019.
Bingyi Kang, Yu Li, Sai Nan Xie, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for representation learning. In ICLR, 2021.
Jackycom Kim, Minjung Kim, Dongyeon Woo, and Gunhee Kim. Drop-bottleneck: Learning discrete com- pressed representation for noise-robust. exploration. Arxio, abs/2103. hts/2103.12300, 12300, 2021.
14

---

Published in Transactions on Machine Learning Research (06/2025)
Artemy Kolchinsky, Brendan D. Tracey, and David H. Wolpert. Nonlinear information bottleneck. Entropy. 21, 2019.
Thomas Laurent, James von Brecht, and Xavier Bresson. Long-tailed learning requires feature learning. In The Eleventh International Conference on Learning Representations, 2022.
Tianhong Li, Peng Cao, Yuan Yuan, Lijie Fan, Yuzhe Yang, Rogério Schmidt Feris, Piotr Indyk, and Dina Katabi. Targeted supervised contrastive learning for long-tailed recognition. 2021.
Tsung- Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object
detection. TEFE Transactions on Pattern Analysis and Machine Intelligence, 11:188327, 2020.
Hong Liu, Jeff Z. HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. ArXiv, abs/2110.05025, 2021.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu. Large-scale long-tailed recognition in an open world. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2532-2541, 2019.
Yawei Luo, Ping Liu, Tao Guan, Junqing Yu, and Yi Yang. Significance-aware information bottleneck for domain adaplive semantic segmentation. 2019 IEEE/CVE International Conference on Compuler Vision (ICCV), pp. 0777-6786, 2019.
Adilya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. ArXit, abs/ /2//000077714,, 2021.
Giung Nam, Sunguk Jang, and Juho Lee. Decoupled training for Long-tailed classification with stochastic representations. arXiv preprint arXiv:2304.094 2023.
Gao Peifeng, Qianqian Xu, Peisong Wen, Zhiyong Yang, Huiyang Shao, and Qingming Huang. Feature directions matter: Long-tailed learning via rotated balanced reproscntation. In Proceedings of the 40th International Conference on Machine Learning, pp. 27542 27563, 2023.
Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, al. Balanced meta-softmax for long- visual recognition. Advances in neural information processing systems, 33:4175-44 -186, 2020.
Noam Slonim. The information bottleneck: Theory and applications. 2006.
Xudong Tian, Zhizhong Zhang, Shaohui Lin, Yanyun Qu, Yuan Xie, and Lizhuang Ma. Farewell to mutual information: Variational distillation for cross-modal person ro-identification. 2021 IEEE/CVF Conference on Compuler Vision and Pallern Recognition (CVPR), pp. 1522-1531, 2021.
N Tishby. The information bottleneck method. In Proc. 37th Annual Allerton Conference on Communicaa- tions, Control and Computing, 1999, pp. 368-377 1999.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. 2015 IEEE Information Theory Workshop (ITW), pp. 1-5, 2015.
Laurens van der Maaten and Geoffrey E. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579-2000 2605, 2008.
Chaozheng Wang, Shuzheng Gao, Cuiyun Gao, Pengyun Wang, Wenjie Pei, Lujia Pan, and Zenglin Xu. 2021a.
Label-aware distribution calibration for long-tailed classification. ArXio,
Jing Wang, Yuanjie Zheng, Jingqi Song, and Sujuan Hou. Cross-view representation learning for multi-view logo classification with information bottleneck. Proceedings of the 29th ACM International Conference on Multimedia, 2021b.
Wei Wang, Haojic Li, Zhengming Ding, F. Nic, Junyang Chen, Xiao Dong, and Zhihui Wang. Rothinking maximum mean discrepancy for visual domain adaptation. IEEE Transactions on Neural Networks and Learning Systems, 34:264-22 277, 2021c
15

---

Published in Transactions on Machine Learning Research (06/2025)
Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X Yu. Long-tailed recognition by routing diverse distribution-aware experts. arXiv preprint 0rx:020010.00 010809, 2020.
Yidong Wang. Bowen Zhang. Wenxin Hou, Zhen Wu, Jindong Wang, and Takahiro Shinozaki. Margin calibration for loiged visual recognition. Arxie, abs/2112.07225, 2021d.
Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class-imbalanced learning. ArXin, abs/2006.07529, 2020.
Songyang Zhang, Zeming Li, Shipeng Yan, Xuming He, and Jian Sun. Distribution alignment: A unified framework for long-tail visual recognition. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2361-2370, 2021.
Yifan Zhang, Bryan Hooi, Lanqing Hong, and Jiashi Feng. Self-supervised aggregation of diverse experts for test-agnostic long-tailed recognition. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 34077-34090. Curran Associates, Inc., 2022.
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. IEEE Transactions on Pullern Analysis and Machine Intelligence, 2023a.
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. IEEE Trans. Pattern Anal. Mach. Intell., 45(9):10795 10816, sep 2023b.
Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recognition. 2021 IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pp. 16484-16499, 2021.
Zhisheng Zhong, Jiequan Cui, Eric Lo, Zeming Li, Jian Sun, and Jiaya Jia. Rebalanced siamese rastivvv mining for long-tailed tailed recognition. ArXiv, 0ttp/2203.11506 2022.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbu: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognation (CVPR), June 2020.
Zhipeng Zhou, Lanqing Li, Peilin Zhao, Pheng-Ann Hong, and Wei Gong. Class-conditional sharpnoss aware minimization for deep long-tailed tailed learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
A Proof of the Re-Balance Technique
According to VIB, (I, y) and -I y) are bounded as Ep(n,) - log (tyl)) v) and Ep(z,)) log -logq(y)))) However, when the labels are long-tailed, we need to re-balance them. Our purpose train an end-to-end model, that is, the output of the model is ()) According to the properties of the conditional probability, we have
(u (v) P. (clw)).
(18)
Pc(y)u)pp(e)) (v))))))(())
(19)
where the subscript s represents the training set distribution, and the subscript t represents the test set distribution. Since the training set and the testing set come from the same image domain, it can be assumed that (v) = (v)) and (u(l))) (.(l)ss) We have
po(ull)
(20)
(v.)
Pe (v(v))
16

---

Published in Transactions on Machine Learning Research (06/2025)
Due to (16)
N and (1())) K. we have (o(l))) x n.pc(ulle). By normalizing (o(lilu), we get
Σ R4
Po(vil0) n:pp(y:lu)
(21)
K
Σ
(3pr(B)lo)
We can obtain Pe(uslu) by the output of model, so that
elo:0
(wilv) xe (e(u)l))
(22)
eli(v;))
We can rewrite Eq. 21 as
n (1e((000)
(u(uil)) =o (v(lo)
(23)
Σ nyeh(v:))
Therefore, we get the first loss as
L0881 -p(ess) log ds (ylv))
(24)
Similarly, for maximizing T(2, y), get the second loss
Loss2 Ep((.)) log qs (v)2).
(25)
B Impact of Different Components of BIB
In order to further understand the influence of different components of the BIB loss function on the experi- mental results, we conducted ablation experiments on the loss function of BIB, As shown in Figure G when the B is 0, the performance of the model can be improved, which indicates that information compression by information bottleneck is conducive to long-tailed visual recognition. On the other hand, if the use of Loss; and Loss has not been re- balanced, even if the information bottleneck is used, it will not achieve satisfactory results.
CE Zband
Method
Balanced 3 0
BSCE 2branch
CF [F__2branch
X
/
SDIB
X
x
BSCE BSCR__brrachh
6.5
BIB
x
100
10
Imbalanced Factor
Figure 6: Ablation results of loss function. The Balanced option is means that the method uses entropy loss for Loss and Loss The B is 0 means that u and are independent.
C Comparison Between BIB and BSCE
From the form perspoctive, Loss and Loss, are consistent with BSCE. order to better understand our proposed BIB, we conducted extensive experiments on the ImageNet-LT with ResNet10. Table shows the comparison results between BIB and BSCE. BSCE indicates that the network structure is the same
17

---

Published in Transactions on Machine Learning Research (06/2025)
branch in BIB, and the loss function is BSCE. BSCE MLP indicates that the network structure is the same branch in BIB, and the loss function is BSCF. BIB and BIB indicates the result obtained by directly using the output of f(e;0) and 9)2 BIB ensemble indicates the result obtained by the mcan of f(u; 0) and g(2; 8), that is, the result we finally use. Table 7 shows that adding MLP only may hurt the performance of the model consistent with findings in Kang el al. (2019). Since IB can remove label-indeennent information from the representation as much as possible, the head classes performance of BIB has been significantly improved. At the same time, the tail classes performance has declined due to the limited number of tail class samples. However, the average performance has been greatly improved. We assume that v can retain all the information in æ, but the information will still be lost from z to is upstream of in the information flow, and the improvement of the quality of = will also lead to the improvement of the quality of v, so the performance of BIB has also been greatly improved. At the same time, the mean of f(u; f) and ((; f) can achieve the best performance.
Table 7: Comparison between BIB and BSCE on ImageNet-LT with ResNet10.
Method
Many
Medium
Few
All
BSCE
53.4
38.5
17.0
41.3
BSCE MLP
51.7
36.6
18.2
39.9
BIB
53.8
40.2
23.1
43.1
BIB_z
54.6
38.5
19.2
42.1
BIB ensemble
54.7
40.0
21.7
43.2
D Analysis of the Posterior Probability Distribution
From equation 14 we can infer that the closer the mean positive posterior probability of per class is to 1,
the better. Figure 7 (a) shows the mean positive posterior probability 75l(z) BIB. Figure 7 (b) presents
the diff (diff= 7, ())a)) 72 (x)(²)) between the posterior probability obtained by BIB and CE. The results show that CE severely over-fitting to the head class and under-fitting the tail class. Figure 7 (c) presents the diff between the posterior probability obtained by BIB and BSCE, revealing BIB's superiority in most classes. (d) shows the mean positive posterior probability of MBIB, (e) presents the diff between the posterior probability obtained by MBIB and BIB, which shows that MBIB behaves better than BIB in the tail classes.
E Analysis of the Learned Representations Showed by t-SNE
We visualize the representation of the test set obtained by different models using I-SNE, as shown in Figure The visualization results show that the separability of inter-class obtained by BIB and MBIB increases, and the representations within a class are more aggregated. Both quantitative analysis and visualization results show that the quality of representations obtained by BIB and MBIB are better than others. It indicates that BIB and MBIB get better classification performance, and the representation spaces become better simultaneously, which is consistent with our expectations.
F Experiments on Two Other Mixture of BIB Networks
We discussed two other Mixture of BIB network in the paper: one structure applied sequential BIB connec- tions between layer pairs (SE-MBIB), while the other inlegrated BIB connections among all feature represen- tations (ALL-MBIB). In this section, we present the experiment results the two alternative MBIB network. Figure T shows the performances of MBIB, SE-MBIB and AL..-MBIB on CIFAR-100-LT (IF=100). It is evident that both SE-MBIB and ALL-MBIB exhibit inferior performances compared MBIB, The analysis can be found in the discussion section of the paper.
18

---

Published in Transactions on Machine Learning Research (06/2025)
Saa
Clan (b)
Cloc
(a)
(c)
(c)
(e)
Figure 7: The mean Positive Posterior probability ImageNet-LT with ResNet10. (a) The mean Positive Posterior probability per class of BIB. (b) Diff of the mean Positive Posterior probability per class between BIB and CE. (c) Diff of the mean Positive Postcrior probability per class between BIB and BSCE.(d) The mean Positive Posterior probability per class MBIB.(e) Diff of the mean Positive Posterior probability per class between MBIB and BIB.
(a) BSCE
(b) BIB
(c) MBIB
(d) BSCE MLP
MLP
BIB
( MBIB
Figure 8: Visualization analysis. The t-SNE used visualize the test feature space on LIFAR-100-I1 (1F=100) and 10 classes are selected. The lower left corner shows the number of samples for each class during training.
19

---

Published in Transactions on Machine Learning Research (06/2025)
47
46
Overall Accuracy
46-
455
4522
45
44
44.8
64.0
428
41-
4.0
40
0
3
The value of B
Figure 9: Overall accuracy of the three MBIB networks CIFAR-100-LT (IF=100).
G
Combining MBIB and MoE
As discussed in Section 5 there is significant potential for integrating MBIB into Mixture--f-Experr (MoE) frameworks. To explore this, we conducted experiments by incorporating MBIB into each expert of MoE model, using SADE the base architecture. The reproduced performance of SADE and the results of the combined model (SADE + MBIB) CIFAR-100-LT (IF=100) are summarized in Table S We observe that this straightforward integration yields an improvement of approximately 1.6% in overall accuracy compared to the vanilla SADE model. We believe that with carefully tailored, expert-specific MBIB configurations, even greater performance gains could be achieved.
Table 8: Top-1 accuracy(%) reproduced SADE and the comebined model (SADE + MBIB).
Method
All
Many
Medium
Few
SADE
48.7
62.4
50.2
30.7
SADE + MBIB
50.3 (+1.6)
59.9
47.4
35.8
H Impact of Different Re-balancing Methods
While prior works, such Focal Loss, have shown the effectiveness of class re-balancing losses, recent studies [Menon et al. 2021) have highlighted their limitations, including the lack of Fisher consistency. To address these issues, logit adjustment methods have been proposed (Menon et al. 2021). Furthermore, these methods can be combined, as demonstrated in Zhang et al. (2023a). As a result, we incorporate both class re-balancing methods (Eq.7) and logits adjustment methods (Eq14) as part of the re-balancing strategy in our approach.
We also conducted further ablation studies on the re-balancing techniques of MBIB to demonstrate the importance of each re-balancing technique, with the overall accuracy shown in the Table 9 MBIB w/o class re-balancing means 0 and the weight in Eq 7 is for all classes. MBIB w/o logits adjustment means removing logits adjustment in MBIB. MBIB w/o re-balancing means not using any re- re-balancing techniques in MBIB. The experimental results reveal that removing either re-balancing component significantly im- pairs performance, especially the tail (few) classes. Using only class re-balancing methods (without logit adjustment) causes substantial 2.2% drop in overall accuracy of all classes. Similarly, employing only logit adjustment (without class balancinng results in 0.9% accuracy reduction of all classes. These findings further demonstrate the necessity of incorporating both techniques jointly.
20

---

Published in Transactions on Machine Learning Research (06/2025)
Table Ablation results of re- re-banann techniques.
Method
All
Many
Medium
Few
MBIB w/o re-balancing
43.5
62.4
44.6
20.1
MBIB w/o class re-balancing
46.1
65.0
46.6
23.5
MBIB w/o logits adjustment
44.8
63.0
45.8
22.3
MBIB
47.0
64.1
47.7
26.1
Tuning of Hyperparameters
While hyperparameters (a, b, B) do influence performance, the variation is within a reasonable range (approx- imately 1.2% overall accuracy). Importantly, across most hyperparameter configurations shown in Figure and Figure 4 MBIB consistently outperforms all one-stage and two-stage baselines, demonstrating the reliability and robustness of our methods.
From the perspective of information theory, the influence of hyperparameters the performance is justifi- able, and we tune the hyperparameters based these insights:
(1) Hyperparameters and b control the relative emphasis on intermediate observations and U2- Since ti contains the most label-related information but also the most redundant information (followed by d2 and then 03), we observe that: should not be too large to avoid optimization disruption from redundant information
can be larger than due to 12 containing less redundant information than on
The ideal configuration should satisfy < <b<
As illustrated in Figure MBIB performs well when and b meet this condition, confirming this theoretical expectation. We selected 0.1 and 6 0.3 within these satisfactory configurations.
(2) Hyperparameter 8 regulates the emphasis on the IB objective optimization. As shown in Figure 3 performance generally improves when B * 0, and we select the value that yields the best results.
(3) For hyperparameter m that regulates the class re- balancingg loss, we follow established settings from prior works (Du et al. 2023).
J Code of BIB Loss
We present the code and comments for BIB loss in Figure
21

---

Published in Transactions on Machine Learning Research (06/2025)
def BIB [iB_loss(labblll v_logits, 2_logits, sample_per_clase reduction):
Compute the Balanced Information Bottleneck (BIB) Loss.
Args:
labels: A int tensor of size [batch] class labels for each sample. v_ logsts: A float tensor size [batch, vo_of_classes
intermediate logéts. final logits. sample_per_casee A int tensor of size [no_ classess class frequencies. reduction: string. One of "none", "mean", "sum" how to reduce the loss.
2_ logits: A float tensor of size Ibatch, no_of_classe
Returns:
loss: float tensor the calculated BIB Loss.
Hyperparameter controlling the emphasis on VSDLoss
beta =
Temperature scaling for logdt adjustment
gamma
N max, torch. www.saalee_prr_cclee class, dim=0)
21
n corch.reciprocal (sample__pg _eciiee_p________ class / N m__aa)
T torch. unsquezze (torch, pow (ow(n, gamma), 0) temperature T. expand as av (oloitt)
24
Variational Self-Distillation Loss (VSDLoss)
36
beta beta_logits_T
ialogits.detache temperature
beta_ @ata_logits_$ z_logits / temperature
P_I coottmaxbbeaa_lggitt.. dim=-1)
# KL divergence between teacher and student
vad loss (p_T P_T.log() P_T (boo__softaaaxetttaaaotttt dim=-1)) (ddmm=-1) mean
Class re-balancing weights
label label_weightt wwightigggggg 0.1 Weighting factor parameter
weights 1.0 (up.arrag) arway _pe__class) ** label_weighting,
weights weights mp. sum (weighta)
weights torch. Flocc.Fooortrrrrr (weights) len @ssample_per_class
Balanced Softmez Cross-Entropy Loss (BSCLoss)
spc sample_ per_ class. type as (v s__logit))
spc spc. unsqueeze (0) expand (v logits shapee(), -1)
Apply logst adjustment
3
v v_logits v_logits log(1 # Logst adjustment based on class frequencies z_logits = 2_logits log() # Logst adjustment based class frequencies
Calculate BSCLoss
loss1 cross ceceessentrppyeeee (input=y_logiiiii targat=labels, reductionfreductiion, weight=weights)
loss2 crossa_entropy (inpropy(inputzz_1ogigaa target=labels, reductionFreduction,, weight=weights)
30
Comdsned loss
loss loss1 + loss2 + beta vad loss
22
return loss
Figure 10: Python code of BIB Loss.
22


Reviews:
[{"review_id": "Hlb675d9RK", "reviewer": "Reviewer_BXxd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1.\tStrengths\n- Well written: The paper is well written with sufficient background information to allow readers appreciate the motivation of the work.\n- Theoretical Grounding: The approach is well-rooted in Information Bottleneck theory, using it to systematically address imbalances in label distributions.\n- Ample Empirical Evidence: Experiments demonstrate improved classification accuracy across multiple benchmarks and confirm representation improvements via distance metrics (e.g., inter-class vs. intra-class distances) and t-SNE plots.\n2.\tWeaknesses\n- Importance of re-balance: The BIB implementation introduced two weighting scheme, one in Eqn (4) scaling the softmax and one in Eqn (7) scaling loss. Many prior work, such as Focal Loss [A], have found re-balancing class losses sufficient. Is Eqn (4) necessary? How effective is this re-balancing?\n- MoE performance: MoE outperforms the proposed approach. Why not combining these techniques?\n- Limited Exploration of Other Architectures/Setup: While the main experiments are thorough, the paper predominantly uses ResNet-based architectures. Demonstrating generalizability to other architectures (e.g., Vision Transformers) might improve the paper’s scope. Similar argument goes for SGD optimizer.\n- Novelty of MBIB: Using per-layer supervision has been a common technique is many representations literature, such as segmentation [B]. What is the novelty here?\n3.\tMinor\n- Section 3.1 the notation `i` is abused, representing both sample i (xi, yi) and class I (ni).\n\n[A] Lin, T. Y., Goyal, P., Girshick, R., He, K., & Dollár, P. (2017). Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision (pp. 2980-2988).\n\n[B] Ronneberger, O., Fischer, P., & Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18 (pp. 234-241). Springer international publishing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript introduces a novel approach—Balanced Information Bottleneck (BIB) and mixture of BIB—to tackle long-tailed visual recognition problems, where data follow highly imbalanced distributions that challenge modern deep networks. \n\nThe authors leverage concepts from the Information Bottleneck (IB) framework, augmenting it with two key ideas arriving at BIB: Loss Function Re-Balancing and Self-Distillation for Preserving Label InformationBuilding on BIB, the paper further proposes Mixture of Balanced Information Bottlenecks (MBIB), which applies IB-based constraints at multiple levels of the network’s feature hierarchy. \n\nExperimental results on well-known benchmarks (CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018) show that the proposed methods (BIB and MBIB):\n- Achieve competitive or superior accuracy compared to single-expert baselines,\n- Competitive with some mixture-of-experts (MoE) solutions while using fewer parameters and FLOPs,\n- Demonstrate improved learned representations, as evidenced by feature-space metrics and visualizations.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.\tCritical\n- Ablation of loss: Provide a more detailed ablation on if both re-balancing techniques are needed.\n- MoE performance comparison: Building on top of MoE and showing improvement could be more convincing.\n- Consider Non-ResNet Architectures: Brief results or references to confirm that the gains hold across transformer-based or MobileNet-like backbones would make the claims more comprehensive.\n2.\tMinor / Enhancements\n- Further Discussion on novelty of MBIB", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "G9RBrRMXuf", "reviewer": "Reviewer_xwiL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. By integrating loss function re-balancing and self-distillation techniques into IB, the proposed approach offers a unique and effective solution for representation learning in the presence of imbalanced data distributions.\n\n2. Through experiments on standard long-tailed datasets like CIFAR100-LT, ImageNet-LT, and iNaturalist 2018, the proposed BIB and MBIB models achieve state-of-the-art performance, showcasing some effectiveness of the approach in handling long-tailed visual recognition tasks.\n\n\nWeaknesses\n\n1. The experimental results in Tables 1 and 2 donnot contain standard deviation making the stability of the proposed methods questionable.\n\n2. Writing should be much clearer. For example, what is the actual implementation of the algorithm, as many approximations are made. Please provide a pseudo code for instance.\n\n3. Lack of theoretical explanations for the benefit of introducing the additional loss term.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces the Balanced Information Bottleneck (BIB) approach, which addresses challenges posed by long-tailed visual recognition data by integrating loss function re-balancing and self-distillation techniques. Additionally, it proposes the Mixture of Multiple Balanced Information Bottlenecks (MBIB), achieving state-of-the-art performance on long-tailed datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. To enhance the credibility of the experimental results, it is essential to include standard deviation values in Tables 1 and 2. This addition will provide insights into the stability and reliability of the proposed methods.\n\n\n2. Enhance the clarity of the paper by providing a more detailed explanation of the algorithm implementation. Consider including a pseudo code snippet to elucidate the workings of the proposed approach and reduce ambiguity stemming from approximations.\n\n\n3. Address the lack of theoretical explanations regarding the benefits of introducing the additional loss term. Clearly articulate the rationale behind incorporating this term and elucidate how it contributes to the overall effectiveness of the method in handling long-tailed visual recognition tasks.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PVAY6cld0w", "reviewer": "Reviewer_UDAS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The text and figures are easy to understand\n\n2. Many experiment settings are tested\n\nCons:\n1. The writing style of the paper seems more suitable to fit in the computer vision venue rather than the machine learning venue — the notations are usually used without definition (e.g. what is $E_{p(z,y)}$ in Eq. 2?);   The main contribution is the carefully designed new loss to train Neural Network models that consists of many terms (e.g. Eq. 11, 12) and the modification of the Neural Network (Fig. 2).\n\n2. The performance of the model is not very convincing. The proposed model introduces too many hyperparameters ($a, b, \\beta, v, m$), and the corresponding accuracy can differ with up to 2% (Fig. 4). This character diminish the reliablility of the model — the reported superior performance (Tab.1, 2, 3) looks more likely to be achieved by careful hyperparameter tuning.\n\n3. The theoretical contribution of the model is not very strong — the introduced “re-balance technique” is just the re-weighting method that have been used widely in previous Long-Tailed models; Eq. 7 is introduced directly without further analysis, with an extra hyperparameter $m$;  The proposed MBIB loss is a simple augmentation of BIB loss over intermediate layer output of the NN model, without further theoretical analysis.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed a Neural Network model for Long-Tailed recognition task, where the information bottleneck method is introduced in the Long-Tailed problem, two losses BIB/MBIB are introduced. Performance of these models on CIFAR100-100-LT, ImageNet-LT and iNaturalist datasets are reported to justify the their effectiveness.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "From the perspective of machine learning research, I am not very convinced by the theoretical validity and the empirical effectiveness of the model. My first recommendation to the authors is considering resubmit this manuscript to a computer vision conference/journal. If that is not preferable, please consider:\n\n1. Define the notations before using them. Formally describe the assumptions, theorems and proofs used and justify the theoretical contribution of the model.\n\n2. Provide theoretical evidence to justify BIB/MBIB model (e.g. a theoretical model that introduce $m$, $\\gamma$ naturally instead of only intuitively like “to mitigate the effect of the long-tailed distribution”).\n\n3. Provide a convincing and effective hyperparameter tuning technique that search for $a,b,\\beta,v,m$ so that the performance looks reliable.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)