Enhancing Adaptive Deep Networks for Image Classification via Uncertainty-aware Decision Fusion

Authors: xu  zhang
zhipeng  xie
haiyang  yu
qitong  wang
peng  wang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 17:27:29

Abstract: Handling varying computational resources is a critical issue in modern AI applications.
\textit{Adaptive deep networks}, featuring the dynamic employment of multiple classifier heads among different layers, have been proposed to address classification tasks under varying computing resources. Existing approaches typically utilize the last classifier supported by the available resources for inference, as they believe that the last classifier always performs better across all classes.
However, our findings indicate that earlier classifier heads can outperform the last head for certain classes. Based on this observation, we introduce the Collaborative Decision Making (CDM) module, which fuses the multiple classifier heads to enhance the inference performance of \textit{adaptive deep networks}. CDM incorporates an uncertainty-aware fusion method based on evidential deep learning (EDL), that utilizes the \textit{reliability} (uncertainty values) from the first $c$-1 classifiers to improve the $c$-th classifier' accuracy. We also design a balance term that reduces fusion \textit{saturation} and \textit{unfairness} issues caused by EDL constraints to improve the fusion quality of CDM. Finally, a regularized training strategy that uses the last classifier to guide the learning process of early classifiers is proposed to further enhance the CDM module's effect, called the Guided Collaborative Decision Making (GCDM) framework. The experimental evaluation demonstrates the effectiveness of our approaches. Results on ImageNet datasets show CDM and GCDM obtain 0.4\% to 2.8\% accuracy improvement (under varying computing resources) on popular adaptive networks. The code is available at the link \url{https://github.com/Meteor-Stars/GCDM_AdaptiveNet}.

Main paper: 
 introduction: 
 Deep convolutional neural networks (CNN) include the traditional architecture of ResNet [11] and DenseNet [15] or the light-weight architecture of MobileNet [12] and CondenseNet [14] . They have promoted the development of many fields such as object detection [4] . The above deep networks that contain only one classifier at the end of the network architecture, and they need to work on high computational costs and will become ineffective when computational resources are insufficient.A popular solution is to transform deep networks into a multiclassifier network, where each classifier works based on different computational resources [13] . As shown in Figure 3 , a deep network consists of 𝑐 blocks, each consisting of different CNNs. Different classifiers are attached to the exits of different blocks. The computational resources required for the 𝑐-th classifier are the sum of all the preceding 𝑐 -1 blocks. This enables different classifiers to work under varying computational resources and if it's insufficient to support the 𝑐-th classifier, we can select a classifier from the first 𝑐 -1 classifiers that meet the requirements. Deep networks with multi-classifiers can be seen as Adaptive Deep Networks. The simplest implementation of it is to add multiple classifiers at different depths in traditional CNN architectures like ResNet [11] . However, traditional architectures may suffer from optimization conflicts between classifiers, leading to poor performance. Encouragingly, a new architecture called MSDNet [13] is proposed and successfully addresses the optimization conflicts among multiple classifiers, subsequently leading to the emergence of a more effective architecture like RANet [28] and their improved versions [1, 20, 29] . Meanwhile, IMTA [19] proposes improved techniques for enhancing adaptive deep networks, which is a gradient equilibrium-based two-stage training algorithm to further enhance the performance of adaptive deep networks like MSDNet and RANet. However, current adaptive deep networks have a common limitation: they assume the last classifier always has the best performance on all classes. Consequently, they only use the last 𝐶-th classifier that computing resources can support (the previous all 𝐶 -1 classifiers are unused). This raises a question: when there are enough computational resources for the 𝑐-th classifier, it is also sufficient to support all 𝑐 -1 classifiers. Can we utilize all the previous 𝑐 -1 classifiers to enhance the performance of the 𝑐-th classifier? The answer is affirmative because we find there is good diversity among the first 𝑐 classifiers in Adaptive Deep Networks, and their decisions can complement each other.As shown in Figure 1 (a), we visualize the accuracy of randomly sampled classes (C5, C53,...) on different classifiers. CF1 and CF10 represent classifiers attached to the 1-th and 10-th block, and they are allocated with the minimal and maximum computational resources. We can observe that the overall performance of CF10 is the best among all classifiers because it utilizes the maximum resources to extract the highest-quality features for classification. However, the accuracy of CF10 in some classes (e.g., C53 and C65) is not the best and is even worse than other classifiers. This suggests that different classifiers have their own advantages (more qualitative analysis refers to the full version paper shown in our code link), and 𝑐 -1 classifiers can provide better decisions for the 𝑐-th classifier to enhance its performance.Based on the above observations, this paper proposes CDM, called Collaborative Decision Making, which fuses the decision information of all 𝑐 -1 classifiers to enhance the performance of the 𝑐-th classifier. CDM works only during the inference stage with no extra model parameters and without obviously increasing inference time. For the design of CDM, we propose a classifier fusion method based on evidential deep learning (EDL) [25] framework and uncertainty. Specifically, classifiers with higher uncertainty are considered to have less reliability, and vice versa. To this end, we first propose an uncertainty-aware attention mechanism-based fusion method to weight and integrate the decision information from different classifiers based on their reliability. However, we find that the prior design in the EDL framework may bring fusion saturation and fusion unfairness issues for the uncertainty-aware fusion, which harms its fusion quality. Hence, a balance term to slow down the changing trend of fusion values is further introduced to alleviate the fusion saturation and fusion unfairness issues, enhancing the fusion quality of CDM.Finally, we make efforts to enhance the performance of the CDM module. The fusion performance of multiple classifiers depends on the accuracy and diversity of them [2, 3, 23] . For accuracy, as described in Figure 1 (a), the overall performance on all classes of the last classifier is better than the early classifiers. Hence, we can increase the accuracy of early classifiers by exerting regularization between the last classifier and early ones (called regularized training). To this end, we propose the Guided Collaborative Decision Making (GCDM) framework to use the last classifier to guide the learning process of early classifiers. For diversity, intuitively, regularized training may decrease the diversity of early classifiers. However, we observe that regularized training doesn't obviously harm and can even improve the diversity among classifiers in experiments. Specifically, we calculate the agreement table [27] ) of 10 classifiers of MSDNet after regularized training on the ImageNet100 dataset, as shown in Figure 1(b) . Therefore, we can enhance the performance of CDM by regularized training, i.e., increasing the accuracy of early classifiers, not obviously harming and even improving their diversity. Our contributions are as follows:(1) A Collaborative Decision Making (CDM) idea is proposed to improve the performance of popular adaptive deep networks.(2) An uncertainty-aware fusion method is proposed to realize CDM, which weights and integrates the decision information from different classifiers based on their reliability (uncertainty values).(3) A Guided Collaborative Decision Making (GCDM) framework is further proposed to enhance CDM, which uses regularized training to increase the accuracy of early classifiers and not obviously harm their diversity.(4) Empirical study on the large scale ImageNet1000, ImageNet100, Cifar100, and Cifar10 datasets shows the good performance of CDM and GCDM, e.g., our method can improve the accuracy of SOTA MSDNet, RANet, and IMTA by approximately 0.8% to 2.8% under various computing resource constraints on the ImageNet datasets. 
related work: 
 Adaptive deep networks. Works can be divided into two categories: focusing on designing effective architectures and training strategies. For the former one, MSDNet [13] creates an innovative multi-scale convolutional network featuring multi-classifiers with different computational budgets. These classifiers can be dynamically chosen during the inference stage. RANet [28] proposes a resolution adaptive network by designing an architecture that can feed features with a suitable resolution for different samples. Then, [1] uses concatenation and strided convolutions to further improve MSDNet. [20] proposes an adaptive router to predict the difficulty scores of the images and achieve automatic classification. [29] regards adaptive deep networks as an additive model, and train it in a boosting manner to address the distribution mismatch problem in the train-test stages. For designing effective training strategies, IMTA [19] proposes improved techniques, such as the gradient equilibrium and forward-backward knowledge transfer based twostage training algorithms (introducing extra model parameters) for improving the performance of adaptive deep networks. However, there is a limitation for current methods: they all assume the last classifier always has the best performance and multi-classifiers work independently during the testing stage. In other words, when computational resources are sufficient to support the 𝑐-th classifier, all 𝑐 -1 classifiers can also be used to improve the performance. However, they ignore the decision information of the 𝑐 -1 classifiers, resulting in computational resource waste and performance limitations. This paper proposes one-stage (end-to-end training) methods to address this limitation, making full use of computational resources and enhancing the accuracy of each classifier.Evidential Deep Learning (EDL). Traditional softmax-based neural networks for single-point estimation of class probability distributions cannot effectively estimate classification uncertainty and are prone to overconfidence in wrong predictions [8] . In contrast, EDL targets knowing "what they don't know" based on a prior belief [21, 26] . Through the Dempster Shafer Theory of Evidence (DST) [6] and Subjective Logic [16] , EDL realizes uncertainty estimation in a single forward pass by collecting evidence for each category and modeling the distribution of class probabilities. In recent years, EDL has successfully been adopted in various tasks, including outof-distribution detection [7, 30] , multiview classification [9, 10] and domain adaptation learning [5] . However, to our knowledge, EDL hasn't been explored in the field of adaptive deep networks. This paper uses evidential uncertainty to quantify the reliability of different classifiers and further develops an uncertainty-aware attention mechanism to fuse the decision information from different classifiers with an emphasis on their reliability. 
methods: 
 Our method is shown in Figure 3 . Unlike traditional adaptive deep networks, our proposed approach CDM differs in that during the inference stage, the 𝑐-th classifier and all 𝑐 -1 classifiers are not independent, thereby making full use of computational resources. Specifically, we enhance the performance of the 𝑐-th classifier through the proposed CDM module and GCDM framework. In CDM, an uncertainty-aware attention mechanism is designed to weight and fuse the decision information from different classifiers based on their reliability (uncertainty values). Further, through regularized training, GCDM enhances the performance of CDM by increasing the accuracy of early classifiers and not obviously harming or even improving their diversity. We will proceed to introduce the details of CDM and GCDM.The adaptive deep network can be seen as a network with 𝐶 classifiers, where these classifiers are attached at varying depths of the network. Given the input image 𝑥 and corresponding true class label 𝑦, the output of the 𝑐-th classifier (1 ≤ 𝑐 ≤ 𝐶) is:𝑝 𝑐 = 𝑓 𝑐 (𝑥; 𝜃 𝑐 ) = [𝑝 𝑐 1 , • • • , 𝑝 𝑐 𝐾 ] ∈ R 𝐾 (1)where 𝐾 is the number of class labels and 𝜃 𝑐 denotes the model parameters of the 𝑐-th classifier and each value 𝑝 𝑐 𝑘 is the logit of the 𝑘-th class on 𝑐-th classifier.The adaptive deep network can realize computationally efficient inference in two forms: anytime prediction (Figure 2 the computational resource is only sufficient for the 𝑐-th classifier. In anytime prediction, all test samples will be classified by the 𝑐-th classifier. In contrast, budgeted batch prediction dynamically selects the suitable classifier for different test samples based on their difficulty. Concretely, early classifiers with fewer computational resource costs are used for the classification of easy images, while classifiers with higher computational demands are used for the classification of harder images. To measure the difficulty of images, we follow [13, 19, 28] to use the classifier confidence of validation set. Hence, the frequency of using classifiers with higher computational resource will be reduced, thereby achieving a further reduction in overall computational resource consumption. We recommend to see [13, 28] for more details about budgeted batch prediction.CDM is proposed to enhance the performance of 𝑐-th classifier by reusing the 𝑐 -1 available classifiers under limited computational resources where 1 < 𝑐 ≤ 𝐶 and 𝐶 is the total number of classifiers. CDM first quantifies the uncertainty (reliability) of different classifiers and uses the uncertainty-aware attention mechanism for collaborative decision-making fusion. We will further discuss them.Evidential uncertainty is derived from the Dempster-Shafer Theory of Evidence (DST) and is further developed by Subjective Logic (SL) [17] based on Dirichlet distribution. SL defines a theoretical framework for obtaining the belief masses of different classes and the uncertainty measurement of the samples based on the 𝑒𝑣𝑖𝑑𝑒𝑛𝑐𝑒 collected from them. For the 𝑐-th classifier, 𝐾 + 1 mass values are all non-negative and their sum is one:𝑢 𝑐 + 𝐾 ∑︁ 𝑘=1 𝑏 𝑐 𝑘 = 1 (2)Figure 3 : Overview of our methods for adaptive deep network. Our proposed CDM fusion is suitable for both anytime prediction and budgeted batch prediction settings as shown in Figure 2 .where 𝑏 𝑐 𝑘 represents belief mass (the probability of the 𝑘-th class on the 𝑐-th classifier) and 𝑢 𝑐 is uncertainty. The Dirichlet parameters 𝛼 𝑐 , evidence 𝑒 𝑐 , belief mass 𝑏 𝑐 𝑘 and uncertainty 𝑢 𝑐 are defined as:𝛼 𝑐 𝑘 = 𝑒 𝑐 𝑘 + 1 = 𝑆𝑜 𝑓 𝑡𝑃𝑙𝑢𝑠 (𝑝 𝑐 𝑘 ) + 1, 𝑏 𝑐 𝑘 = 𝑒 𝑐 𝑘 𝑆 𝑐 , 𝑢 𝑐 = 𝐾 𝑆 𝑐 (3)where 𝑝 𝑐 𝑘 is the output of these classifiers, as described in Eq. 1 and𝑆 𝑐 = 𝐾 𝑘=1 (𝑒 𝑐 𝑘 + 1) is Dirichlet strength. 𝑆𝑜 𝑓 𝑡𝑃𝑙𝑢𝑠 (•) is a smoothed ReLU activation function.We can quantify the uncertainty 𝑢 𝑐 by optimizing the Dirichlet distributions [30] of different classifiers:EQUATIONwhere 𝑉 𝑎𝑟 (𝑓 𝑐 (𝑥; 𝜃 𝑐 )) denotes the variance of the Dirichlet distribution.The obtained uncertainty 𝑢 𝑐 can measure the reliability of 𝑐-th classifier. Then, we design an uncertainty-aware attention mechanism to weight and fuse the decisions from different classifiers based on their reliability (uncertainty values). The key design is to focus more on classifiers with higher reliability (lower uncertainty) during decision fusion. For simplicity, we take the fusion of the two classifiers for instance and the proposed uncertainty-aware fusion process is formulated by:û = 𝑢 𝑐 𝑢 𝑐+1 (5) b𝑘 = 𝑏 𝑐 𝑘 𝑏 𝑐+1 𝑘 + 𝑏 𝑐 𝑘 (1 -𝑢 𝑐 ) + 𝑏 𝑐+1 𝑘 (1 -𝑢 𝑐+1 ) (6) Ŝ𝑐 = 𝐾/ û𝑐 , ê𝑐 𝑘 = Ŝ𝑐 • b𝑘 (7)We first fuse uncertainty 𝑢 and belief mass 𝑏 through elementwise multiplication. Such form (𝑢 𝑐 𝑢 𝑐+1 ) and 𝑏 𝑐 𝑘 𝑏 𝑐+1 𝑘 ) has the advantage of enhancing compatible information while suppressing conflicting portions, helping gather information where the two classifiers reach a consensus. Then, we use 1 -𝑢 𝑐 and 1 -𝑢 𝑐+1 as attention weights (also called attention term) to weight the decision information from 𝑏 𝑐 𝑘 and 𝑏 𝑐+1 𝑘 . When the uncertainty value is high, we reduce the contribution of this classifier to the fusion because it shows low reliability, and vice versa. Further, the fused evidence ê𝑐 𝑘 is used for final classification.However, the accumulative multiplication form in Eq. 5 and 6 will cause two issues. First, in Eq. 2, it holds that 𝑢 𝑐 < 1 and 𝑏 𝑐 𝑘 < 1. If one of them approaches 1 prematurely (e.g., it approaches 1 only in the first 3 classifiers after fusion), the fusion value will change very slowly, which means the fusion between classifiers will be invalid prematurely and the 𝑐-th classifier cannot obtain useful knowledge from the (𝑐 -1) classifiers. We call this issue as fusion saturation, as shown in Figure 6 (a) . If it appears earlier, the accuracy of the 𝑐-th classifier will decrease after fusion with (𝑐 -1) classifiers and even be worse than a single 𝑐-th classifier without fusion.Second, before the appearance of the fusion saturation, the fusion values will change sharply due to the accumulative multiplication form of fusion. In Eq. 5-7, taking 𝑏 𝑐 k for instance here, it will increase sharply after fusion if the classifier to be fused shows the high confidence of the k-th class (presents high 𝑏 𝑐 -1 k ). Moreover, 𝑏 𝑐 0 , 𝑏 𝑐 1 , ..., 𝑏 𝑐 𝑘 where 𝑘! = k will decrease sharply after fusion at the same time. we call this issue as fusion unfairness. Hence, if the (𝑐 -1)-th classifier presents the fake high confidence towards the k-th class during fusion, the current sample almost cannot be classified correctly in the latter fusion. One reason is that the belief mass of k class is too high to be adjusted. Another reason is the fusion saturation issue mentioned above, which leads to invalid fusion because the fusion value changes very slowly. Hence, fusion saturation and fusion unfairness issues may result in generating an overconfident fusion result, leading to wrong classification.To relieve the fusion saturation and fusion unfairness issues, we introduce the balance term into Eq. 6 and Eq. 5. Specifically, we introduce weighting and sum operation into the fusion process for the purpose of slowing down the changing trend of fusion parameters uncertainty and belief mass:EQUATIONEQUATIONEQUATIONwhere newly added Eq. 10, coefficient 0.5 in Eq. 5 and Eq. 6 is the balance term to improve the fusion quality. For obtaining the fused decision of 𝑐-th classifier, all 𝑐 -1 classifiers will be sent to the CDM for fusion based on uncertainty-aware attention mechanism, where 𝑐 ⩾ 2. Finally, the fused evidence 𝑒 𝑐 𝑘 based on balance term can be obtained through Eq. 7 for final classification. The algorithm pseudocode of uncertainty-aware fusion is shown in Algorithm 1.The fusion performance of multiple classifiers depends on the accuracy and diversity of them [2, 3, 23] . Hence, if we can improve the accuracy of early classifiers and their diversity, the performance of CDM can be enhanced. Considering that the overall performance of the last classifier across all classes is better than early classifiers, we exert regularization between the last classifier and others (called regularized training), using the last one to guide the learning process of early classifiers. Specifically, the Jensen-Shannon divergence [22] is used to pull close the distribution of the last classifier and early ones. We name the CDM based on regularized training as Guided Collaborative Decision Making (GCDM). To make the distribution of the last classifier and the early one more distinguishable, we use temperature-scaled distribution [8] of the classifier logits 𝑓 𝑐 (𝑥, 𝜃 𝑐 ; 𝜏) instead of original distribution 𝑓 𝑐 (𝑥; 𝜃 𝑐 ):EQUATION𝑓 𝐶 (𝑥, 𝜃 𝐶 ; 𝜏) = 𝑆𝑜 𝑓 𝑡𝑃𝑙𝑢𝑠 (𝑝 𝐶 )/𝜏 = 𝑒 𝐶 /𝜏 (12) where 𝑐 ≠ 𝐶 and 𝐽𝑆 (•∥•) denotes the Jensen-Shannon divergence. We minimize the Jensen-Shannon divergence between the last classifier (𝑓 𝐶 (𝑥, 𝜃 𝐶 ; 𝜏)) and early classifiers. The final loss function to optimize GCDM is:EQUATIONL 𝐽 𝑆 = 𝐶 -1 𝑐=1 (𝐽𝑆 (𝑥; 𝑐, 𝜏 obtain fused 𝑢 2 based on (𝑢 1 , 𝑢 2 ) and Eq. 85obtain fused 𝑏 2 based on (𝑏 1 , 𝑏 2 ) and Eq. 96obtain 𝑒 2 based on ( 𝑢 2 , 𝑏 2 ) and Eq. 7 obtain fused 𝑢 𝑐+1 based on ( 𝑢 𝑐 , 𝑢 (𝑐+1) ) and Eq. 810obtain fused 𝑏 𝑐+1 based on ( 𝑏 𝑐 , 𝑏 (𝑐+1) ) and Eq. 911obtain 𝑒 (𝑐+1) based on ( 𝑢 𝑐+1 , 𝑏 𝑐+1 ) and Eq. 7  
experiments 4.1 experimental setup: 
 Datasets. We use large-scale ImageNet1000, ImageNet100, CIFAR10, and CIFAR100 datasets in experiments. Following [13, 19, 28] , all datasets are divided into training, validation, and testing sets. The batch size for the ImageNet100 dataset is fixed at 64 for all methods. Other settings about datasets are as same as in previous work based on their public source codes.Baselines. For adaptive deep networks, we use advanced MS-DNet [13] , RANet [28] and IMTA [19] to create strong baselines. IMTA is an advanced improved technique for adaptive deep networks. For MSDNet, according to the number of blocks between classifiers, there exist two different structures: "E" structure MSDNet 𝐸 (the number of blocks between classifiers is equidistant) and "LG" structure MSDNet 𝐿𝐺 (the number of blocks between classifiers is linearly growing). For RANet, similarly, if the number of layers in each 𝐶𝑜𝑛𝑣𝐵𝑙𝑜𝑐𝑘 is the same or linearly growing, called RANet 𝐸 or RANet 𝐿𝐺 respectively. Detailed model structures refer to the appendix. We don't compare with ensembling multiple independent networks because it performs worse than MSDNet and RANet, as proved in their papers. For fusion methods baselines, we select averaging fusion, weighted averaging fusion, voting fusion, neural network fusion, and multiview fusion [10] based on EDL.Hyperparameters. CDM doesn't involve hyperparameters. For GCDM, the hyperparameters 𝜏 1 and 𝜏 2 in Eq. 14 are set to 0.5 and 1, respectively. Overall, our method involves only a few hyperparameters. For hyperparameters of baselines, we directly use the setting in their public source codes. To ensure a fair comparison, our method is used with the same hyperparameters when combined with other methods. The only difference is whether applying our CDM or GCDM to them. All experiments in this study are conducted on NVIDIA GeForce RTX 3090 GPU based on PyTorch. More experimental details, results, and discussion can be seen in the full version paper (can be found in our code link). We combine our GCDM with popular adaptive deep networks on various datasets and two settings. The results of anytime prediction setting are shown in Table 1 . Our method brings a stable performance improvement to the current popular networks, whether on large-scale datasets ImageNet1000, or other datasets (CIFAR10, CAIFR100, and ImageNet100). For RANet, the average improvement in accuracy on CIFAR10, CIFAR100, ImageNet100, and ImageNet1000 is 0.479%, 2.44%, 1.86%, and 0.84%, respectively. The maximum accuracy improvement is 1.04%, 3.99%, 2.83%, and 1.46%, respectively. Moreover, the results of budgeted batch prediction setting in Figure 4 (ImageNet100 and ImageNet) and Figure 5 (CIFAR10 and CIFAR100) also prove that our GCDM consistently improves the classification accuracy of popular adaptive deep networks such as MSDNet and RANet by a large margin under the same computational resources (measured by FLOPs). The consistent improvements in the above two settings demonstrate the effectiveness of GCDM. Besides, although the experiments were conducted on the "𝐿𝐺" network structures, we still observed consistent performance improvements with our GCDM on the "𝐸" structures, which can be found in the full version paper in our code link.Ablation of each component. We conduct the ablation study on MSDNet with CIFAR100 and ImageNet100 datasets to explore the effectiveness of the proposed CDM and GCDM. The results are shown in Table 2 . The proposed CDM significantly improves the performance of the baseline method MSDNet. Moreover, most classifiers among MSDNet equipped with regularized training (G + ) perform better than the original one (0 + ), denoting G + improves the accuracy of early classifiers. Furthermore, because regularized training doesn't obviously reduce the diversity of the early classifiers, it can further enhance the fusion performance of CDM. Consequently, we observe that MSDNet equipped with GCDM (𝐺 + + 𝐶𝐷𝑀 + ) obtains better accuracy than 𝐺 + and 𝐶𝐷𝑀 + . The above results validate the effectiveness of the design of CDM and GCDM. Diversity of early classifiers after regularization (Eq. 14). We have proved that regularized training can raise the accuracy of early classifiers. Whether it can improve CDM critically depends on whether it would harm the diversity of early classifiers. Hence, we calculate the diversity metrics of all classifiers of MSDNet 𝐸 on CI-FAR100 and ImageNet100. The results shown in Table 3 demonstrate that regularized training even can slightly increase the diversity Table 3 : Diversity metrics of correlation coefficient (Cor.), Q-statistic(Q-sta.), Kohavi-Wolpert variance (Var.) [18] and agreement value(Agr.) [27] on MSDNet 𝐸 (equipped with 10 classifiers). 𝐺 + denotes regularized training; 0 + is the opposite. ↓ means lower is better and vice versa. Stable Training Strategy (STS) in loss function Eq. 14. Results in Figure 8 (b) reveal an interesting observation. When using either 𝜏 = 1 or 𝜏 = 0.5, individually, RANet 𝐸 cannot consistently achieve better performance. For example, RANet 𝐸 trained with 𝜏 = 1 performs better for the first four classifiers while it trained with 𝜏 = 0.5 performs better for the latter four classifiers. This shows performance instability issues. However, after introducing STS (using both 𝜏 = 1 and 𝜏 = 0.5 for weighted JS loss) during training, RANet 𝐸 exhibits better performance in most classifiers. This suggests that the introduced STS is effective in relieving the performance instability issue. we compare our method with different decision fusion methods on large-scale ImageNet1000 dataset, including traditional average, weighted average, voting, neural network fusion method, as well as multi-view fusion method [9] based on evidential learning. For the weighted average fusion, we normalize the accuracy of the classifiers on the validation set to obtain the weights for each classifier.For Neural Network (NN) weighted fusion, we allocate an MLP for 𝑐-th (𝑐 ≥ 2) classifier for fine-tuning based on the well-trained adaptive deep networks. The comparison results are shown in Table 4. We observe that traditional and multi-view fusion methods are even poorer than the original RANet while our fusion is better than all other methods. This is because traditional fusion methods don't take into account uncertainty, and multi-view fusion doesn't consider the issues of fusion saturation and fusion unfairness. Effectiveness of designed balance term and attention term. We conduct the fusion experiment on ImageNet1000 for CDM with balance term (abbreviation is GCDM 𝑒+ ) and without balance term (abbreviation is GCDM 𝑒 ). We visualize the changing trend of the believe mass b𝑘 under different times of fusion, as shown in Figure 6(a) . We find that GCDM 𝑒+ successfully slows down the changing trend of the fusion process, relieving the fusion saturation and fusion unfairness issues. In other words, GCDM 𝑒+ won't lead to prematurely fusion saturation. The high belief mass of a certain class (e.g., 1) won't lead to a sharp decrease in the belief mass of other classes (e.g., 0 and 2), which means that the fusion unfairness issue is relieved. We also record the belief mass increment between 𝑐-th and (𝑐 -1)-th fusion for both GCDM 𝑒+ and GCDM 𝑒 . Finally, we calculate the increment ratio between GCDM 𝑒+ and GCDM 𝑒 , as shown in Figure 6(b) . We find that the belief mass increment ratio is larger than GCDM 𝑒 with the increase of fusion times, which ensures the effectiveness in the following fusion operations. Hence, GCDM 𝑒+ can obtain better performance than GCDM 𝑒 , proving the 4 , we also can observe that the performance of RANet-CDM decreases after removing the balance term, further proving the effectiveness of designed balance term. Moreover, performance sharply declines after further removing the attention term, especially for the last classifier CF8. This indicates that the basic attention term is crucial in the uncertainty-aware fusion.Interested regions visualization of different blocks in MS-DNet. As shown in Figure 7 , based on the cat and dog dataset, we visualize the interested regions of different blocks in the welltrained MSDNet 𝐸 . The top row is visualized on Grad-CAM and the bottom row is on Guided Grad-CAM [24] for pixel-level visualization. We can find that different classifiers capture different regions. Even more interesting is that we use the fusion result of the total 6 classifiers to finish the Guided Grad-CAM visualization and find that the final classifier after fusion can capture more features of the input image. It means that the 𝑐-th classifier can fuse the knowledge of (𝑐 -1) classifiers by using CDM and hence improve the performance of the 𝑐-th classifier.To further validate the effectiveness of CDM and GCDM, we conduct experiments by combining them with improved techniques. IMTA [19] is advanced two-stage improved techniques, which proposes the Gradient Equilibrium, and Forward-backward Knowledge Transfer (FKT) algorithms for improving training of adaptive deep networks. We name the model that uses only Gradient Equilibrium as GE, and both GE and FKT (whole two-stage training) as IMTA. The results of budgeted batch prediction on ImageNet100 shown in Figure 8 (a) can be analyzed as follows: (1) MSDNet 𝐸-𝐼 𝑀𝑇 𝐴 is better than MSDNet 𝐸-𝐺𝐸 , proving the two-stage training method further enhancing the performance of MSDNet 𝐸-𝐺𝐸 . (2) MSDNet 𝐸-𝐼 𝑀𝑇 𝐴 -CDM is better than MSDNet 𝐸-𝐼 𝑀𝑇 𝐴 and MSDNet 𝐸-𝐺𝐸 -CDM is better than MSDNet 𝐸-𝐺𝐸 , demonstrating CDM is effective and can be combined with current techniques for better performance.(3) MSDNet 𝐸-𝐺𝐸 -GCDM outperforms both MSDNet 𝐸-𝐼 𝑀𝑇 𝐴 and MSDNet 𝐸-𝐼 𝑀𝑇 𝐴 -CDM and obtains the best performance, indicating the combination of GCDM and GE algorithm can form a good single-stage method to replace existing two-stage training IMTA.Discussions on different loss functions. We use L 𝑢 = L 𝐽 𝑆 + 𝐶 𝑐=1 𝜆 1 L 𝑐 𝐶𝐸 (𝑝 𝑐 , 𝑦) (cross-entropy loss) to replace the loss function (Eq. 13) and observe the accuracy changes. The results are shown in Figure 9 . The conclusion is our CDM isn't sensitive to the choice of loss function, as it consistently yields stable performance improvements when using other loss functions, e.g., cross-entropy loss. This further proves our ideas' effectiveness. Discussions on Calculation Costs. During the training stage, the additional cost is the added loss function Eq. 14, which can be negligible. During the inference stage, the extra computation cost comes from our uncertainty-aware fusion scheme between two adjacent classifiers. In fact, such extra computation costs also are negligible. Specifically, for inferring 50,000 images on the Ima-geNet1000 dataset, MSDNet with no fusion (MSD) requires 330.16 seconds, while our method MSDNet-CDM with fusion (MSD-CDM) takes 330.37 seconds. Similarly, RANet with no fusion (RAN ) requires 342.15 seconds, while our method RANet-CDM with fusion (RAN-CDM) takes 342.53 seconds. 
conclusion: 
 In this paper, we propose Collaborative Decision Making (CDM) and Guided Collaborative Decision Making (GCDM) to improve the classification performance of adaptive deep networks. CDM incorporates an uncertainty-aware fusion method to fuse decisions of different classifiers based on their reliability (uncertainty values). We also introduce a balance term to alleviate the fusion saturation and unfairness issues caused by the evidential deep learning framework, hence enhancing CDM's fusion quality. GCDM is designed to further improve CDM's performance through regularized training over earlier classifiers using the last classifier. Extensive experiments on CIFAR10, CIFAR100, ImageNet100 and ImageNet1000 show that our proposed CDM module and GCDM framework can consistently improve the performance of adaptive networks. 
 
 Appendix:  


Decision: Excellent work on uncertainty-aware decision fusion albeit that some minor improvements are still possible.

Reviews: review: paper_summary: paper_summary: This manuscript presents two methods, Collaborative Decision Making (CDM) and Guided Collaborative Decision Making (GCDM), to improve the performance of adaptive deep neural networks. CDM works by fusing the decisions of multiple classifiers within the network based on their uncertainty, using an uncertainty-aware attention mechanism to give more weight to more reliable classifiers. GCDM further enhances CDM by using regularized training, where the final classifier in the network guides the training of earlier classifiers for improving their accuracy without significantly impacting their diversity. The experiments are conducted on four datasets: CIFAR-10, CIFAR-100, ImageNet-100, and ImageNet-1000 and show that the proposed methods enhance the performance of the selection of adaptive deep networks involved in the experiments.
strength_weakness: strength_weakness: The concepts presented in the manuscript are comprehensible and straightforward. The writing quality is excellent. The paper appropriately references and utilizes relevant sources. The treatment of the subject is complete.

The approaches are orthogonal. Different from existing work in the literature, the method fuses the decisions of multiple classifiers using an uncertainty-aware approach and regularized training for optimizing the use of all classifiers' decision information.

The introduction of the balance term seems to have a positive impact on the fusion saturation and unfairness issues in CDM by slowing down the fusion process and improve the performance in subsequent fusion operations.

Ablation study is performed. The manuscript tested: 1) the impact of CDM and GCDM over the MSDNet literature approach, showing that both improve the baseline results, 2) the impact of the regularized training, showing that it doesn’t negatively influence the diversity of early classifiers and 3) the impact of the stable training strategy, showing that it can counteract the performance instability issues in training caused by either too long or too short distances between early classifiers and the final classifier.

The manuscript also experimented combining the proposed CDM and GCDM with Gradient Equilibrium and Forward-backward Knowledge Transfer literature approaches, showing their compatibility with current techniques for improved performance.

The source code is provided for reproducibility.
limitations: limitations: Missing insights into potential failure scenarios where the proposed methods may not perform optimally would enable a better understanding of the limitation of the approaches.

Minors
A few typographical errors exist in the manuscript, for example:
Row 919: “ ... their reliability(uncertainty values)” -> missing space between reliability and "(".
Row 836: missing before the reference;
etc.
paper_summary: paper_summary: This paper proposes methods to improve the classification performance of adaptive deep networks, namely Collaborative Decision Making (CDM) and Guided Collaborative Decision Making (GCDM). Experiments on ImageNet, CIFAR10 and CIFAR100 datasets show that CDM and GCDM can consistently improve the classification accuracy of state-of-the-art adaptive networks like MSDNet and RANet.
strength_weakness: strength_weakness: + The paper is clearly written and easy to follow. The key ideas and contributions are described in detail.
+ The related work review is concise and accurate.
+ The authors publicize the training and evaluation code.
+ Consistent performance improvement can be observed from the experiments for different baseline models.
+ Sufficient ablation studies for different modules and factors.
limitations: limitations: - The illustration of CDM and GCDM in Figure 3 could be improved. They are not clear enough to show the idea.
paper_summary: paper_summary: The paper introduces a novel method, the Collaborative Decision Making (CDM) module, aimed at enhancing the inference performance of adaptive deep networks for image classification. This is achieved through an uncertainty-aware decision fusion technique that improves the accuracy of a selected classifier by leveraging the reliability (uncertainty values) from other classifiers in the network.
strength_weakness: strength_weakness: 1.The introduction of the Collaborative Decision Making (CDM) module is a significant advancement in adaptive deep networks. It uniquely addresses the limitations of existing methods by utilizing earlier classifiers when they are more accurate, thus optimizing computational resources and improving performance.
2. Technical Correctness and Evaluation: The paper's methodology is technically sound, evidenced by thorough experimental evaluations on the ImageNet datasets. The results demonstrate an accuracy improvement of 0.4% to 2.8% under varying computational constraints, which substantiates the effectiveness of the proposed approach.
3. Theoretical Approach: The use of evidential deep learning (EDL) to quantify uncertainty and manage decision fusion is a robust theoretical addition. It not only enhances classifier reliability but also introduces a balance term to mitigate fusion saturation and unfairness, which are common in EDL applications.
4 .Clarity: The paper is well-organized and clearly written, making complex concepts accessible to readers. Figures and tables effectively illustrate the methodology and results, enhancing the paper's overall clarity and impact.
limitations: limitations: Lack of Comparative Analysis: While the paper provides a comparison with state-of-the-art models, it primarily focuses on improvements over traditional single-head classifiers. A broader comparison with other multi-head approaches or networks utilizing different types of decision fusion could strengthen the argument.
paper_summary: paper_summary: This paper comploys multiple classiffer heads among different layers to improve decision making. this is an interesting work.
strength_weakness: strength_weakness: A Collaborative Decision Making idea is proposed to improve the performance of popular adaptive deep networks and experiments demonstrate that is is effective.
limitations: limitations: the word size in figure is too bold.