user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Efficient Inference on Deep Neural Networks by Dynamic Representations and Decision Gates

Paper content:
The current trade-off between depth and computational cost makes it difficult to adopt deep neural networks for many industrial applications, especially when computing power is limited.
Here, we are inspired by the idea that, while deeper embeddings are needed to discriminate difficult samples, a large number of samples can be well discriminated via much shallower embeddings.
In this study, we introduce the concept of decision gates (d-gate), modules trained to decide whether a sample needs to be projected into a deeper embedding or if an early prediction can be made at the d-gate, thus enabling the computation of dynamic representations at different depths.
The proposed d-gate modules can be integrated with any deep neural network and reduces the average computational cost of the deep neural networks while maintaining modeling accuracy.
Experimental results show that leveraging the proposed d-gate modules led to a ~38% speed-up and ~39% FLOPS reduction on ResNet-101 and ~46% speed-up and $\sim$36\% FLOPS reduction on DenseNet-201 trained on the CIFAR10 dataset with only ~2% drop in accuracy.
Past studies such as BID15 have shown that deeper architectures often lead to better modeling performance; however, deeper architectures also pose a number of issues.
Besides becoming more prone to overfitting and becoming more difficult to train, the trade-off between depth and computational cost makes it difficult to adopt deeper architectures for many industrial applications.He et al. BID6 tackled the former issue of degradation in learning deeper neural networks (e.g., vanishing gradient) by introducing the concept of residual learning, where learning is based on the residual mapping rather than directly on the unreferenced mapping.
Following that, Xie et al. BID18 took advantage of the inception idea (i.e, split-transform-merge strategy) within a residual block structure to provide better subspace modeling while resolving the degradation problem at the same time, resulting in a ResNext architecture with improved modeling accuracy.
To tackle the issue of computational cost, a wide variety of methods have been proposed, including: precision reduction BID9 , model compression BID5 , teacher-student strategies BID7 , and evolutionary algorithms BID12 BID13 .More
recently, conditional computation BID0 BID3 BID11 BID17 BID1 and early prediction BID16 methods have been proposed to tackle this issue, which involve the dynamic execution of different modules within a network. Conditional
computation methods have largely been motivated by the idea that residual networks can be considered as an ensemble of shallower networks. As such, these
methods take advantage of skip connections to determine which residual modules are necessary to be executed, with most leveraging reinforcement learning.In this study, we explore the idea of early prediction but instead draw inspiration from the soft-margin support vector BID2 theory for decision-making. Specifically,
we introduce the concept of decision gates (d-gate), modules trained to decide whether a sample needs to be projected into a deeper embedding or if an early prediction can be made at the d-gate, thus enabling the conditional computation of dynamic representations at different depths. The proposed
d-gate modules can be integrated with any deep neural network without the need to train networks from scratch, and thus reduces the average computational complexity of the deep neural networks while maintaining modeling accuracy.
The efficacy of the proposed d-gate modules is examined with two different network architectures (ResNet101 BID6 and DenseNet201 BID8 ) on the CIFAR10 dataset.
A key benefit of the proposed d-gate modules is that it enables fine control over the trade-off between modeling accuracy and computational cost by adjusting the d-gate decision thresholds.
By decreasing the d-gate decision thresholds, the number of samples undergoing early prediction increases, thus reducing the average computational cost of network predictions greatly.
For this study, we integrated two d-gate modules in ResNet-101 (after the first and second main blocks) and DenseNet-201 (after the first and second dense blocks), and explore different d-gate configurations.
The networks are implemented in the Pytorch framework and the prediction speeds are reported based on single Nvidia Titan Xp GPU.It can be observed from TAB0 that the computational cost of ResNet network can be reduced by 67 MFLOPS while maintaining the same level of accuracy as to the original ResNet-101 by integrating two d-gate modules with decision thresholds of (t1, t2) = (2.5, 2.5).
The integration of d-gate modules can reduce the computational cost of ResNet-101 network by ∼39% (i.e., lower by 1.95 GFLOPS) with 1.7% drop in accuracy compared to the original ResNet-101 (with distance thresholds (t1, t2) = (1.0, 2.0) in d-gate1 and d-gate2), resulting in a ∼38% speed-up.
The experiments for DenseNet-201 show that it is possible to reduce the number of FLOPs by 970 MFLOPs (∼36% reduction) with only a ∼2% drop in accuracy, leading to a ∼46% speed-up.
Furthermore, a 2.3× speed-up can be achieved with d-gate modules compared to the original DenseNet-201 within a 3% accuracy margin.
Based on the experimental results, the proposed d-gate modules lead to a significant increase in prediction speed, making it well-suited for industrial applications.In addition to the d-gate modules being proposed, one of the key contributions of this paper is the introduction of a hinge loss for training the d-gate modules.
Past studies BID10 have argued that crossentropy results in a small margin between the decision boundaries and the training data.
As such, it is very difficult to trust the confidence values of the Softmax layer to decide about the sample since there is no valuable information in the Softmax output.
To demonstrate the effectiveness of the hinge loss leveraged in the proposed d-gates compared to the cross-entropy loss, an additional comparative experiment was conducted.
More specifically, two decision gates were added to ResNet101 in the same way as reported.
However, rather than train using the proposed hinge loss, the decision gates were instead trained via a cross-entropy loss.
This enables us to compare the effect of hinge loss vs. cross-entropy loss on decision gate functionality.
FIG1 demonstrates the accuracy vs. number of FLOPs for the network where the decision gates were trained based on the proposed hinge loss approach compared to trained using a regular cross-entropy training procedure.
It can be observed that, with the same number of FLOPs in the network, the network where the decision gates were trained based on the proposed hinge loss provides much higher modeling accuracy compared to that trained via cross-entropy loss.
The accuracy gap increases exponentially when the decision gates are configured such that the network uses fewer number of FLOPs.
What this illustrates is the aforementioned issue with the use of cross-entropy loss and decision boundaries.

assistant: text: This paper introduces a new dynamic feature representation approach to provide a more efficient way to do inference on deep neural networks.