user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Doubly Nested Network for Resource-Efficient Inference

Paper content:
We propose a new anytime neural network which allows partial evaluation by subnetworks with different widths as well as depths.
Compared to conventional anytime networks only with the depth controllability, the increased architectural diversity leads to higher resource utilization and consequent performance improvement under various and dynamic resource budgets.
We highlight architectural features to make our scheme feasible as well as efficient, and show its effectiveness in image classification tasks.
When we deploy deep neural network models on resource-constrained mobile devices or autonomous vehicles with a strict real-time latency requirement, it is essential to develop a model which makes the best of available resources.
Although many network compaction techniques including distillation, pruning and quantization have been proposed BID7 BID12 BID4 BID3 BID9 BID6 BID13 , this goal is still challenging because (1) the resource availabilities are continuously changing over time while these resources are being shared with other program instances BID1 , and (2) multiple resources with different characteristics (e.g. computational capacity, memory usage) should be considered together.Anytime machine learning algorithms have addressed the first issue, how to get the optimal performance under dynamic resource budgets, by allowing us to activate only a part of the model with graceful output quality degradation BID14 BID2 .
Most anytime algorithms based on deep neural networks appear in the form of early termination of forward processing according to the current resource budget or the difficulty of the given task BID8 BID0 BID10 .
In other words, the conventional anytime networks are trained to embed many potential sub-networks with different effective depths so that the best one can be chosen according to the current budget.In this work, we propose a new type of the anytime neural network, doubly nested network, to solve the other issue, more efficient utilization of multiple heterogeneous resources.
The proposed network can be sliced along the width as well as the depth to generate more diverse sub-networks than the conventional anytime networks allowing the sub-network extraction only along the depth-wise direction.
As depicted in FIG0 , the increased degree of freedom enables us to get higher resource utilization in the devices constrained by dynamically changing resource budget with multiple criteria.
Causal convolution It is straightforward to form a sub-network along the depth by appending a separate output generation stage to the final convolution layer of the sub-network.
Since one specific layer's output does not depend on the following (upper) layers' outputs, the jointly trained subnetwork does not suffer from the performance degradation even after the extraction.
However, this approach does not work along the width because of the interdependence between two nodes at different horizontal locations (e.g. different channels).
To address this issue, we propose a channelcausal convolution where i-th channel group in one layer is calculated only with activation values from the channel groups from the first to i-th channel group in the previous layer as shown in the right of FIG1 .
The circle indicates the feature map while the square indicates the classifier.
Color refers each channel.
Our network based on the causal convolution allows us to extract the sub-network easily along any directions by making both horizontal and vertical data flow unidirectionally.Output generation stage sharing fully-connected layers Rather than having a separate fullyconnected (FC) layer for one sub-network to generate the final output (e.g. a predicted class given image input), our network is designed to have the FC layers each of which takes only a part of activations from the preceding convolution layers and produce the final output for one sub-network by averaging multiple FC layers' outputs as depicted in FIG1 .
Sharing the FC layers between the sub-networks at the same depth helps us to have similar computational and memory costs of the FC layers in the depth-controllable anytime network BID8 even with much more possible output locations.
We revealed that resource-constrained devices could benefit from the architectural diversity enriched by our anytime prediction scheme.
Our future works include adding adaptive conditioning BID11 which modulates intermediate activations or weight parameters depending on the current sub-network configuration to improve the performance only with a small increase of conditioning parameters.

assistant: text: We propose a new anytime neural network which allows partial evaluation by subnetworks with different widths as well as depths.