ItNet: iterative neural networks for fast and efficient anytime prediction
Authors: Thomas Pfeil
Venue:  (2021)
Keywords: efficient deep neural network
semantic segmentation
parameter sharing
anytime prediction
tiny network graph
massively parallel hardware systems
recurrent convolutional network

Abstract: Deep neural networks have usually to be compressed and accelerated for their usage in low-power, e.g. mobile, devices. Common requirements are high accuracy, high throughput, low latency, and a small memory footprint. A good trade-off between accuracy and latency has been shown by networks comprising multiple intermediate outputs. In this study, we introduce a multi-output network that has a tiny memory footprint in terms of its computational graph, which allows its execution on novel, massively-parallel hardware accelerators designed for extremely high throughput. To this end, the graph is designed to contain loops by iteratively executing a single network building block. These so-called iterative neural networks enable stateof-the-art results for semantic segmentation on the CamVid and Cityscapes datasets that are especially demanding in terms of computational resources. In ablation studies, the improvement of network training by intermediate network outputs as well as the trade-off between weight sharing over iterations and the network size are investigated.

Under review as a conference paper at ICLR 2021
ITNET: ITERATIVE NEURAL NETWORKS FOR FAST AND EFFICIENT ANYTIME PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural networks have usually to be compressed and accelerated for their usage in low-power, c.g. mobile, devices. Common requirements are high accuracy, high throughput, low latency, and a small memory footprint. A good trade-off between accuracy and latency has been shown by networks comprising multiple intermediate outputs. In this study, we introduce multi-output network that has tiny memory footprint in terms of its computational graph, which allows its execution on novel, massively-parallel hardware accelerators designed for extremely high throughput. To this end, the graph is designed to contain loops by iteratively executing single network building block. These so-called iterative neural networks enable state- of-the-art results for semantic segmentation on the CamVid d and Ci tascapes datasets that are especially demanding in terms of computational resources. In ablation studies, the improvement of network training by intermediate network outputs as well as the trade-off between weight sharing over iterations and the network size are investigated.
INTRODUCTION
For massively-paras hardware accelerators (Schemmel et al.) 2010, Merolla et al. 2014; Yao al. 2020 Graphcore| 2020aa), every neuron and synapse in the network model has its physical counterpart on the hardware system. Usually, by design, memory and computation is not separated anymore, but neuron activations are computed next to the memory, i.c. the parameters, and fully in parallel. This is in contrast to the rather sequential data processing of CPUs and GPUs, for which the computation of a network model is tiled and the same arithmetic unit re-used multiple times for different neurons. Since the computation is performed fully in parallel and in memory, the throughput of sassivvely-praalle accelerators is usually much higher than for CPUs and GPUs. This can be attributed to the fact that the latency and power consumption for accessing local memory, like for in-memory computing, are much lower than for computations on CPUs and GPUs that require the frequent access to non-local memory like DRAM (Sze et al. 2017). However, the network graph hardware accelerators to allow for maximal
has to fit into the memory of the mnassivelypparalee throughput. If the network graph exceeds the available memory, in principle, the hardware has to be re-configured at high frequency as it is the case for CPUs and GPUs and the throughput would be substantially reduced. Mixed-signal massively-parallel hardware systems usually operate on shorter time scales than digital ones, e.g. compare Schemmel et al. (2010) and Yao et al. (2020) to Merolla al. 22014) and Graphcore (0220),, and would allow for even higher throughputs.
In order to achieve neural networks with tiny computational graphs, in which nodes are operations and edges are activations, we heavily re-use single building block of the network (see the iterative block in Figure [1). Not only the structure of computations, i.c. the type of network layers including their dimensions and connectivity, is identical for each iteration of this building block, but also the weights are shared between iterations. In the computational graphs of these so- so-called iterative neural networks (ItNet), the re-used building blocks with shared weights can be represented by nodes with self-loops. Compared to conventional feed-forwan networks, loops simplify the graph by reducing the number of unique nodes and, consequently, its computational footprint. However, the restriction of sharing weights usually decreases the number of free parameters and, hence, the accuracy of networks. To isolate and quantify this effect we compare networks with weight sharing to networks, for which the parameters of the building blocks are chosen be independent between iterations of

---

Under review as a conference paper at ICLR 2021
b)
0.6
Semantic m.
Data Block
Iterative Block
Class. Block
mlou
Image
Ittlet ESPNetv2 ReckiNet E
weights are
shared:
0.4
20 MACs
10
1000
2000
(10")
graph size (MB)
latency (a.u.)
5550
00112
81576
91596
SV602
Image
Label
output
output
output
output
output
output
Figure This study in a nutshell. a) The iterative neural network (ItNet); first, images are pre- processed and potentially down-scaled by sub-network called data block. Then, the output of this data block is processed by another sub-network that is iteratively executed. After every iteration n, the output of this iterative block is fed back to its imput and, at the same time, is further processed by the classification block to predict the semantic map m,, This network generates multiple outputs m, with increasing quality and computational costs and heavily re-uses intermediate network activations. While the parameters are shared between iterations the iterative block, the parameters of the classification block are independent for each n. b-d) Intersection-over-une accumulate operations (MACs), the size of the computational graph, and the latency, respectively. on the validation
(mIoU) over multiply-
the Ci sitysscapes dataset. To our knowledge ESPNetv2 (Mehta al. 2019 the state-of -oc--of-hie af-thh-art in terms of mloU over MACs, for which we show the mloU if not using pre- traitre weights (Ruiz& & Verbeek, 2019). Wang et al. (2019) share the weights of U-Nets and recurrently connect their bottlenecks. Note that the ItNet requires more MACs, but has substantially smaller size of the computational graph, and lower latency than the reference networks. The throughput of ItNets massively-parally
is potentially also higher, since the small graph size allows for the execution hardware systems. e) Image, label, and network predictions (with the mloU in their corner) over the network outputs of the ItNet in (c). The data sample with the 75th-percentile mIoU is shown.
the building block. In contrast the above proposal, conventional deep neural networks for image processing usually do not share weights and have no (e.g. [Hang et al. (207)) or few (e.g. one building block for each scale like by Greff et al. (017) layers of identical structure. Liao Poggio (2016) share weights between re-used building blocks, but use multiple unique building blocks.
To improve the training of networks, which contain loops in their graphs, and to reduce the latency of multiple intermediate outputs. Multi-output networks that heavily
networks during inference re-use intermediate activations are beneficial for a wide range of applications, especially in the mobile domain. In online manner, they allow to trade off latency versus accuracy with barely any overhead (e.g. Huang et al., 2018). From an application point of view, the benefit this trade-off can be best deseribed in the following two scenarios (Huang al. 2018): In the so-called anytime prediction scenario, the prediction of a network is progressively updated, whereas the first output defines the second scenario, limited computational budget can unevenly
initial latency of the network. distributed over a set of samples with different "difficulties" in order to increase the average accuracy,
Since all nodes in the network graph are computed in parallel on massively-paralle hardware systems (e.g. [Esser et al., 2016), the latency for inference is dominated by the depth of the network, i.e. the longest path from input to output (Fischer et al.) 2018). Consequently, we prefer networks that compute all scales in parallel (similar to Huang et al. 2018 Ke et al. (017)) and increase their depth for each additional intermediate output to networks that keep the depth constant and progressively increase their width (Yu al. 2019). Furthermore, multi-scale networks integration of global information, as especially required by dense prediction tasks like semantic segmentation (Zhao et al.) (017). To further reduce the latency we also reduce the depth of the building blocks for each scale.
also beneficial for the
In deep learning literature, the computational costs usually quantified by counting the parameters and/or the coltiply-accumuute operations (MACs) required for the inference single sample. For fully convolutional networks, the number of parameters independent of the spatial resolution
2

---

Under review as a conference paper at ICLR 2021
of the network's input and the intermediate feature maps. Especially for large inputs commonly used for semantic segmentation, the number of parameters does not cover the main workload and is, hence, not suited as a measure for computational costs. MACs have the advantage that they can be easily calculated and are usually a good approximation for the latency and throughput on CPUs and even GPUs. However, for most novel hardware accelerators, not the MACs, but memory transfers are dominating the computational costs in terms of power consumption (Chen et al. 2016) Sze al. 2017: Chao et al. 2019). These memory transfers are minimized on massivly-parallel hardware systems as long as the network graph fits into the in-computation memory of these systems, ie. the memory of their arithmetic units. Since both the power consumption during inference and the production cost scale with the size of this memory, in addition to MACS, we also compare the size of the computational graphs between networks. Note that the practical benefits of ItNets cannot be demonstrated on conventional CPUs or GPUs, since these hardware systems do not support the processing of neural networks in a Sally-paralll and, hence, low-latency fashion.
Note that, in this study, we focus network models in the low-power regime of only few billion MACs, while processing large-scale images as commonly used for semantic segmentation (for datasets, see Section 2.4). The key contributions of this study are:
We introduce efficient networks with tiny computational graphs that heavily re-use single building block and provide multiple intermediate outputs (Sections 2.1
2.2 and 2.5
We search for the best hyperparameters of this model and investigate the effect of multiple outputs and weight sharing on the network training (Sections 2.3 I. and 3.2).
To our knowledge, we the state-of-the-art in terms of accuracy over the size of the computational graph and discuss the potential benefits for low-power applicatios.
We will release the source code upon acceptance for publication.
2
METHODS
The following networks process images of size x x y X 3 and output N semantic maps m. of size x x y x C with C' being the number of classes.
2.1 NETWORK ARCHITECTURE
We are interested in network architectures with a small computational footprint facilitating their application in mobile devices. To this end, we design neural network that heavily re-uses the intermediate activations and weights (Figure (p)). Conceptionally. the network model can be split into three main building blocks: the data block, the iterative block and the classification block (for an overview and details, see Figure 1 and Figure respectively). While the data block is executed only once for each image, the iterative block can be executed multiple times in row by feeding back its output as the input for the next iteration. The classification block outputs the prediction of the semantic map by processing the intermediate activations of the feedback signal. While the weights of the iterative block are shared between iterations, the weights of the classification block are unique for each iteration. We consider the size of the computational graph and the number of MACs meaningful indicators for the computational footprint (see also Section I). To obtain high-acc gggh-accurayy network architecture under these objectives we reduce the size of the computational graph by introducing loops and optimize the following architectural hyperparameters: the number of scales L,, the number of iterations N, and the number of bottleneck residual blocks K.
2.2 NETWORK TRAINING
For training, we use a joint cost function for all outputs of the network:
Σ Cn Cn (mn, mn); n
where c, is the categorical cross entropy between the true labels m, and the network predictions m,, The weight factors an normalized as follows: a. an (E, az).
3

---

Under review as a conference paper at ICLR 2021
a)
b)
Conpert
Idontity
UpSama
UpSamo
ConveR
Add-R
->0
BR6
Add-R
BN
Comp Comp-BNaa BN-R
Cony BIVR E.a:3
Imago
Maxeool
Max
+
ConVeR
02
AvgPoo Identity
UpSamp
Conv-R f.5mk
Add-R
BR
Add-R
BN
Reggh
MSR
Figure 2: Detailed description of the data block (a) and iterative block (b) as shown in Figure ! a) First, images are nown-sampled in two stages, each using stride and pooling of size s (inspired by Paszke et al.] (016). Then, the output of the second stage is processed on I. different scales. For clarity, an example with L is shown. Convolutional layers (with kernel size 3), batch- normalization layers and ReLU activation functions are denoted with Conv, BN and R. respectively, b) The iterative block receives input at L different scales denoted as A, B and C. order to mix the information between the different scales (e.g. Zhao et al. (017)) the inputs A, B and C are concatenated and processed with residual block (He et al. 2016). Then, an additional K bottleneck residual blocks (BRB; with expansion factor 6: Sandler et al. 2018) applied obtain the output of the iterative block consisting of feature maps at again 7. different scales. This output is both fed back input for the next iteration n and forwarded the classification block. The classification block consists of a 1 x 1 convolutional layer with C' output channels and an 8-fold bilinear up-sampling to the original spatial dimensions. In the first iteration (n 1) of the iterative block, the mixing of the scales is skipped and the input is directly processed by the K bottleneck residual blocks (see injection of input as A, and C). The number of output channels are denoted by ʃ and f. = //2, respectively. The batch-normalization layers within the iterative blocks (dashed boxes) are never shared between iterations.
We use the Adam optimizer with 31 = 0.9, B2 = 0.999 and a learning rate 0.001 that we multiply with 0.1 after 70% and 85% of the number of overall training epochs. We use batch size of 8 and train the network for 2000 (4000 for Figure 5) and 900 for the CamVid and Cityscapes datasets, respectively. For Figure Figure 4 and the appendix, we report the mean values and the errors the means across 5 trials. For Figures and we report the trial with the highest peak accuracy over 3 trials.
5
For the results shown in Figures in the bottleneck residual blocks and an L2 weight decay 10 all convolutional layers. For all other results, we do not dropout and weight decay.
T and 5 we use dropout with rate 0.1 after the depth-wise convolutions
2.3 NETWORK EVALUATION
Throughout this study, we measure the quality of semantic segmentation by calculating the mean intersectionn-ovr-uniio (mloU Jaccard 1912). which is the ratio of the of overlap and the area of union
(accar
label prediction
ToU
label U prediction
averaged over all classes.
We consider network to perform well it achieves a high mloU while requiring few MACs. this end, calculate the area under the curve of the mloU (y.) over MACs (αx) with output index as follows:
N Σ n=
(In+1 xn)) (yn+1 y0) + (yn Moo)
AUC
2
4

---

Under review as a conference paper at ICLR 2021
with (zu, yo) (0, 0.00828), where Yo denotes the mloU at chance level for the CamVid dataset. To compensate for different maximum numbers n of MACs for different sets of hyperparameters, we normalize as follows: AUC
AUC
ON
The size of the computational graph is computed by accumulating the memory requirements of all nodes, i.e. network layers, the network graph. For each layer, the total required memory the sum of the memory for parameters, input feature maps and output feature maps.
The theoretical latency of network executed fully parallel is determined by the length, i.e. the depth, of the path from input to output of this network (see also Section D
For both the size of the computational graph and the latency, we only consider convolutional layers like commonly done in literature (e.g- Passke et al. 2016 Wu et al. 2018 Mehta et al. (2019). This means, we ignore other network layers like normalizations, activations, concatenations, additions and spatial resizing, for which assume that they can be fused with the convolutional layers.
2.4 DATASETS
The CamVic d dataset (Brostow et al. 2008) consists of 701 (367 for training, 101 for validation, 233 for testing) annotated images filmed from moving We use the same static pre-processing in Badrinarayanan et al. (2017) to obtain images and semantic labels of size 480 x 360 and normalize the pixel values to the interval [0, 1] by dividing all pixel values by 255. For online data augmentation during training, we horizontally flip the pairs of images and labels at random and use random crops of size 480 352.
The Cityscapes dataset (Cordts et al. 2016) consists of 3475 (2975 for training, 500 for validation) annotated images and we use the validation set for testing. We resize the original images and semantic labels to 1024 x 512 and divide all pixel values by 127.5 and subtract 1 to obtain values in the interval [-1,1]. 1]. For online data augmentation during training, we horizontally flip the pairs of images and labels at random.
For both datasets, pixels with class labels not marked for training are ignored the cost function.
2.5 BATCH NORMALIZATION
In case of independent parameters between iterations of the iterative block, i.c. Wa+1 is independent from w, in Figure Ta batch normalization improves the network training. However, case of weight sharing, also sharing the parameters of batch normalization between iterations significantly worsens network training. Since not sharing batch-normalizztion parameters would violate our idea re-use the identical building block again and again, we place batch-normalizatioo layers between the iterations of the iterative block (see Figure ()). For comparability, we use the same setup also for networks without weight sharing, although the average validation mloU is slightly decreased compared to networks that instead use batch normalization after each convolution (Figure 3
3 RESULTS
In order to obtain accurate networks with low computational costs, we first search for the best set of the architectural hyperparameters L, N and K (Section B.I.. Since we are also interested in the trade-off between weight sharing and the networks size, we also consider networks without weight sharing this search. Then, we investigate the impact of intermediate losses on the network performance to find the best set of weight factors for the loss function (Section 3.2). Finally, for the found of hyperparameters and weight factors, we show results of ItNets on the CamVic d and Ci tttyccappe dataset (Section 3.3
3.1 SEARCH FOR ARCHITECTURAL HYPERPARAMETER
In order to compare ItNets with and without weight sharing between iterations of the iterative block goal is to find common set of architectural hyperparameters for both scenarios. We perform grid search over the hyperparameters our network model as described in Figures [ndd]] 2 For each set of hyperparameters, choose the largest possible number of channels that results in network
5

---

Under review as a conference paper at ICLR 2021
8)
mlou
miuu
0.55
0.6
0.50
block wise BN
non shared
shared
0.5
MACS (10")
MACs (G)
Figure 3: a) Comparison of mloU over MACs between networks with batch normalization applied between iterative blocks (purple curve) and applied after each convolution (green curve) on the test set of the CamVid d dataset. Note that, for both cases, batch normalization is always applied after each convolution in the data block. b) Peak mloU over MACs for networks with different widths on the validation set of the CamVid dataset. Each data point represents the peak mloU over all network outputs for one specific network width and the corresponding MACs for this output. For following studies, selected the network highlighted in red that achieves, out of 5 trials, maximum mloU of 72.7 and 72.8 for independent and shared weights, respectively.
a)
b}
d)
mlou
0.65
unifomm
0.55
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
9)
h)
mloU (shared)
000000
0.70
unifom
6
6
# MACS (10%)
MACS (1.0)
MACS (1.09)
MACS (109)
Figure 4: Ablation studies for multi-output training on the validation set of the CamVic d' dataset: The mloU over MACs is shown for the same architectural hyperparameters, but for different sets of an a-c and e-g) The weight factors On are set for the shown outputs and to 0 otherwise. Each different set of weight factors is highlighted with a different color, and h) The weight factors a₂ modulated over the network outputs. We increase the weight factor from 16 (iner) or keep the weights uniform over all outputs.
with less than 2.2 billion MACs for the last output of the network on single samples of the CamVid dataset.
Since we are interested network architectures with a high mean intersection-over-unioo (mloU) and low number of MACs, we sort the architectures by their area under the curve (see AUC in Sec- tion 2.3) for both networks with independent and shared weights. Networks with a peak performance max, (mloU,) < mIoUrotmax 0.02, where mIoU is the highest peak performance all sets of hyperparameters, are considered as not suitable for applications and are discarded. For comparability between networks with independent and shared weights, choose the architecture with the lowest average rank across the two classes of networks and obtain the following set of hyperparameters: N 16 iterations. L scales and K bottleneck residual blocks (for the results of the grid search, see Figures [nd in the appendix). Note that many iterations (N = 16) result high re-usage of the iterative block, a low latency and small computational graph.
3.2 MULTI-OUTPUT TRAINING
We study the impact of additional intermediate network outputs on the network performance by training the same network architecture with different sets of weight factors a, of the loss function

---

Under review as a conference paper at ICLR 2021
-)
b)
mlou
Iteeti
ItNet-S
0.55
Itteb
0.50
OGNet
0.0 0.2
0.4
100
200
300
400
20
40
60
80
# MACS (10")
# parameters (10")
graph size (MB)
latency (a.u.)
Figure 5: Network performance in terms of mIoU over MACs (a), mloU over number parameters (b), mloU over memory bandwidth (c), and mloU over memory footprint of the intermediate state (d) on the test set of the CamVic dataset. In case of weight sharing, we increased the width of the network (compare orange to green curve) to recover the mloU of the network with independent weights (blue curve). According to Figure 4e and g we thin out the losses. ENet (Paszke al. 2016) and CGNet (Wu et al., (018)) denote efficien reference networks. Note that the MACs of these reference data points are normalized to the actual input size (480 x 360) that was used to obtain the reported mloU values.
(for details, see Section 2.2 In summary. later network outputs benefit from the additional loss applied to earlier outputs. This is supported by ablation studies, in which we trained networks with single, only late thinned out outputs. The following results and conclusions are similar for both network types with and without weight sharing and, hence, we discuss them jointly.
The training of single outputs is worse than jointly training all outputs, except for only training the first output (see Figure 4 and c). However, a network with only the first output has no practical relevance due to its tiny network size and, consequently, low mloU.
The intermediate activations of the early layers are optimized for the potentially conflicting tasks of providing the basis for high accuracy early outputs and, the same time, top accuracy at late outputs. However, our experiments suggest that especially the optimization of the early outputs improves the performance of late outputs (see Figure (4p and f). Our observation that early losses do not conflict with late losses allows to keep these early losses resulting in networks with very low latency (see Figgue(5tt) In addition, keeping the first output allow for low latency, but decreasing the density of outputs, removes unnecessary constraints and slightly improves the mIoU (see Figure 4 and g). The removed outputs are likely required in applications, since the improvement of the mloU between consecutive outputs is rather small (see Figure [ee..
So far, we removed sets of outputs from the loss function, but kept the weight factors a, identical for all remaining outputs. For linearly increasing the weight factors over the outputs, the mloU jointly decreases for all outputs (compare different colors in Figure [ and h). This unexpected decrease in the mloU for late outputs may be attributed to the observed effect that earlier outputs are crucial for late performance as discussed above.
3.3 THE COST OF WEIGHT SHARING
In the following, we address the cost of sharing weights in the iterative block. Specifically, we compare the number MACs, the number of parameters, the size of the computational graph and the latency between networks with and without weight sharing.
Compared to networks with independent weights, networks with shared weights have substantially less free parameters that, consequently, results in worse mloUs. To compensate for this decrease in the peak mloUs we increase the width of the latter by factor 1.76 (ʃ = 51 with weight sharing compared
29 without weight sharing), which corresponds approximately three times the number MACs (FigureBp)
Although the number of MACs has to be increased for weight sharing (Figure 5.). the number of parameters is substantially smaller (44%), since parameters are highly re-used (Figure 5p). The size of the computational graph is dominated by the size of the feature maps due their large spatial dimensions. If the weights are shared between iterations of the iterative block the previously unrolled network graph be collapsed into single node with self-loop reducing the size of the
7

---

Under review as a conference paper at ICLR 2021
computational graph (Figure 5p). The latency is identical for all networks, since we use the same architectural hyperparameters (Figure(5p)
DISCUSSION
This study investigates the trade-off between the size of the computational graphs and the accuracies of so-called iterative neural networks that show state-of-the the-art performance for this trade-off. Iterative networks require less than half the size for the computational graph compared to networks that are state-of-the-ard in terms of the number of MACs (compare 276MB for ItNet to 639MB for ESPNetv2 to achieve an mloU of 61.1 in Figure Ip In addition, the multiple intermediate network outputs allow for any-time prediction and lower latency that is further reduced by the shallow network design (Figure [d). While targeting the same accuracy with an identical network structure, the sharing of weights reduces the size of the graph by factor larger than 4 (compare 83MB 384MB to achieve an mloU of approximately 64.2 in Figure (5)) but increases the number of MACs by a factor of approximately 3 (compare 2.0 to 6.2 billion MACs in Figure However, this increase in MACs is expected to be smaller than the increase in throughput by novel, maasiveelypparale hardware accelerators, especially if the computational graphs fit into the in-computation memory of these hardware systems (see also Section D. For example, Graphcore (2020b) reports 300-fold increase in throughput compared to GPUs for recurrent networks of small size and Fsssr et al. (2016) report more than 5000 frames per second and Watt for computer vision tasks. During the training of ItNets, we observe that additional and especially early network outputs improve the performance of late outputs, which was also observed by Zhou et al. (2019) on pre-trained networks for age estimation. In principle, the presented methods could also be applied to different tasks, like object classification and detection, for which expect similar observations. However, these datasets are usually provided in a lower spatial resolution, which reduces the challenge and the need to reduce the size of these already rather small computational graphs.
Since saaiiveely-prrallle hardware systems are not easily accessible at the moment, we cannot provide real benchmark data, but will in the following exemplarily discuss the potential benefits of our network models if executed on such hardware systems. To this end, we describe the execution of our model on the Graphcore (2020a) system, since this system seems be the most promising in terms of commercialization and availability. If the full network graph fits into the in- inmoouuttiioon memory of the Graphcore (2020a) system (3.6GB for the IPU-M2 000 machine), the input data can be pipelined through this graph in streaming fashion without the need to reconfigure the hardware. Then, by design, independent nodes this graph are executed fully in parallel (see also Fischer et al.) 2018). This results in substantially lower latencies during network inference compared to systems that do not execute network graphs fully in parallel like CPUs and GPUs (an up to 25 times lower latency is reported by Graphcore, 2020b). In contrast, for CPUs and GPUs, the workload of the network graph has to be tiled and the arithmetic units are continuously reconfigured for each tile. ItNets significantly reduce the footprint of their graphs by introducing loops and, hence, improve the network's s accuracy for the same footprint. Although the in-computation memory is huge for the TPU-M2000 machine that is optimized for data centers, expect embedded versions of similar systems to have a considerably smaller in-computation memory that, then, require networks with tiny computational graphs. Other pusssively-paaalle hardware designs increase the density of the memory in terms of synapses per chip area by aggressive quantization (Merolla et al. 2014) or mixed-signal implementations (Schemmel al. 2010 Yao et al., 2020) and impose additional challenges for the development of network models.
For the Ci Cityscapes dataset, our ItNet already requires two Nvidia V100 GPU with 32GB memory if the standard training procedure is not modified. Consequently, exploring large-scale ItNets is difficult by using backpropagation on GPUs. However, since ItNets are optimized for their execution massively parallel hardware systems, this demonstrates the fundamentally different operation principles between these systems and GPUs as well as highlights the need to think beyond workloads that are tailored for GPUs.
In parallel to this study, a novel training technique was introduced by Bai et al. (020)) that significantly improves the scaling of vision networks composed of iteratively executed building blocks with weight sharing. Instead of using backpropagation. they optimize the equilibrium point of a fix fix-point process described by the iterative execution of the building block. They achieved task accuracies comparable
8

---

Under review as a conference paper at ICLR 2021
to of-te-of-the-at networks and, to this end, required approximately the same number of parameters compared to the baselines. In line with our study, this results in comparably large building blocks that are executed many times and, consequently, require significantly more MACs than the baseline networks. However, the training method of Bai et al. (020) can not be applied to networks, for which the weights are allowed be updated between iterations and, hence, the effect of weight sharing can not be studied in isolation. In addition, their method includes the raw image (8MB in float 32 for Cityscapes) into the state of the iterative block, which prevents to shrink their networks to the regime of few billion MACs as shown by our method, for which the state has size of only 4.8MB.
The concept of recurrent building blocks has also been previously applied to semantic segmentation (Pinheiro & Collobert 2014) and to generate super-resolution images (Cheng et al. 2018). However, these works share the same network building block between different scales of the feature maps, which does not allow for anytime predictions. Instead of processing all scales at once like in Bai et al. (2020) and the work at hand, a full encoder-decoder network can be shared over time. To predict semantic maps for single images (Wang al. 2019) videos (Valipour et al. 2017) the intermediate feature maps of this encoder-decoder network are connected by recurrent units over time. Ballas et al. (2016) apply similar technique to networks with encoders only video tasks.
As an extension of this study, partly releasing the constraint of weight sharing may allow for a significant reduction in the required size of the iterative block. To this end, ratio between shared and free weights may be chosen close to, but smaller than one. This will require to reconfigure the network graph hardware, which may be costly or possible at all depending on the hardware system. An alternative approach to release constraints would be to replace the weights of the building block by functions that modulates the weights over the iterations of the building block. Another interesting open question for further studies is the root cause for the observation that early outputs improve the performance of late outputs. This effect may be attributed to some kind of knowledge distillation and / or to the shortcut for the gradients from network output to input.
REFERENCES
V. Badrinarayanan, A. Kendall, and R. Cipolla. SegNet: A deep convolutional encoder-decoder archi- tecture for image segmentation. IEEE Transactions Pattern Analysis and Machine Intelligence, 39(12):2481-2495 2017.
Shaojie Bai, Vladlen Koltun, and J. Zico Kolter. Multiscale deep equilibrium models. In Advances in Neural Information Processing Systems, 2020.
Nicolas Ballas, Li Yao, Chris Pal, and Aaron C. Courville. Delving deeper into convolutional networks for learning video representations. In International Conference on Learning Representations (ICLR), 2016.
Gabriel J. Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla. Segmentation and recogni- tion using structure from motion point clouds. In ECCV, pp. 44-57, 2008
Ping Chao, Chao-Yare Kao, Yu-Shan Ruan, Chien-Hsiang Huang, and Youn-Long Lin. HarDNet: A low memory traffic network. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2019.
Y. Chen, J. Emer, and V. Sze. Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks. In 2016 ACM/IEEE 43rd Annual International Symposiun Computer Architecture (ISCA), pp. 367-379, 2016.
X. Cheng, X. Li, J. Yang, and Y. Tai. SESR: Single image super resolution with recursive squeeze and excitation networks. In 2018 24th International Conference Pattern Recognition (ICPR), pp. 157-152, 2018.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.

---

Under review as a conference paper at ICLR 2021
Steven K. Esser, Paul A. Merolla, John V. Arthur, Andrew S. Cassidy, Rathinakumar Appuswamy, Alexander Andreopoulos, David J. Berg, Jeffrey L. McKinstry, Timothy Melano, Davis R. Barch, Carmelo di Nolfo, Pallab Datta, Armnn Amir, Brian Taba, Myron D. Flickner, and Dharmendra S. Modha. Convolutional networks for fast, energy-efficient neuromorphic computing. Proceedings of the National Academy of Sciences, 113(41) ): 1441-1144444 2016.
Volker Fischer, Jan Koehler, and Thomas Pfeil. The streaming rollout of deep networks towards fully model-parallel execution. In Advances in Neural Information Processing Systems 31, pp. 4039-4050. 2018.
Graphcore. The IPU-M2000 machine, 2020a.
URL https///w.graphcortt
Graphcore. Graphcore benchmarks, 2020b. URL https: www.wwwwwr.....o......... htw..gaaphooreee./uubtss///
Bttps///ppe/GcBBeccmiee liiissoctt20000 pdf PhsLang=en
Klaus Greff, Rupesh K Srivastava, and Jürgen Schmidhuber. Highway and residual networks learn unrolled iterative estimation. In International Conference Learning Representations (ICLR), 2017.
K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261-2269, 2017.
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q Weinberger. Multi-scale dense networks for resource efficient image classification. In International Conference on Learning Representations (ICLR), 2018.
Paul Jaccard. The distribution of the flora in the alpine zone. New Phytologist, 11(2):37-50, 1912.
Tsung- Wei Ke, Michael Maire, and Stella X. Yu. Multigrid neural architectures. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Qianli Liao and Tomaso A. Poggio. Bridging the gaps between residual learning, recurrent neural networks and visual cortex. CoRR, abs/1604.03644 2016. URL
Sachin Mehta, Mohammad Rastegari, Linda Shapiro, and Hannaneh Hajishirzi. ESPNetv2: A liggh- power efficient, and general purpose convolutional neural network. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2019.
Paul A. Merolla, John V. Arthur, Rodrigo Alvarez-Icaza, Andrew S. Cassidy, Jun Sawada, Filipp Akopyan, Bryan L. Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, Bernard Brezzo, Ivan Vo, Steven K. Esser, Rathinakumar Appuswamy, Brian Taba, Arnon Amir, Myron D. Flickner, William P. Risk, Rajit Manohar, and Dharmendra S. Modha. A million spiking-neuron integrated circuit with a scalable communication network and interface. Science, 345(6197):668 673, 2014.
Adam Paszke, Abhishek Chaurasia, Sangpil Kim, and Eugenio Culurciello. ENet: A deep neural network architecture for real-time semantic segmentation. CoRR, abs/1606.001177, 2016. URL
http: wwtp://w..w./a///....o6 http:///axx...orr/aa///66...... 02147
Pedro Pinheiro and Ronan Collobert. Recurrent convolutional neural networks for scene labeling. volume 32 of Proceedings of Machine Learning Research, pp. 82-90, Bejing, China, 22-24 Jun 2014. PMLR.
Adria Ruiz and Jakob Verbeek. Adaptative inference cost with convolutional neural mixture models. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2019.
M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L. Chen. MobileNetV2: Inverted residuals and linear bottlenecks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4510-4520, 2018
10

---

Under review as a conference paper at ICLR 2021
J. Schemmel, D. Brüderle, A. Grübl, M. Hock, K. Meier, and S. Millner. A wafer-scale neuromorphic hardware system for large-scale neural modeling. In 2010 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1947-1950, 2010.
V. Sze, Y. Chen, T. Yang, and J. S. Emer. Efficient processing of deep neural networks: A tutorial and survey. Proceedings of the IEEE, 105(12):225--2322 2017.
S. Valipour, M. Siam, M. Jagersand, and N. Ray. Recurrent fully convolutional networks for video segmentation. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 29-36, 2017.
Wei Wang, Kaicheng Yu, Joachim Hugonot, Pascal Fua, and Mathieu Salzmann. Recurrent U-Net for resource-constrained segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Tianyi Wu, Sheng Tang, Rui Zhang, and Yongdong Zhang, CGNet: light- context guided network for semantic segmentation. CoRR, abs/1811.08201, 2018. URL http::t/txxx...oo/
abs/1811 08201
Peng Yao, Huaqiang Wu, Bin Gao, Jianshi Tang, Qingtian Zhang, Wenqiang Zhang. J. Joshua Yang. and He Qian. Fully hardware-implemented memristor convolutional neural network. Nature, 577 (7792):641-66) 1-66,, 2020.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. In International Conference Learning Representations (ICI.R, 2019.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang. and Jiaya Jia. Pyramid scene parsing network. In Conference on Computer Vision and Pattern Recognition (CVPR). 2017.
Y. Zhou, Y. Bai, S. S. Bhattacharyya, and H. Huttunen. Elastic neural networks for classification. In 2019 IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS), pp. 251-255, 2019.

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
0.7
0.6
mlou
0.5
0.4
K=1
Km1
mloU
0.5
0.4
K=-5
Km-5
K=1
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
# MACS (B) (N=4)
# MMCS (B) (N=8)
MACs (B) (N-16)
MACs (B) (N-32)
Grid search over the following hyperparameters the validation set of the CamVic i dataset: number layers N, number blocks L and number of residual blocks Parameters are not shared between iterations of the iterative block. The best set of hyperparameters (for details about the selection, see Section 3.1 depicted in red.
Figure 6:
0.5
K=5
K+5
0.4
Kw
Kw
Kw1
0.6
mloU
0.5
KW3
KW3
0.7
0.6
mlou
Km5
Km5
Km5
0.4
K-3
K-3
0.3
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
MACS (B) (N=4)
(B) (N=8)
MACS (B) (N=16)
MACS (B) (N=32)
Figure 7: Like Figure 6 but the parameters are shared between iterations of the iterative block.
12


Reviews:
[{"review_id": "mFw8UNvlYBW", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a homogeneous network structure, where a single building block is iteratively executed to refine the outputs, in order to achieve a good trade-off between latency and accuracy. Empirical studies are conducted to show the superiority of such homogeneous networks over ENet and CGNet on CamVid and Cityscapes datasets.\n\nConcerns: \n\n- The paper is not well organized which makes it hard to follow. The authors should revise the writing and clearly introduce the challenges, the contributions, the proposed architecture structure, the design insights, the experiment settings, and so on.\n\n- The key concern about the paper is the lack of novelty and design insights. Why are homogeneous networks more beneficial for massively-parallel hardware systems? It would better to see the empirical evaluation and comparison against those non-homogeneous networks, such as multi-scale DenseNet, on a real hardware platform. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "XguDW6xFPmi", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies homogenious networks, which is defined by the paper as networks that reuse building blocks with shared or different weights multiple times during the inference of the network. During the inference, the network iteratively use the same set of blocks to process input feature maps with different resolutions, and each step, the output feature map can be used by the prediction head to generate the output. This paper studies the cost of the network, in terms of MACs, parameters, memory footprints, and the accuracy vs. the number of iterations. The author noted that for the studied network, they need to increase the MACs by 3x in order to match the performance of regular networks. Despite this, this kind of homogeneous networks can be useful for novel hardware architectures with limited memory bandwidth. \n\nStrength of the paper: this paper presents an interesting study over a novel type of network and studied its accuracy-cost trade-off. \n\nWeaknesses ofthe paper: \n1/ The organization of the paper should be improved significantly. For example, the architecture of the homogeneous network is a central part of the paper, yet the introduction to it is only provided in two figures, and not in the method description. \n2/ It is not clear what should be counted as the contribution of the paper. I am assuming a few possibilities: a) proposing the new homogeneous networks? However, there are already several previous papers mentioning this type of networks. [cite] b) Better performance achieved by using homogenous networks? However, the experiments of the paper show that the network under study is not as competitive as previous baselines cited in this paper. Also, though the authors suggest this type of network can be useful for future hardware processors, but without explicitly mentioning the processor, it is hard to justify this. c) The analysis of the homogeneous network and their accuracy/cost trade-off. This might be counted as a contribution, but it is not clear if this analysis can, say, help us improve the performance of homogeneous networks. \n\nOverall, I think this paper is not complete and has not met the standard of acceptance. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "BG6ospN2E1o", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "=========\nSummary:\n\nThis paper proposes a homogeneous network structure for semantic segmentation, which optimizes for prediction accuracy, latency as well as memory footprint.  The paper studies anytime prediction setting and designs a re-usable single building block to reduce the memory footprint. Experimental results on CamVid data shows that it's possible to use a homogeneous network architecture to achieve competitive mIoU compared to previous work at the cost of increased MACs.  Experimental evaluation on larger datasets such as Cityscapes is prohibited due to memory constraints of the available GPUs. \n\n=========\nPros:\n*  The idea of using a homogeneous network is interesting and re-usable blocks are reasonable for the anytime prediction setting. \n*  The paper is easy to follow. \n\n=========\nConcerns:\n\n1.  Is the parameters of the re-usable building block updated iteratively during inference? The paper described two scenarios where the parameters of the re-usable building block can be updated or shared across multiple iterations.  My first question is,  is this for training time only or for both training and inference? \n\n a) At inference time if the parameter updates are allowed at each iteration, what is the mechanism to decide the new weights?  If an independent block was used and only the structure of the building block is reused, there would not be any benefit for latency or memory footprint. \n \n b) If the parameters of the building blocks are fixed during inference,  then it falls into the weight-sharing scenario where the model performance is much lower according to Figure 1 c). \n\n2. Lack of anytime prediction experiments.  The paper claims it studies anytime prediction setting, however, I couldn't find explicit descriptions about it in the experiment section and descriptions about how the model choose the number of iterations during inference.  \n\n3. Missing Cityscapes experiments.  The paper claims in the abstract that the method was evaluated on cityscapes but in the discussion section, it says they couldn't provide cityscapes experiments due to memory issues.  The inconsistency in the paper should be fixed.  \n\n4. What is the exact benefit of the homogenous network?  The paper shows this structure requires at least 3 times larger MACs in order to achieve competitive performance. It also says this network has potential for novel, massively-parallel hardware accelerators.  This is a bit vague to me, could you provide a concrete example? \n\n5. Related work on image pyramid and feature pyramid.  In this paper,  the data block preprocesses the images into multiple scales and the re-usable building block concatenates the features from multiple scales.  A discussion/revisit on the literature about the image pyramid and feature pyramid is recommended here and it might give some inspirations to address the memory issues of training the model on Cityscapes. \n\n========\n\nOverall, I think the idea of building a homogenous network is reasonable to me and may lead to a better trade-off between latency, accuracy, and memory. However, the current manuscript still needs much improvement. \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7SSzHDP8S2O", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the influence of the use of shared vs independent parameters in re-used blocks of neural networks. This is achieved for the task of semantic segmentation, with a network architecture that iteratively refines its prediction.\n\nStrengths:\n- Studying the effect of parameter sharing vs the use of independent parameters in recurrent types of architecture is interesting and could lead to a better understanding of these architectures\n- The paper is clearly written, and the methodology would be reproducible\n\n\nWeaknesses:\n\nContribution:\n- While I do believe that such a study could provide the community with a better understanding of architectures with re-used blocks, the scale of the study performed in this paper is too small to draw conclusions. The paper tackles a single task (semantic segmentation), and, more critically, evaluates a single architecture. There is therefore no evidence that the conclusions drawn here will generalize to other architectures/tasks, which significantly limits the potential impact of this paper on the community.\n\nRelated work:\n- While I am not aware of similar studies, several important references tackling the task of recurrent semantic segmentation are missing, e.g., Pinheiro & Collobert, ICML 2014; Wang et al., ICCV 2019. These methods rely on different architectures, and studying the effect of parameter sharing in their frameworks would broaden the scope of this work.\n- Similarly, video segmentation has also been tackled with recurrent architectures, e.g., Ballas et al., ICLR 2016; Valipour et al., WACV 2017. Studying the use of parameter sharing in this context would thus also increase the potential impact of this work.\n\nEmpirical results:\n- While the results indeed evaluate the effect of parameter sharing on the chosen architecture, they seem to be somewhat disappointing in terms of absolute performance. In particular, on CityScapes (Fig. 1(b)), CNMM yields significantly higher mIoUs than the proposed method for the same number of MACs. Wouldn't it be possible to also extend this study to the CNMM architecture?\n\nMinor comments:\n- In the caption of Fig. 1, the authors mention that the parameters of the classification block are never shared. Does this mean that there is one classification block for each recurrent iteration?\n- In Section 2.2, when explaining the normalization of the weight factors, I suggest using a different notation for the weights before and after normalization, e.g., w_n = w_n'/(\\sum_i w_i').\n- The title of Section 3.1 (Neural Architecture Search) is a bit misleading, as the authors simply perform a grid search of a few hyper-parameters, not a proper search over a large space of different architectures as in the NAS sub-field.\n- The meaning of the different colors in Fig. 4 is not always explicitly defined.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that the paper is well written and some of the experiments are interesting. However, the paper did not clearly highlight how this work fits in with prior research, neither did it show what the advantages of the presented homogeneous network are. The authors addressed some of these concerns in the rebuttal, but not enough to sway the reviewers. In the end all reviewers recommend rejection, and the AC sees no evidence to overturn this recommendation.
Decision: Reject