Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: An approach to adjust inference speed, power consumption or latency by using incomplete dot products McDanel et al. (2017) is investigated.

The approach is based on `profile coefficients’ which are learned for every channel in a convolution layer, or for every column in the fully connected layer. Based on the magnitude of this profile coefficient, which determines the importance of this `filter,’ individual components in a neural net are switched on or off. McDanel et al. (2017) propose to train such an approach in a stage-by-stage manner.

Different from a recently proposed method by McDanel et al. (2017), the authors of this submission argue that the stage-by-stage training doesn’t fully utilize the deep net performance. To address this issue a `loss aggregation’ is proposed which jointly optimizes a deep net when multiple fractions of incomplete products are used.

The method is evaluated on the MNIST and CIFAR-10 datasets and shown to outperform work on incomplete dot products by McDanel et al. (2017) by 32% in the low resource regime.

Summary:
——
In summary, I think the paper proposes an interesting approach but more work is necessary to demonstrate the effectiveness of the discussed method. The results are preliminary and should be extended to CIFAR-100 and ImageNet to be convincing. In addition, the writing should be improved as it is often ambiguous. See below for details.

Review:
—————
1. Experiments are only provided on very small datasets. According to my opinion, this isn’t sufficient to illustrate the effectiveness of the proposed approach. As a reader I wouldn’t want to see results on CIFAR-100 and ImageNet using multiple network architectures, e.g., AlexNet and VGG16.

2. Usage of the incomplete dot product for the fully connected layer and the convolutional layer seems inconsistent. More specifically, while the profile coefficient is applied for every input element in Eq. (1), it’s applied based on output channels in Eq. (2). This seems inconsistent and a comment like `These two approaches, however, are equivalent with negligible difference induced by the first hidden layer’ is more confusing than clarifying.

3. The writing should be improved significantly and statements should be made more precise, e.g., `From now on, x% DP, where \leq x \geq 100, means the x% of terms used in dot products’. While sentences like those can be deciphered, they aren’t that appealing.

4. The loss functions in Eq. (3) should be made more precise. It remains unclear whether the profile coefficients and the weights are trained jointly, separately, incrementally etc.

5. Algorithm 1 and Algorithm 2 call functions that aren’t described/defined.

6. Baseline numbers for training on datasets without incomplete dot products should be provided.

review_title: review

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a modification of a numeric solution: Incomplete Dot Product (IDP), that allows a trained network to be used under different hardware constraints. The IDP method works by incorporating a 'coefficient' to each layer (fully connected or convolution), which can be learned as the weights of the model are being optimized. These coefficients can be used to prune subsets of the nodes or filters, when hardware has limited computational capacity. 

The original IDP method (cited in the paper) is based on iteratively training for higher hardware capacities. This paper improves upon the limitation of the original IDP by allowing the weights of the network be trained concurrently with these coefficients, and authors present a loss function that is linear combination of loss function under original or constrained network setting. They also present results for a 'harmonic' combination which was not explained in the paper at all.

Overall the paper has very good motivation and significance. 
However the writing is not very clear and the paper is not self-contained at all. I was not able to understand the significance of early stopping and how this connects with loss aggregation, and how the learning process differs from the original IDP paper, if they also have a scheduled learning setting. 

Additionally, there were several terms that were unexplained in this paper such as 'harmonic' method highlighted in Figure  3. As is, while results are promising, I can't fully assess that the paper has major contributions.  
review_title: TESLA: Task-wise Early Stopping and Loss Aggregation for Dynamic Neural Network Inference 

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose a method for reducing the computational burden when performing inference in deep neural networks. The method is based a previously-developed approach called incomplete dot products, which works by pruning some of the inputs in the dot products via the introduction of pre-specified coefficients. The authors of this paper extend the method by introducing a task-wise learning procedure that sequentially optimizes a loss function for decreasing percentage of included features in the dot product. 

Unfortunately, this paper was hard to follow for someone who does not actively work in this field, making it hard to judge if the contribution is significant or not. While the description of the problem itself is adequate, when it comes to describing the TESLA procedure and the alternative training procedure, the relevant passages are, in my opinion, too vague to allow other researchers to implement this procedure.

Positive points:
- The application seems relevant, and the task-wise procedure seems like an improvement over the original IDP proposal.
- Application to two well-known benchmarking datasets.

Negative points:
- The method is not described in sufficient detail to allow reproducibility, the algorithms are no more than sketches.
- It is not clear to me what the advantage of this approach is, as opposed to alternative ways of compressing the network (e.g. via group lasso regularization), or training an emulator on the full model for each task.

Minor point:
- Figure 1 is unclear and requires a better caption. 
review_title: A potentially useful method, but not well motivated or explained

Meta-review: General consensus among reviewers that paper does not meet criteria for publication.

Pro:
- Improvement over the original IDP proposal.
- Some promising preliminary results.

Con:
- Insufficient comparison to other methods of network compression,
- Insufficient comparison to other datasets (such as ImageNet)
- Insufficient evaluation on variety of other models
- Writing could be more clear
Decision: Reject