Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper mostly reads well. It proposes to use statistics from previous activations to compute and adaptive scaling of the loss such that the amount of underflow is minimized. The scaling is defined per layer. Experiments are carried for various model sizes and datasets. 

If anything I think the paper can do a better job at centralizing (maybe in an appendix) the gritty details (e.g. how the stats are computed etc). Unfortunately, the best way of doing this might be in the form of code or maybe pseudocode, but being quite explicit in all technical details. 

Right now this is mentioned in the text (same way as batch norm stats if I understood correctly, based on the current minibatch). Though is not clear how the variance on w is treated. How is the variance on dirac delta (backpropagated error) is converted into a scalar (that will be for the entire loss).
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors propose a method to train models in FP16 precision. The authors show that the key reason of training performance drop is the overflow or underflow of back propagation information. Instead of using a fixed value or dynamic value proposed by a previous work, this paper adopts a more elaborate way to minimize underflow
in every layer simultaneously and automatically based on the current layer statistics. Experiment results on CIFAR10, ImageNet and Object Detection models are conducted to demonstrate the effectiveness of the proposed method.

There are some concerns about this paper:
1. This paper tries to solve a very practical problem which is good, however, the stability of this method which is very important for real applications remains unclear. More networks such as VGG/ResNet/depthwise-conv based networks, more initialization methods (such as gaussian, xavier, kaiming), w/o bn layers,  and more tasks such as segmentation and detection with different batch sizes are strongly recommended to make this work more solid.
2. In the experiments, it seems that dynamic loss scaling method works well too except on the configuration of SSD batchsize=8. Why dynamic loss scaling fails on this case? More detailed analysis are recommended to show the advantage of the proposed method.
3. In  many experiments, it seems adaptive loss scaling with FP16 is even better than FP32, is this stable? Could we further improve the FP32 results if using dynamic / adaptive loss scaling? 
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The authors propose an adaptive loss scaling method during the backpropagation stage for the mix precision training to reduce the underflow. Compared with the previous work, which scales the loss by human design, and needs to be consistent in all layers. The authors state that they can decide the scale rate layer by layer automatically to reduce the underflow in a low precision situation. 

They calculate the scale rate using the statistic information of the layer weight and gradient. By adaptively scale each layer’s loss and gradient, this method can reduce the underflow rate better than the previous work. Additionally, the authors claim that the computation overhead is not significant, so it is efficient to use rather than searching from a set of fix scale rates. 

The experiments present on image classification and objective detection benchmarks. From the result, we can see that the adaptive loss scale reaches a relatively high point on all the tasks.
 
Pros:
 
- The method is straight forward and easy to understand. The motivation is good. They get some impressive results on ResNet110 and SSD512 comparing with the fixed scaling method. Besides, they give some analysis of their advantages and disadvantages in different networks, which looks promising to me.
 
Cons:

- One question in Section 3.2.1, the assumption that w, g, p can be treated as the random variable with Gaussian distribution seems not natural in the training process. Especially p is a zero-mean distribution. The cited paper uses this assumption in a more convincing case, such as the weight initialization task. Notice that He et al., 2015 claim that the product of weight and gradient can be a zero-mean normal distribution is based on the weight is a symmetric distribution around zero, which is not true in neither this paper’s assumption nor the real training situation. 

- In the objective detection part, I can not find which dataset the authors use. Though the author state that they follow Liu et al., 2016 ’s work, there are also several tasks in Liu’s paper, and I can not directly match the resulting point with any of those tasks, which makes me hard to confirm the experiment result.

- The experiment setting is unclear. Here are two questions. 1, What is the initial scale at the last layer? It should be manually designed in the experiment, and I think this value may affect the other layer’s scale as well. If the algorithm is robust for this scale, it is better to show some study on that. 2, What update frequency is used in the experiment? The authors say that the overhead can be reduced by reducing the frequency, but they do not clearly show which frequency they use in their experiment, if the frequency does not affect the performance, it is also better to claim or show some study on that.

Minor comments:
- Figures can use a larger font.  Figures 4a and 4b can be aligned better. 
review_title: Official Blind Review #4

Meta-review: This work proposes to improve mixed precision training by adaptively scaling the loss based on statistics from previous activations to minimize underflow during training. However, the method is designed rather heuristically and can be improved with stronger theoretical support and improved representation of the paper. 

Decision: Reject