In this task, you are given a paper review. Based on the review, your job is to identify language and generate "en" if the review is in English or generate "es" if the review is in Spanish. Note that URLs in the text have been replaced with [Link].

[Q]: The primary point made by this paper is that given certain architectural characteristics of multi-GPU systems, namely the use of bi-directional PCI-E for communication and the integration of two independent DMA engines on recent GPU devices (providing support for simultaneous independent communications), and given the characteristics of the communications patterns required by synchronous SGD trainers for deep neural networks, namely that the messages are large, dense, and have a fixed length, it makes sense to design communication collectives such as broadcast, reduce, and allreduce specifically for the use case of synchronous SGD training on a multi-GPU system.  The paper describes the implementation of these three collectives (broadcast, reduce, and allreduce) using a linear pipelining (LP) scheme on a (logical) ring topology.  The paper compares the LP collectives to two alternatives:  collectives based on a minimal spanning tree (MST) topology and collectives based on bidirectional exchange (BE).  First, a theoretical comparison is made using a standard cost model used in the high performance computing community.  When assumptions based on multi-GPU system architecture (very low latency for messages) and on the communication characteristics of synchronous SGD training (very large messages) are integrated into the model, the paper finds that the LP collectives should be less costly than BE collectives by a factor of 2 and less costly than MST collectives by a factor of log(p), where p is the number of GPUs being used.  Second, an empirical comparison is performed in which (1) the time required to perform each of the different collectives on a 4-device (k40m) system is measured as a function of message size and (2) the time required to perform each of the different collectives with a 200 MB message length is measured as a function of the number of devices in the system.  These measurements show that the LP-based collectives are consistently the fastest.  Third, DNN training experiments with AlexNet and GoogLeNet are performed on a 4-device system using three different synchronous SGD algorithms with the different implementations of the collectives (a total of 6 different algorithms in all).  Measurements of the communication and computation costs show that the LP collectives reduce communication costs without affecting computation costs (as expected).  Measurements of the convergence of the training loss as a function of time for the two DNN architectures show that use of the LP collectives leads to faster training.  While the theory says that the costs of LP collectives should be invariant to the number of devices in a multi-GPU system, the empirical work shows that in practice this does not hold going from 4 to 5 devices (in the tested configuration) because in a 5-device system messages must traverse the QPI.  Are there other practical considerations that the authors are aware of that affect the scaling of the LP collectives?  If so, these should be mentioned in the paper.  In the sentence "Worringen (2003) proposed a pipeline collective model in shared memory environment for CPU data, but communications of different MPI processes sharing the same CPU memory bus within the same CPU socket." I really can't figure out what the words after "but communications of different MPI processes" are trying to convey.  This sentence is not comprehensible.  "Please note the latency term is log pα, which is the smallest among algorithms in Table.1. Therefore, MST only suits for high frequent short messages."  The claim that MST collectives are only suitable for high-frequency, short messages does not follow from the statement that MST collectives have the smallest latency term.  You also need to consider the way the cost scales with message size (the bandwidth term).  If the MST collectives had a better bandwidth term than the other collectives, then they would also be superior for large messages.  "Let’s take an appropriate block size b to ensure n/b ≪ α."  This looks wrong, since n > b.  Should it be b/n ≪ α?  "However, the parameters are not consistent after several iterations due to the precision issues of float multiplications in Gradient Update."  Are you sure the inconsistency in weight estimates across devices is due to multiplication?  I would expect that it would be due to gradients being accumulated in different orders; that is, because floating point addition is not commutative.  I recommend replacing the term "sub-gradients" in this paper with "partial gradients."  In the optimization literature, the term "sub-gradient" has a very specific meaning that differs from this paper's use of the term (see
[A]: en


[Q]: __Note__: An earlier version of the review (almost identical to the present one) for an earlier version of the paper (available on arXiV) can be found here:
[A]: en


[Q]: Studying the Hessian in deep learning, the experiments in this paper suggest that the eigenvalue distribution is concentrated around zero and the non zero eigenvalues are related to the complexity of the input data. I find most of the discussions and experiments to be interesting and insightful. However, the current paper could be significantly improved.  Quality: It seems that the arguments in the paper could be enhanced by more effort and more comprehensive experiments. Performing some of the experiments discussed in the conclusion could certainly help a lot. Some other suggestions: 1- It would be very helpful to add other plots showing the distribution of eigenvalues for some other machine learning method for the purpose of comparison to deep learning. 2- There are some issues about the scaling of the weights and it make sense to normalize the weights each time before calculating the Hessian otherwise the result might be misleading. 3- It might worth trying to find a quantity that measures the singularity of Hessian because it is difficult to visually conclude something from the plots. 4- Adding some plots for the Hessian during the optimization is definitely needed because we mostly care about the Hessian during the optimization not after the convergence.  Clarity: 1- There is no reference to figures in the main text which makes it confusing for the reading to know the context for each figure. For example, when looking at Figure 1, it is not clear that the Hessian is calculated at the beginning of optimization or after convergence. 2- The texts in the figures are very small and hard to read.
[A]:
en