Task: Language Identification
Input (Spanish, English): This paper proposes a novel approach ParMAC, a parallel and distributed framework of MAC (the Method of Auxiliary Coordinates) to learn nested and non-convex models which is based on the composition of multiple processing layers (i.e., deep nets). The basic idea of MAC to optimise the nested objective function, which is traditionally learned using methods based on the chain-rule gradients but inconvenient and is hard to parallelise, is to break nested functional relationships judiciously by introducing new variables ( the auxiliary coordinates) as equality constraints, and then to optimise a penalised function using alternating optimisation over the original parameters (W step) and over the coordinates (Z step).  The minimisation (W step) updates the parameters by splitting the nested model into independent submodels and training them using existing algorithms, and the coordination (Z step) ensures that corresponding inputs and outputs of submodels eventually match.  In this paper, the basic assumptions of ParMAC are that with large datasets in distributed systems, it is imperative to minimise data movement over the network because of the communication time generally far exceeds the computation time in modern architectures. Thus, the authors propose the ParMAC to translate the parallelism inherent in MAC into a distributed system by data parallelism and model parallelism. They also analyse its parallel speedup and convergence, and demonstrated it with MPI-based implementation to optimise binary autoencoders. The proposed ParMAC is tested on 3 colour image retrieval datasets.   The organization of the paper is well written, and the presentation is clear. My questions are included in the following: - The MAC framework solves the original problem approximately. If people use the sigmoid function to smooth the stepwise function, the naive optimization methods can be easier applied. What is the difference between these two? Or why do we want to use a new approach to solve it? - The authors do not compare their ParMAC model with other distributed approaches for the same nested function optimization problem.
Output (English): en