This paper describes an approach to predict (unseen) future frames of a video given a set of known past frames. The approach is based on a CNN that, in contrast to most related papers, work in the space of affine transformations (instead of pixels or flow). Said another way, the network takes as input a set of affine transforms that describe the motion of patches in the past frames, and likewise, outputs a set of affine transforms that predict future patch motion.  To that aim, the authors make a few simplifying hypotheses, namely, that a sequence of frames can be modeled accurately enough in their patch-affine framework. This is not unreasonable. A lot of papers in the optical flow community are based on similar hypotheses, i.e. model the flow as a smoothly varying affine field (for instance see "Locally affine sparse-to-dense matching for motion and occlusion estimation" by Leordeanu et al., "EpicFlow: Edge-Preserving Interpolation of Correspondences for Optical Flow" by Revaud et al., "Optical Flow With Semantic Segmentation and Localized Layers" by Sevilla-Lara et al.). These methods are state of the art, which gives a hint about the validity of this kind of approach.  In addition, it also seems very reasonable to reformulate the prediction task as predicting motion rather than predicting raw pixels. Indeed, the (patch-affine) motion space is considerably smaller than the image space, making the problem much more tractable and amenable to high-resolution videos.  While I agree with the authors on these points, I also find that the paper suffer from important flaws. Specifically:    - the choice of not comparing with previous approaches in term of pixel prediction error seems very "convenient", to say the least. While it is clear that the evaluation metric is imperfect, it is not a reason to completely dismiss all quantitative comparisons with previous work. The frames output by the network on, e.g. the moving digits datasets (Figure 4), looks ok and can definitely be compared with other papers. Yet, the authors chose not to, which is suspicious.      - The newly proposed metric poses several problems. First, action classification is evaluated with C3D, which is not a state-of-the-art approach at all for this task. Second, this metric actually *does not* evaluate what the network is claimed to do, that is, next frame prediction. Instead, it evaluates if another network, which was never trained to distinguish between real or synthetic frames by the way, can accurately classify an action from the predicted frames. I find that this proxy metric is only weakly related to what is supposed to be measured. In adition, it does not really make sense to train a network for something else that the final task it is evaluated for.      - how is the affine motion of patches estimated? It is only explained that the problem is solved globally (not treating each patch independently) in a pretty vague manner. Estimating the motion of all patches is akin to solving the optical flow, which is still an active subject of research. Therefore, an important flaw of the paper lies in the potentially erroneous etimation of the motion input to the network. In the videos made available, it is clear that the motion is wrongly estimated sometimes. Since the entire approach depends on this input, I find it important to discuss this aspect. How do motion estimation failures impact the network? Also, the patch-affine hypothesis does not hold when patches are large enough that they cover several objects with contradictory motion. Which appears to be the case on UCF101 videos.      - Even ignoring the weird proxy-evaluation part, the network is still not trained end-to-end. That is, the network is trained to minimize the difference between (noisy) ground-truth and output affine transforms, instead of minimizing a loss in the actual output space (frame pixels) for which an (exact) ground-truth is available. It is true that the MSE loss on raw pixels leads to blurry results, but other types of losses do exist, for instance the gradient loss introduced by Mathieu et al. was shown to solve this issue. As noted by the authors themselves, minimizing a loss in the transformation space, where affine parameters are harder to intepret, introduces unexpected artifacts. The motion is often largely underestimated, as is obvious in Figure 5 where it is hard to tell the difference between the input and output frames.       - The proposed approach is not sufficiently compared to previous work. In particular, the approach is closely related to "SPATIO-TEMPORAL VIDEO AUTOENCODER WITH DIFFERENTIABLE MEMORY" of Taraucean et al, ICLR'15. This paper also output prediction in the motion space. Experimental results should compare against it.    - The comparison with optical flow is unfair. First, the approach of Brox et al. is more than 10 years old. Second, it is not really fair to assume a constant flow for all frames. At least some basic extrapolation could be done to take into account the flow of all pairs of input frames and not just the last one. Overall, the approach is not compared to very challenging baselines.    - I disagree with the answer that the authors gave to a reviewer's question. Denote ground-truth frames as {X_0, X_1 ...} and predicted frames as {Y_1, Y_2, ...}. When asked if the videos at
Output: en