reviews: 
Comments: The paper presents an approach for learning video-based policies in robot manipulation settings. The key benefit of the approach is training on actionless video data across human and robot embodiments. The method, termed Actions from Video Dense Correspondences (AVDC), consists of three stages: (1) diffusion-based video prediction given a text-based goal and starting image, (2) optical flow prediction creating dense correspondences, and (3) executed on a robot platform using off-the-shelf inverse kinematics and motion planners. AVDC uses the ability to project a 3D point onto the image plane both from depth and optical flow to compute the transformation of rigid objects across the predicted video frames. These transformations allow AVDC to infer actions in the environment. Then off-the-shelf robotics primitives can be used to enact the planned trajectory. The approach is benchmarked on the Meta-World and iTHOR simulation platforms and on a real-world robot platform, outperforming the considered baselines.
Strengths: * The general problem of making use of actionless human video data is of interest and importance to the research community.
* The problem is well-motivated and the literature review does a good job of contextualizing the paper in prior work.
* The paper is strong, well-written and easy to follow.
* The use of geometry to reconstruct the transformation of the predicted objects (stationary camera) or embodiment (moving camera) which can be derived simultaneously from the optical flow and depth camera during deployment is clever. This allows the training data for the video prediction and optical flow not to require depth, with depth only being necessary during deployment. The transformations of either the objects or the embodiment then can be used in conjunction with off-the-shelf inverse kinematics, motion planners, grasp point predictors, etc. This also allows for learning from human videos and then zero-shot deploying to the robot, which is very impressive.
* The figures are informative and effectively illustrate the benefits of the proposed approach.
* The experiments consider both simulation and real robot evaluation, as well as an ablation study, demonstrating AVDC's superior performance as compared to the considered baselines and support for AVDC's design choices. In particular, I appreciated the discussion and later the results for why not to directly predict the optical flow without the intermediate step of video prediction.
* The discussion did a good job of describing the weaknesses and failure modes of the proposed method.
Weaknesses: * The literature review is missing a number of relevant works.
  * V-PTR: similar high-level motivation of using video-based, prediction-focused pre-training and then action-based finetuning. This should have likely served as a baseline for the proposed method. 
  * [A] Bhateja, Chethan, et al. "Robotic Offline RL from Internet Videos via Value-Function Pre-Training." arXiv preprint arXiv:2309.13041 (2023).
  * Diffusion policy: diffusion policy has shown very good results in terms of multi-task, low-data regime performance. 
    * [B] Chi, Cheng, et al. "Diffusion policy: Visuomotor policy learning via action diffusion." arXiv preprint arXiv:2303.04137 (2023). 
    * [C] Ha, Huy, Pete Florence, and Shuran Song. "Scaling up and distilling down: Language-guided robot skill acquisition." arXiv preprint arXiv:2307.14535 (2023).
* In particular, my biggest concern with the paper is the lack of comparison to a strong BC baseline. As the AVDC method uses diffusion to predict images, it seems natural to baseline against a diffusion policy (e.g., [B, C]). R3M is a fairly old representation at this point (e.g., Voltron [D] would be a better representation) and the simple MLP-based BC policy would strongly underperform diffusion policy. This is confirmed by, for example, the very poor baseline performance in Tables 1 and 2. In Sec. 4.3, it is mentioned that since 'R3M is pretrained on robot manipulation tasks ... it might not be suitable for visual navigation tasks'. Something like [E] could be a better baseline here.
  * [D] Karamcheti, Siddharth, et al. "Language-driven representation learning for robotics." arXiv preprint arXiv:2302.12766 (2023).
  * [E] Shah, Dhruv, et al. "ViNT: A Foundation Model for Visual Navigation." arXiv preprint arXiv:2306.14846 (2023).
* In Sec. 4.5 'Results', the paper states that Fig. 10 presents screenshots of robot trajectories, but I believe that is Fig. 9? Fig. 10 shows human predicted trajectories.

Some typos and points of confusion are listed below:
1. Page 3 - 'Unipi'.
2. Sec. 4.1:  'compare AVDC to its [variants] that also predict dense correspondence'.
3. Sec. 4.2: 'maximum number of planning affects' -> 'maximum number of replanning steps affects'.

**Post-rebuttal: Most of my concerns have been addressed! I am raising my score as such.
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 4 excellent