A method implemented by one or more processors of a robot, the method comprising: processing an instance of sensor data using an environment-conditioned action sequence prediction model, wherein the sensor data includes an instance of vision data captured by a vision component of the robot, wherein the vision data comprises an image of at least one object in the environment of the robot, and wherein the environment-conditioned action sequence prediction model is a trained machine learning model; determining, based on output generated based on the processing using the environment-conditioned action sequence prediction model, a set of predicted actions for an object manipulation robotic task associated with the at least one object, and a particular order for performing the predicted actions of the set; controlling the robot to perform the predicted actions of the set in the particular order, wherein controlling the robot to perform each of the predicted actions of the set in the particular order comprises: for each of the predicted actions, and in the particular order: selecting a corresponding action model that corresponds to the predicted action; until determining that the predicted action is complete: processing corresponding additional instances of sensor data, of the robot, using the corresponding action model, and controlling the robot based on action output, generated based on the processing using the corresponding action model. processing an instance of sensor data using an environment-conditioned action sequence prediction model, wherein the sensor data includes an instance of vision data captured by a vision component of the robot, wherein the vision data comprises an image of at least one object in the environment of the robot, and wherein the environment-conditioned action sequence prediction model is a trained machine learning model; determining, based on output generated based on the processing using the environment-conditioned action sequence prediction model, a set of predicted actions for an object manipulation robotic task associated with the at least one object, and a particular order for performing the predicted actions of the set; controlling the robot to perform the predicted actions of the set in the particular order, wherein controlling the robot to perform each of the predicted actions of the set in the particular order comprises: for each of the predicted actions, and in the particular order: selecting a corresponding action model that corresponds to the predicted action; until determining that the predicted action is complete: processing corresponding additional instances of sensor data, of the robot, using the corresponding action model, and controlling the robot based on action output, generated based on the processing using the corresponding action model. selecting a corresponding action model that corresponds to the predicted action; until determining that the predicted action is complete: processing corresponding additional instances of sensor data, of the robot, using the corresponding action model, and controlling the robot based on action output, generated based on the processing using the corresponding action model. processing corresponding additional instances of sensor data, of the robot, using the corresponding action model, and controlling the robot based on action output, generated based on the processing using the corresponding action model.