A method performed by one or more data processing apparatus for selecting actions to be performed by an agent interacting with an environment to accomplish a goal, the method comprising: generating a respective planning embedding corresponding to each of multiple experience tuples in an external memory, wherein each experience tuple characterizes interaction of the agent with the environment at a respective previous time step, wherein generating the respective planning embedding comprises appending a representation of a goal state of the environment to the experience tuple, and wherein the goal state represents a state of the environment when the goal of the agent has been accomplished; processing the planning embeddings using a planning neural network to generate an implicit plan for accomplishing the goal; and selecting the action to be performed by the agent at the time step using the implicit plan; and wherein i) the agent is a mechanical agent and the environment is a real-world environment, and the actions include control signals for actions of the mechanical agent in the real-world environment to accomplish the goal, or ii) the environment is a real-world environment including items of equipment in a plant or facility and the agent controls actions in the environment, the actions controlling or imposing operating conditions on the items of equipment of the plant/facility, or iii) the environment is a real-world environment and the agent manages distribution of tasks across computing resources in the environment, and the actions include assigning tasks to particular computing resources. generating a respective planning embedding corresponding to each of multiple experience tuples in an external memory, wherein each experience tuple characterizes interaction of the agent with the environment at a respective previous time step, wherein generating the respective planning embedding comprises appending a representation of a goal state of the environment to the experience tuple, and wherein the goal state represents a state of the environment when the goal of the agent has been accomplished; processing the planning embeddings using a planning neural network to generate an implicit plan for accomplishing the goal; and selecting the action to be performed by the agent at the time step using the implicit plan; and wherein i) the agent is a mechanical agent and the environment is a real-world environment, and the actions include control signals for actions of the mechanical agent in the real-world environment to accomplish the goal, or ii) the environment is a real-world environment including items of equipment in a plant or facility and the agent controls actions in the environment, the actions controlling or imposing operating conditions on the items of equipment of the plant/facility, or iii) the environment is a real-world environment and the agent manages distribution of tasks across computing resources in the environment, and the actions include assigning tasks to particular computing resources. i) the agent is a mechanical agent and the environment is a real-world environment, and the actions include control signals for actions of the mechanical agent in the real-world environment to accomplish the goal, or ii) the environment is a real-world environment including items of equipment in a plant or facility and the agent controls actions in the environment, the actions controlling or imposing operating conditions on the items of equipment of the plant/facility, or iii) the environment is a real-world environment and the agent manages distribution of tasks across computing resources in the environment, and the actions include assigning tasks to particular computing resources.