A method of training a computer-implemented reinforcement learning neural network system, the reinforcement learning neural network system comprising: an input to receive observation data defining an observation of a real-world environment, wherein the observation comprises an image frame comprising one or more images; an input to receive environmental reward data from the real-world environment defining an extrinsic reward received in response to performing an action; an output to provide action data for selecting an action to be performed by a mechanical agent acting in the real-world environment to perform a task; a visual entity identifying neural network subsystem configured to process the observation data to determine a set of spatial maps for the one or more images, each spatial map representing one of a set of learned discrete visual entities, each spatial map comprising map pixels, wherein each map pixel identifies whether a corresponding region of the one or more images is associated with the discrete visual entity for the spatial map; an action encoding neural network configured to encode the action data to provide encoded action data for training the visual entity identifying neural network subsystem; and a reinforcement learning neural network subsystem configured to process data from the set of spatial maps and the environmental reward data to provide the action data; the method comprising: receiving observation data defining an observation comprising a current image frame; processing the observation data using the visual entity identifying neural network subsystem to determine the set of spatial maps for the reinforcement learning neural network subsystem; using the action data from the reinforcement learning neural network subsystem to select and perform an action using the mechanical agent in the real-world environment; encoding the action data using the action encoding neural network to provide encoded action data; and training the visual entity identifying neural network subsystem to distinguish between frames that are inside a time window number of frames distant from the current frame and frames that are outside the time window by training the visual entity identifying neural network subsystem dependent upon i) a difference between image frames within and outside the time window and ii) the encoded action data. an input to receive observation data defining an observation of a real-world environment, wherein the observation comprises an image frame comprising one or more images; an input to receive environmental reward data from the real-world environment defining an extrinsic reward received in response to performing an action; an output to provide action data for selecting an action to be performed by a mechanical agent acting in the real-world environment to perform a task; a visual entity identifying neural network subsystem configured to process the observation data to determine a set of spatial maps for the one or more images, each spatial map representing one of a set of learned discrete visual entities, each spatial map comprising map pixels, wherein each map pixel identifies whether a corresponding region of the one or more images is associated with the discrete visual entity for the spatial map; an action encoding neural network configured to encode the action data to provide encoded action data for training the visual entity identifying neural network subsystem; and a reinforcement learning neural network subsystem configured to process data from the set of spatial maps and the environmental reward data to provide the action data; the method comprising: receiving observation data defining an observation comprising a current image frame; processing the observation data using the visual entity identifying neural network subsystem to determine the set of spatial maps for the reinforcement learning neural network subsystem; using the action data from the reinforcement learning neural network subsystem to select and perform an action using the mechanical agent in the real-world environment; encoding the action data using the action encoding neural network to provide encoded action data; and training the visual entity identifying neural network subsystem to distinguish between frames that are inside a time window number of frames distant from the current frame and frames that are outside the time window by training the visual entity identifying neural network subsystem dependent upon i) a difference between image frames within and outside the time window and ii) the encoded action data. receiving observation data defining an observation comprising a current image frame; processing the observation data using the visual entity identifying neural network subsystem to determine the set of spatial maps for the reinforcement learning neural network subsystem; using the action data from the reinforcement learning neural network subsystem to select and perform an action using the mechanical agent in the real-world environment; encoding the action data using the action encoding neural network to provide encoded action data; and training the visual entity identifying neural network subsystem to distinguish between frames that are inside a time window number of frames distant from the current frame and frames that are outside the time window by training the visual entity identifying neural network subsystem dependent upon i) a difference between image frames within and outside the time window and ii) the encoded action data.