A computing system for performing distributed large scale reinforcement learning with improved efficiency, the computing system comprising: a plurality of actor devices , wherein each actor device locally stores a local version of a machine-learned model , and wherein each actor device is configured to implement the local version of the machine-learned model at the actor device to determine an action to take in an environment to generate an experience, and wherein the plurality of actor devices are organized into a plurality of cells and the respective actor devices in each cell are managed as a single unit; a server computing system configured to perform one or more learning algorithms to learn an updated version of the machine-learned model based on the experiences generated by the plurality of actor devices; a plurality of sharded replay buffers for providing the experiences to the server computing system; and a hierarchical and distributed data caching system comprising a plurality of layers of data caches that propagate data descriptive of the updated version of the machine-learned model from the server computing system to the plurality of actor devices to enable each actor device to update its respective local version of the machine-learned model; wherein the plurality of layers of data caches comprise at least a first layer that receives the data descriptive of the updated version of the machine-learned model from the server computing system and a final layer that provides the data descriptive of the updated version of the machine-learned model to the plurality of actor devices, and wherein, for each of the plurality of cells, the final layer comprises one or more cell-specific data caches that service only the respective actor devices in the cell. a plurality of actor devices , wherein each actor device locally stores a local version of a machine-learned model , and wherein each actor device is configured to implement the local version of the machine-learned model at the actor device to determine an action to take in an environment to generate an experience, and wherein the plurality of actor devices are organized into a plurality of cells and the respective actor devices in each cell are managed as a single unit; a server computing system configured to perform one or more learning algorithms to learn an updated version of the machine-learned model based on the experiences generated by the plurality of actor devices; a plurality of sharded replay buffers for providing the experiences to the server computing system; and a hierarchical and distributed data caching system comprising a plurality of layers of data caches that propagate data descriptive of the updated version of the machine-learned model from the server computing system to the plurality of actor devices to enable each actor device to update its respective local version of the machine-learned model; wherein the plurality of layers of data caches comprise at least a first layer that receives the data descriptive of the updated version of the machine-learned model from the server computing system and a final layer that provides the data descriptive of the updated version of the machine-learned model to the plurality of actor devices, and wherein, for each of the plurality of cells, the final layer comprises one or more cell-specific data caches that service only the respective actor devices in the cell. <EOS>