A computer-implemented method comprising: receiving a request to perform an inference using a particular machine learning model; determining, from among a plurality of groups of hosts, a group of hosts to route the request to, the group of hosts to host a plurality of machine learning models including the particular machine learning model; determining a path to the determined group of hosts; determining a particular host of the group of hosts to perform an analysis of the request based on the determined path, the particular host having the particular machine learning model in memory; routing the request to the particular host of the group of hosts; performing inference on the request using the particular host; and providing a result of the inference to a requester, wherein each host of the determined group of hosts caches a first plurality of machine learning models loaded in random access memory and caches a second, different plurality of machine learning models according to a least frequently used caching model in disk, wherein the method further comprises storing machine learning models in a model artifact storage which is an available and scalable data store, wherein the group of hosts is an autoscaling group that scales a number of hosts based on throughput of machine learning models it hosts, wherein the method comprises, when new hosts are added to the autoscaling group to increase the number of hosts in the autoscaling group: sending a request to the plurality of groups of hosts for a list of machine learning models that each host has cached and utilization statistics of these machine learning models; receiving the list of machine learning models that each host has cached and utilization statistics of these machine learning models; and determining, based on the list and the utilization statistics of the machine learning models in the list, which machine learning models to load into the new hosts from the model artifact storage, and fetching those machine learning models from the model artifact storage. receiving a request to perform an inference using a particular machine learning model; determining, from among a plurality of groups of hosts, a group of hosts to route the request to, the group of hosts to host a plurality of machine learning models including the particular machine learning model; determining a path to the determined group of hosts; determining a particular host of the group of hosts to perform an analysis of the request based on the determined path, the particular host having the particular machine learning model in memory; routing the request to the particular host of the group of hosts; performing inference on the request using the particular host; and providing a result of the inference to a requester, wherein each host of the determined group of hosts caches a first plurality of machine learning models loaded in random access memory and caches a second, different plurality of machine learning models according to a least frequently used caching model in disk, wherein the method further comprises storing machine learning models in a model artifact storage which is an available and scalable data store, wherein the group of hosts is an autoscaling group that scales a number of hosts based on throughput of machine learning models it hosts, wherein the method comprises, when new hosts are added to the autoscaling group to increase the number of hosts in the autoscaling group: sending a request to the plurality of groups of hosts for a list of machine learning models that each host has cached and utilization statistics of these machine learning models; receiving the list of machine learning models that each host has cached and utilization statistics of these machine learning models; and determining, based on the list and the utilization statistics of the machine learning models in the list, which machine learning models to load into the new hosts from the model artifact storage, and fetching those machine learning models from the model artifact storage. sending a request to the plurality of groups of hosts for a list of machine learning models that each host has cached and utilization statistics of these machine learning models; receiving the list of machine learning models that each host has cached and utilization statistics of these machine learning models; and determining, based on the list and the utilization statistics of the machine learning models in the list, which machine learning models to load into the new hosts from the model artifact storage, and fetching those machine learning models from the model artifact storage.