A computer-implemented method, comprising: obtaining, by a computing system comprising one or more processors, state data descriptive of at least a current or past state of a plurality of objects (411 A- J) that are perceived by an autonomous vehicle, wherein the plurality of objects are within a surrounding environment of the autonomous vehicle; determining, by the computing system, a priority classification for each object in the plurality of objects based at least in part on the respective state data for each object; determining, by the computing system, an order at which the computing system determines a predicted future location for each object based at least in part on the priority classification for each object; determining, by the computing system, the predicted future location for each object based at least in part on the determined order; wherein determining, by the computing system, a predicted future location for an object based at least in part on the priority classification for the object comprises: selecting, by the computing system, a future location prediction system from a plurality of different location prediction systems (104, 124) based at least in part on the priority classification for the object, wherein the plurality of different location prediction systems comprises at least a low-fidelity prediction system and at least a high-fidelity prediction system; determining, by the computing system, the predicted future location for the object using the selected future location prediction system, wherein a higher fidelity prediction system is used to determine predicted future locations for higher-priority objects and a lower-fidelity prediction system is used to determine predicted future locations for lower priority objects; determining, by the computing system, a motion plan for the autonomous vehicle based at least in part on the predicted future location for at least one of the plurality of objects; and controlling a motion of the autonomous vehicle according to the motion plan. obtaining, by a computing system comprising one or more processors, state data descriptive of at least a current or past state of a plurality of objects (411 A- J) that are perceived by an autonomous vehicle, wherein the plurality of objects are within a surrounding environment of the autonomous vehicle; determining, by the computing system, a priority classification for each object in the plurality of objects based at least in part on the respective state data for each object; determining, by the computing system, an order at which the computing system determines a predicted future location for each object based at least in part on the priority classification for each object; determining, by the computing system, the predicted future location for each object based at least in part on the determined order; wherein determining, by the computing system, a predicted future location for an object based at least in part on the priority classification for the object comprises: selecting, by the computing system, a future location prediction system from a plurality of different location prediction systems (104, 124) based at least in part on the priority classification for the object, wherein the plurality of different location prediction systems comprises at least a low-fidelity prediction system and at least a high-fidelity prediction system; determining, by the computing system, the predicted future location for the object using the selected future location prediction system, wherein a higher fidelity prediction system is used to determine predicted future locations for higher-priority objects and a lower-fidelity prediction system is used to determine predicted future locations for lower priority objects; selecting, by the computing system, a future location prediction system from a plurality of different location prediction systems (104, 124) based at least in part on the priority classification for the object, wherein the plurality of different location prediction systems comprises at least a low-fidelity prediction system and at least a high-fidelity prediction system; determining, by the computing system, the predicted future location for the object using the selected future location prediction system, wherein a higher fidelity prediction system is used to determine predicted future locations for higher-priority objects and a lower-fidelity prediction system is used to determine predicted future locations for lower priority objects; determining, by the computing system, a motion plan for the autonomous vehicle based at least in part on the predicted future location for at least one of the plurality of objects; and controlling a motion of the autonomous vehicle according to the motion plan.