A server system comprising: a plurality of servers, each server including at least one hardware acceleration device and at least one processor communicatively coupled to the hardware acceleration device by an internal data bus and executing a host server instance, the host server instances of the plurality of servers collectively providing a software plane, and the hardware acceleration devices of the plurality of servers collectively providing a hardware acceleration plane that implements a plurality of hardware accelerated services; wherein each hardware acceleration device maintains in memory a data structure that contains load data indicating a load of each of a plurality of target hardware acceleration devices implementing a designated hardware accelerated service of the plurality of hardware accelerated services, wherein the memory is a local memory of a server from the plurality of servers that contains the hardware acceleration device; and wherein, when a requesting hardware acceleration device routes a request for the designated hardware accelerated service, the requesting hardware acceleration device routes the request to a target hardware acceleration device that is indicated by the load data in the data structure to have a lower load than other of the target hardware acceleration devices; wherein each hardware acceleration device periodically sends load data indicating its current load and load data from its data structure for the plurality of target hardware acceleration devices to one or more other hardware acceleration devices that have previously sent requests to that hardware acceleration device. a plurality of servers, each server including at least one hardware acceleration device and at least one processor communicatively coupled to the hardware acceleration device by an internal data bus and executing a host server instance, the host server instances of the plurality of servers collectively providing a software plane, and the hardware acceleration devices of the plurality of servers collectively providing a hardware acceleration plane that implements a plurality of hardware accelerated services; wherein each hardware acceleration device maintains in memory a data structure that contains load data indicating a load of each of a plurality of target hardware acceleration devices implementing a designated hardware accelerated service of the plurality of hardware accelerated services, wherein the memory is a local memory of a server from the plurality of servers that contains the hardware acceleration device; and wherein, when a requesting hardware acceleration device routes a request for the designated hardware accelerated service, the requesting hardware acceleration device routes the request to a target hardware acceleration device that is indicated by the load data in the data structure to have a lower load than other of the target hardware acceleration devices; wherein each hardware acceleration device periodically sends load data indicating its current load and load data from its data structure for the plurality of target hardware acceleration devices to one or more other hardware acceleration devices that have previously sent requests to that hardware acceleration device.