A system to configure distributed training of a neural network, the system comprising: memory to store a library to facilitate data transmission during distributed training of the neural network; a network interface to enable transmission and receipt of configuration data associated with a set of worker nodes (2216A, 2216B, 2236A, 2236B), the worker nodes (2216A, 2216B, 2236A, 2236B) configured to perform distributed training of the neural network, wherein each worker node (2216A, 2216B, 2236A, 2236B) includes one or more graphics processors to perform compute operations associated with a machine learning framework workflow; and a processor to execute instructions for flexible node grouping provided by the library, the instructions to cause the processor to create multiple groups (2210, 2230) of the worker nodes (2216A, 2216B, 2236A, 2236B), the multiple groups (2210, 2230) of the worker nodes (2216A, 2216B, 2236A, 2236B) to be created based on a network topology for a network that interconnects the worker nodes (2216A, 2216B, 2236A, 2236B) such that the communication latency between the worker nodes within a group is lower than the communication latency observed when communicating with worker nodes of other groups, and instantiate one or more parameter servers, the one or more parameter servers to enable inter-group communication between the multiple groups (2210, 2230) of the worker nodes (2216A, 2216B, 2236A, 2236B), characterised in that nodes within a group communicate with each other using an all-reduce operation. memory to store a library to facilitate data transmission during distributed training of the neural network; a network interface to enable transmission and receipt of configuration data associated with a set of worker nodes (2216A, 2216B, 2236A, 2236B), the worker nodes (2216A, 2216B, 2236A, 2236B) configured to perform distributed training of the neural network, wherein each worker node (2216A, 2216B, 2236A, 2236B) includes one or more graphics processors to perform compute operations associated with a machine learning framework workflow; and a processor to execute instructions for flexible node grouping provided by the library, the instructions to cause the processor to create multiple groups (2210, 2230) of the worker nodes (2216A, 2216B, 2236A, 2236B), the multiple groups (2210, 2230) of the worker nodes (2216A, 2216B, 2236A, 2236B) to be created based on a network topology for a network that interconnects the worker nodes (2216A, 2216B, 2236A, 2236B) such that the communication latency between the worker nodes within a group is lower than the communication latency observed when communicating with worker nodes of other groups, and instantiate one or more parameter servers, the one or more parameter servers to enable inter-group communication between the multiple groups (2210, 2230) of the worker nodes (2216A, 2216B, 2236A, 2236B), characterised in that nodes within a group communicate with each other using an all-reduce operation. create multiple groups (2210, 2230) of the worker nodes (2216A, 2216B, 2236A, 2236B), the multiple groups (2210, 2230) of the worker nodes (2216A, 2216B, 2236A, 2236B) to be created based on a network topology for a network that interconnects the worker nodes (2216A, 2216B, 2236A, 2236B) such that the communication latency between the worker nodes within a group is lower than the communication latency observed when communicating with worker nodes of other groups, and instantiate one or more parameter servers, the one or more parameter servers to enable inter-group communication between the multiple groups (2210, 2230) of the worker nodes (2216A, 2216B, 2236A, 2236B), characterised in that nodes within a group communicate with each other using an all-reduce operation.