A method of performing distributed training of a neural network using multiple interconnected worker nodes of a distributed training network, the multiple worker nodes of the distributed training network being interconnected via a high-performance communication fabric and comprising each a fabric interface configured to connect to the communication fabric, the method comprising: storing a messaging library in memory, the messaging library for facilitating transmission of data during distributed training of the neural network, the data associated with trainable parameters of the neural network, in each worker node: transmitting and receiving gradient data associated with the trainable parameters via the fabric interface of the worker node, wherein during training, the worker node transmits and receives the gradient data associated with the trainable parameters via the fabric interface; executing, via a general-purpose processor of the worker node, instructions provided by the messaging library, the instructions to cause the general-purpose processor to configure the fabric interface to transmit and receive the gradient data associated with the trainable parameters during a workflow of a machine learning framework; and performing, via at least one graphics processor of the worker node, compute operations associated with the machine learning framework workflow to generate the gradient data associated with the trainable parameters, the compute operations interleaved with transmission and receipt of gradient data via the fabric interface; wherein the at least one graphics processor includes the fabric interface and the communication fabric is a mesh of point to point interconnects between the multiple worker nodes, wherein the fabric interface comprises point-to-point primitive logic to enable data exchange with other worker nodes in a synchronized manner and execute point-to-point primitives for point-to-point communications, the point-to-point primitives comprising one or more of following operations: a remote atomic operation, a remote procedure call, RPC, and a store with notify operation. transmitting and receiving gradient data associated with the trainable parameters via the fabric interface of the worker node, wherein during training, the worker node transmits and receives the gradient data associated with the trainable parameters via the fabric interface; executing, via a general-purpose processor of the worker node, instructions provided by the messaging library, the instructions to cause the general-purpose processor to configure the fabric interface to transmit and receive the gradient data associated with the trainable parameters during a workflow of a machine learning framework; and performing, via at least one graphics processor of the worker node, compute operations associated with the machine learning framework workflow to generate the gradient data associated with the trainable parameters, the compute operations interleaved with transmission and receipt of gradient data via the fabric interface; wherein the at least one graphics processor includes the fabric interface and the communication fabric is a mesh of point to point interconnects between the multiple worker nodes, wherein the fabric interface comprises point-to-point primitive logic to enable data exchange with other worker nodes in a synchronized manner and execute point-to-point primitives for point-to-point communications, the point-to-point primitives comprising one or more of following operations: a remote atomic operation, a remote procedure call, RPC, and a store with notify operation.