user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: AMPNet: Asynchronous Model-Parallel Training for Dynamic Neural Networks

Paper content:
New types of compute hardware in development and entering the market hold the promise of revolutionizing deep learning in a manner as profound as GPUs.
However, existing software frameworks and training algorithms for deep learning have yet to evolve to fully leverage the capability of the new wave of silicon.
In particular, models that exploit structured input via complex and instance-dependent control flow are difficult to accelerate using existing algorithms and hardware that typically rely on minibatching.
We present an asynchronous model-parallel (AMP) training algorithm that is specifically motivated by training on networks of interconnected devices.
Through an implementation on multi-core CPUs, we show that AMP training converges to the same accuracy as conventional synchronous training algorithms in a similar number of epochs, but utilizes the available hardware more efficiently, even for small minibatch sizes, resulting in shorter overall training times.
Our framework opens the door for scaling up a new class of deep learning models that cannot be efficiently trained today.
An emerging category of neural networks show the common trait of reacting in dynamic and unique ways to properties of their input.
Networks like tree-structured recursive neural networks BID35 BID36 and graph neural networks (GNNs) BID31 BID20 BID12 take structured data types as input and and execute a computation that depends on the input structure.
This defies the moden GPU-driven paradigm of minibatch-based processing, and we refer to this new class of models with dynamic control flow as dynamic neural networks.The development of dynamic neural network frameworks -Chainer BID37 , DyNet BID24 , and PyTorch (PyTorch core team) -speaks to the importance of this class of models and highlights the challenge of how to make it easy for users to describe them.
Yet there is another big challenge: how can we train these models efficiently?Managing
minibatches to keep GPUs fully utilized is typically considered a user's responsibility in these dynamic frameworks (with the exception of DyNet's autobatching feature; see Sec. 7). This means
that users have to think about how to change their data feeding pipeline or even the model itself to run efficiently on GPUs, rather spending time innovating to improve the model accuracy.What if we had a hypothetical device with low memory overhead that allows perfect scaling without batching; i.e., processing 1 item is simply 100x faster than processing 100 items? Recent work
on FPGAs and other specialized hardware BID10 BID4 BID17 for deep learning encourages us to investigate this question. Our premises
are 1. No batching
is required for efficient processing.2. Each device
may not have enough memory to hold the entire model (this is a realistic constraint for current memory systems that approach the perfect scaling we require)Based on these premises, we propose an asynchronous model-parallel (AMP) training algorithm. Our idea is
illustrated in FIG0 . We need model
parallelism because each device may be too small to hold the entire model (premise 2). However, if we
perform synchronous parameter updates following the full forward and backward propagations, the only way to increase device utilization is by pipelining multiple instances into the system (see e.g., . Pipeline parallelism
with synchronous updates is at odds with convergence speed due to a decreased parameter update frequency; compare FIG0 (a) and (b).To overcome
this problem
, we
propose asynchronous parameter updates that occur without global synchronization whenever a pre-specified number of gradients have been accumulated; see Fig. 1 (c) . With this design we aim
for both
high device utilization and update frequency.In this setting, however, model parameters may be updated between the forward and the backward computation of an instance, introducing gradient "staleness". Despite staleness, we show that
AMP training can converge fast with good hardware utilization. Specifically, our contributions
are:• We present the asynchronous model parallel training algorithm for efficient distributed training of dynamic networks.• We present an intermediate representation
(IR) with explicit constructs for branching and joining control flow that supports AMP training. Unlike previous work that considers static
computation graphs for static control flow (e.g., Caffe), and dynamic computation graphs for dynamic control flow (e.g., Chainer), our IR encodes a static computation graph to execute dynamic control flow 1 . This makes training easy to distribute and
parallelize.• We show that our IR can readily encode replicas
, a form of data parallelism (see Sec. 5). In addition, our IR includes operators for data
aggregation, which recover a form of batching, enabling our methods to be applied even on hardware where batching is beneficial.• We implement AMP training on a multi-core CPU
and empirically demonstrate that AMP training converges to similar accuracies as synchronous algorithms on a variety of dynamic neural network models including Tree RNN and gated graph neural networks (GGNN).In summary, our work demonstrates the benefits of
AMP training and gives a novel way to design and deploy neural network libraries with dynamic control flow. In addition, we use our implementation to estimate
the performance on a hypothetical device satisfying premises 1 & 2, with 1TFLOPS compute capability (see Appendix C). Together, these contributions open up new ways to
scale up dynamic networks on interconnected compute devices.
We have presented an asynchronous model-parallel SGD algorithm for distributed neural network training.
We have described an IR and multi-core CPU runtime for models with irregular and/or instance-dependent control flow.
Looking forward, we aim to deploy our system on specialized hardware.
Equally importantly, we plan to build a compiler that automatically deduces the information to be placed in the states and generates state keying functions from a higher-level description of the models.
By unlocking scalable distributed training of dynamic models, we hope to enable exploration of this class of models that are currently only on the horizon but may become more mainstream in the future.A AMPNET RUNTIME IMPLEMENTATIONWe have implemented an AMPNet runtime for multi-core CPUs.
Our runtime spawns multiple workers each associated with a hardware thread and hosting one or more IR nodes -in a more general setting each worker corresponds to a compute device.
To remain faithful to a distributed environment communication is only through message passing.
Each worker is equipped with a multiple-producer single-consumer queue that can accept messages for any IR node hosted on that worker.The main worker loop periodically offloads messages from the concurrent queue to a worker-local priority queue that assigns higher priority to backward messages.
Backward prioritization is designed for situations when multiple IR nodes with a dependency on the IR graph end up hosted on the same worker.
As a consequence, backpropagation can complete faster and new instances can be pumped in by the controller.
We dequeue the top message and invoke the forward or backward method of the target IR node.
These methods may update internal IR node state (such as cache the state of the incoming message and wait for more messages) or post new forward or backward messages.How to update the parameters using the gradients is a configuration option that selects amongst a range of optimization algorithms.
We have implemented runtime configuration options for selecting several well-known schemes such as (momentum-)SGD and Adam BID18 , and for controlling the training hyper-parameters.

assistant: text: Using asynchronous gradient updates to accelerate dynamic neural network training