File size: 929 Bytes
7934b29 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 |
.. _parallelisms:
Parallelisms
------------
NeMo Megatron supports 4 types of parallelisms (can be mixed together arbitraritly):
Distributed Data parallelism
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. image:: images/ddp.gif
:align: center
:width: 800px
:alt: Distributed Data Parallel
Tensor Parallelism
^^^^^^^^^^^^^^^^^^
.. image:: images/tp.gif
:align: center
:width: 800px
:alt: Tensor Parallel
Pipeline Parallelism
^^^^^^^^^^^^^^^^^^^^
.. image:: images/pp.gif
:align: center
:width: 800px
:alt: Pipeline Parallel
Sequence Parallelism
^^^^^^^^^^^^^^^^^^^^
.. image:: images/sp.gif
:align: center
:width: 800px
:alt: Sqeuence Parallel
Parallelism nomenclature
^^^^^^^^^^^^^^^^^^^^^^^^
When reading and modifying NeMo Megatron code you will encounter the following terms.
.. image:: images/pnom.gif
:align: center
:width: 800px
:alt: Parallelism nomenclature
|