Package: hetseq
HetSeq: Distributed GPU Training on Heterogeneous InfrastructureThis is our coding implementation for the paper:Yifan Ding, Nicholas Botzer, Tim Weninger.
HetSeq: Distributed GPU Training on Heterogeneous Infrastructure, Proc. of Association for the Advancement of Artificial Intelligence (AAAI) Innovative Application of Artificial Intelligence, February 2021.Author: Yifan Ding (yding4@nd.edu)arxiv paper available:https://arxiv.org/abs/2009.14783Documentation available:https://hetseq.readthedocs.ioMedium towards data science Post:Training BERT at a UniversityDocumentation includesDistributed Setting,Scripts to Run HetSeq,Extending HetSeq,Parameter ExplanationandCode Reference.OverviewHetSeq is a distributed neural network platiform designed to run on Heterogeneous Infrastructure with common scientific shared file system.
It can be run directly on command line with SSH or task queen submission system without privilege or any extra packages. It takes care of the data index randomization and assignment to different GPUs in the multi-node and multi-GPU setting. Users can easily extend HetSeq to many other models with minimum effort.HetSeq requires installation ofPyTorchwith GPU support andNCCL.Installationcreate and activate conda virtual environment with Python 3.7.4 (recommended)$ conda create --name hetseq
$ conda activate hetseq
$ conda install python=3.7.4Git clone directory and install nessasory package$ git clone https://github.com/yifding/hetseq.git
$ cd /path/to/hetseq
$ pip install -r requirements.txt 
$ pip install --editable .To Run BERT:Download data files including training corpus, model configuration, and BPE dictionary. Test corpus fromhere, full data fromthis link. Download test_DATA.zip for test or DATA.zip for full run, unzip it and place thepreprocessing/directory inside the package directory. Available corpus underpreprocessing/,phase one of BERT training corpus :preprocessing/hdf5_lower_case_1_seq_len_128.../wikicorpus_en/phase two of BERT training corpus :preprocessing/hdf5_lower_case_1_seq_len_512.../wikicorpus_en/sample test for phase one :preprocessing/test128/sample test for phase two :preprocessing/test512/seeNVIDIA-pytorch-BERT,google_original_BERTandBERT paperfor more information.current provided is generated fromNVIDIA-pytorch-BERTwith wikipedia data (book data is not available)Running HetSeq script is available athttps://hetseq.readthedocs.io/en/master/examples.html,Distributed ConfigurationHetSeq can be executed on single GPU on a single node, multiple GPUs on a single node, or multiple GPUs across multiple nodes. Main logic is defined attrain.py.--distributed-init-method: defines an initialization. e.g.: "tcp://10.32.82.207:11111" (tcp for multiple nodes) or "file:///hetseq/communicate.txt" (shared file for multiple nodes).--distributed-world-size: total number of GPUs used in the training.--distributed-gpus: the number of GPUs on the current node.--distributed-rank: represents the rank/index of the first GPU used on current node.Performance tableRunning BERT on nodes with 4 GPUs each.nodesGPUsepochsbatch sizestepsavg. time per steptraining timetraining lossexpansionspeedup145128267,1392.60s7.19d0.02611285256133,5702.69s4.19d0.0280.861.72416551266,7852.7942.23d0.0310.813.228325102433,3933.1261.21d0.0550.745.94Notice and tipsloading BERT data takes a while.Known issuescurrently not supporting continue trainingmnist datasets download does not support multiple GPUsfuture patchbert processing pipeline not includedinterface of datasets/transformers not includedhetseq not supporting download from pipevaluation separate/combined not includedfp16 supportLicensethis repository is MIT-licensed. It is created based onfairseq,NVIDIA-BERT, andpytorchPlease send us e-mail or leave comments on github if have any questions.Copyright (c) 2020 Yifan Ding andWeninger Lab