Package: oneccl-binding-pt
Intel® oneCCL Bindings for PyTorch (formerly known as torch_ccl)This repository holds PyTorch bindings maintained by Intel for the Intel® oneAPI Collective Communications Library (oneCCL).IntroductionPyTorchis an open-source machine learning framework.Intel® oneCCL(collective communications library) is a library for efficient distributed deep learning training implementing such collectives likeallreduce,allgather,alltoall. For more information on oneCCL, please refer to theoneCCL documentation.oneccl_bindings_for_pytorchmodule implements PyTorch C10D ProcessGroup API and can be dynamically loaded as external ProcessGroup and only works on Linux platform now.CapabilityThe table below shows which functions are available for use with CPU / Intel dGPU tensors.CPUGPUsend×√recv×√broadcast√√all_reduce√√reduce√√all_gather√√gather√√scatter××reduce_scatter√√all_to_all√√barrier√√Pytorch API AlignWe recommend using Anaconda as Python package management system. The followings are the corresponding branches (tags) ofoneccl_bindings_for_pytorchand supported Pytorch.torchoneccl_bindings_for_pytorchmastermasterv2.2.0ccl_torch2.2.0+cpuv2.1.0ccl_torch2.1.0+cpuv2.0.1ccl_torch2.0.100v1.13ccl_torch1.13v1.12.1ccl_torch1.12.100v1.12.0ccl_torch1.12v1.11.0ccl_torch1.11v1.10.0ccl_torch1.10v1.9.0ccl_torch1.9v1.8.1ccl_torch1.8v1.7.1ccl_torch1.7v1.6.0ccl_torch1.6v1.5-rc3beta09The usage details can be found in the README of corresponding branch. The following part is about the usage of v1.9 tag. if you want to use other version of torch-ccl please checkout to that branch(tag). For pytorch-1.5.0-rc3, the#PR28068and#PR32361are need to dynamicall register external ProcessGroup and enablealltoallcollective communication primitive. The patch file about these two PRs is inpatchesdirectory and you can use it directly.RequirementsPython 3.8 or later and a C++17 compilerPyTorch v2.2.0Build Option ListThe following build options are supported in Intel® oneCCL Bindings for PyTorch*.Build OptionDefault ValueDescriptionCOMPUTE_BACKENDSet oneCCLCOMPUTE_BACKEDN,set todpcppand use DPC++ Compiler to enable support for Intel XPUUSE_SYSTEM_ONECCLOFFUse oneCCL library in systemCCL_PACKAGE_NAMEoneccl-bind-ptSet Wheel NameONECCL_BINDINGS_FOR_PYTORCH_BACKENDcpuSet BACKENDCCL_SHA_VERSIONFalseadd git head sha version to Wheel nameLunch Option ListThe following lunch options are supported in Intel® oneCCL Bindings for PyTorch*.Lunch OptionDefault ValueDescriptionONECCL_BINDINGS_FOR_PYTORCH_ENV_VERBOSE0Set verbose level in ONECCL_BINDINGS_FOR_PYTORCHONECCL_BINDINGS_FOR_PYTORCH_ENV_WAIT_GDB0Set 1 to force the oneccl_bindings_for_pytorch wait for GDB attachingInstallationInstall from Sourceclone theoneccl_bindings_for_pytorch.gitclonehttps://github.com/intel/torch-ccl.git&&cdtorch-ccl
gitsubmodulesync
gitsubmoduleupdate--init--recursiveInstalloneccl_bindings_for_pytorch# for CPU Backend Onlypythonsetup.pyinstall# for XPU Backend: use DPC++ Compiler to enable support for Intel XPU# build with oneCCL from third partyCOMPUTE_BACKEND=dpcpppythonsetup.pyinstall# build without oneCCLexportINTELONEAPIROOT=${HOME}/intel/oneapiUSE_SYSTEM_ONECCL=ONCOMPUTE_BACKEND=dpcpppythonsetup.pyinstallInstall PreBuilt WheelWheel files are avaiable for the following Python versions.Extension VersionPython 3.6Python 3.7Python 3.8Python 3.9Python 3.10Python 3.112.2.0√√√√2.1.0√√√√2.0.100√√√√1.13√√√√1.12.100√√√√1.12.0√√√√1.11.0√√√√1.10.0√√√√python-mpipinstalloneccl_bind_pt==2.0.100-fhttps://developer.intel.com/ipex-whl-stable-xpuRuntime Dynamic LinkingIf oneccl_bindings_for_pytorch is built without oneCCL and use oneCCL in system, dynamic link oneCCl from oneAPI basekit (recommended usage):source$basekit_root/ccl/latest/env/vars.shNote: Make sure you have installedbasekitwhen using Intel® oneCCL Bindings for Pytorch* on Intel® GPUs.If oneccl_bindings_for_pytorch is built with oneCCL from third party or installed from prebuilt wheel:
Dynamic link oneCCL and Intel MPI libraries:source$(python-c"import oneccl_bindings_for_pytorch as torch_ccl;print(torch_ccl.cwd)")/env/setvars.shDynamic link oneCCL only (not including Intel MPI):source$(python-c"import oneccl_bindings_for_pytorch as torch_ccl;print(torch_ccl.cwd)")/env/vars.shUsageexample.pyimporttorch.nn.parallelimporttorch.distributedasdistimportoneccl_bindings_for_pytorch...os.environ['MASTER_ADDR']='127.0.0.1'os.environ['MASTER_PORT']='29500'os.environ['RANK']=str(os.environ.get('PMI_RANK',0))os.environ['WORLD_SIZE']=str(os.environ.get('PMI_SIZE',1))backend='ccl'dist.init_process_group(backend,...)my_rank=dist.get_rank()my_size=dist.get_world_size()print("my rank =%dmy size =%d"%(my_rank,my_size))...model=torch.nn.parallel.DistributedDataParallel(model,...)...(oneccl_bindings_for_pytorch is built without oneCCL, use oneCCL and MPI(if needed) in system)source$basekit_root/ccl/latest/env/vars.shsource$basekit_root/mpi/latest/env/vars.shmpirun -n  -ppn  -f  python example.py## Performance Debugging

For debugging performance of communication primitives PyTorch's [Autograd profiler](https://pytorch.org/docs/stable/autograd.html#profiler)
can be used to inspect time spent inside oneCCL calls.

Example:

profiling.py

```python

import torch.nn.parallel
import torch.distributed as dist
import oneccl_bindings_for_pytorch
import os

os.environ['MASTER_ADDR'] = '127.0.0.1'
os.environ['MASTER_PORT'] = '29500'
os.environ['RANK'] = str(os.environ.get('PMI_RANK', 0))
os.environ['WORLD_SIZE'] = str(os.environ.get('PMI_SIZE', 1))

backend = 'ccl'
dist.init_process_group(backend)
my_rank = dist.get_rank()
my_size = dist.get_world_size()
print("my rank = %d  my size = %d" % (my_rank, my_size))

x = torch.ones([2, 2])
y = torch.ones([4, 4])
with torch.autograd.profiler.profile(record_shapes=True) as prof:
    for _ in range(10):
        dist.all_reduce(x)
        dist.all_reduce(y)
dist.barrier()
print(prof.key_averages(group_by_input_shape=True).table(sort_by="self_cpu_time_total"))mpirun-n2-lpythonprofiling.py[0]myrank=0mysize=2[0]-------------------------------------------------------------------------------------------------------------------------------------------------[0]NameSelfCPU%SelfCPUCPUtotal%CPUtotalCPUtimeavg# of Calls          Input Shapes[0]-------------------------------------------------------------------------------------------------------------------------------------------------[0]oneccl_bindings_for_pytorch::allreduce91.41%297.900ms91.41%297.900ms29.790ms10[[2,2]][0]oneccl_bindings_for_pytorch::wait::cpu::allreduce8.24%26.845ms8.24%26.845ms2.684ms10[[2,2],[2,2]][0]oneccl_bindings_for_pytorch::wait::cpu::allreduce0.30%973.651us0.30%973.651us97.365us10[[4,4],[4,4]][0]oneccl_bindings_for_pytorch::allreduce0.06%190.254us0.06%190.254us19.025us10[[4,4]][0]-------------------------------------------------------------------------------------------------------------------------------------------------[0]SelfCPUtimetotal:325.909ms[0][1]myrank=1mysize=2[1]-------------------------------------------------------------------------------------------------------------------------------------------------[1]NameSelfCPU%SelfCPUCPUtotal%CPUtotalCPUtimeavg# of Calls          Input Shapes[1]-------------------------------------------------------------------------------------------------------------------------------------------------[1]oneccl_bindings_for_pytorch::allreduce96.03%318.551ms96.03%318.551ms31.855ms10[[2,2]][1]oneccl_bindings_for_pytorch::wait::cpu::allreduce3.62%12.019ms3.62%12.019ms1.202ms10[[2,2],[2,2]][1]oneccl_bindings_for_pytorch::allreduce0.33%1.082ms0.33%1.082ms108.157us10[[4,4]][1]oneccl_bindings_for_pytorch::wait::cpu::allreduce0.02%56.505us0.02%56.505us5.651us10[[4,4],[4,4]][1]-------------------------------------------------------------------------------------------------------------------------------------------------[1]SelfCPUtimetotal:331.708ms[1]Known IssuesFor Point-to-point communication, directly call dist.send/recv after initializing the process group in launch script will trigger runtime error. Because all ranks of the group are expected to participate in this call to create communicators in our current implementation, while dist.send/recv only has a pair of ranks' participation. As a result, dist.send/recv should be used after collective call, which ensures all ranks' participation. The further solution for supporting directly call dist.send/recv after initializing the process group is still under investigation.LicenseBSD License