neulab/docprompting-conala huggingface dataset:
torch.distributed.reduce_scatter_multigpu(output_tensor_list, input_tensor_lists, op=<ReduceOp.SUM: 0>, group=None, async_op=False) [source]
 
Reduce and scatter a list of tensors to the whole group. Only nccl backend is currently supported. Each tensor in output_tensor_list should reside on a separate GPU, as should each list of tensors in input_tensor_lists.  Parameters 
 
output_tensor_list (List[Tensor]) – 
Output tensors (on different GPUs) to receive the result of the operation. Note that len(output_tensor_list) needs to be the same for all the distributed processes calling this function.  
input_tensor_lists (List[List[Tensor]]) – 
Input lists. It should contain correctly-sized tensors on each GPU to be used for input of the collective, e.g. input_tensor_lists[i] contains the reduce_scatter input that resides on the GPU of output_tensor_list[i]. Note that each element of input_tensor_lists has the size of world_size * len(output_tensor_list), since the function scatters the result from every single GPU in the group. To interpret each element of input_tensor_lists[i], note that output_tensor_list[j] of rank k receives the reduce-scattered result from input_tensor_lists[i][k * world_size + j] Also note that len(input_tensor_lists), and the size of each element in input_tensor_lists (each element is a list, therefore len(input_tensor_lists[i])) need to be the same for all the distributed processes calling this function.  
group (ProcessGroup, optional) – The process group to work on. If None, the default process group will be used. 
async_op (bool, optional) – Whether this op should be an async op.   Returns 
Async work handle, if async_op is set to True. None, if not async_op or if not part of the group.
class torch.distributed.rpc.BackendType  
An enum class of available backends. PyTorch ships with two builtin backends: BackendType.TENSORPIPE and BackendType.PROCESS_GROUP. Additional ones can be registered using the register_backend() function.
torch.distributed.rpc.get_worker_info(worker_name=None) [source]
 
Get WorkerInfo of a given worker name. Use this WorkerInfo to avoid passing an expensive string on every invocation.  Parameters 
worker_name (str) – the string name of a worker. If None, return the the id of the current worker. (default None)  Returns 
WorkerInfo instance for the given worker_name or WorkerInfo of the current worker if worker_name is None.
torch.distributed.rpc.init_rpc(name, backend=None, rank=-1, world_size=None, rpc_backend_options=None) [source]
 
Initializes RPC primitives such as the local RPC agent and distributed autograd, which immediately makes the current process ready to send and receive RPCs.  Parameters 
 
name (str) – a globally unique name of this node. (e.g., Trainer3, ParameterServer2, Master, Worker1) Name can only contain number, alphabet, underscore, colon, and/or dash, and must be shorter than 128 characters. 
backend (BackendType, optional) – The type of RPC backend implementation. Supported values include BackendType.TENSORPIPE (the default) and BackendType.PROCESS_GROUP. See Backends for more information. 
rank (int) – a globally unique id/rank of this node. 
world_size (int) – The number of workers in the group. 
rpc_backend_options (RpcBackendOptions, optional) – The options passed to the RpcAgent constructor. It must be an agent-specific subclass of RpcBackendOptions and contains agent-specific initialization configurations. By default, for all agents, it sets the default timeout to 60 seconds and performs the rendezvous with an underlying process group initialized using init_method = "env://", meaning that environment variables MASTER_ADDR and MASTER_PORT need to be set properly. See Backends for more information and find which options are available.
class torch.distributed.rpc.ProcessGroupRpcBackendOptions  
The backend options class for ProcessGroupAgent, which is derived from RpcBackendOptions.  Parameters 
 
num_send_recv_threads (int, optional) – The number of threads in the thread-pool used by ProcessGroupAgent (default: 4). 
rpc_timeout (float, optional) – The default timeout, in seconds, for RPC requests (default: 60 seconds). If the RPC has not completed in this timeframe, an exception indicating so will be raised. Callers can override this timeout for individual RPCs in rpc_sync() and rpc_async() if necessary. 
init_method (str, optional) – The URL to initialize ProcessGroupGloo (default: env://).     
property init_method  
URL specifying how to initialize the process group. Default is env:// 
  
property num_send_recv_threads  
The number of threads in the thread-pool used by ProcessGroupAgent. 
  
property rpc_timeout  
A float indicating the timeout to use for all RPCs. If an RPC does not complete in this timeframe, it will complete with an exception indicating that it has timed out.
property num_send_recv_threads  
The number of threads in the thread-pool used by ProcessGroupAgent.
property rpc_timeout  
A float indicating the timeout to use for all RPCs. If an RPC does not complete in this timeframe, it will complete with an exception indicating that it has timed out.
class torch.distributed.rpc.RpcBackendOptions  
An abstract structure encapsulating the options passed into the RPC backend. An instance of this class can be passed in to init_rpc() in order to initialize RPC with specific configurations, such as the RPC timeout and init_method to be used.  
property init_method  
URL specifying how to initialize the process group. Default is env:// 
  
property rpc_timeout  
A float indicating the timeout to use for all RPCs. If an RPC does not complete in this timeframe, it will complete with an exception indicating that it has timed out.
torch.distributed.rpc.rpc_async(to, func, args=None, kwargs=None, timeout=-1.0) [source]
 
Make a non-blocking RPC call to run function func on worker to. RPC messages are sent and received in parallel to execution of Python code. This method is thread-safe. This method will immediately return a Future that can be awaited on.  Parameters 
 
to (str or WorkerInfo or int) – name/rank/WorkerInfo of the destination worker. 
func (callable) – a callable function, such as Python callables, builtin operators (e.g. add()) and annotated TorchScript functions. 
args (tuple) – the argument tuple for the func invocation. 
kwargs (dict) – is a dictionary of keyword arguments for the func invocation. 
timeout (float, optional) – timeout in seconds to use for this RPC. If the RPC does not complete in this amount of time, an exception indicating it has timed out will be raised. A value of 0 indicates an infinite timeout, i.e. a timeout error will never be raised. If not provided, the default value set during initialization or with _set_rpc_timeout is used.   Returns 
Returns a Future object that can be waited on. When completed, the return value of func on args and kwargs can be retrieved from the Future object.    Warning Using GPU tensors as arguments or return values of func is not supported since we don’t support sending GPU tensors over the wire. You need to explicitly copy GPU tensors to CPU before using them as arguments or return values of func.   Warning The rpc_async API does not copy storages of argument tensors until sending them over the wire, which could be done by a different thread depending on the RPC backend type. The caller should make sure that the contents of those tensors stay intact until the returned Future completes.   Example::

Make sure that MASTER_ADDR and MASTER_PORT are set properly on both workers. Refer to init_process_group() API for more details. For example, >>> export MASTER_ADDR=localhost
>>> export MASTER_PORT=5678
 Then run the following code in two different processes: >>> # On worker 0:
>>> import torch
>>> import torch.distributed.rpc as rpc
>>> rpc.init_rpc("worker0", rank=0, world_size=2)
>>> fut1 = rpc.rpc_async("worker1", torch.add, args=(torch.ones(2), 3))
>>> fut2 = rpc.rpc_async("worker1", min, args=(1, 2))
>>> result = fut1.wait() + fut2.wait()
>>> rpc.shutdown()
 >>> # On worker 1:
>>> import torch.distributed.rpc as rpc
>>> rpc.init_rpc("worker1", rank=1, world_size=2)
>>> rpc.shutdown()
 Below is an example of running a TorchScript function using RPC. >>> # On both workers:
>>> @torch.jit.script
>>> def my_script_add(t1, t2):
>>>    return torch.add(t1, t2)
 >>> # On worker 0:
>>> import torch.distributed.rpc as rpc
>>> rpc.init_rpc("worker0", rank=0, world_size=2)
>>> fut = rpc.rpc_async("worker1", my_script_add, args=(torch.ones(2), 3))
>>> ret = fut.wait()
>>> rpc.shutdown()
 >>> # On worker 1:
>>> import torch.distributed.rpc as rpc
>>> rpc.init_rpc("worker1", rank=1, world_size=2)
>>> rpc.shutdown()
torch.distributed.rpc.rpc_sync(to, func, args=None, kwargs=None, timeout=-1.0) [source]
 
Make a blocking RPC call to run function func on worker to. RPC messages are sent and received in parallel to execution of Python code. This method is thread-safe.  Parameters 
 
to (str or WorkerInfo or int) – name/rank/WorkerInfo of the destination worker. 
func (callable) – a callable function, such as Python callables, builtin operators (e.g. add()) and annotated TorchScript functions. 
args (tuple) – the argument tuple for the func invocation. 
kwargs (dict) – is a dictionary of keyword arguments for the func invocation. 
timeout (float, optional) – timeout in seconds to use for this RPC. If the RPC does not complete in this amount of time, an exception indicating it has timed out will be raised. A value of 0 indicates an infinite timeout, i.e. a timeout error will never be raised. If not provided, the default value set during initialization or with _set_rpc_timeout is used.   Returns 
Returns the result of running func with args and kwargs.    Warning Using GPU tensors as arguments or return values of func is not supported since we don’t support sending GPU tensors over the wire. You need to explicitly copy GPU tensors to CPU before using them as arguments or return values of func.   Example::

Make sure that MASTER_ADDR and MASTER_PORT are set properly on both workers. Refer to init_process_group() API for more details. For example, >>> export MASTER_ADDR=localhost
>>> export MASTER_PORT=5678
 Then run the following code in two different processes: >>> # On worker 0:
>>> import torch
>>> import torch.distributed.rpc as rpc
>>> rpc.init_rpc("worker0", rank=0, world_size=2)
>>> ret = rpc.rpc_sync("worker1", torch.add, args=(torch.ones(2), 3))
>>> rpc.shutdown()
 >>> # On worker 1:
>>> import torch.distributed.rpc as rpc
>>> rpc.init_rpc("worker1", rank=1, world_size=2)
>>> rpc.shutdown()
 Below is an example of running a TorchScript function using RPC. >>> # On both workers:
>>> @torch.jit.script
>>> def my_script_add(t1, t2):
>>>    return torch.add(t1, t2)
 >>> # On worker 0:
>>> import torch.distributed.rpc as rpc
>>> rpc.init_rpc("worker0", rank=0, world_size=2)
>>> ret = rpc.rpc_sync("worker1", my_script_add, args=(torch.ones(2), 3))
>>> rpc.shutdown()
 >>> # On worker 1:
>>> import torch.distributed.rpc as rpc
>>> rpc.init_rpc("worker1", rank=1, world_size=2)
>>> rpc.shutdown()
backward(self: torch._C._distributed_rpc.PyRRef, dist_autograd_ctx_id: int = -1, retain_graph: bool = False) → None   Runs the backward pass using the RRef as the root of the backward pass. If dist_autograd_ctx_id is provided, we perform a distributed backward pass using the provided ctx_id starting from the owner of the RRef. In this case, get_gradients() should be used to retrieve the gradients. If dist_autograd_ctx_id is None, it is assumed that this is a local autograd graph and we only perform a local backward pass. In the local case, the node calling this API has to be the owner of the RRef. The value of the RRef is expected to be a scalar Tensor.  Parameters 
 
dist_autograd_ctx_id (int, optional) – The distributed autograd context id for which we should retrieve the gradients (default: -1). 
retain_graph (bool, optional) – If False, the graph used to compute the grad will be freed. Note that in nearly all cases setting this option to True is not needed and often can be worked around in a much more efficient way. Usually, you need to set this to True to run backward multiple times (default: False).     Example::

>>> import torch.distributed.autograd as dist_autograd
>>> with dist_autograd.context() as context_id:
>>>     rref.backward(context_id)
confirmed_by_owner(self: torch._C._distributed_rpc.PyRRef) → bool  
Returns whether this RRef has been confirmed by the owner. OwnerRRef always returns true, while UserRRef only returns true when the owner knowns about this UserRRef.
is_owner(self: torch._C._distributed_rpc.PyRRef) → bool  
Returns whether or not the current node is the owner of this RRef.
local_value(self: torch._C._distributed_rpc.PyRRef) → object  
If the current node is the owner, returns a reference to the local value. Otherwise, throws an exception.
owner(self: torch._C._distributed_rpc.PyRRef) → torch._C._distributed_rpc.WorkerInfo  
Returns worker information of the node that owns this RRef.
owner_name(self: torch._C._distributed_rpc.PyRRef) → str  
Returns worker name of the node that owns this RRef.
remote(self: torch._C._distributed_rpc.PyRRef, timeout: float = -1.0) → object  
Create a helper proxy to easily launch a remote using the owner of the RRef as the destination to run functions on the object referenced by this RRef. More specifically, rref.remote().func_name(*args, **kwargs) is the same as the following: >>> def run(rref, func_name, args, kwargs):
>>>   return getattr(rref.local_value(), func_name)(*args, **kwargs)
>>>
>>> rpc.remote(rref.owner(), run, args=(rref, func_name, args, kwargs))
  Parameters 
timeout (float, optional) – Timeout for rref.remote(). If the creation of this RRef is not successfully completed within the timeout, then the next time there is an attempt to use the RRef (such as to_here), a timeout will be raised. If not provided, the default RPC timeout will be used. Please see rpc.remote() for specific timeout semantics for RRef.    Example::

>>> from torch.distributed import rpc
>>> rref = rpc.remote("worker1", torch.add, args=(torch.zeros(2, 2), 1))
>>> rref.remote().size().to_here()  # returns torch.Size([2, 2])
>>> rref.remote().view(1, 4).to_here()  # returns tensor([[1., 1., 1., 1.]])
rpc_async(self: torch._C._distributed_rpc.PyRRef, timeout: float = -1.0) → object  
Create a helper proxy to easily launch an rpc_async using the owner of the RRef as the destination to run functions on the object referenced by this RRef. More specifically, rref.rpc_async().func_name(*args, **kwargs) is the same as the following: >>> def run(rref, func_name, args, kwargs):
>>>   return getattr(rref.local_value(), func_name)(*args, **kwargs)
>>>
>>> rpc.rpc_async(rref.owner(), run, args=(rref, func_name, args, kwargs))
  Parameters 
timeout (float, optional) – Timeout for rref.rpc_async(). If the call does not complete within this timeframe, an exception indicating so will be raised. If this argument is not provided, the default RPC timeout will be used.    Example::

>>> from torch.distributed import rpc
>>> rref = rpc.remote("worker1", torch.add, args=(torch.zeros(2, 2), 1))
>>> rref.rpc_async().size().wait()  # returns torch.Size([2, 2])
>>> rref.rpc_async().view(1, 4).wait()  # returns tensor([[1., 1., 1., 1.]])
rpc_sync(self: torch._C._distributed_rpc.PyRRef, timeout: float = -1.0) → object  
Create a helper proxy to easily launch an rpc_sync using the owner of the RRef as the destination to run functions on the object referenced by this RRef. More specifically, rref.rpc_sync().func_name(*args, **kwargs) is the same as the following: >>> def run(rref, func_name, args, kwargs):
>>>   return getattr(rref.local_value(), func_name)(*args, **kwargs)
>>>
>>> rpc.rpc_sync(rref.owner(), run, args=(rref, func_name, args, kwargs))
  Parameters 
timeout (float, optional) – Timeout for rref.rpc_sync(). If the call does not complete within this timeframe, an exception indicating so will be raised. If this argument is not provided, the default RPC timeout will be used.    Example::

>>> from torch.distributed import rpc
>>> rref = rpc.remote("worker1", torch.add, args=(torch.zeros(2, 2), 1))
>>> rref.rpc_sync().size()  # returns torch.Size([2, 2])
>>> rref.rpc_sync().view(1, 4)  # returns tensor([[1., 1., 1., 1.]])
to_here(self: torch._C._distributed_rpc.PyRRef, timeout: float = -1.0) → object  
Blocking call that copies the value of the RRef from the owner to the local node and returns it. If the current node is the owner, returns a reference to the local value.  Parameters 
timeout (float, optional) – Timeout for to_here. If the call does not complete within this timeframe, an exception indicating so will be raised. If this argument is not provided, the default RPC timeout (60s) will be used.
torch.distributed.rpc.shutdown(graceful=True) [source]
 
Perform a shutdown of the RPC agent, and then destroy the RPC agent. This stops the local agent from accepting outstanding requests, and shuts down the RPC framework by terminating all RPC threads. If graceful=True, this will block until all local and remote RPC processes reach this method and wait for all outstanding work to complete. Otherwise, if graceful=False, this is a local shutdown, and it does not wait for other RPC processes to reach this method.  Warning For Future objects returned by rpc_async(), future.wait() should not be called after shutdown().   Parameters 
graceful (bool) – Whether to do a graceful shutdown or not. If True, this will 1) wait until there is no pending system messages for UserRRefs and delete them; 2) block until all local and remote RPC processes have reached this method and wait for all outstanding work to complete.    Example::

Make sure that MASTER_ADDR and MASTER_PORT are set properly on both workers. Refer to init_process_group() API for more details. For example, >>> export MASTER_ADDR=localhost
>>> export MASTER_PORT=5678
 Then run the following code in two different processes: >>> # On worker 0:
>>> import torch
>>> import torch.distributed.rpc as rpc
>>> rpc.init_rpc("worker0", rank=0, world_size=2)
>>> # do some work
>>> result = rpc.rpc_sync("worker1", torch.add, args=(torch.ones(1), 1))
>>> # ready to shutdown
>>> rpc.shutdown()
 >>> # On worker 1:
>>> import torch.distributed.rpc as rpc
>>> rpc.init_rpc("worker1", rank=1, world_size=2)
>>> # wait for worker 0 to finish work, and then shutdown.
>>> rpc.shutdown()
class torch.distributed.rpc.TensorPipeRpcBackendOptions(*, num_worker_threads=16, rpc_timeout=60.0, init_method='env://', device_maps=None, _transports=None, _channels=None) [source]
 
The backend options for TensorPipeAgent, derived from RpcBackendOptions.  Parameters 
 
num_worker_threads (int, optional) – The number of threads in the thread-pool used by TensorPipeAgent to execute requests (default: 16). 
rpc_timeout (float, optional) – The default timeout, in seconds, for RPC requests (default: 60 seconds). If the RPC has not completed in this timeframe, an exception indicating so will be raised. Callers can override this timeout for individual RPCs in rpc_sync() and rpc_async() if necessary. 
init_method (str, optional) – The URL to initialize the distributed store used for rendezvous. It takes any value accepted for the same argument of init_process_group() (default: env://). 
device_maps (Dict[str, Dict]) – Device placement mappings from this worker to the callee. Key is the callee worker name and value the dictionary (Dict of int, str, or torch.device) that maps this worker’s devices to the callee worker’s devices. (default: None)     
property device_maps  
The device map locations. 
  
property init_method  
URL specifying how to initialize the process group. Default is env:// 
  
property num_worker_threads  
The number of threads in the thread-pool used by TensorPipeAgent to execute requests. 
  
property rpc_timeout  
A float indicating the timeout to use for all RPCs. If an RPC does not complete in this timeframe, it will complete with an exception indicating that it has timed out. 
  
set_device_map(to, device_map) [source]
 
Set device mapping between each RPC caller and callee pair. This function can be called multiple times to incrementally add device placement configurations.  Parameters 
 
worker_name (str) – Callee name. 
device_map (Dict of python:int, str, or torch.device) – Device placement mappings from this worker to the callee. This map must be invertible.     Example::

>>> # both workers
>>> def add(x, y):
>>>     print(x)  # tensor([1., 1.], device='cuda:1')
>>>     return x + y, (x + y).to(2)
>>>
>>> # on worker 0
>>> options = TensorPipeRpcBackendOptions(
>>>     num_worker_threads=8,
>>>     device_maps={"worker1": {0, 1}}
>>>     # maps worker0's cuda:0 to worker1's cuda:1
>>> )
>>> options.set_device_map("worker1", {1, 2})
>>> # maps worker0's cuda:1 to worker1's cuda:2
>>>
>>> rpc.init_rpc(
>>>     "worker0",
>>>     rank=0,
>>>     world_size=2
>>>     backend=rpc.BackendType.TENSORPIPE,
>>>     rpc_backend_options=options
>>> )
>>>
>>> x = torch.ones(2)
>>> rets = rpc.rpc_sync("worker1", add, args=(x.to(0), 1))
>>> # The first argument will be moved to cuda:1 on worker1. When
>>> # sending the return value back, it will follow the invert of
>>> # the device map, and hence will be moved back to cuda:0 and
>>> # cuda:1 on worker0
>>> print(rets[0])  # tensor([2., 2.], device='cuda:0')
>>> print(rets[0])  # tensor([2., 2.], device='cuda:1')
property num_worker_threads  
The number of threads in the thread-pool used by TensorPipeAgent to execute requests.
set_device_map(to, device_map) [source]
 
Set device mapping between each RPC caller and callee pair. This function can be called multiple times to incrementally add device placement configurations.  Parameters 
 
worker_name (str) – Callee name. 
device_map (Dict of python:int, str, or torch.device) – Device placement mappings from this worker to the callee. This map must be invertible.     Example::

>>> # both workers
>>> def add(x, y):
>>>     print(x)  # tensor([1., 1.], device='cuda:1')
>>>     return x + y, (x + y).to(2)
>>>
>>> # on worker 0
>>> options = TensorPipeRpcBackendOptions(
>>>     num_worker_threads=8,
>>>     device_maps={"worker1": {0, 1}}
>>>     # maps worker0's cuda:0 to worker1's cuda:1
>>> )
>>> options.set_device_map("worker1", {1, 2})
>>> # maps worker0's cuda:1 to worker1's cuda:2
>>>
>>> rpc.init_rpc(
>>>     "worker0",
>>>     rank=0,
>>>     world_size=2
>>>     backend=rpc.BackendType.TENSORPIPE,
>>>     rpc_backend_options=options
>>> )
>>>
>>> x = torch.ones(2)
>>> rets = rpc.rpc_sync("worker1", add, args=(x.to(0), 1))
>>> # The first argument will be moved to cuda:1 on worker1. When
>>> # sending the return value back, it will follow the invert of
>>> # the device map, and hence will be moved back to cuda:0 and
>>> # cuda:1 on worker0
>>> print(rets[0])  # tensor([2., 2.], device='cuda:0')
>>> print(rets[0])  # tensor([2., 2.], device='cuda:1')
class torch.distributed.rpc.WorkerInfo  
A structure that encapsulates information of a worker in the system. Contains the name and ID of the worker. This class is not meant to be constructed directly, rather, an instance can be retrieved through get_worker_info() and the result can be passed in to functions such as rpc_sync(), rpc_async(), remote() to avoid copying a string on every invocation.  
property id  
Globally unique id to identify the worker. 
  
property name  
The name of the worker.
torch.distributed.scatter(tensor, scatter_list=None, src=0, group=None, async_op=False) [source]
 
Scatters a list of tensors to all processes in a group. Each process will receive exactly one tensor and store its data in the tensor argument.  Parameters 
 
tensor (Tensor) – Output tensor. 
scatter_list (list[Tensor]) – List of tensors to scatter (default is None, must be specified on the source rank) 
src (int) – Source rank (default is 0) 
group (ProcessGroup, optional) – The process group to work on. If None, the default process group will be used. 
async_op (bool, optional) – Whether this op should be an async op   Returns 
Async work handle, if async_op is set to True. None, if not async_op or if not part of the group