neulab/docprompting-conala huggingface dataset:
torch.distributed.broadcast_object_list(object_list, src=0, group=None) [source]
 
Broadcasts picklable objects in object_list to the whole group. Similar to broadcast(), but Python objects can be passed in. Note that all objects in object_list must be picklable in order to be broadcasted.  Parameters 
 
object_list (List[Any]) – List of input objects to broadcast. Each object must be picklable. Only objects on the src rank will be broadcast, but each rank must provide lists of equal sizes. 
src (int) – Source rank from which to broadcast object_list. 
group – (ProcessGroup, optional): The process group to work on. If None, the default process group will be used. Default is None.   Returns 
None. If rank is part of the group, object_list will contain the broadcasted objects from src rank.    Note For NCCL-based processed groups, internal tensor representations of objects must be moved to the GPU device before communication takes place. In this case, the device used is given by torch.cuda.current_device() and it is the user’s responsiblity to ensure that this is set so that each rank has an individual GPU, via torch.cuda.set_device().   Note Note that this API differs slightly from the all_gather() collective since it does not provide an async_op handle and thus will be a blocking call.   Warning broadcast_object_list() uses pickle module implicitly, which is known to be insecure. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling. Only call this function with data you trust.   Example::

>>> # Note: Process group initialization omitted on each rank.
>>> import torch.distributed as dist
>>> if dist.get_rank() == 0:
>>>     # Assumes world_size of 3.
>>>     objects = ["foo", 12, {1: 2}] # any picklable object
>>> else:
>>>     objects = [None, None, None]
>>> dist.broadcast_object_list(objects, src=0)
>>> broadcast_objects
['foo', 12, {1: 2}]
class torch.distributed.FileStore  
A store implementation that uses a file to store the underlying key-value pairs.  Parameters 
 
file_name (str) – path of the file in which to store the key-value pairs 
world_size (int) – The total number of processes using the store     Example::

>>> import torch.distributed as dist
>>> store1 = dist.FileStore("/tmp/filestore", 2)
>>> store2 = dist.FileStore("/tmp/filestore", 2)
>>> # Use any of the store methods from either the client or server after initialization
>>> store1.set("first_key", "first_value")
>>> store2.get("first_key")
torch.distributed.gather(tensor, gather_list=None, dst=0, group=None, async_op=False) [source]
 
Gathers a list of tensors in a single process.  Parameters 
 
tensor (Tensor) – Input tensor. 
gather_list (list[Tensor], optional) – List of appropriately-sized tensors to use for gathered data (default is None, must be specified on the destination rank) 
dst (int, optional) – Destination rank (default is 0) 
group (ProcessGroup, optional) – The process group to work on. If None, the default process group will be used. 
async_op (bool, optional) – Whether this op should be an async op   Returns 
Async work handle, if async_op is set to True. None, if not async_op or if not part of the group
torch.distributed.gather_object(obj, object_gather_list=None, dst=0, group=None) [source]
 
Gathers picklable objects from the whole group in a single process. Similar to gather(), but Python objects can be passed in. Note that the object must be picklable in order to be gathered.  Parameters 
 
obj (Any) – Input object. Must be picklable. 
object_gather_list (list[Any]) – Output list. On the dst rank, it should be correctly sized as the size of the group for this collective and will contain the output. Must be None on non-dst ranks. (default is None) 
dst (int, optional) – Destination rank. (default is 0) 
group – (ProcessGroup, optional): The process group to work on. If None, the default process group will be used. Default is None.   Returns 
None. On the dst rank, object_gather_list will contain the output of the collective.    Note Note that this API differs slightly from the gather collective since it does not provide an async_op handle and thus will be a blocking call.   Note Note that this API is not supported when using the NCCL backend.   Warning gather_object() uses pickle module implicitly, which is known to be insecure. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling. Only call this function with data you trust.   Example::

>>> # Note: Process group initialization omitted on each rank.
>>> import torch.distributed as dist
>>> # Assumes world_size of 3.
>>> gather_objects = ["foo", 12, {1: 2}] # any picklable object
>>> output = [None for _ in gather_objects]
>>> dist.gather_object(
        gather_objects[dist.get_rank()],
        output if dist.get_rank() == 0 else None,
        dst=0
    )
>>> # On rank 0
>>> output
['foo', 12, {1: 2}]
torch.distributed.get_backend(group=None) [source]
 
Returns the backend of the given process group.  Parameters 
group (ProcessGroup, optional) – The process group to work on. The default is the general main process group. If another specific group is specified, the calling process must be part of group.  Returns 
The backend of the given process group as a lower case string.
torch.distributed.get_rank(group=None) [source]
 
Returns the rank of current process group Rank is a unique identifier assigned to each process within a distributed process group. They are always consecutive integers ranging from 0 to world_size.  Parameters 
group (ProcessGroup, optional) – The process group to work on. If None, the default process group will be used.  Returns 
The rank of the process group -1, if not part of the group
torch.distributed.get_world_size(group=None) [source]
 
Returns the number of processes in the current process group  Parameters 
group (ProcessGroup, optional) – The process group to work on. If None, the default process group will be used.  Returns 
The world size of the process group -1, if not part of the group
class torch.distributed.HashStore  
A thread-safe store implementation based on an underlying hashmap. This store can be used within the same process (for example, by other threads), but cannot be used across processes.  Example::

>>> import torch.distributed as dist
>>> store = dist.HashStore()
>>> # store can be used from other threads
>>> # Use any of the store methods after initialization
>>> store.set("first_key", "first_value")
torch.distributed.init_process_group(backend, init_method=None, timeout=datetime.timedelta(seconds=1800), world_size=-1, rank=-1, store=None, group_name='') [source]
 
Initializes the default distributed process group, and this will also initialize the distributed package.  There are 2 main ways to initialize a process group:

 Specify store, rank, and world_size explicitly. Specify init_method (a URL string) which indicates where/how to discover peers. Optionally specify rank and world_size, or encode all required parameters in the URL and omit them.    If neither is specified, init_method is assumed to be “env://”.  Parameters 
 
backend (str or Backend) – The backend to use. Depending on build-time configurations, valid values include mpi, gloo, and nccl. This field should be given as a lowercase string (e.g., "gloo"), which can also be accessed via Backend attributes (e.g., Backend.GLOO). If using multiple processes per machine with nccl backend, each process must have exclusive access to every GPU it uses, as sharing GPUs between processes can result in deadlocks. 
init_method (str, optional) – URL specifying how to initialize the process group. Default is “env://” if no init_method or store is specified. Mutually exclusive with store. 
world_size (int, optional) – Number of processes participating in the job. Required if store is specified. 
rank (int, optional) – Rank of the current process (it should be a number between 0 and world_size-1). Required if store is specified. 
store (Store, optional) – Key/value store accessible to all workers, used to exchange connection/address information. Mutually exclusive with init_method. 
timeout (timedelta, optional) – Timeout for operations executed against the process group. Default value equals 30 minutes. This is applicable for the gloo backend. For nccl, this is applicable only if the environment variable NCCL_BLOCKING_WAIT or NCCL_ASYNC_ERROR_HANDLING is set to 1. When NCCL_BLOCKING_WAIT is set, this is the duration for which the process will block and wait for collectives to complete before throwing an exception. When NCCL_ASYNC_ERROR_HANDLING is set, this is the duration after which collectives will be aborted asynchronously and the process will crash. NCCL_BLOCKING_WAIT will provide errors to the user which can be caught and handled, but due to its blocking nature, it has a performance overhead. On the other hand, NCCL_ASYNC_ERROR_HANDLING has very little performance overhead, but crashes the process on errors. This is done since CUDA execution is async and it is no longer safe to continue executing user code since failed async NCCL operations might result in subsequent CUDA operations running on corrupted data. Only one of these two environment variables should be set. 
group_name (str, optional, deprecated) – Group name.    To enable backend == Backend.MPI, PyTorch needs to be built from source on a system that supports MPI.
torch.distributed.irecv(tensor, src=None, group=None, tag=0) [source]
 
Receives a tensor asynchronously.  Parameters 
 
tensor (Tensor) – Tensor to fill with received data. 
src (int, optional) – Source rank. Will receive from any process if unspecified. 
group (ProcessGroup, optional) – The process group to work on. If None, the default process group will be used. 
tag (int, optional) – Tag to match recv with remote send   Returns 
A distributed request object. None, if not part of the group
torch.distributed.isend(tensor, dst, group=None, tag=0) [source]
 
Sends a tensor asynchronously.  Parameters 
 
tensor (Tensor) – Tensor to send. 
dst (int) – Destination rank. 
group (ProcessGroup, optional) – The process group to work on. If None, the default process group will be used. 
tag (int, optional) – Tag to match send with remote recv   Returns 
A distributed request object. None, if not part of the group
torch.distributed.is_available() [source]
 
Returns True if the distributed package is available. Otherwise, torch.distributed does not expose any other APIs. Currently, torch.distributed is available on Linux, MacOS and Windows. Set USE_DISTRIBUTED=1 to enable it when building PyTorch from source. Currently, the default value is USE_DISTRIBUTED=1 for Linux and Windows, USE_DISTRIBUTED=0 for MacOS.
torch.distributed.is_initialized() [source]
 
Checking if the default process group has been initialized
torch.distributed.is_mpi_available() [source]
 
Checks if the MPI backend is available.
torch.distributed.is_nccl_available() [source]
 
Checks if the NCCL backend is available.
torch.distributed.new_group(ranks=None, timeout=datetime.timedelta(seconds=1800), backend=None) [source]
 
Creates a new distributed group. This function requires that all processes in the main group (i.e. all processes that are part of the distributed job) enter this function, even if they are not going to be members of the group. Additionally, groups should be created in the same order in all processes.  Warning Using multiple process groups with the NCCL backend concurrently is not safe and the user should perform explicit synchronization in their application to ensure only one process group is used at a time. This means collectives from one process group should have completed execution on the device (not just enqueued since CUDA execution is async) before collectives from another process group are enqueued. See Using multiple NCCL communicators concurrently for more details.   Parameters 
 
ranks (list[int]) – List of ranks of group members. If None, will be set to all ranks. Default is None. 
timeout (timedelta, optional) – Timeout for operations executed against the process group. Default value equals 30 minutes. This is only applicable for the gloo backend. 
backend (str or Backend, optional) – The backend to use. Depending on build-time configurations, valid values are gloo and nccl. By default uses the same backend as the global group. This field should be given as a lowercase string (e.g., "gloo"), which can also be accessed via Backend attributes (e.g., Backend.GLOO).   Returns 
A handle of distributed group that can be given to collective calls.
class torch.distributed.optim.DistributedOptimizer(optimizer_class, params_rref, *args, **kwargs) [source]
 
DistributedOptimizer takes remote references to parameters scattered across workers and applies the given optimizer locally for each parameter. This class uses get_gradients() in order to retrieve the gradients for specific parameters. Concurrent calls to step(), either from the same or different clients, will be serialized on each worker – as each worker’s optimizer can only work on one set of gradients at a time. However, there is no guarantee that the full forward-backward-optimizer sequence will execute for one client at a time. This means that the gradients being applied may not correspond to the latest forward pass executed on a given worker. Also, there is no guaranteed ordering across workers. DistributedOptimizer creates the local optimizer with TorchScript enabled by default, so that optimizer updates are not blocked by the Python Global Interpreter Lock (GIL) during multithreaded training (e.g. Distributed Model Parallel). This feature is currently in beta stage, enabled for optimizers including Adagrad, Adam, SGD, RMSprop, AdamW and Adadelta. We are increasing the coverage to all optimizers in future releases.  Parameters 
 
optimizer_class (optim.Optimizer) – the class of optimizer to instantiate on each worker. 
params_rref (list[RRef]) – list of RRefs to local or remote parameters to optimize. 
args – arguments to pass to the optimizer constructor on each worker. 
kwargs – arguments to pass to the optimizer constructor on each worker.     Example::

>>> import torch.distributed.autograd as dist_autograd
>>> import torch.distributed.rpc as rpc
>>> from torch import optim
>>> from torch.distributed.optim import DistributedOptimizer
>>>
>>> with dist_autograd.context() as context_id:
>>>   # Forward pass.
>>>   rref1 = rpc.remote("worker1", torch.add, args=(torch.ones(2), 3))
>>>   rref2 = rpc.remote("worker1", torch.add, args=(torch.ones(2), 1))
>>>   loss = rref1.to_here() + rref2.to_here()
>>>
>>>   # Backward pass.
>>>   dist_autograd.backward(context_id, [loss.sum()])
>>>
>>>   # Optimizer.
>>>   dist_optim = DistributedOptimizer(
>>>      optim.SGD,
>>>      [rref1, rref2],
>>>      lr=0.05,
>>>   )
>>>   dist_optim.step(context_id)
    
step(context_id) [source]
 
Performs a single optimization step. This will call torch.optim.Optimizer.step() on each worker containing parameters to be optimized, and will block until all workers return. The provided context_id will be used to retrieve the corresponding context that contains the gradients that should be applied to the parameters.  Parameters 
context_id – the autograd context id for which we should run the optimizer step.
step(context_id) [source]
 
Performs a single optimization step. This will call torch.optim.Optimizer.step() on each worker containing parameters to be optimized, and will block until all workers return. The provided context_id will be used to retrieve the corresponding context that contains the gradients that should be applied to the parameters.  Parameters 
context_id – the autograd context id for which we should run the optimizer step.
forward(input) [source]
 
Processes a single input mini-batch through the pipe and returns an RRef pointing to the output. Pipe is a fairly transparent module wrapper. It doesn’t modify the input and output signature of the underlying module. But there’s type restriction. Input and output have to be a Tensor or a sequence of tensors. This restriction is applied at partition boundaries too. The input tensor is split into multiple micro-batches based on the chunks parameter used to initialize Pipe. The batch size is assumed to be the first dimension of the tensor and if the batch size is less than chunks, the number of micro-batches is equal to the batch size.  Parameters 
input (torch.Tensor or sequence of Tensor) – input mini-batch  Returns 
RRef to the output of the mini-batch  Raises 
TypeError – input is not a tensor or sequence of tensors.
class torch.distributed.pipeline.sync.skip.skippable.pop(name) [source]
 
The command to pop a skip tensor. def forward(self, input):
    skip = yield pop('name')
    return f(input) + skip
  Parameters 
name (str) – name of skip tensor  Returns 
the skip tensor previously stashed by another layer under the same name
torch.distributed.pipeline.sync.skip.skippable.skippable(stash=(), pop=()) [source]
 
The decorator to define a nn.Module with skip connections. Decorated modules are called “skippable”. This functionality works perfectly fine even when the module is not wrapped by Pipe. Each skip tensor is managed by its name. Before manipulating skip tensors, a skippable module must statically declare the names for skip tensors by stash and/or pop parameters. Skip tensors with pre-declared name can be stashed by yield stash(name, tensor) or popped by tensor = yield
pop(name). Here is an example with three layers. A skip tensor named “1to3” is stashed and popped at the first and last layer, respectively: @skippable(stash=['1to3'])
class Layer1(nn.Module):
    def forward(self, input):
        yield stash('1to3', input)
        return f1(input)

class Layer2(nn.Module):
    def forward(self, input):
        return f2(input)

@skippable(pop=['1to3'])
class Layer3(nn.Module):
    def forward(self, input):
        skip_1to3 = yield pop('1to3')
        return f3(input) + skip_1to3

model = nn.Sequential(Layer1(), Layer2(), Layer3())
 One skippable module can stash or pop multiple skip tensors: @skippable(stash=['alice', 'bob'], pop=['carol'])
class StashStashPop(nn.Module):
    def forward(self, input):
        yield stash('alice', f_alice(input))
        yield stash('bob', f_bob(input))
        carol = yield pop('carol')
        return input + carol
 Every skip tensor must be associated with exactly one pair of stash and pop. Pipe checks this restriction automatically when wrapping a module. You can also check the restriction by verify_skippables() without Pipe.
class torch.distributed.pipeline.sync.skip.skippable.stash(name, tensor) [source]
 
The command to stash a skip tensor. def forward(self, input):
    yield stash('name', input)
    return f(input)
  Parameters 
 
name (str) – name of skip tensor 
input (torch.Tensor or None) – tensor to pass to the skip connection
torch.distributed.pipeline.sync.skip.skippable.verify_skippables(module) [source]
 
Verifies if the underlying skippable modules satisfy integrity. Every skip tensor must have only one pair of stash and pop. If there are one or more unmatched pairs, it will raise TypeError with the detailed messages. Here are a few failure cases. verify_skippables() will report failure for these cases: # Layer1 stashes "1to3".
# Layer3 pops "1to3".

nn.Sequential(Layer1(), Layer2())
#               └──── ?

nn.Sequential(Layer2(), Layer3())
#                   ? ────┘

nn.Sequential(Layer1(), Layer2(), Layer3(), Layer3())
#               └───────────────────┘       ^^^^^^

nn.Sequential(Layer1(), Layer1(), Layer2(), Layer3())
#             ^^^^^^      └───────────────────┘
 To use the same name for multiple skip tensors, they must be isolated by different namespaces. See isolate().  Raises 
TypeError – one or more pairs of stash and pop are not matched.
class torch.distributed.PrefixStore  
A wrapper around any of the 3 key-value stores (TCPStore, FileStore, and HashStore) that adds a prefix to each key inserted to the store.  Parameters 
 
prefix (str) – The prefix string that is prepended to each key before being inserted into the store. 
store (torch.distributed.store) – A store object that forms the underlying key-value store.
torch.distributed.recv(tensor, src=None, group=None, tag=0) [source]
 
Receives a tensor synchronously.  Parameters 
 
tensor (Tensor) – Tensor to fill with received data. 
src (int, optional) – Source rank. Will receive from any process if unspecified. 
group (ProcessGroup, optional) – The process group to work on. If None, the default process group will be used. 
tag (int, optional) – Tag to match recv with remote send   Returns 
Sender rank -1, if not part of the group
torch.distributed.reduce(tensor, dst, op=<ReduceOp.SUM: 0>, group=None, async_op=False) [source]
 
Reduces the tensor data across all machines. Only the process with rank dst is going to receive the final result.  Parameters 
 
tensor (Tensor) – Input and output of the collective. The function operates in-place. 
dst (int) – Destination rank 
op (optional) – One of the values from torch.distributed.ReduceOp enum. Specifies an operation used for element-wise reductions. 
group (ProcessGroup, optional) – The process group to work on. If None, the default process group will be used. 
async_op (bool, optional) – Whether this op should be an async op   Returns 
Async work handle, if async_op is set to True. None, if not async_op or if not part of the group
class torch.distributed.ReduceOp  
An enum-like class for available reduction operations: SUM, PRODUCT, MIN, MAX, BAND, BOR, and BXOR. Note that BAND, BOR, and BXOR reductions are not available when using the NCCL backend. Additionally, MAX, MIN and PRODUCT are not supported for complex tensors. The values of this class can be accessed as attributes, e.g., ReduceOp.SUM. They are used in specifying strategies for reduction collectives, e.g., reduce(), all_reduce_multigpu(), etc. Members: SUM PRODUCT MIN MAX BAND BOR BXOR
torch.distributed.reduce_multigpu(tensor_list, dst, op=<ReduceOp.SUM: 0>, group=None, async_op=False, dst_tensor=0) [source]
 
Reduces the tensor data on multiple GPUs across all machines. Each tensor in tensor_list should reside on a separate GPU Only the GPU of tensor_list[dst_tensor] on the process with rank dst is going to receive the final result. Only nccl backend is currently supported tensors should only be GPU tensors  Parameters 
 
tensor_list (List[Tensor]) – Input and output GPU tensors of the collective. The function operates in-place. You also need to make sure that len(tensor_list) is the same for all the distributed processes calling this function. 
dst (int) – Destination rank 
op (optional) – One of the values from torch.distributed.ReduceOp enum. Specifies an operation used for element-wise reductions. 
group (ProcessGroup, optional) – The process group to work on. If None, the default process group will be used. 
async_op (bool, optional) – Whether this op should be an async op 
dst_tensor (int, optional) – Destination tensor rank within tensor_list
   Returns 
Async work handle, if async_op is set to True. None, otherwise
class torch.distributed.reduce_op  
Deprecated enum-like class for reduction operations: SUM, PRODUCT, MIN, and MAX. ReduceOp is recommended to use instead.
torch.distributed.reduce_scatter(output, input_list, op=<ReduceOp.SUM: 0>, group=None, async_op=False) [source]
 
Reduces, then scatters a list of tensors to all processes in a group.  Parameters 
 
output (Tensor) – Output tensor. 
input_list (list[Tensor]) – List of tensors to reduce and scatter. 
group (ProcessGroup, optional) – The process group to work on. If None, the default process group will be used. 
async_op (bool, optional) – Whether this op should be an async op.   Returns 
Async work handle, if async_op is set to True. None, if not async_op or if not part of the group.