akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
There are many processes have been created on each node when I use DDP package

username: khalil
cooked: <p>Today I want to use DISTRIBUTED COMMUNICATION PACKAGE to train the imagenet, however I found there are many processes have been created on the node.</p>
<p>script.py:</p>
<pre><code class="lang-auto">def main():
    ...
    model.to(device)
    model = torch.nn.parallel.DistributedDataParallel(model, device_ids=None, output_device=None, find_unused_parameters=True)
    ...

if __name__ == '__main__':
    print('Use {back} as backend.'.format(back=args.backend))
    dist.init_process_group(backend=args.backend, init_method='env://', timeout=datetime.timedelta(seconds=1000))
    main()

</code></pre>
<p>bash.sh:</p>
<pre><code class="lang-auto">CUDA_VISIBLE_DEVICES=0,1,2,3
export GLOO_SOCKET_IFNAME=ib0
export OMP_NUM_THREADS=24
NPROC_PER_NODE=4
SLURM_JOB_NUM_NODES=4
...

COMMAND="script.py -a inception_v3 --print-freq 1000 --backend gloo --nproc-per-node 4 --pretrained  --multiprocessing-distributed  $HOME/ImageNet"

python -m torch.distributed.launch \
--nproc_per_node=$NPROC_PER_NODE \
--nnodes=$SLURM_JOB_NUM_NODES \
--node_rank=$SLURM_NODEID \
--master_addr=$MIP \
--master_port=$MPORT \
$COMMAND &gt; $HOME/thesis/PCL/log/"log_v1_inception_"${SLURM_JOB_ID}"_"${SLURM_NODEID}".out"
</code></pre>
<p>I have trained it in 4 nodes and there are 4 gpus on each node.</p>
<p>I log in the node0 then I found there are 16 processes in this node<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/e/dec13b418a19edcfb1a5796edde87a78bd8d7d18.png" alt="image" data-base62-sha1="vMzZGAZ1cKvQjkVNCAdy1O6mGYU" width="666" height="350"></p>
<p>Is it abnormal? Or DDP would create nproc_per_nodee * node_num processes on each node?</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/khalil">@khalil</a>, sorry about the delay. Looking at the screenshot, they are actually the same set of processes, but each process used all GPUs.  Any reason for setting <code>device_ids</code> to <code>None</code>, instead of <code>device_ids=[local_rank]</code>? If it is None, by default, DDP will use all visible CUDA devices.</p>
Efficiently Generate Sequence from Transformer Decoder

username: peterpane
cooked: <p>I’m currently using a Transformer decoder as an autoregressive model to generate a sequence. (So each element within a sequence depends on all the previously generated elements)</p>
<p>Now if I want to generate a sequence, I have to generate each element one by one in sequence. When I use the forward pass of the transformer decoder, the embeddings for all the previous elements in the sequence are always recomputed, but I actually only need to compute the very last element to attach to the sequence (since the previous elements are unchanged).</p>
<p>Is there any way to do this more efficiently in PyTorch using the MultiheadAttention/TransformerDecoder module? (Let me know if my question is unclear)</p>
Make parameter gradient a function of another parameter

username: afps
cooked: <p>How can I make the gradient of a parameter a function itself?</p>
<pre><code class="lang-auto">import torch

def fun(q):
    
    def result(w):
        l = w * q
        l.backward()
        return w.grad
    
    return result

w = torch.tensor((2.), requires_grad=True)
q = torch.tensor((3.), requires_grad=True)

f = fun(q)

print(f(w))
</code></pre>
<p>In the code above, how can I make f(w) to have gradient with respect to q?</p>

username: albanD
cooked: <p>Hi,</p>
<p>If you use <code>f(w).sum().backward()</code> then it will actually populate the <code>.grad</code> field for both w and q.<br>
Or you can ask for it as <code>autograd.grad(f(w).sum(), q)</code>.</p>
Change h finite diff step size for grad calc

username: shaunster
cooked: <p>I’m using Pytorch on data which is much noisier than normal image data, and I want to change the step size (h or eps) which autograd uses in the finite difference calc:<br>
grad = f(x + h) - f(x) / h</p>
<p>I read somewhere the default eps or h = 1e-6, I want to use 1e-3 or -4.<br>
I think I could override backwards() in my layer (which is a convolution which subclasses torch.nn.Conv1d) but defining my own backwards() seems overkill when I only want to change a step size … then backwards() would be a bit strange, it would define a finite diff calc when generally I believe backwards() is used to replace the finite diff calc (eg. by providing an analytical function).<br>
Note: I’m using a small CNN with SGD or Adam.</p>
<p>whats the best way to do this?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Pytorch does not use finite difference during the backward pass but automatic differentiation (a chain of analytical functions).<br>
Only the gradcheck tool we use to test our backward pass actually does finite difference. But this is only used for testing purposes.</p>
OOMKilled with exit code 137 with pytorch xla

username: Rabeeh_Karimi
cooked: <p>Hi<br>
I am running a small model with small batch size on TPU with 8 cores, and when it calls</p>
<p><code> xmp.spawn(mod._mp_fn, args=(), nprocs=args.num_cores)</code></p>
<p>It always goes to out of memory on google cloud, and I am not sure where is the issue, I tried to reduce the batch size as much as possible still the error is there. thanks</p>

username: agolynski
cooked: <p>Hi,<br>
Could you provide more details on your setup as it is hard to reproduce it ASIS. CC: <a class="mention" href="/u/ailzhang">@ailzhang</a> on XLA.</p>
Appearance-based hashing for similarity detection for picking the 100 most distinct images out 500 images

username: Mona_Jalal
cooked: <p>I would like to perform appearance-based hashing for similarity detection. I have 500 photos for each of my categories but I only want to maintain the 100 of them that are most distinct. How should I go about this? Are there already well-known baselines for this?</p>
<p>Also, what are the other known methods for this task? I was thinking I could also run the resnet50 on images, extract a 2048 dimension vector and do a cosine similarity but I wonder if there are deep neural network that can do the task I have in mind end-to-end or be more efficient than what I have in mind.</p>