akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Exporting pytorch models for use in Ruby

username: Parkz
cooked: <p>I made a LSTM model in a python notebook. For production, I’d like to save and export the model for use in Ruby. I have no familiarity with Ruby whatsoever. Are there any guides out there for this sort of thing?</p>
<p>What format should I save the model as? What libraries are needed to accomplish this?</p>

username: pascal_notsawo
cooked: <p>I have never coded in Rudy (I have no knowledge about it). And I don’t think Rudy is part of the environment (I prefer this word to language) where we can manipulate pytorch/tensorflow models (js, C++, …)</p>
<p>In this kind of case you can deploy your model in a python (or compatible) environment, for example a Django/Flask (torch) or nodejs (torch-js) API… And query it from Rudy</p>
How to do DataParallel when input is a list

username: Boltzmachine
cooked: <p>I have an input, which is a list of tensors. the element of list is edge list of a graph. since the number of edge is different. I cannot convert it into a tensor. So how to do DataParallel so I can split the list.</p>

username: huahuanZ
cooked: <blockquote>
<p>Arbitrary positional and keyword inputs are allowed to be passed into DataParallel EXCEPT Tensors. All tensors will be scattered on dim specified (default 0). Primitive types will be broadcasted, but all other types will be a shallow copy and can be corrupted if written to in the model’s forward pass.</p>
</blockquote>
How to calculate train accuracy with DDP

username: 111519
cooked: <p>I have read some tutorials about Distributed data parallel, however, I didn’t find out how to calculate train loss and accuracy after training one epoch correctly.</p>
<p>With DataParallel, we can easily calculate loss and accuracy since there is only one process. But with DDP, every gpu is running its own process and training its own data. The problem is,</p>
<ol>
<li>How to evaluate the training accuracy correctly?</li>
<li>I follow the example here. <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" rel="noopener nofollow ugc">ImageNet Example</a><br>
Does the code redundantly calculate the same test accuray across multiple gpus? If so, is there any way to sample the testloader just like trainloader and avoid repetitive computing?</li>
</ol>

username: 111519
cooked: <p>Finally find out. It seems that it can not be easily done. We need to send message from other process and gather information together.</p>

username: sio277
cooked: <aside class="quote no-group" data-username="111519" data-post="2" data-topic="119768">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/111519/48/37718_2.png" class="avatar"> 111519:</div>
<blockquote>
<p>We need to send message from other process and gather information together.</p>
</blockquote>
</aside>
<p>Yes. I use all-reduce function something like this:</p>
<pre><code class="lang-auto">import torch
import torch.distributed as dist


def global_meters_all_avg(args, *meters):
    """meters: scalar values of loss/accuracy calculated in each rank"""
    tensors = [torch.tensor(meter, device=args.gpu, dtype=torch.float32) for meter in meters]
    for tensor in tensors:
        # each item of `tensors` is all-reduced starting from index 0 (in-place)
        dist.all_reduce(tensor)

    return [(tensor / args.world_size).item() for tensor in tensors]
</code></pre>

username: huahuanZ
cooked: <p>For your questions:</p>
<ol>
<li>Use <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.all_reduce" rel="noopener nofollow ugc">all reduce method</a> to communicate across processes;</li>
<li>Yes. And if you want to distributedly conducting evaluation, just follow how the example deal with training data. e.g. create a test_sampler for distribute data into GPUs.</li>
</ol>