akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Dataloading for text/ audios/ videos/ procedurally generated streaming

Etienne_Perot: <p>is there a way with iterable dataset or dataloader to stream multiple videos in a temporally coherent manner?</p>
<p>I have this piece of code that seems to work but it is completely custom so i guess it has many caveats.</p>
<pre><code class="lang-auto">class MultiStreamer(object):
"""
Multithreaded Streaming for Temporally Coherent Batches 

uses the multiprocessing package
expects the "data" in tensor form with array_dim shape per thread.
"""
def __init__(self, make_env, array_dim, batchsize, max_q_size, num_threads, max_iter=int(1e6)):
    self.readyQs = [mp.Queue(maxsize=max_q_size) for i in range(num_threads)]
    self.array_dim = array_dim
    self.num_threads = num_threads
    self.num_videos_per_thread = batchsize // num_threads
    self.max_q_size = max_q_size
    self.batchsize = batchsize
    self.make_env = make_env
    self.batch = np.zeros((self.num_threads, self.num_videos_per_thread,
                           *array_dim), dtype=np.float32) 

    array_dim2 = (self.max_q_size, self.num_videos_per_thread,
                  *array_dim)

    self.m_arrays = (mp.Array('f', int(np.prod(array_dim2)), lock=mp.Lock()) for _ in range(num_threads))
    self.arrays = [(m, np.frombuffer(m.get_obj(), dtype='f').reshape(array_dim2)) for m in self.m_arrays]
    self.max_iter = max_iter

def multi_frame_stream(self, i, m, n, shape):
    group = self.make_env(num=self.num_videos_per_thread)
    j = 0
    while 1:
        m.acquire()
        info = group.next(n[j])
        self.readyQs[i].put((j, info))
        j = (j+1)%self.max_q_size

def __iter__(self):
    procs = [mp.Process(target=self.multi_frame_stream, args=(i, m, n, self.array_dim), daemon=True) for i, (m, n) in
             enumerate(self.arrays)]
    [p.start() for p in procs]
    _utils.signal_handling._set_worker_pids(id(self), tuple(w.pid for w in procs))
    _utils.signal_handling._set_SIGCHLD_handler()
    print('Start Streaming')
    for i in range(self.max_iter):
        start = time.time()
        batch = defaultdict(list)
        for n in range(self.num_threads):
            j, infos = self.readyQs[n].get() 
            m, arr = self.arrays[n]
            self.batch[n] = arr[j]
            for k, v in infos.items():
                batch[k] += v
            m.release()
        batch['data'] = self.batch.reshape(self.batchsize, *array_dim)
        yield batch
    [p.terminate() for p in procs]
</code></pre>
How to add neurons of 1 manually to each layer of the network without messing up dimension matching input and output of a layer

gebrahimi: <p>Hi,</p>
<p>For some reasons I need to remove the bias in torch.nn.Linear and add it manually myself. I did something like this:</p>
<p>class LeNet_300_100(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()</p>
<pre><code>    self.fc1 = nn.Linear(785, 300, bias = False) 
    self.relu1 = nn.ReLU()
    self.fc2 = nn.Linear(301, 100, bias = False) 
    self.relu2 = nn.ReLU()
    self.fc3 = nn.Linear(101, 10, bias = False) 

def forward(self, x):
    batch_size = x.size()[0]
    x = torch.cat((x.view(-1, 784), torch.ones(batch_size,1)), 1) #batch_size*785
    x = self.fc1(x)
    x = self.relu1(x)
    x = torch.cat((x, torch.ones(batch_size,1)), 1)
    x = self.fc2(x)
    x = self.relu2(x)
    x = torch.cat((x, torch.ones(batch_size,1)), 1)
    x = self.fc3(x)
    return F.log_softmax(x, dim=1)
</code></pre>
<p>However the problem is mismatch between input and output of layers. For instance layer one has [batch_size *300] input, but [batch_size *301] output. How can I avoid this. One possible way is adding a tensor of 0 weights to the first layer and freeze them to zeros. Any idea how to do that? or any other reasonable solution?</p>
<p>Thanks.</p>
Fit gradient of the network using a function of inputs

Mrad: <p>Hi all,<br>
I want to train a network which first of all instead of the inputs itself I want to pass a function of inputs to the network and also I want to fit the gradient of the network. I have written a code for that but the code doesn’t work. Here is a simple code for what I want to do</p>
<pre><code class="lang-python">import torch
import torch.nn as nn


x_data = torch.rand((20, 3))
y_data = torch.rand(20)
f_data = torch.rand((20,3))

def dist1(x, data):
    return (torch.norm(data - x, dim=-1))

net = torch.nn.Sequential(
        torch.nn.Linear(20,25),
        torch.nn.Sigmoid(),
        torch.nn.Linear(25,15),
        torch.nn.Sigmoid(),
        torch.nn.Linear(15,1))

optimizer = torch.optim.Adam(net.parameters(), lr= 0.1)

for i in range(10):
    n=torch.randint(0,20, (1,))
    xi = x_data[n]
    yi = y_data[n]
    xi.requires_grad_()
    env = dist1(xi, x_data)
    
    
    energy = net(env)
    energy.backward(create_graph=True)
    force = xi.grad
    
    loss = torch.mean((force - f_data[n])**2)
    loss.backward()
    optimizer.step()
    
    print('=============================')
    for name, param in net.named_parameters():
        print('name: ', name)
        print('param: ', param)
    print('=============================')
</code></pre>

albanD: <p>Hi,</p>
<p>I think you’re missing a <code>optimizer.zero_grad()</code> before the last <code>.backward()</code> that computes your gradient step.<br>
Also a slight optimization to avoid computing all the gradients in the network in the when computing the force: <code>force, = autograd.grad(energy, xi, create_graph=True)</code>.</p>

Mrad: <p>I tried both of your suggestions but still I have “nan” values for my parameters.</p>

albanD: <p>You can add <code>torch.autograd.set_detect_anomaly(True)</code> at the beginning of your script to know where nan appear in the backward pass.<br>
In your particular case, the problem is that in your <code>dist1</code> function, you take the second derivative of the <code>norm</code> at <code>0</code>. Since this derivative includes a division by the norm value, you get <code>nan</code>s.<br>
You will need to make sure that <code>data - x != 0</code> before taking the norm.</p>
RuntimeException: empty not implemented for TensorTypeSet

Chame_call: <p>I’ve exported Pytorch model for android and trying to load Module I’ve got the following error:</p>
<blockquote>
<pre><code>java.lang.RuntimeException: Unable to start activity ComponentInfo{org.pytorch.demo/org.pytorch.demo.vision.FrameProcessingActivity}: com.facebook.jni.CppException: empty not implemented for TensorTypeSet(VariableTensorId, CUDATensorId) (empty at aten/src/ATen/Functions.h:3679)
(no backtrace available)
</code></pre>
</blockquote>
<p>whole stack trace is:</p>
<blockquote>
<p>E/AndroidRuntime: FATAL EXCEPTION: main<br>
Process: org.pytorch.demo, PID: 19189<br>
java.lang.RuntimeException: Unable to start activity ComponentInfo{org.pytorch.demo/org.pytorch.demo.vision.FrameProcessingActivity}: com.facebook.jni.CppException: empty not implemented for TensorTypeSet(VariableTensorId, CUDATensorId) (empty at aten/src/ATen/Functions.h:3679)<br>
(no backtrace available)<br>
at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2379)<br>
at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2442)<br>
at android.app.ActivityThread.access$800(ActivityThread.java:156)<br>
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1351)<br>
at android.os.Handler.dispatchMessage(Handler.java:102)<br>
at android.os.Looper.loop(Looper.java:211)<br>
at android.app.ActivityThread.main(ActivityThread.java:5389)<br>
at java.lang.reflect.Method.invoke(Native Method)<br>
at java.lang.reflect.Method.invoke(Method.java:372)<br>
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1020)<br>
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:815)<br>
Caused by: com.facebook.jni.CppException: empty not implemented for TensorTypeSet(VariableTensorId, CUDATensorId) (empty at aten/src/ATen/Functions.h:3679)<br>
(no backtrace available)<br>
at org.pytorch.NativePeer.initHybrid(Native Method)<br>
at org.pytorch.NativePeer.(NativePeer.java:18)<br>
at org.pytorch.Module.load(Module.java:23)<br>
at org.pytorch.demo.models.BeautyDefiner.(BeautyDefiner.java:23)<br>
at org.pytorch.demo.vision.FrameProcessingActivity.onCreate(FrameProcessingActivity.java:83)<br>
at android.app.Activity.performCreate(Activity.java:5990)<br>
at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1106)<br>
at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2332)<br>
… 10 more</p>
</blockquote>
<p>How to solve that error?</p>

Chame_call: <p>The problem was that I ran the model on the gpu before exporting with jit.<br>
So I move the model to cpu and exported the model again.<br>
And it worked.</p>
Best activation or input normalization for autoencoder denoiser?

Jean-Eric_Campagne: <p>Looking at several denoiser autoencoder, one trains the network with images to which one adds noise, these images serve as input to the network while the original images are the target. Well, to prepare the data set, usually the images are normalized to [0,1], and the noisy images are cliped (clamped) to [0,1] also. And this normalization is done in conjunction of the fact that the last layer of the network is using ‘torch.sigmoid’.</p>
<p>So far so good, but the sigmoid(x) function reaches 0, for large negative value of x, and symetricaly, the function reaches  1 for large positive value of x. In a certain sense, reaching 0,1 demand +/- infinity values.</p>
<p>So, my question, does the normalization [0,1] suited to sigmoid(last layer) and vice versa if one uses sigmoid does the [0,1] is the best normalisation (eg. 0.02, 0.98 may be better ?) ?</p>
<p>It is up to you to comment…<br>
Best.</p>
Visualizing attention heads in Graph Attention Network

arijitthegame: <p>Hi everyone,</p>
<p>I used the DGL to build a Graph Attention Network and I am play with adding some edge features. But I am interested in doing some of the visualizations in the original GAT paper and also in <a href="https://docs.dgl.ai/en/latest/tutorials/models/1_gnn/9_gat.html?highlight=visualization" rel="nofollow noopener">https://docs.dgl.ai/en/latest/tutorials/models/1_gnn/9_gat.html?highlight=visualization</a></p>
<p>Is there some tool or repo that I use to visualize various attention heads at various layers?</p>
<p>Thank you for your responses.</p>
ByteTensor to FloatTensor is slow?

jingweiz: <p>Hey,<br>
So I when I try to convert the resulting ByteTensor out of <a href="http://torch.ge" rel="nofollow noopener">torch.ge</a>() to FloatTensor by:<br>
my_tensor.type(torch.FloatTensor)<br>
it took around 2.5e-4<br>
while when I check the converting of other types to FloatTensor they all took on the magnitude of e-5, why is this converting from Byte to Float relatively slow? Are there any other ways to do this?<br>
Thanks a lot!</p>

smth: <p>right now there’s no faster way than just typecasting (as you did).</p>

smth: <p>With the latest 0.4 release, you can use the <code>.to</code> function along with <code>device</code> objects to make this much cleaner.</p>
<pre><code class="lang-auto"># on top of your script somewhere
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

# in your code, use .to(device)
(x_u == 0).to(device, dtype=torch.float32)
</code></pre>
How to convert cv::Mat to torch::Tensor?

dalalaa: <p>The value of matrix in my code changed after converting from cv::Mat to torch::Tensor:</p>
<pre><code class="lang-auto">torch::Tensor get_rotate_matrix(double cx, double cy, double angle, double scale)
{
    cv::Point2f center = cv::Point2f((float)cx, (float)cy);
    cv::Mat matrix = cv::getRotationMatrix2D(center, angle, scale);

    std::cout &lt;&lt; "matrix \n"
              &lt;&lt; matrix &lt;&lt; std::endl;

    torch::Tensor result = torch::from_blob(matrix.ptr&lt;float&gt;(), {2, 3});
    std::cout &lt;&lt; "result" &lt;&lt; std::endl;
    for (int i = 0; i &lt; 2; i++)
    {
        for (int j = 0; j &lt; 3; j++)
        {
            std::cout &lt;&lt; result[i][j].item().toFloat() &lt;&lt; "\t";
        }
        std::cout &lt;&lt; std::endl;
    }
    return result.clone();
}
</code></pre>
<p>I got the following output:</p>
<pre><code class="lang-auto">
matrix
[0.898794046299167, 0.4383711467890774, -7.755363398773218;
  -0.4383711467890774, 0.898794046299167, 6.407830541907433]
result
-0.00326877     1.8497  -2.67433e+16
1.71919 -1.59202e+36    -2.48471
</code></pre>
<p>How can I convert cv::Mat to torch::Tensor correctly?</p>
<p>Looking forward to your replay.</p>

21fa417e3fb06e56040c: <p>you try this</p>
<pre><code class="lang-auto">torch::tensor(at::ArrayRef&lt;float&gt;(img.data, h* w)).view({ h, w });
</code></pre>
Dataloader error after using torch.utils.data.ConcatDataset

Deeply: <p>I am using <code>torch.utils.data.ConcatDataset</code> to merge 7 different datasets and I am getting this error in <code>dataloader.py</code> file.</p>
<p>The error happens at this line:</p>
<p><code>for imgs, target, word_str, weight in test_loader:</code></p>
<p>…</p>
<pre><code class="lang-auto">def default_collate(batch):
...
...
    numel = sum([x.numel() for x in batch])
    AttributeError: 'numpy.ndarray' object has no attribute 'numel'
</code></pre>
<p>Clearly, the code expects a <code>tensor</code> format. All my images have <code>tensor</code> format, but the multihot encoding are in numpy. However, when I try the same code it works well when I merge 6 datasets. What is happening?</p>
<p>When I try the seventh data set alone, it also works well; sampler is not used here.</p>
<p>Any ideas what is happening?</p>
<p>NB. The test_loader is based on a <code>Sampler</code></p>
<pre><code class="lang-auto">def get_sampled_loader(cf, test_set):
        no_of_samples  = len(test_set)
        sample_idx = np.random.permutation(np.arange(0, no_of_samples))[:cf.no_of_sampled_data]                     
        if len(sample_idx) ==0:  
            exit('exiting function get_the_sampler(), sample_idx size is 0')    
        my_sampler = torch.utils.data.sampler.SubsetRandomSampler(sample_idx)  
        test_loader = torch.utils.data.DataLoader(test_set, batch_size=cf.batch_size_test,
                          shuffle= False, num_workers=cf.num_workers, sampler=my_sampler)
        return test_loader
</code></pre>
<p><strong>Update</strong>: Removing the sampler did not resolve  the  error.</p>

Deeply: <p>The targets for the six dataset seem to be defined as <code>tensor</code>, while the seventh dataset is <code>numpy</code>. The problem was resolved after converting the target of the seventh dataset to the <code>tensor</code>.</p>
Compute gradient of bitwise >

xianshunw: <p>I need to compare two tensors element-wise and get a binary mask. e.g.</p>
<pre><code class="lang-auto">c = a &gt; b
c = c.float()
</code></pre>
<p>However, this operation is not differentiable. Does someone know how I can calculate the element-wise<code>&gt;</code> and still be able to backprop?</p>

xianshunw: <p><code>torch.sign()</code> works for me</p>

KFrank: <p>Hi Xianshun!</p>
<aside class="quote no-group quote-modified" data-username="xianshunw" data-post="1" data-topic="63872" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/xianshunw/48/18700_2.png" class="avatar"> xianshunw:</div>
<blockquote>
<p>However, this operation is not differentiable. Does someone know how I can calculate the bitwise <code>&gt;</code> and still be able to backprop?</p>
</blockquote>
</aside>
<p>The greater-than operator (<code>&gt;</code>) is basically a step function.<br>
A common differentiable approximation to a step function<br>
is a sigmoid.  (There are others.)</p>
<p>You could try:</p>
<pre data-code-wrap="python"><code class="lang-python">c = nn.sigmoid (a - b)
</code></pre>
<p>You can sharpen (or smooth out) the sigmoid with a parameter:</p>
<pre data-code-wrap="python"><code class="lang-python">c = nn.sigmoid (s * (a - b))
</code></pre>
<p>(By the way, are you sure you mean <em>bitwise</em> <code>&gt;</code>?)</p>
<p>Good luck.</p>
<p>K. Frank</p>

xianshunw: <p>sorry for the mistake. It should be <strong>element-wise</strong>. Thanks for your help.</p>
AttributeError: module 'torchaudio.transforms' has no attribute 'MelScale'

ThisuriCham: <p>I used this line in my colab. for all torchaudio transformations same Attribute error is showing.</p>
<p>link to my colab -&gt; <a href="https://github.com/ThisuriCham/Pytorch/blob/master/audio_classification.ipynb" rel="nofollow noopener">https://github.com/ThisuriCham/Pytorch/blob/master/audio_classification.ipynb</a></p>
Fill a 3D tensor in an arbitrary dimension given 2D indices

JuanFMontesinos: <p>Is there any faster way to using advanced indexing?</p>
<pre><code class="lang-auto">import torch

sp = torch.rand(3, 4, 4)
indices = sp.max(dim=0)[1]
gt = torch.zeros_like(sp)

for idx, sp_i in enumerate(gt):
    sp_i.masked_fill_(indices == idx, 1.)
</code></pre>
Correctly profiling the whole loading-training-writing pipeline

jj0mst: <p>I am training my networks on two different machines.</p>
<p>They have pretty much the same GPU: the first one uses tesla v100 16GB with nvlink, the second uses tesla v100 32GB with pcie. The main differences stand in CPUs and storage, but I can see that the same script is much slower on the second machine (2x or even more sometimes).</p>
<p>I would like to understand where is the bottleneck in the second machine. In particular I would like to find precisely how much time these operations require: data loading, CPU-&gt;GPU, gpu computation as a whole (forward+backward+update), GPU-&gt;CPU, data writing (e.g. dumping the output images).</p>
<p>Can I rely on simply measuring the time between the calls in my script?</p>
<p>I have already tried this method and it seems that the GPU computations on the second machine are much slower, but this makes no sense to me since the specs say they should achieve almost the same performance.</p>
<p>Is it possible that the asynchronism caused by CUDA streams is giving me fake results?</p>
Libtorch how to generate random within 0-1?

jinfagang: <p>Say this one:</p>
<pre><code class="lang-auto">    at::Tensor scores = torch::randn({34}, torch::kCUDA);

</code></pre>
<p>I want score actually between 0-1. how to specific this?</p>
<p>it generates -1-1 now.</p>

21fa417e3fb06e56040c: <p>Try this.<br>
Not recommended…</p>
<pre><code class="lang-auto">at::Tensor scores = torch::randn({ 34 }).clamp_max_(1).clamp_min_(0);
</code></pre>

Haydnspass: <p>Note that randn draws from a unit normal (Gaussian) distribution! It will also not be between -1,1 but just be in ~70% of all cases in this range.</p>
<p>torch::rand or torch.rand (without the trailing n) is for uniform distributed random numbers between 0…1</p>

LeviViana: <p>This generates i.i.d. uniform random numbers between 0 and 1.</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;iostream&gt;

int main(){
        torch::Tensor tensor = torch::rand({10});
        std::cout&lt;&lt; "Tensor " &lt;&lt; tensor &lt;&lt; std::endl;
        return 0;
}

</code></pre>
Screen becomes laggy when using GPU

Beinan_Wang: <p>This is not a PyTorch specific problem but I don’t know where to ask this question. I run PyTorch on Ubuntu because it is the one of the most popular platform for ML. However, one problem has been bothering me. When I train neural networks using GPU (which uses CUDA), my screen responds very slowly, such as scrolling up and down in a web browser. This is not the case if I use Ubuntu on Wayland instead of Xorg but I can’t find the option to use Wayland for Ubuntu 19,10. The cog at logging in is not there. I can’t be the only person having this problem right? I wonder how other people deal with this problem.</p>

albanD: <p>Unfortunately, that happens if your rendering uses the GPU and you do GPU compute at the same time.<br>
What I used to do on my local machine is to get an old and cheap GPU that I use only for rendering the screen: attache your screen to it and hide it with <code>CUDA_VISIBLE_DEVICES=1</code> when doing pytorch job.<br>
That way the two will be independent.</p>

Beinan_Wang: <p>Makes sense. But why Windows does not have this issue? If I run PyTorch on Windows, rendering is very smooth but I think PyTorch on Windows uses more memory?</p>

albanD: <p>It’s not a problem of memory but compute on the GPU. If the rendering does not use the GPU, then it will run smoothly. That is very OS dependent I’m afraid.</p>

alex.veuthey: <p>If you have that possibility, you can also use the motherboard’s direct output for rendering, which then uses more CPU, but doesn’t require having an additional GPU, if available space is limited for example… As always, it’s a tradeoff…</p>
<p>You can then block Xorg from using the GPU and free a bit of GPU memory this way (I think this was what I followed but can’t remember for sure: <a href="https://gist.github.com/alexlee-gk/76a409f62a53883971a18a11af93241b" rel="nofollow noopener">https://gist.github.com/alexlee-gk/76a409f62a53883971a18a11af93241b</a>).</p>
Training continuously on real-time data

akshayb7: <p>Hi everyone.<br>
I have to create a model which is to be trained online (after a batch of data is collected) and get the loss error for that batch of training.</p>
<p>I was saving the model after each batch of training and outputting the loss error for that batch, before reloading the model again for next batch of training. This seems very inefficient to me and I think there should be a better method to do so.</p>
<p>Anyone has any pointers for me?</p>
<p>Thanks in advance!</p>

akshayb7: <p>I think I should clarify a little more. What I’m after is an autoencoder for anomaly detection. Where the recreation error (loss function error, mse, mae, etc) between the input and output (which are same) is an indicator of presence of anomaly. Now this system is to be deployed online in such a way that it is to be processed after every batch, but this requires constant training for the algorithm in deployment itself. It’s here where I’m not sure how to do this. I’ve been able to do this for a csv based file but the online scenario is where I can’t figure out the process.</p>
<p>My theory was to save the model back to a file after each training batch, and then reload the model before the next training batch session and then train it on the new batch (repeat the process). To get the error I either get the loss from the model itself (pytorch) or use a custom history callback (in keras). But this process of training over batches and reloading model each time seems redundant and inefficient.</p>
<p>Would be very thankful if you have any ideas on how to tackle such a problem. I prefer Pytorch because this model is so much faster in Pytorch than in Keras.</p>

akshayb7: <p>I’m aware of the <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset" rel="nofollow noopener">IterableDataset</a> but am unsure how to use this in this scenario.</p>
<p>Please help.</p>
Question about the behavior of torch.utils.data.distributed.DistributedSampler

coincheung: <p>Hi,</p>
<p>I am currently playing around distributed training, and I have some trouble trying to understand it:</p>
<p>When using <code>torch.utils.data.distributed.DistributedSampler</code>, does it mean to split the whole dataset <code>D</code> to <code>D1</code> and <code>D2</code>(considering the one node two gpus scenario), and each process only iterates data from one of <code>D1</code> or <code>D2</code>, thus there will never be a batch generated from a mixed of <code>D1</code> data and <code>D2</code> data? Or will the split operation be committed automatically each time one epoch is done and another epoch is started?</p>

Junwu_Weng: <p>I have a similar question here. If this is the way the DistributedSampler sample data, and if the portion of the data is quite smal on each single GPU, the network on each single GPU may overfit trained on the small portion of the dataset.</p>
<p>Will the portion of the dataset on each single GPU update after each epoch?</p>

viperit: <p>Yes, l also have a question about it. Is there someone have any insight for this?</p>

viperit: <p>Hi, I have found that your question can be solved by <code>sampler.set_epoch(e)</code>. Just try my code, each gpu will get shuffled and different data in a different epoch.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
from torch.utils.data.distributed import DistributedSampler


torch.distributed.init_process_group(backend="nccl")

input_size = 5
output_size = 2
batch_size = 2
data_size = 16

local_rank = torch.distributed.get_rank()
torch.cuda.set_device(local_rank)
device = torch.device("cuda", local_rank)

class RandomDataset(Dataset):
    def __init__(self, size, length, local_rank):
        self.len = length
        self.data = torch.stack([torch.ones(5), torch.ones(5)*2,
                                 torch.ones(5)*3,torch.ones(5)*4,
                                 torch.ones(5)*5,torch.ones(5)*6,
                                 torch.ones(5)*7,torch.ones(5)*8,
                                 torch.ones(5)*9, torch.ones(5)*10,
                                 torch.ones(5)*11,torch.ones(5)*12,
                                 torch.ones(5)*13,torch.ones(5)*14,
                                 torch.ones(5)*15,torch.ones(5)*16]).to('cuda')

        self.local_rank = local_rank
    def __getitem__(self, index):

        return self.data[index]

    def __len__(self):
        return self.len
    
dataset = RandomDataset(input_size, data_size, local_rank)
sampler = DistributedSampler(dataset)
rand_loader = DataLoader(dataset=dataset,
                         batch_size=batch_size,
                         sampler=sampler)

e = 0
while e &lt; 2:
    t = 0
    sampler.set_epoch(e)
    for data in rand_loader:
        print(data)
    e+=1

</code></pre>
DistributedSampler: the result of every epoch and even every time I run it is same

viperit: <p>Hi, I have a bug when I test a simple code of <strong>DistributedDataParallel</strong>.</p>
<p>In summary, I just want to figure out whether the subset dataset split by <strong>DistributedSampler</strong> can be shuffle as it should be. However, the result of every epoch and even every time I run it is the same.</p>
<p>Is someone have any insight into this?</p>
<p>The code is shown below, pytorch=1.1.0 or 1.3.1:</p>
<p>run</p>
<pre><code class="lang-auto">CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node=2 test.py
</code></pre>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
from torch.utils.data.distributed import DistributedSampler


torch.distributed.init_process_group(backend="nccl")

input_size = 5
output_size = 2
batch_size = 2
data_size = 16

local_rank = torch.distributed.get_rank()
torch.cuda.set_device(local_rank)
device = torch.device("cuda", local_rank)

class RandomDataset(Dataset):
    def __init__(self, size, length, local_rank):
        self.len = length
        self.data = torch.stack([torch.ones(5), torch.ones(5)*2,
                                 torch.ones(5)*3,torch.ones(5)*4,
                                 torch.ones(5)*5,torch.ones(5)*6,
                                 torch.ones(5)*7,torch.ones(5)*8,
                                 torch.ones(5)*9, torch.ones(5)*10,
                                 torch.ones(5)*11,torch.ones(5)*12,
                                 torch.ones(5)*13,torch.ones(5)*14,
                                 torch.ones(5)*15,torch.ones(5)*16]).to('cuda')
        
        self.local_rank = local_rank
    def __getitem__(self, index):
        
        return self.data[index]
    
    def __len__(self):
        return self.len
    

dataset = RandomDataset(input_size, data_size, local_rank)
sampler = DistributedSampler(dataset)
rand_loader = DataLoader(dataset=dataset,
                         batch_size=batch_size,
                         sampler=sampler)
    
class Model(nn.Module):
    def __init__(self, input_size, output_size):
        super(Model, self).__init__()
        self.fc = nn.Linear(input_size, output_size)

    def forward(self, input):
        output = self.fc(input)
        return output
    
model = Model(input_size, output_size)

model.to(device)
if torch.cuda.device_count() &gt; 1:
    model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[local_rank], output_device=local_rank)   

e = 0
while e &lt; 2:
    t = 0
    for data in rand_loader:
        if torch.cuda.is_available():
            input_var = data
        else:
            input_var = data
        _ = model(input_var)
        print(input_var)
    e+=1

</code></pre>

viperit: <p>I have solved this. Just add <code>sampler.set_epoch(e)</code> after  <code>while e &lt; 2:</code>, just like:</p>
<pre><code class="lang-auto">while e &lt; 2:
    t = 0
    sampler.set_epoch(e)
    for data in rand_loader:
        print(data)
    e+=1
</code></pre>
Iterator expression is expected to be a list, iterable, or range, found value of type 'Tensor'

dalalaa: <p>I met this error when converting my model to torchscript.</p>
<pre><code class="lang-auto">RuntimeError: 
iterator expression is expected to be a list, iterable, or range, found value of type 'Tensor':
at /home/dai/scripts/card_ocr_cpu/detector/model_torchscript.py:39:28
def standard_nms(S, thres):
    
    order = torch.argsort(S[:, 8]).flip(0)
    keep:List[int] = []
    while order.size(0) &gt; 0:
        i = order[0]
        keep.append(i.long().item())
        ovr = torch.tensor([intersection(S[i], S[t]) for t in order[1:]])
                            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
        inds = torch.where(ovr &lt;= thres)[0]
        order = order[inds + 1]
    return S[keep]
'standard_nms' is being compiled since it was called from 'nms_locality'
at /home/dai/scripts/card_ocr_cpu/detector/model_torchscript.py:63:11
            p = weighted_merge(g, p)
        else:  
            if p.size(0) &gt; 1:
                S.append(p)
            p = g
    if p is not None:
        S.append(p)
    if len(S) == 0:
        return torch.tensor([0])
    return standard_nms(torch.stack(S), thres)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
'nms_locality' is being compiled since it was called from 'get_boxes_torch'
at /home/dai/scripts/card_ocr_cpu/detector/model_torchscript.py:247:4
        return None
    boxes = torch.zeros((polys_restored.shape[0],9)).float()
    boxes[:,:8] = polys_restored
    xs = xy_text[index,0]
    ys = xy_text[index,1]
    boxes[:,8] = score[xs,ys]
    
    boxes = nms_locality(boxes,nms_thresh)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
    return boxes
'get_boxes_torch' is being compiled since it was called from '__torch__.EAST.forward'
at /home/dai/scripts/card_ocr_cpu/detector/model_torchscript.py:216:12
    def forward(self, x, train:bool=False):
        x1,x2,x3,x4 = self.extractor(x)
        x = self.merge(x1,x2,x3,x4)

        # del x1,x2,x3,x4
        # collect()

        score,geo = self.output(x)
        if not train:
            boxes = get_boxes_torch(score,geo,score_thresh=0.95,nms_thresh=0.2)
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE

            return boxes
        return score,geo
</code></pre>
<p>How can I  iterate through the tensor like a list. Looking forward for your replay.</p>

wanchaol: <p>This should already get supported. it might not be in v1.3 yet, can you try update your pytorch version to nightly build to see if it works?</p>
How to normalize sequence input data

gilad: <p>Hi</p>
<p>What is the recommended way to normalize sequence input data (3d tensors).<br>
Should I use batchnorm layer? create my own transform on Dataloader?<br>
Is there any example of normalizing sequence data which is not word embedding?</p>
<p>Thanks!<br>
Gilad</p>

smth: <p>What kind of normalization are you thinking of applying?</p>
<p>You can compute the torch.mean / torch.std over the sequence and center it by subtracting / dividing…</p>

Royi: <p>That makes me wonder.<br>
Let’s say the input sample is $ x $.<br>
So we feed the net with $ \frac{x - m}{v} = \frac{x}{v} - \frac{m}{v} $.<br>
Now, if the first layer of the net is Linear Layer we have the Bias term (Assuming it is turned on).<br>
If there is need to subtract a value from all samples, like $ \frac{m}{v} $, I’d assume it will be learned.</p>
<p>So I see the point in dividing, but what’s the point in centering?</p>

smth: <p>yes the centering “might” be learnt by the bias term, if all stars align (and your biases are closely initialized). Giving a centering prior based on dataset statistics just helps. Scaling can also technically be learnt by the convolution operation (the weights can learn to scale up or down), but whether the weights can learn good scaling depends on initialization, how activation dynamic range changes over the network depth etc.</p>

Royi: <p>Well, I’m not experience in Deep Learning.<br>
The question is if the Bias term is indeed (For centered data sets) approaches zero.<br>
I will check on MNIST just for self knowledge.</p>

billtubbs: <p>I have the same question.  Can’t understand why there aren’t more examples of normalizing the inputs (and outputs potentially).  Looking at <code>torchvision.transforms.Normalize</code> it says it is for normalizing “a tensor image with mean and standard deviation” which I don’t think is the same as what we’re talking about here.</p>
<p>In Scikit-Learn you simply add a <code>sklearn.preprocessing.StandardScaler</code> to your pipeline and it normalizes your dataset before training starts.  As far as I can see there are no ‘built-in’ ways to do this in PyTorch or am I missing something?</p>
<p>Are there any options other than these:</p>
<ol>
<li>Manually calculate the mean and std deviation of your data and convert it at the beginning and keep a record of those parameters.</li>
<li>Build above into your <code>torch.utils.data.dataset</code> if you are using a custom dataset and data loader.</li>
<li>For output normalization, attach the parameters to your model and build the conversion back to original units into your models <code>forward()</code> method (remembering to feed un-normalized target values to your test criterion).</li>
<li>Or, do all your predictions and test evaluations in normalized units and only convert them back when you finally plot/save the results.</li>
<li>Or import and use <code>sklearn.preprocessing.StandardScaler</code>?</li>
</ol>
What's the difference between scriptmodule.graph() vs. scriptmodule.graph_for()

Jonson: <p>I noticed that <a href="https://pytorch.org/docs/stable/jit.html?highlight=scriptmodule#torch.jit.ScriptModule" rel="nofollow noopener"><code>scriptmodule</code> have graph() and graph_for() methods</a>, but I am confused about those two methods. Does anyone can tell me the difference? really thanks!</p>

ptrblck: <p><code>graph_for</code> should try to optimize or fuse the graph for the provided inputs, while <code>graph</code> should print the IR directly.</p>

Jonson: <p>Great! now I see, and I have another question, will <code>torch.jit.script()</code> try to optimize or fuse the graph when exporting a pytorch native model to a scriptmodule?</p>

driazati: <p>It will automatically optimize/fuse for you when you run it with some inputs, <code>graph_for</code> is showing you a preview of what will be executed for a certain set of inputs.</p>
Video dataloader - recover video labels

Luca: <p>I need to extract some features from HMDB51/UCF101 dataset for a video classification task using a pretrained 3D CNN. From what I understand, the dataloaders available in Pytorch divide each video in a certain number of subclips (which I cannot set), separated by x frames (which I can set), and each subclip is made up of a set number of frames (which again I can set).</p>
<p>Let’s assume that I use the following code:</p>
<pre><code class="lang-auto">from torchvision.datasets import HMDB51

root = "&lt;path_to_videos&gt;"
annotation_path =  "&lt;path_to_annotations&gt;"
frames_per_clip = 32
step_between_clips=50
fold=1
num_workers = 12
norm_value = 255


normalize = T.Normalize(mean=[114.7748 / norm_value, 107.7354 / norm_value, 99.4750 / norm_value],
                        std=[0.22803, 0.22145, 0.216989])
height, width = 224, 224
transform_test = transforms.Compose([
    T.ToFloatTensorInZeroOne(),
    T.Resize((height, width)),
    normalize
])

dataset_test = HMDB51(root, annotation_path, frames_per_clip, step_between_clips=step_between_clips, 
                 fold=fold, train=False, transform=transform_test, num_workers=num_workers)
</code></pre>
<p>The above code produces 2528 datapoints for the test split 1 of HMDB51. I would like to average the prediction for the clips belonging to the same video, so that I can measure the accuracy of my classifier at the video level and not only at the clip one.</p>
<p>To do so I thought about using <code> dataset_test.video_clips.get_clip_location()</code> to get the video indeces, and then pick the labels in order according to the video index. The loader is not shuffled, so the video with index 0 gets the first label, the next one gets the second and so on.<br>
In doing so I noticed that some videos are missing: Suppose that I try this:</p>
<pre><code class="lang-auto">for i in range(300,303):
    print(i, dataset_test.video_clips.get_clip_location(i))
</code></pre>
<p>This gives me</p>
<pre><code class="lang-auto">300 (137, 0)
301 (138, 0)
302 (141, 0)
</code></pre>
<p>Where are videos 139 and 140? Moreover, if I call <code>dataset_test.video_clips.cumulative_sizes[137:142]</code>,  I  get <code>[301, 302, 302, 302, 303]</code>. Why do I get thos 3 numbers equal?<br>
Also, what happens if a video is shorter than the <code>frames_per_clip</code> I selected? Will the video be ignored?<br>
Thanks in advance for the help!</p>
Pytorch model deployed within signal processing block written in C

arijit17: <p>Hi all:</p>
<p>I would like to deploy my PyTorch model (written in usual “research friendly eager mode” with <strong>Python API</strong>) within a signal processing framework (<strong>written in C code</strong>) and do inference.</p>
<p>What are the steps I should take to make it happen?</p>
<p>Basically what I want is:<br>
Signal Processing Block (C code) --&gt; Pytorch Model (takes in signal and outputs signal) --&gt; Signal Processing Block (C code).</p>

albanD: <p>Is <code>C</code> a strong requirement or is cpp ok?<br>
If you can do cpp, then you can use libtorch and the cpp api to run a jit model without python. <a href="https://pytorch.org/tutorials/advanced/cpp_export.html">This doc</a> should be a good introduction about this.</p>

arijit17: <p>Unfortunately, the PyTorch model has to be integrated within C code.</p>
<p>Thanks for pointing to the doc.</p>

albanD: <p>In that case, I guess you can always run a python interpreter in your C code and run the python model directly in that interpreter <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Differentiation in PyTorch

Johannes: <p>Hi all,</p>
<p>I have been reading up on differentiation techniques in DL frameworks. I came across the terms <em>symbol-to-number differentiation</em> which is used by Caffe and <em>symbol-to-symbol differentiation</em>. The former describes the approach of setting actual numeric values to the computational graph and computing the derivative in terms of those numeric values. The later describes an approach of adding additional nodes to the computational graph (providing a description of the derivates) which are used to compute the gradients. This later approach is used by TensorFlow. I was wondering which approach PyTorch here takes.</p>
<p>Best and thank you.</p>

albanD: <p>Hi,</p>
<p>I am not familiar with these terms so I cannot really tell you directly.<br>
In pytorch, we create a graph that contains functions to call to compute the backward.<br>
Note that contrary to tensorflow, the graph is created dynamically during the forward. So we don’t really have a concept of “symbol” everything is already a Tensor with values in it.</p>

Johannes: <p>Hi,</p>
<p>Thank you very much for your fast response. As far as I understand your answer, this would fall into the category of <em>symbol-to-number differentiation</em> since the numeric values are inserted directly into the graph without adding symbolic nodes that would <em>represent</em> the numeric values.</p>

albanD: <p>I would agree yes.<br>
What I understand is that the original Tensorflow cannot do this because they build the graph ahead of time. So there are no "number"s available yet.</p>

Johannes: <p>Oh, now I see what you aimed at by pointing out the difference in graph creation between PyTorch and TensorFlow. Thank you very much this helped a lot.</p>
Simple tutorial of pytorch avoid use of inner function

Rocket: <p>I look for full simple tutorial/example of pytorch avoid use of inner function except autograd such as…</p>
<p>Linear.</p>
<p>I have shown tutorials which are most works are automated as below.</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super.__init__()
        fc1 = linear(10,20)
   
    def forward(x):
        return fc1(x)

net =Net()

...
loss.backward()
opim.step()

</code></pre>
<p>I want to learn manual code(hard code) as below.</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super.__init__()
        w1 = torch.rand(10)
        b1 = torch.zeros(10)
   
    def forward(x):
        return torch.mm(x, w1) + b1

net =Net()
...
loss.backward()
opim.step()

</code></pre>
<p>etc…</p>

albanD: <p>Hi,</p>
<p>Sorry I’m not sure to understand your question.<br>
Could you rephrase it please?</p>

Rocket: <p>sorry for that<br>
additionally described</p>

albanD: <p>So first, you never want to use <code>.data</code>.</p>
<p>If you want to do your weight update manually, you can do:</p>
<pre><code class="lang-auto">loss.backward()
with torch.no_grad():
  w1 -=  learning_rate * w1.grad

</code></pre>
Backpropagation through time when to do .backward()?

bkoyuncu: <p>So I am training a convLSTM network, however, I decided to check if BPTT differs in results when it is called on a cumulative loss or on each individual loss object. I experimented on the gradient over the weight w1 and it not exactly the same in two cases. Code is below, anyone can comment on that difference</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torch.autograd import Variable
import numpy

w1 = torch.nn.Parameter(torch.tensor(([[1,2],[4,1]]),dtype=torch.float))
x = torch.nn.Parameter(torch.tensor(([[1,5],[4,5]]),dtype=torch.float),requires_grad=False)
y = torch.nn.Parameter(torch.tensor(([[2,7],[3,4]]),dtype=torch.float),requires_grad=False)
loss_fn = torch.nn.MSELoss(reduction='mean')

t= 20

#initial grad
print(w1.grad)

#case 1 call backward outside of the loop on a cumulative loss
loss = 0
x1=x.clone()
for i in range(t):
  y_pred = w1.mm(x1)
  loss += loss_fn(y_pred, y)
  print(i, loss_fn(y_pred, y).item())
  x1= y_pred.detach()
loss.backward()

case1 = (w1.grad).data

#reset grad of w1
w1.grad= None

#case two : backward in the loop
loss = 0
x1=x.clone()
for i in range(t):
  y_pred = w1.mm(x1)
  loss += loss_fn(y_pred, y)
  loss_fn(y_pred, y).backward(retain_graph=True)
  print(i, loss_fn(y_pred, y).item())
  x1= y_pred.detach()

case2 = (w1.grad).data

print(torch.all(torch.eq(case1, case2))) #this gives false
numpy.isclose(case1, case2) #this gives true, they are close but not same

</code></pre>

albanD: <p>Hi,</p>
<p>Please avoid using <code>.data</code>. Not sure why you use it here as <code>w1.grad</code> does not require gradient. If you want a copy, you need <code>.clone()</code>.</p>
<p>These gradients are expected to be close but not equal I’m afraid.<br>
Floating point operations are not exact and can differ based on their order. Here when you backward every time or if you sum the result and then do one backward, you will perform all operations in different orders.</p>

bkoyuncu: <p><a class="mention" href="/u/alband">@albanD</a> thanks so much. I also thought that it may be because of the floating points since gradient gets too large in this particular example. Since I wanted to compare numeric values I used <code>.data</code>. I will also consider your other advices, so the conclusion is they are same right?</p>

albanD: <p><code>.data</code> is not a thing anymore since we removed <code>Variable</code>s. It still does things but mostly buggy things so you should avoid it <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<blockquote>
<p>the conclusion is they are same right ?</p>
</blockquote>
<p>They are as close as we can get them. So here yes they are “the same” up to floating point precision.</p>
Efficient matching of two tensors

smonsays: <p>In an attempt to implement the <code>log_prob()</code> method for an <code>Empirical</code> distribution function, I need to match the entries of a big tensor with samples. My naive implementation is terribly slow:</p>
<pre><code class="lang-python">samples = torch.randn(10)
value = torch.randn(10000)

num_matches = torch.zeros_like(values)
for i, v in enumerate(value):
    num_matches[i] = torch.isclose(samples, v).sum()
</code></pre>
<p>Does somebody have a suggestion, how I could speed it up?</p>

albanD: <p>You want to count how many samples each value is close to?</p>
<p>You can do</p>
<pre><code class="lang-auto">expanded_values = value.unsqueeze(1).expand(10000, 10)
expanded_samples = samples.unsqueeze(0).expand(10000, 10)
num_matches = torch.isclose(expanded_values, expanded_samples).sum(-1)
</code></pre>

smonsays: <p>Perfect, thank you! I didn’t realize that <a href="https://pytorch.org/docs/stable/tensors.html?highlight=expand#torch.Tensor.expand" rel="nofollow noopener">torch.Tensor.expand</a> creates a view on the existing tensor without allocating new memory. Very neat!</p>

albanD: <p>It does !<br>
The result of <code>isclose</code> will be quite large though in this case <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"> But if it fits in memory that is definitely going to be the fastest.</p>
Convolution with Sparse image data

davidlee: <p>Hi all,</p>
<p>I’m wondering how i can implement convolution layer with sparse 2D image data.<br>
The example is attached.<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/6/269cdf0a4e51087012c915bd8b75d1b434110cd0.png" alt="1" data-base62-sha1="5vAeOEaCjQMm1U5OOn87yMWvm7u" width="385" height="341"><br>
It has patterns in a very little portion of the entire image and<br>
the rest pixels of the image is zero.</p>
<p>These kinds of images are stacked in multiple channels.</p>
<p>Would the dilated convolution or submanifold convolution be helpful?<br>
Or, Can i just use normal 2D convoluton layer for this data?</p>
<p>Please help me!</p>
<p>Thank you very much in advance.</p>