akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Unexpected model output

WAN: <p>Hello,<br>
I have a multi classification model ( with 5 classes). The model looks like this:</p>
<pre><code class="lang-auto">classifier(
  (embedding): Embedding(28564, 100)
  (lstm): LSTM(100, 32, num_layers=2, batch_first=True, dropout=0.2, bidirectional=True)
  (fc): Linear(in_features=64, out_features=5, bias=True)
  (act): Sigmoid()
)
</code></pre>
<p>But when I do predict to a new  sampel, I have an output like this:</p>
<pre><code class="lang-auto">tensor([[ 4.9442, -4.3852,  0.8900, -2.8472, -4.5958]], device='cuda:0',
       grad_fn=&lt;SliceBackward&gt;)
</code></pre>
<p>What should I modify to have an output as a label (0,1,2,3,4)?<br>
Thanks in advance.</p>

ptrblck: <p>I assume you are working on a multi-label classification, where each sample might belong to zero, one, or multiple classes?<br>
If so, you could use a threshold and get the predicted label for an active/inactive class as:</p>
<pre><code class="lang-python">out = ...
preds = out &gt; 0.5
# preds would now indicate which class is active (1) or inactive (0) for the sample
</code></pre>
<p>In your code snippet your model uses <code>nn.Sigmoid</code> as the last layer, which doesn’t fit the output as it’s negative.<br>
In case you are indeed using <code>nn.Sigmoid</code>, a default threshold of <code>0.5</code> could be a good starting point. On the other hand, if your model outputs logits, you could use <code>0.0</code> as the default threshold.</p>

WAN: <p>Thanks, ptrblck, for your reply.<br>
No, I’m working on multiclass classification where there is no overlap between the classes.<br>
In my case, what do you suggest?<br>
Also, I can not use softmax since my loss function is CrossEntropyLoss which has built-in softmax (as I understand from the documentation of pytorch).</p>
<p>Thanks,</p>

ptrblck: <p>Ah OK, in that case remove the <code>nn.Sigmoid</code> and pass the raw logits in the shape <code>[batch_size, nb_classes]</code> to <code>nn.CrossEntropyLoss</code>. The targets should have the shape <code>[batch_size]</code> and contain class indices in the range <code>[0, nb_classes-1]</code>. To get the predicted class from the model output use <code>preds = torch.argmax(output, dim=1)</code>.</p>
Non-maximum suppression(NMS) with IoU threshold

iamexperimentingnow: <p>Hi, I have a doubt in this method.</p>
<p>from torchvision.ops import nms</p>
<p>Here, in nms method, the IoU threshold value is 0.05, does it mean, it is considering the predicted bbox which is greater than 5 %?</p>
<p>or</p>
<p>is it 1 - 0.05 = 0.95 → 95 % percent of overlap?</p>
<pre><code class="lang-auto">from torchvision.ops import nms
nms(torch.tensor(bbs.astype(np.float32)), torch.tensor(confs), 0.05)
</code></pre>
`torch.multiprocessing.SimpleQueue` on an `IterableDataset`

remiliani93: <p>Hi, I wanted to know if it is possible to use a <code>torch.multiprocessing.SimpleQueue</code> other than the two examples provided in the <code>IterableDataset</code> documentation to split the work among the <code>num_worker</code>. Let me set up a dummy example. Assume the following map-style <code>Dataset</code>:</p>
<pre><code class="lang-auto">class MyMapDataset(Dataset):
    def __init__(self, length: int = 100):
        self.length = length

    def __getitem__(self, index) -&gt; torch.Tensor:
        return torch.arange(0, index + 1)

    def __len__(self):
        return self.length
</code></pre>
<p>Now we have an iterable-style dataset that uses the <code>MyMapDataset</code>:</p>
<pre><code class="lang-auto">class MyIterableDataset(IterableDataset):
    def __init__(self, dataset: MyMapDataset):
        self.dataset = dataset
        self.queue = SimpleQueue()

        # Put the indices to process in a shared queue.
        for index in range(len(self.dataset)):
            self.queue.put(index)

    def __iter__(self):
        while not self.queue.empty():
            yield from self.dataset[self.queue.get()]
</code></pre>
<p>And finally to run:</p>
<pre><code class="lang-auto">map_dataset = MyMapDataset()
iterable_dataset = MyIterableDataset(dataset=map_dataset)
dl = DataLoader(iterable_dataset, num_workers=4)
for sample in dl:
    pass
</code></pre>
<p>Will the <code>SimpleQueue</code> be created on the main process and then all the workers will have access to it or will this just create <code>num_workers</code> instances of a <code>SimpleQueue</code>?</p>
Cuda out of memory. Memory leaks on pytorch

Sanjana_Mudduluru: <p>Hello,<br>
I am very new to PyTorch. I have been working on UNET segmentation on retinal fundus images(RGB) + masks(Grayscale) for hemorrhage segmentation. I am trying to train on GPU + High-RAM on google colab pro. I am constantly running into ‘RunTimeError: Cuda out of memory’. I tried torch.cuda.empty_cache() for each X,y training loop. I also tried gc.collect() for X,y,y_pred after computing loss each time. I am still not able to figure out why the almost 13GB of GPU is getting used for just first epoch.<br>
Here is the complete notebook: <a href="https://colab.research.google.com/drive/1yEkSAAg0XsT9KUBy0aSiirOzans7pJYY?usp=sharing" rel="noopener nofollow ugc">UNET_train</a><br>
Any help would be great.</p>

Sanjana_Mudduluru: <p>I think I figured it out. The images and masks are not resized properly leading to out of memory issues.</p>
How to create a c10::List<IValue>?

Bycob: <p>I try to create a List from C++ API, but I keep getting a compilation error for code as simple as this one:</p>
<pre><code class="lang-auto">c10::List&lt;c10::IValue&gt; list;
</code></pre>
<p>I get this error:</p>
<pre><code class="lang-auto">In file included from /usr/local/include/ATen/core/List.h:472:0,
                 from /usr/local/include/ATen/core/ivalue_inl.h:13,
                 from /usr/local/include/ATen/core/ivalue.h:895,
                 from /usr/local/include/ATen/ATen.h:33,
                 from /usr/local/include/torch/csrc/api/include/torch/types.h:3,
                 from /usr/local/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                 from /usr/local/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                 from /usr/local/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                 from /usr/local/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                 from /usr/local/include/torch/csrc/api/include/torch/data.h:3,
                 from /usr/local/include/torch/csrc/api/include/torch/all.h:8,
                 from /usr/local/include/torch/csrc/api/include/torch/torch.h:3,
                 from /home/aaaa/bbbbb/test.cc:2:
/usr/local/include/ATen/core/List_inl.h: In instantiation of ‘c10::List&lt;T&gt;::List() [with T = c10::IValue]’:
/usr/local/include/ATen/core/ivalue_inl.h:976:62:   required from ‘c10::IValue::IValue(const std::vector&lt;T&gt;&amp;) [with T = c10::IValue; typename std::enable_if&lt;std::is_constructible&lt;c10::IValue, T&gt;::value, std::nullptr_t&gt;::type &lt;anonymous&gt; = nullptr]’
/home/aaaa/bbbbb/torch_exp/test.cc:89:49:   required from here
/usr/local/include/ATen/core/List_inl.h:20:3: error: static assertion failed: This constructor is not valid for List&lt;IValue&gt;. Please use c10::impl::GenericList(elementType) instead.
   static_assert(!std::is_same&lt;T, IValue&gt;::value, "This constructor is not valid for List&lt;IValue&gt;. Please use c10::impl::GenericList(elementType) instead.");
   ^~~~~~~~~~~~~
</code></pre>
<p>Am I trying to create a list the wrong way? In which case, what is the correct way of creating list of IValues? I want to pass them to a jit module that takes a List of Tensor as input</p>
<p>And I don’t understand why this assert is triggering, since <code>std::is_same&lt;T, IValue&gt;::value</code> should be true for T == c10::IValue ?</p>

Bycob: <p>I’m still interested by a reply.</p>

Jungmo_Ahn: <p>did you solve this problem?</p>

Bycob: <p>No<br>
I worked my way around by passing a tensor instead of a list, so I wouldn’t have to create a list.</p>

tom: <p>Does <code>std::vector&lt;IValue&gt;</code> not work?</p>

inou: <p>Any solution regarding this?</p>

Mikhail_Miha: <p>Hi! May be u can show some code?</p>
List element wise addition

cbd: <p>I have list “c=[[1, 2], [6, 7]]”. I want element wise sum of list(0) and list(1) such that output is<br>
“out=[7,9]”.</p>

albanD: <p>Hi,</p>
<p>Are you asking in python in general or using pytorch?<br>
Using PyTorch, you can simply do <code>torch.tensor(your_list).sum(1)</code>.</p>

cbd: <p>Thanks that’s solved my issue.</p>
Help!Why grad is always NA

Nid_Carry: <blockquote>
<p>It is a <em>[Multiple](javascript:<img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"> [Linear](javascript:<img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"> [Regression](javascript:<img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></em> model<br>
but it seems grad  is not worked</p>
</blockquote>
<pre data-code-wrap="python"><code class="lang-python">theta.requires_grad=True
for eproch in range(epochs):
    y_pred=X_b.mm(theta)
    loss=torch.sqrt((y_pred-y).sum())
    loss.backward()
    print(theta.grad)
#     theta-=lr*theta.grad
</code></pre>
<p>tensor([[nan],<br>
[nan],<br>
[nan],<br>
[nan]], device=‘cuda:0’, dtype=torch.float64)<br>
tensor([[nan],<br>
[nan],<br>
[nan],<br>
[nan]], device=‘cuda:0’, dtype=torch.float64)<br>
tensor([[nan],<br>
[nan],<br>
[nan],<br>
[nan]], device=‘cuda:0’, dtype=torch.float64)<br>
tensor([[nan],<br>
[nan],<br>
[nan],<br>
[nan]], device=‘cuda:0’, dtype=torch.float64)<br>
tensor([[nan],<br>
[nan],<br>
[nan],<br>
[nan]], device=‘cuda:0’, dtype=torch.float64)<br>
tensor([[nan],<br>
[nan],<br>
[nan],<br>
[nan]], device=‘cuda:0’, dtype=torch.float64)<br>
tensor([[nan],<br>
[nan],<br>
[nan],<br>
[nan]], device=‘cuda:0’, dtype=torch.float64)<br>
tensor([[nan],<br>
[nan],<br>
[nan],<br>
[nan]], device=‘cuda:0’, dtype=torch.float64)<br>
tensor([[nan],<br>
[nan],<br>
[nan],<br>
[nan]], device=‘cuda:0’, dtype=torch.float64)<br>
tensor([[nan],<br>
[nan],<br>
[nan],<br>
[nan]], device=‘cuda:0’, dtype=torch.float64)<br>
tensor([[nan],<br>
[nan],<br>
[nan],<br>
[nan]], device=‘cuda:0’, dtype=torch.float64)<br>
tensor([[nan],<br>
[nan],<br>
[nan],<br>
[nan]], device=‘cuda:0’, dtype=torch.float64)<br>
tensor([[nan],<br>
[nan],<br>
[nan],<br>
[nan]], device=‘cuda:0’, dtype=torch.float64)</p>
<pre><code class="lang-auto">type or paste code here
</code></pre>

albanD: <p>Hi,</p>
<p>What are the values of y and y_pred?<br>
The gradient of sqrt at 0 is going to be infinite, leading to nans.</p>
Question about an error when loading pretrained models

Ford: <p>When I loaded the pretrained model using the following code:<br>
<code>ckp = torch.load(checkpoint, map_location='cpu')</code><br>
I meet an error which is “UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xb0 in position 3: invalid start byte”</p>
<p>Can anyone fix this problem? Thanks.</p>
What's the most convenient way to rebuild for a small change

mszhanyi: <p>If the pytorch source code has been already build with ninja, and there’s a very small change, such as printf,  in one c++ code, not headers. what’s the most convenient way to rebuild it? What’s your best practices.</p>

ptrblck: <p>The <a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md">CONTRIBUTING</a> docs mention some useful workflows, such as using <code>ccache</code> etc. for incremental builds, which can speedup the development significantly.</p>

llan-ml: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I notice that it still says that we would need to reinstall if we modify non-Python files (e.g., <code>.cpp</code>) in <a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md#developing-pytorch" rel="noopener nofollow ugc">CONTRIBUTING</a>, (just above <a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md#tips-and-debugging" rel="noopener nofollow ugc">here</a>).</p>
<p>However, as you commented in other posts, after we modify some <code>cpp</code> files, running again <code>python setup.py develop</code> is enough to rebuild the changes.</p>
<p>Is the guide in CONTRIBUTING out-dated?</p>
How to specify PyTorch version with the cuda version when building from source?

daimingyang: <p>When exporting <code>PYTORCH_BUILD_VERSION=1.10.0</code> and <code>PYTORCH_BUILD_NUMBER=1</code>, I could build locally version <code>1.10.0</code> with PyTorch 1.10.0-rc3 from source. However, the cuda113 torchvision rc requests PyTorch version <code>1.10.0+cu113</code>, and the PyTorch build fails if I export <code>PYTORCH_BUILD_VERSION=1.10.0+cu113</code>. I looked into the python script, and yet could only find this one method to specify the PyTorch version. I wondering how PyTorch team built the wheel with the cuda version tag. Similarly for CPU binaries, how did the public binary come with <code>1.10.0+cpu</code>? Could someone help here? Thanks!</p>

ptrblck: <p>I guess the suffix might be added <a href="https://github.com/pytorch/builder/blob/1c4c239f02873521b2bbfdd17cf73bf02c8bf5a7/release/promote.sh">here</a>. Also, you could check the <code>builder</code> repository for any usage of a suffix tag and would find e.g. <a href="https://github.com/pytorch/builder/blob/1c4c239f02873521b2bbfdd17cf73bf02c8bf5a7/conda/build_vision.sh#L71">this for <code>torchvision</code></a>.</p>

daimingyang: <p>Update:<br>
Exporting <code>PYTORCH_BUILD_VERSION=1.10.0+cpu</code> and <code>1.10.0+cu113</code> works without issues in my current linux container now.<br>
Regarding the scripts mentioned by <a class="mention" href="/u/ptrblck">@ptrblck</a> , I think these scripts are run when promoting built binaries, not when building them. The vision script is under conda, and there isn’t one for torch. Thanks for the help though.</p>
Reset DataLoader

Odin_NI: <p>I’m using 4 DataLoaders for optimized BERT training. I have 4 datasets with different sequence lengths (8, 16, 32, 64), batch sizes are (512, 256, 128, 64) correspondingly. All this done for memory consumption control.</p>
<p>I’m calculating backward for each batch from my DataLoaders and then I’m making an optimizer step. Each DataLoader has different length. How can I reset all DataLoaders in case of using zip function?</p>
<p>I will provide some example code:</p>
<pre><code class="lang-auto">data_loaders = []
for size in [8, 16, 32, 64]:
    data_loaders.append(
        DataLoader(
            dataset=dataset[size],
            collate_fn=collate_fn,
            batch_size=num_samples // size,
            num_workers=8,
            shuffle=True,
            pin_memory=True,
            drop_last=True,
        )
    )
</code></pre>
<pre><code class="lang-auto">for epoch in range(1, num_epochs + 1):
    for i, batches in enumerate(zip(*data_loaders)):
        for batch, labels in batches:
            batch = batch.to(device)
            labels = labels.to(device)
            
            with autocast(enabled=use_amp):
                predict = _model(
                    batch
                )
                loss = criterion(
                    predict,
                    labels,
                )

            scaler.scale(loss).backward()

        scaler.step(_optimizer)
        scaler.update()
        optimizer.zero_grad(set_to_none=True)
</code></pre>
Any plan for Symmetric padding?

hideinshadow: <p>The paper <a href="https://arxiv.org/abs/2010.02178" rel="noopener nofollow ugc">[2010.02178] Mind the Pad – CNNs can Develop Blind Spots (arxiv.org)</a> clarified that Symmetric padding is helpful to tasks that require high precision at the periphery. But there is no implementation of this padding algorithm in PyTorch now (In fact, there is a implementation of it - <a href="https://discuss.pytorch.org/t/symmetric-padding/19866">Symmetric Padding - PyTorch Forums</a>, but I think an official implementation can make PyTorch better.). So is there any plan for Symmetric padding?</p>
Generate a mask for all duplicate grey-scale pixels in a minibatch

char_c: <p>Hello guys, I want to find all duplicate elements in a minibatch tensor, specifically:<br>
given a tensor X=[B, 1, H, W], which can be regarded as B grey-scale images with size H*W. I want to find duplicate pixel in each image, thus, the desired output is a  binary mask Y=[B, 1, H, W], where every unique pixel is assigned 1 while all duplicate pixel assigned 0.</p>
<p>Currently I am able to achieve part of the above function via torch.unique+torch.split (for a single grey-image rather than a minibatch), and I show my implementation here:</p>
<pre><code class="lang-auto">import torch

W, H = 3, 3
image_flatten = torch.tensor([2, 3, 4, 1, 6, 2, 2, 5, 3])
print(image_flatten.view(W, H))
sorted_image_flatten, idx_sort = torch.sort(image_flatten)
unique_values, _, count = torch.unique(sorted_image_flatten, return_counts=True, return_inverse=True)
repetitive_idx_groups = torch.split(idx_sort, list(count))
rep_idx_flatten = [rep_idx \
        for rep_idx_group in repetitive_idx_groups if len(rep_idx_group) &gt; 1 \
        for rep_idx in rep_idx_group]
rep_idx_flatten = torch.tensor(rep_idx_flatten)
mask = torch.ones((W*H,1))
mask[rep_idx_flatten] = 0
mask = mask.view(H, W)
print(mask)
</code></pre>
<p>the outputs:</p>
<pre><code class="lang-auto">tensor([[2, 3, 4],
        [1, 6, 2],
        [2, 5, 3]])
tensor([[0., 0., 1.],
        [1., 1., 0.],
        [0., 1., 0.]])
</code></pre>
<p>The problem is that this implementation costs too much time even when running on CUDA, the time budget is too high to allow training my network. The reasons are 1) torch.split is very time-consuming, 2) torch.unique does not support batch operation.</p>
<p>So I wonder if there is something in PyTorch Library that can achieve the above function for a minibatch? I really appreaciate your time on helping me solve this problem.</p>

tumble-weed: <p>upon trying your code, it seems that sorting before unique is eating some time. unique takes a <strong>sorted</strong> argument of its own that takes less time (i think sorting before unique runs 2 passes through the data, while sorting during unique does it in 1 pass). also you can use the inverse indexes returned by unique to simplify the code:</p>
<pre><code class="lang-auto">import torch
import time
W, H = 3, 3
image_flatten = torch.tensor([2, 3, 4, 1, 6, 2, 2, 5, 3])
print(image_flatten.view(W, H))
# sorted_image_flatten, idx_sort = torch.sort(image_flatten)
unique_values, inverse_indexes, count = torch.unique(image_flatten, sorted = True,return_counts=True, return_inverse=True)
non_repeated_vals = (count == 1).float()
mask_flatten = non_repeated_vals[inverse_indexes]
mask = mask_flatten.view(H,W)
</code></pre>

char_c: <p>Brilliant <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=10" title=":+1:" class="emoji" alt=":+1:"> I tried your code and found the time budget is almost affordable now.<br>
Actually when tracing the time cost, I found the most time is spent on torch.split, and creating and assigning the mask.<br>
Thank you~</p>
CUDA graph with object as Model's input

gemsanyou: <p>As I see from the examples in <a href="https://pytorch.org/docs/master/notes/cuda.html#cuda-graphs" class="inline-onebox" rel="noopener nofollow ugc">CUDA semantics — PyTorch master documentation</a> ,  the real input has the exact same shape of the static input of the initial recording of the graph.</p>
<p>Can we use torch tensor of different shape (i.e, different batch size) for the real input?</p>
<p>Moreover, can we use an object instead of torch.Tensor for the input? The object have several fields which are of torch.Tensor</p>
<p>Does the concern of using different shape of data have anything to do with the graph memory management? I can’t really understand the part in the <a href="https://pytorch.org/docs/master/notes/cuda.html#graph-memory-management" rel="noopener nofollow ugc">Graph Memory Management Section</a></p>

ptrblck: <p>No, as described in the docs:</p>
<blockquote>
<p>Replaying a graph sacrifices the dynamic flexibility of typical eager execution in exchange for <strong>greatly reduced CPU overhead</strong>. A graph’s arguments and kernels are fixed, so a graph replay skips all layers of argument setup and kernel dispatch, including Python, C++, and CUDA driver overheads. Under the hood, a replay submits the entire graph’s work to the GPU with a single call to <a href="https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__GRAPH.html#group__CUDART__GRAPH_1g1accfe1da0c605a577c22d9751a09597">cudaGraphLaunch</a>. Kernels in a replay also execute slightly faster on the GPU, but eliding CPU overhead is the main benefit.<br>
You should try CUDA graphs if all or part of your network is graph-safe (usually this means static shapes and static control flow, but see the other <a href="https://pytorch.org/docs/master/notes/cuda.html#capture-constraints">constraints</a>) and you suspect its runtime is at least somewhat CPU-limited.</p>
</blockquote>
<p>you would have to use static shapes since the arguments and kernels are fixed and just replayed.</p>

gemsanyou: <p>I see, so maybe CUDA graph is not the approach to improve performance for such learning framework with variable input size, e.g. RL environment. Thank you so much</p>

gemsanyou: <p>However, maybe a part of the module’s can be recorded, which have static input size. I’ll check on that.</p>

ptrblck: <p>Yes, that might certainly be the case. The <a href="https://pytorch.org/docs/master/notes/cuda.html#partial-network-capture">Partial-network capture section</a> would show an example.</p>
Time Series variable length

sesale: <p>I am trying to run an SVM on time series data. My dataset is represented by multiple pandas dataframes and one sample is represented by one dataframe row where each column represents one time step. Unfortunately, the samples have different length. Does anyone know how to deal with that issue?</p>

ptrblck: <p>Could you describe your model a bit more, i.e. what kind of architecture you are using?<br>
A common approach would be to pad your sequences to the same lengths.</p>

sesale: <p>This is the architecture I am using:</p>
<pre><code class="lang-auto">class SVM_Net(nn.Module):
    def __init__(self,n_feature,n_class):
        super(SVM_Net, self).__init__()
        self.fc=nn.Linear(n_feature,n_class)
        nn.init.kaiming_uniform(self.fc.weight)
        nn.init.constant(self.fc.bias,0.1)
        
    def forward(self,x):
        output = self.fc(x)
        return output
</code></pre>
<p>In the meantime I already found out that I can use padding in order to solve my problem.<br>
I am using a custom collate_fn method, however is there a way to pad them to a fixed, predefined length?</p>
<pre><code class="lang-auto">def pad_collate(batch):
    xx = []
    for elt in batch:
        xx.append(elt['data'])
    
    xx_pad = torch.nn.utils.rnn.pad_sequence(xx, batch_first=True, padding_value=0)
    return xx_pad
</code></pre>

ptrblck: <p>If you want to pad the sequences to a pre-defined length, you could add the padding in the <code>__getitem__</code> and return equal sequence lengths.</p>

sesale: <p>I am not exactly sure how to achieve this. I tried adding the following line to the <strong>getitem</strong> method:</p>
<pre><code class="lang-auto">max_len=50
data = nn.ConstantPad1d((0, max_len - data.shape[0]), 0)(data)
</code></pre>
<p>where data represents one sequence (sample) of my data. However, one time step has multiple values, meaning that data looks something like this:</p>
<pre><code class="lang-auto">data = [[0,1,30,40,56], [5,4,8,13,15], [90,81,76,3,22], ...]
</code></pre>
<p>and I would like to pad it to a maximum sequence length, meaning to add as much timesteps as needed in order to receive the maximum length. What the code I added does, is padding each time step (so each array inside the sequence) to the desired length.</p>

ptrblck: <p>Given your input data the constant padding would be added in <code>dim1</code>. I’m not sure if this is what you want or if you are hitting an issue with this approach.</p>
Does torch function call 'cudaEventRecord' internally?

111480: <p>I am currently using the RTX Quadro 6000 for the DNN provided by torchvision.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/6/46cf49ada7fdb2ff09a0a5a7eec50a734b1a6e94.png" data-download-href="https://discuss.pytorch.org/uploads/default/46cf49ada7fdb2ff09a0a5a7eec50a734b1a6e94" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/46cf49ada7fdb2ff09a0a5a7eec50a734b1a6e94_2_517x202.png" alt="image" data-base62-sha1="a6pxaziCf0gKCtpUZd1sZVZmLre" width="517" height="202" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/46cf49ada7fdb2ff09a0a5a7eec50a734b1a6e94_2_517x202.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/46cf49ada7fdb2ff09a0a5a7eec50a734b1a6e94_2_775x303.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/46cf49ada7fdb2ff09a0a5a7eec50a734b1a6e94_2_1034x404.png 2x" data-dominant-color="D7E3D5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1463×573 37.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>When I got the profile result with nsight system from NVIDIA , I found that ‘cudaEventRecord’ that I didn’t call was called.</p>
<p>Are these calls automatically called by the torch function?</p>
Output shape of mini batch for graph neural network

MD_TOKI_TAHMID: <p>I have a dataset containing 100 graph data. Each has different number of nodes, but 1024 features. And each graph has a target 0 or 1.</p>
<p>Now while using DataLoader to load the data using a batch size of 32, I am getting x shape of (something, 1024). I understand the value of "something " depends on which 32 graphs are being concatenated in the batch matrix. But the output shape (y) remains (32,2).</p>
<p>Now while training I am getting error that, expected input batch size (7557) to batch target batch size (32). How can I solve this issue?</p>
<p>Ik this is a very basic qus. But I am having some issues to deal with it.<br>
Thanks!</p>
Use validation to adjust parameters of an Autoencoder

papyrus: <p>Hello everyone,</p>
<p>Let’s say I have data with shape <code>N</code>, and an Autoencoder that takes as input this type of data that are normalized, so the general steps are:</p>
<p>1- Data lies in the original domain<br>
2- Data is normalized<br>
3- Data passes through the Autoencoder<br>
4- Reconstructed data lies in the normalized domain<br>
5- Reconstructed data is “denormalized” in order to lie in the original domain.</p>
<p>The thing is I need to stay in the normalized domain during training. So, the loss is computed in the normalized domain: the Autoencoder learns to reconstruct the data with equal weights to the <code>N</code> values. But some of those <code>N</code> values are more important than other when lying in the original domain, so the Autoencoder should learn to reconstruct the data with different weights.</p>
<p>So my question is: how can I learn those weights?</p>
<p>Is it possible to make a “double” learning that would go this way:</p>
<ul>
<li>initialize two neural networks: an Autoencoder and another one with learnable parameters (the weights)</li>
<li>train the Autoencoder. The loss is computed with weights from the other network</li>
<li>when validating (validation is done in the original domain), use the results in the original domain to adjust the parameters corresponding to the weights</li>
</ul>
<p>Or is there a simpler way of doing this?</p>
<p>I can’t find examples of code doing those kind of things. Does someone have some resources?</p>
<p>Thanks a lot for reading</p>
How to use more than one cpu to run pytorch inference in the arrch64 architecture

qingye_meng: <p>I am running a Transformers BertForTokenClassification model in a  arrch64 architecture pc.In the inference stage, I found that the speed is very slow.<br>
The speed mainly consumes in the following code:</p>
<blockquote>
<pre><code>   outputs = self.model(input_ids, input_mask, segment_ids)
</code></pre>
</blockquote>
<p>Then I used the watch command to find that only 1 cpu was used in the inference stage but this machine has 8 cpus.</p>
<p>Generally speaking, pytorch under the x86 architecture will use all cpu by default, so under the arrch64 architecture, what additional settings do I need to use all the cpu?</p>

ptrblck: <p>I don’t know how you’ve installed PyTorch, but would assume you’ve built it from source (or did you find pip wheels for aarch64?).<br>
If so, did you install NEON on your system and was PyTorch detecting it during the build? This should use vectorized operations and could yield a speedup. Also, OpenBLAS should be able to use all your cores.</p>

qingye_meng: <p>Hi，thank you for your reply.I found the cause of the problem. After I upgraded the pytorch version from 1.4.0 to 1.8.2, the problem was solved. <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=10" title=":grin:" class="emoji" alt=":grin:"></p>
When using torch.sort(), how do I assign the minimal index to those same elements?

Weize: <p>Like if I have a Tensor</p>
<pre><code class="lang-auto">a = [1, 2, 2, 3]
</code></pre>
<p>If I perform torch.sort(a), then the index it returns will be</p>
<pre><code class="lang-auto">[0, 1, 2, 3]
</code></pre>
<p>What if I want to get the result like</p>
<pre><code class="lang-auto">[0, 1, 1, 3]
</code></pre>
<p>I know there’s a function exists in scipy “scipy.data.rankdata”, in the function, I can specify the method it uses when ranking the data. But how can I achieve the similar goal with pytorch?</p>

smth: <pre><code class="lang-auto">torch.unique(x, sorted=True, return_inverse=True)
(tensor([1, 2, 3]), tensor([0, 1, 1, 2]))
</code></pre>

Weize: <p>Thanks!<br>
But the result I expect to get is</p>
<pre><code class="lang-auto">[0, 1, 1, 3]
</code></pre>
<p>That is, when same elements appear in an array, the returned index will not be “continous”. Are there any other ways to do it?</p>

smth: <p>I dont think there’s a direct way to do <code>[0, 1, 1, 3]</code>. Unique gives you what you want from an ordering perspective, but does not account for the right magnitude,</p>

Weize: <p>It’s also useful. Thanks anyway!</p>

tumble-weed: <p>For anyone who stumbles over this thread again, a small modification to <a class="mention" href="/u/smth">@smth</a> 's answer works:</p>
<pre><code class="lang-auto">import torch
x = torch.tensor([1, 2, 2, 3])
uq_vals,inverse_ixs,counts = torch.unique(x, sorted=True, return_inverse=True,return_counts=True)
cumsum_counts = counts.cumsum(dim=0)
possible_ranks = torch.zeros_like(counts)
possible_ranks[1:] = cumsum_counts[:-1]
ranks = possible_ranks[inverse_ixs]
print(x,
      ranks)
</code></pre>
Techniques for multiclass semantic segmentation with large number of output classes

scribblepad: <p>I’m working on a custom dataset which has over 100 classes (10K images), trying to train the models using the deeplabv3plus and unetplusplus (also tried the base versions). The metrics I’m using are dice loss and iou score. It seems to be converging at a very early epochs and don’t see any improvements to loss or iou post that.</p>
<p>What are some of the best practices or techniques to improve the models iou when working large number of output classes? Choosing the model, focusing on particular hyperparameters?</p>
<p>Thanks</p>
Tf.contrib.resampler.resampler() alternative in PyTorch

vinaykumar2491: <p>Hi everyone,<br>
What is the PyTorch alternative for <code>tf.contrib.resampler.resampler</code>. I tried to use <code>torch.nn.functional.interpolate</code> but it does not seem to work the same.<br>
Thanks.</p>

Yiru_Shen: <p>You can use <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.functional.grid_sample" rel="noopener nofollow ugc">grid_sample</a> as an alternative</p>

vinaykumar2491: <p>Unfortunately this is not working for me… In tensorflow, the two tensors <code>input</code> and <code>grid</code> can have different number of dimensions<br>
for eg:<br>
<code>input.get_shape() = [N, H, W, C]</code> and<br>
<code>grid.get_shape() = [N, dim_0, ... , dim_n, 2]</code>…</p>
<p>But <code>torch.grid_sample()</code> requires the <code>input</code> and <code>grid</code> to have the same number of dimensions.</p>
<p>Is there any solution to this.</p>
<p>My <code>input</code> is 4D and <code>grid</code> is 3D.</p>
<p>Thanks.</p>
Use_deterministic_algorithms INTERNAL ASSERT FAILED

LearnedLately: <p>I get the following internal error when I try to use <code>torch.use_deterministic_algorithms(True)</code>. The code runs fine without that line.</p>
<blockquote>
<p>RuntimeError: linearIndex.numel()<em>sliceSize</em>nElemBefore == value.numel()INTERNAL ASSERT FAILED at “/pytorch/aten/src/ATen/native/cuda/Indexing.cu”:253, please report a bug to PyTorch. number of flattened indices did not match number of elements in the value tensor71</p>
</blockquote>
<p>At this line of code:<br>
<code>l1_values[torch.arange(len(max_idxs), device="cuda"), max_idxs] = 1</code></p>
<p>l1_values.shape is torch.Size([71, 1500])<br>
max_idxs.shape is torch.Size([71])</p>
<p><strong>My environment:</strong><br>
$ python3 -m torch.utils.collect_env<br>
Collecting environment information…<br>
PyTorch version: 1.9.1+cu111<br>
Is debug build: False<br>
CUDA used to build PyTorch: 11.1<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Ubuntu 18.04.6 LTS (x86_64)<br>
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0<br>
Clang version: Could not collect<br>
CMake version: version 3.13.4<br>
Libc version: glibc-2.25</p>
<p>Python version: 3.6 (64-bit runtime)<br>
Python platform: Linux-5.4.0-87-lowlatency-x86_64-with-Ubuntu-18.04-bionic<br>
Is CUDA available: True<br>
CUDA runtime version: Could not collect<br>
GPU models and configuration:<br>
GPU 0: GeForce GTX 980 Ti<br>
GPU 1: GeForce GTX TITAN X</p>
<p>Nvidia driver version: 460.73.01<br>
cuDNN version: Probably one of the following:<br>
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5<br>
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn.so.8.2.1<br>
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.2.1<br>
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.2.1<br>
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.2.1<br>
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.2.1<br>
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.2.1<br>
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.2.1<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.19.5<br>
[pip3] torch==1.9.1+cu111<br>
[pip3] torchaudio==0.9.1<br>
[pip3] torchtext==0.8.1<br>
[pip3] torchvision==0.10.1+cu111<br>
[conda] blas                      1.0                         mkl<br>
[conda] mkl                       2020.2                      256<br>
[conda] mkl-service               2.3.0            py38he904b0f_0<br>
[conda] mkl_fft                   1.2.0            py38h23d657b_0<br>
[conda] mkl_random                1.1.1            py38h0573a6f_0<br>
[conda] numpy                     1.19.2           py38h54aff64_0<br>
[conda] numpy-base                1.19.2           py38hfa32c7d_0<br>
[conda] numpydoc                  1.1.0              pyhd3eb1b0_1</p>

ptrblck: <p>This issue should have been already fixed in the current master branch. Could you update to the nightly release and rerun your code? The upcoming <code>1.10</code> release would also ship with the fix.</p>
Load multiple Image files in dataloader to improve the storage I/O traffic

mahb324: <p>I am trying to run pytorch using 100G dataset with 1M image size,<br>
I wrote a mapstyle dataset to get the image from storage, I have a powerful system (96 core)<br>
so I launched a dataset with 90 workers and 1024 batch size,<br>
but my dataloader is still slow and when look at the i/o traffic does not go beyond 250M/sec<br>
Seems like at the moment each worker only get one index and load one data point,<br>
I tried passing sampler to fetch multiple data points but did not help the performance  stays the same.</p>
<p>Can some one help me to make the worker fetch more files quickly and improve the performance of the dataloader?</p>

ptrblck: <p>Depending on your SSD and its connection 250MB/sec might be expected.<br>
Did you profile your SSD and if so what’s the max. read speed?</p>

mahb324: <p>I tried FIO on my SSD it is around 2G/sec throughput.<br>
but when I go to the dataloader it decreases so much.<br>
I want to know how can I load more image per each worker?<br>
should I use map style or iterative style, to increase the worker reading load?</p>
<p>Thanks</p>

ptrblck: <p>The difference between map style and iterative datasets is that the former uses indices to load samples while the latter can be used for streaming data.<br>
You could search for 3rd party Python libraries yielding a high throughput and check if you could use them. The PyTorch <code>Dataset</code> and <code>DataLoader</code> are utilities to load samples, shuffle and batch them. The data loading itself can be done by any library which would be compatible with multiprocessing.<br>
If you think that loading multiple samples in a single <code>__getitem__</code> call might be beneficial, you could check the <code>BatchSampler</code>, which would pass a batch of indices to this method.</p>
Gpu utilization too low - best practices to build an efficient datapipeline?

scribblepad: <p>Hello,</p>
<p>I’m trying to run a segmentation task, following the below example notebook,<br>
<a href="https://github.com/qubvel/segmentation_models.pytorch/blob/master/examples/cars%20segmentation%20(camvid).ipynb" rel="noopener nofollow ugc">https://github.com/qubvel/segmentation_models.pytorch/blob/master/examples/cars%20segmentation%20(camvid).ipynb</a></p>
<p>The GPU utilization seems to be low, it fluctuates between (0% → 30% -&gt;45%)<br>
I tried to increase the batch_size that can fit in the memory and also try to increase the num_workers in DataLoader. Nothing seems to be helping, I see a lot of time in 0%, hence its using CPU more. What would be the best practice for data pipeline to improve GPU utilization in pytorch?</p>
<p>I’m also suspecting the lower GPU utilization could be because of data augmentation which might be happening on CPU instead of GPU.</p>
<p>Code segment from example notebook:</p>
<p>train_dataset = Dataset(<br>
x_train_dir,<br>
y_train_dir,<br>
augmentation=get_training_augmentation(),<br>
preprocessing=get_preprocessing(preprocessing_fn),<br>
classes=CLASSES,<br>
)</p>
<p>valid_dataset = Dataset(<br>
x_valid_dir,<br>
y_valid_dir,<br>
augmentation=get_validation_augmentation(),<br>
preprocessing=get_preprocessing(preprocessing_fn),<br>
classes=CLASSES,<br>
)</p>
<p>train_loader = DataLoader(train_dataset, batch_size=8, shuffle=True, num_workers=12)<br>
valid_loader = DataLoader(valid_dataset, batch_size=1, shuffle=False, num_workers=4)</p>