akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pytorch version of my numpy code not running vectorized

sleepyo: <p>Hi, I’m currently converting my numpy vectorized code running a fft2 to pytorch, but somehow the result is different. Am I doing it wrong?</p>
<p><strong>numpy version:</strong></p>
<pre><code class="lang-auto">def apply_filter_vector(_filter, arr):
    tc= arr.copy()
    temp = apply_filter(tc, _filter[:, :, None])
    return temp 

def apply_filter(arr, _filter, axes=(0,1)):
    return ifft2(
        np.multiply(ifftshift(1 - _filter), fft2(arr, axes=axes)),
        axes=axes,
    ).real
</code></pre>
<p>pytorch version :</p>
<pre><code class="lang-auto">def apply_filter_vector_pytorch(_filter, arr, cuda=False):
    tc= arr.copy()
    
    if cuda == True:
        temp = apply_filter_pytorch(tc, _filter[:, :, None], cuda=True)        
    else:
        temp = apply_filter_pytorch(tc, _filter[:, :, None], cuda=False) 
    return temp

def apply_filter_pytorch(arr, _filter, axes=(0,1), cuda=False):
    _filter = torch.from_numpy(_filter)
    arr = torch.from_numpy(arr)
    if cuda == True:
        _filter=_filter.cuda()
        arr=arr.cuda()
    result = torch.fft.ifft2(torch.multiply(torch.fft.ifftshift(1.0-_filter),torch.fft.fft2(arr, dim =axes)))
    if cuda == True:
        result= result.cpu()
    result= result.numpy().real
    return result
</code></pre>
<p>the data is a 3D array (2D square images with z axis variation in time).</p>

ptrblck: <p>Could you post the shapes for all inputs, as the code fails in a broadcast:</p>
<pre><code class="lang-python">    np.multiply(np.fft.ifftshift(1 - _filter), np.fft.fft2(arr, axes=axes))

ValueError: operands could not be broadcast together with shapes (24,24,1) (100,100,100) 
</code></pre>
<p>using:</p>
<pre><code class="lang-auto">filt = np.random.randn(24, 24)
arr = np.random.randn(100, 100, 100)

out_np = apply_filter_vector(filt, arr)
</code></pre>
<p>Also, how large is the <code>abs().max()</code> error you are seeing?</p>

sleepyo: <p>hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , Thanks, I already solved it, turns out I only had to add the dim to the <code>ifft2</code>!</p>
<pre><code class="lang-auto">torch.fft.ifft2(
        torch.multiply(
            torch.fft.ifftshift(1.0 - _filter), torch.fft.fft2(arr, dim=axes)
        ),
        dim=axes,
    )
</code></pre>
[onnx export]UserWarning: Exporting aten::index operator with indices of type Byte

bigtree: <p>I am trying to convert a model to onnx in pytorch. But I got warning :<br>
<code>C:\Users\xiaoj\.conda\envs\yolact_clone\lib\site-packages\torch\onnx\symbolic_opset9.py:2691: UserWarning: Exporting aten::index operator with indices of type Byte. Only 1-D indices are supported. In any other case, this will produce an incorrect ONNX graph. warnings.warn("Exporting aten::index operator with indices of type Byte. "</code></p>
<p>After some research i found this <a href="https://github.com/pytorch/pytorch/blob/master/torch/onnx/symbolic_opset9.py#L2700" class="inline-onebox" rel="noopener nofollow ugc">pytorch/symbolic_opset9.py at master · pytorch/pytorch · GitHub</a><br>
But I still not sure how to fix this warning.<br>
Please help.</p>
DDP with AMD ROCm

davidshisui: <p>Does Pytorch 1.8 binaries support distributed data parellel on AMD?<br>
What should I use as the communication backend, nccl or gloo?</p>

pritamdamania87: <p>If you’re using the ROCm binaries, using the “nccl” backend would work since it would transparently use <a href="https://github.com/ROCmSoftwarePlatform/rccl" rel="noopener nofollow ugc">rccl</a> under the hood.</p>

davidshisui: <p>Thank you, but I still get NCCL error while initializing model with model = DDP(model, …)<br>
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:825, internal error, NCCL version 2.7.8<br>
ncclInternalError: Internal check failed. This is either a bug in NCCL or due to memory corruption</p>

jithunnair-amd: <p>The error is unfortunately too generic to indicate the root cause. A few sanity checks:</p>
<ol>
<li>What command did you use to install the pytorch 1.8 binaries?</li>
<li>Are you ensuring that each rank in your distributed model is being assigned to a different device ID?</li>
</ol>
What is a better way to save/load a temporary dataset?

huangdi: <p>Hi.</p>
<p>I’m training a model with two parts: part A randomly generates data, and part B consumes data. Part B reuses the data generated for several times, and the data will then be regenerated by part A. The pseudo code is shown as following:</p>
<pre><code class="lang-auto">for epoch in range(n_epochs):
    if epoch % 10 == 0:
        dataset = partA()
    for i, (data, label) in enumerate(dataset):
        partB(data)  # training partB
</code></pre>
<p>The dataset generated by part A is pretty large (2,000,000x16x16x16 in total), so I think I need to save it somewhere. I try to use json/ujson. Is there any way to do this faster?</p>
<p>Thanks!</p>

TheOraware: <p>I think dataloader can serve as it is stream based loading, load objects one by one using next(iterator)</p>
<blockquote>
<p>num_epochs=20</p>
<p>class partA(Dataset):<br>
def <strong>init</strong>(self):<br>
# data loading<br>
xy = np.loadtxt(‘file.csv’,delimiter=“,”,dtype=np.float32,skiprows=1)</p>
<pre><code>    self.x = torch.from_numpy(xy[:,1:])
    self.y = torch.from_numpy(xy[:,[0]])
    
    self.n_samples = xy.shape[0]

def __getitem__(self,index):
    # dataset[0]
    return self.x[index], self.y[index]
    
def __len__(self):
    return self.n_samples
</code></pre>
<p>dataset= partA()</p>
<p>dataloader= DataLoader(dataset=dataset, batch_size=40,shuffle=True,num_workers=2)</p>
<p>for epoch in range(num_epochs):<br>
for i, (inputs,labels) in enumerate(dataloader):<br>
# forward backward , update weights here<br>
if (i+1) % 10 ==0:<br>
print(f’epoch {epoch+1}/{num_epochs}, step{i+1}/{n_iterations},inputs {inputs.shape}’</p>
</blockquote>
<p>Apologies , if i misunderstood your question. I am very naive in pytorch</p>

huangdi: <p>I’m sorry but it is not the case.</p>
<p>Note that <code>dataset</code> is dynamically generated by part A <strong>during training</strong>. It is not fixed.</p>
<p>In my description, <code>dataset</code> changes its data every 10 epochs. However, in your code, <code>dataset</code> is unchanged.</p>
<p>Thanks for your reply!</p>
How to normalize model parameters

Qizheng_Gu: <p>I want to make weight of certain layer to have a norm of 1, so how can I implement it? Will the following work properly?</p>
<pre><code>class mylin(nn.Module):
    def __init__(self):
        super(mylin, self).__init__()
        self.lin= nn.Parameter(torch.Tensor(20,1,5))
        nn.init.xavier_uniform_(self.lin)
    
    def forward(self, x):
        setattr(self.lin, 'data', self.lin / self.lin.norm(dim=(1,2)))
        x = torch.bmm(self.lin, x)
        return x</code></pre>
Using softplus to learn sigma

vr308: <p>Hi,</p>
<p>I have a model which fits a mean and sigma for some data.</p>
<p>I have parameterised sigma using softplus:</p>
<p><code>q_sigma = torch.nn.Parameter(torch.nn.functional.softplus(torch.randn(n, dim)))</code></p>
<p>but encounter this error because the q_sigma value become a small negative number (-0.0015)</p>
<pre><code class="lang-auto">
  File "/home/vidhi/anaconda3/lib/python3.8/site-packages/torch/distributions/distribution.py", line 53, in __init__
    raise ValueError("The parameter {} has invalid values".format(param))

ValueError: The parameter scale has invalid values
</code></pre>
<p>I’m guessing this is classic numerical instability issue, but is there a better transform to learn sigma?</p>
<p>Thanks.</p>

KFrank: <p>Hi Vidhi!</p>
<p>It’s not clear to me what your are trying to do.  But I will cheerfully<br>
guess!</p>
<aside class="quote no-group" data-username="vr308" data-post="1" data-topic="118765" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vr308/48/22157_2.png" class="avatar"> vr308:</div>
<blockquote>
<p><code>q_sigma = torch.nn.Parameter(torch.nn.functional.softplus(torch.randn(n, dim)))</code></p>
</blockquote>
</aside>
<p>Because you make <code>q_sigma</code> a <code>Parameter</code> I assume that you are<br>
training it with some optimizer.</p>
<p>You initialize it with a positive value (the result of <code>softplus()</code>).</p>
<aside class="quote no-group">
<blockquote>
<p>the q_sigma value become a small negative number (-0.0015)</p>
</blockquote>
</aside>
<p>Even though <code>q_sigma</code> started out positive, the optimizer has know way<br>
of knowing that <code>q_sigma</code> is supposed to remain positive, so at some<br>
point it takes an optimizer step that makes <code>q_sigma</code> negative.</p>
<p>I would suggest instead that you train a parameter that is allowed to<br>
be negative, and then derive a positive <code>q_sigma</code> from it:</p>
<pre data-code-wrap="python"><code class="lang-python">q_parameter = torch.nn.Parameter (torch.randn (n, dim))
</code></pre>
<p>Then later in your code when you need (the positive) <code>q_sigma</code>, simply<br>
derive it from <code>q_parameter</code> (with something like <code>softplus()</code> that<br>
guarantees a positive result).</p>
<p>The point is, instead of training <code>q_sigma</code> and trying to figure out some<br>
way to constrain it to be positive – even though pytorch doesn’t naturally<br>
do constrained optimizations – you train the possibly-negative (and<br>
unconstrained) <code>q_parameter</code>, but then map it to an always-positive<br>
<code>q_sigma</code>.</p>
<p>Good luck.</p>
<p>K. Frank</p>

vr308: <p>Thank you, it does indeed make sens</p>
Taking the element-wise average of a 2-D tensor

cpsanders: <p>Hello, I have a tensor <code>O_k</code> of shape <code>[M,4,2]</code>. I need to get an average across <code>M</code> for these tensors element-wise to obtain <code>ave_O_k</code> of shape <code>[4,2]</code>. I initially tried <code>sum(O_k)/len(O_k)</code>, but this is very inefficient, I imagine it would be better to use torch’s <code>torch.mean(O_k)</code>, but this function seems to take the row-wise average of <code>O_k</code>.</p>

AlphaBetaGamma96: <p>Have you tried doing <code>torch.mean(O_k, dim=0)</code>?</p>
<pre><code class="lang-auto">x = torch.randn(1000,4,2)
mean = torch.mean(x, dim=0)
mean.shape() #returns torch.Size([4, 2])
</code></pre>
Two processes: inference and backward running at the same time?

metal_ball: <p>My goal is to achieve real-time inference and backward in PyTorch.</p>
<p>Suppose I have a stream of input tensors arriving at different times. I created two torch.multiprocessing.Process called inference_process and backward_process.<br>
I would like to make these two processes run simultaneously: after inference_process forwards an input tensor, it will send the output tensor to the backward_process. The backward_process will compute the gradient, while the inference_process continues to forward the next tensor.</p>
<p>In inference_process, I want to do this:<br>
output_buffer = my_model(input_buffer)<br>
output_buffer_stream.put(output_buffer)</p>
<p>And in backward_process, I want to get the output buffer from the inference_process:<br>
output_buffer = output_buffer_stream.get()</p>
<p>output_buffer_stream is a torch.multiprocessing.Queue()</p>
<p>However, I get the following error at output_buffer_stream.put(output_buffer):<br>
<code>RuntimeError: Cowardly refusing to serialize non-leaf tensor which requires_grad, since autograd does not support crossing process boundaries. If you just want to transfer the data, call detach() on the tensor before serializing (e.g., putting it on the queue).</code></p>
<p>Any ideas on how to solve it? How can I share a tensor with gradient between two processes? Should I try torch.distributed?</p>

metal_ball: <p>I suspect the reason is that the computation graph(<code>.grad_fn</code>) cannot be serialized. So tensor with a computation graph cannot share between processes.</p>
Training Neural Networks with Posit<8,1>

rodrilag: <p>Hello, I’m trying to train Neural Networks using format datatype <strong>Posit8</strong> in Pytorch.</p>
<p>To start, I’ve tried to train LeNet5 with MNIST dataset.<br>
My problem is located when I try to get the train and valid datasets. Firstly, I get them in tensors of <strong>float32</strong> datatype with the next code:</p>
<pre><code class="lang-auto">transforms = transforms.Compose([transforms.Resize((32, 32)),
                                 transforms.ToTensor()]) #Variable que tendrá las especificaciones de los tensores de datos

# download and create datasets
train_dataset = datasets.MNIST(root='mnist_data', 
                               train=True, 
                               transform=transforms,
                               download=True) #Descargamos y creamos el dataset de entrenamiento

valid_dataset = datasets.MNIST(root='mnist_data', 
                               train=False, 
                               transform=transforms) #Descargamos y creamos el dataset de validación

# define the data loaders
train_loader = DataLoader(dataset=train_dataset, 
                          batch_size=BATCH_SIZE, 
                          shuffle=True)  #Definimos el dataset de entrenamiento

valid_loader = DataLoader(dataset=valid_dataset, 
                          batch_size=BATCH_SIZE, 
                          shuffle=False)  #Descargamos y creamos el dataset de validación
</code></pre>
<p>After getting the datasets, I use softposit in order to change the datatype. I didn’t find a way to change the entire tensor, because softposit only change the dataype of a number, not a tensor or array. So, to change the datasets I used 5 <em>for loops</em> to get each one of the numbers of the tensor and change the datatype. Here is my code:</p>
<pre><code class="lang-auto">def TensorToP8(X_array, Y_array, narray):
    c1=0
    c2=0
    c3=0
    c4=0
    c5=0
    k=0

    for i1,y in narray:
        for i2 in i1:
            for i3 in i2:
                for i4 in i3:
                    for i5 in i4:
                        X_array[c1][c2][c3][c4][c5] = sp.posit8(np.double(i5))
                        c5+=1
                    c4+=1
                    c5=0
                c3+=1
                c4=0
            c2+=1
            c3=0
        for y2 in y:
            Y_array[c1][k] = y2.numpy()
            k+=1
        c1+=1
        c2=0
        k=0
    return X_array,Y_array                    
</code></pre>
<p>And this:</p>
<pre><code class="lang-auto">X_train= np.ndarray([1875,32,1,32,32])
Y_train = np.ndarray([1875,32])
X_validate = np.ndarray([313,32,1,32,32])
Y_validate = np.ndarray([313,32])

X_train, Y_train = TensorToP8(X_train,Y_train,train_loader)

X_validate, Y_validate = TensorToP8(X_validate,Y_validate, valid_loader)
</code></pre>
<p>My problem is the way I cast the dataset, because when I try to use it, the datatype of the arrays is <strong>float64</strong>, so it means that the np.ndarray I use doesn’t save the <strong>posit</strong> datatype.</p>
<p>Is there a better way of cast the dataset to <strong>Posit8</strong> ?</p>
<p>Thank you</p>
Correct way to calculate Dice Loss for binary segmentation

marcoramos: <p>On some examples I see Dice Loss calculated leaving the batch out (inputs.reshape(inputs.shape[0], -1):</p>
<pre><code class="lang-auto">def dice_loss(inputs, target):
    num = target.size(0)
    inputs = inputs.reshape(num, -1)
    target = target.reshape(num, -1)


    intersection = (inputs * target).sum(1)
    union = inputs.sum(1) + target.sum(1)
    dice = (2. * intersection) / (union + 1e-8)
    dice = dice.sum() / num
    return 1 - dice
</code></pre>
<p>Other examples they flatten the entire tensor (inputs.reshape(-1)):</p>
<pre><code class="lang-auto">def dice_loss(inputs, target):
   
    inputs = inputs.reshape( -1)
    target = target.reshape(-1)


    intersection = (inputs * target).sum()
    union = inputs.sum() + target.sum()
    dice = (2. * intersection) / (union + 1e-8)
    dice = dice.sum() / num
    return 1 - dice
</code></pre>
<p>and I am confused</p>
Embed_dim , hidden_size , num_subspace, num_heads,emphasis

TheOraware: <p>I am reading one of notebook DAE for tabular data , i know AE but not DAE. In AE i did not see any embed_dim and hidden_size parameter. I just saw hidden layer (bottleneck) layer in AE and this hidden layer is a latent space which is normally less than in number than original input layer.</p>
<p>But here while seeing DAE from one of notebook i see hidden_size , not sure what is it ? If it is hidden layer as i learnt in AE then why it is bigger than original input layer?</p>
<blockquote>
<p>DAE_CFG = {‘batch_size’ : 384,<br>
‘init_lr’ : 3e-4,<br>
‘lr_decay’ : .998,<br>
‘noise_decay’ : 0.999,<br>
‘max_epochs’ : 600,<br>
‘save_freq’:50,<br>
‘hidden_size’:1024,<br>
‘num_subspaces’:8,<br>
‘embed_dim’:128,<br>
‘num_heads’:8,<br>
‘dropout’:0,<br>
‘feedforward_dim’:512,<br>
‘emphasis’:.75,<br>
‘task_weights’:[len(fts_categorical), len(fts_continuous)],<br>
‘mask_loss_weight’:2,<br>
‘prob_categorical’ : 0.5,<br>
‘prob_continuous’ : 0.5,}</p>
</blockquote>
<p>Any could please share some info about</p>
<p>What is hidden_size?<br>
What is embed_dim?<br>
What is num_subspace?<br>
Whats is num_heads?<br>
Whats is emphasis,mask_loss_weight,prob_categorical,prob_continuous?</p>
CosineAnnealingLR step size (T_max)

Karan_Chhabra: <p>Hi,<br>
I was trying to use the cosineAnnealing Learning rate but I was confused about what should be the <strong>T_max</strong> parameter be.</p>
<p>Whether it should be <em>number of epochs</em>, <em>length of train_loader</em>  or multiple of the two?</p>
<p><code>torch.optim.lr_scheduler.</code> <code>CosineAnnealingLR</code> (<em>optimizer</em> , <em>T_max</em> , <em>eta_min=0</em> , <em>last_epoch=-1</em> , <em>verbose=False</em> )</p>

Karan_Chhabra: <p>Also can someone help me understand how it is different from CyclicLR</p>

ptrblck: <p>The <a href="https://pytorch.org/docs/stable/optim.html?highlight=cosineannealinglr#torch.optim.lr_scheduler.CosineAnnealingLR">docs</a> give you the applied formula and show how <code>T_max</code> is used. In particular it’s used to divide the current epoch by its value, which would thus anneal the change in the learning rate and end with the max. learning rate.</p>
<p><code>CyclicLR</code> cycles the learning rate between two boundaries with a constant frequency.</p>

ntomita: <p>The original implementation in the paper updates <code>T_cur</code> at every iteration, which means your <code>scheduler.step()</code> should be located at the end of every batch iteration, instead of every epoch, and thus <code>T_max</code> value should be <code>num_epochs_before_restart * len(dataloader)</code> to make it consistent with pytorch implementation of <code>_LRScheduler</code>. Haven’t tested if updating lr every iteration is empirically recommended though.</p>
<p>Ref:</p>
<blockquote>
<p>SinceT_cur is updated at each batch iteration t, it can take discredited values such as 0.1, 0.2, etc.</p>
</blockquote>
My learnable vector cannot be updated

111462: <p>I want a learnable vector in ResNet to get<br>
In <strong>init</strong> function of ResNet, I insert some code.</p>
<pre><code class="lang-auto">        weight = torch.rand((5, 1), device='cuda:0')
        self.weight = weight.requires_grad_(True)
</code></pre>
<p>And in <strong>forward</strong> function, I use matrix multiplication between out and self.weight<br>
out is a result of original ResNet. (self._forward_impl)</p>
<pre><code class="lang-auto">    def forward(self, x):
        out = self._forward_impl(x)                 #(batch_size * 5, 1)

        out = out.view(-1, 5)                       #(batch_size, 5)
        out = torch.mm(out, self.weight)            #(batch_size, 5)x(5, 1)=(batch_size, 1)

        return out
</code></pre>
<p>When I get a loss value using Focal Loss and run loss.backward(), it <strong>doesn’t change the self.weight</strong> value and sometimes <strong>self.weight.grad is None</strong>…</p>
<pre><code class="lang-auto">print(self.weight)
print(self.weight.grad)
</code></pre>
<pre><code class="lang-auto">tensor([[0.2026],
        [0.6027],
        [0.2351],
        [0.0260],
        [0.2805]], device='cuda:0', requires_grad=True)
tensor([[-0.1866],
        [-0.2713],
        [-0.2793],
        [-0.1892],
        [-0.1899]], device='cuda:0')
tensor([[0.2026],
        [0.6027],
        [0.2351],
        [0.0260],
        [0.2805]], device='cuda:0', requires_grad=True)
tensor([[-0.2546],
        [-0.3381],
        [-0.3341],
        [-0.2421],
        [-0.2273]], device='cuda:0')
</code></pre>
<p>I don’t know why self.weight cannot be updated…</p>

sio277: <p>Wrap your weight with <code>nn.Parameter</code>. After that, the tensor is registered as one of model parameters. Otherwise, it is not visible to <code>model.parameters()</code>, which are optimizer tries to update.</p>

111462: <p>Thank you! I change my code as you said, now it works well!</p>
Torchviz showing layer names with nn.Module

barclay742: <p>Hey, is there a way on getting torchviz to show the layer names while using nn.Module,<br>
All the examples are I’ve found are using nn.Sequential! for example <a href="https://github.com/szagoruyko/pytorchviz/blob/master/examples.ipynb" rel="noopener nofollow ugc">here</a> where the blue boxes show W0.weight, W0.bias etc.</p>

floriandonhauser: <p>What exactly are you trying to achieve?<br>
Are you trying to have a sequential model where some layers are custom?<br>
Or will the whole model not use sequential?<br>
What might be interesting is to look at how torchviz gets the names.<br>
What I e.g. found is the method <code>get_var_name</code> in dot.py<br>
You then have to make sure that the names you want are returned by the functions (or stored in the variables) that torchviz is using.</p>
How to use VOCSegmentation from torchvision?

seyeeet: <p>I have downloaded the VOC2012 dataset and want to use <code>VOCSegmentation </code> for my dataloader, but what ever I do I get the error <code>RuntimeError: Dataset not found or corrupted. You can use download=True to download it </code>.</p>
<p>my dataset looks like this:</p>
<pre><code class="lang-auto">/VOCdevkit
└── VOC2012
    ├── Annotations
    ├── ImageSets
    │   └── Segmentation
    ├── JPEGImages
    ├── SegmentationObject
    └── SegmentationClass
</code></pre>
<p>can someone tell me how should I set us the VOCSegmentation  inputs? I have it like this but it is not working.</p>
<pre><code class="lang-auto">dataset = torchvision.datasets.VOCSegmentation(root='/path/VOCdevkit/', year='2012', image_set='trainval', download=False)
</code></pre>
<p>the complete error that I am getting is:</p>
<pre><code class="lang-auto">~/anaconda3/envs/pytorch/lib/python3.6/site-packages/torchvision/datasets/voc.py in __init__(self, root, year, image_set, download, transform, target_transform, transforms)
    103         if not os.path.isdir(voc_root):
    104             raise RuntimeError('Dataset not found or corrupted.' +
--&gt; 105                                ' You can use download=True to download it')
    106 
    107         splits_dir = os.path.join(voc_root, 'ImageSets/Segmentation')

RuntimeError: Dataset not found or corrupted. You can use download=True to download it
</code></pre>

seyeeet: <p>solution is <a href="https://github.com/pytorch/vision/issues/3707" rel="noopener nofollow ugc">here</a></p>
Kfold Pytorch Custom Dataset

Jam: <p>I implemented a custom dataset class in PyTorch.<br>
Currently I am reading the data likes this:</p>
<pre><code class="lang-auto"># paths
paths =  [x for x in glob.glob("polyp_db_sequences/*/*") if os.path.isdir(x)] #all paths that contain images

# splits
train_paths = random.sample(paths, int(len(paths) * 0.8)) #input for train dataset
test_paths = [x for x in paths if x not in train_paths] #input for test dataset

train_set = MyDataset(train_paths)
train_loader = DataLoader(train_set, batch_size = 16)
</code></pre>
<p>However, instead of the 80/20 splitting above, I’d like to use Kfold to split the data into train and test set but since I only have the paths, I don’t know how to do that.</p>

ElPolloDiablo: <p>Hi <a class="mention" href="/u/jam">@Jam</a> ,</p>
<p>I think the following should do the trick for you:</p>
<pre><code class="lang-auto">from sklearn.model_selection import KFold
import pandas as pd
import os

myData = os.listdir(myrootdirectory) 
kf = KFold(n_splits=5, shuffle=True)

train_index, test_index = next(kf.split(myData), None)
train_data, test_data = myData [train_index], myData [test_index]
</code></pre>
<p>If you use pandas dataframes, you can then easily pass the train/test dataframe object to your custom Dataset <code>MyDataset</code>. Dataframes make sampling and shuffling much easier in the training cohort.</p>
<p>Hope this helps!</p>
<p>Edit: A split number of n=5 in KFold corresponds to a split ratio 4:1, a.k.a. 20/80</p>

Jam: <p>Thanks a lot! This helped a lot, I could make it work now.</p>
Ignore_index in Adaptive Softmax?

luuksuurmeijer: <p>I am training a large scale neural language model with pytorch and would like to use an Adaptive Softmax (because my vocabulary is very large) function over the outputs, this is provided in pytorch by <code>torch.nn.AdaptiveSoftMaxWithLoss</code>. This function computes the loss for me as well as the adaptive softmax. However, my input data is padded to a certain length with a padding token. Normally when I use <code>CrossEntropyLoss</code> or <code>NLLLoss</code> there is the parameter <code>ignore_index</code> which allows me to ignore the padding token when computing the loss. It does not seem like such a parameter is available for the adaptive softmax implementation, how do I work around this?</p>
Will eval mode disable this dropout?

Omroth: <p>In my descendent of torch.nn.Module this is the start of my forward function:</p>
<pre><code class="lang-auto">def forward(self, x):
            dropout_out = self.dropout(x.float())
            layer1_out = self.linear1(dropout_out)
</code></pre>
<p>Will self.dropout automatically do nothing if self.eval() has been called?</p>

thomasjo: <p>Assuming that <code>self.dropout</code> is <a href="https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html#torch.nn.Dropout" rel="noopener nofollow ugc"><code>torch.nn.Dropout</code></a>, then your <code>dropout_out</code> variable will be equal to <code>x.float()</code> in evaluation mode. During evaluation the dropout function is an identity function.</p>
<p>If you want to apply dropout during evaluation (e.g. to implement <em>Monte Carlo Dropout</em> or similar), then you can use the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html#torch.nn.Dropout" rel="noopener nofollow ugc"><code>torch.nn.functional.dropout</code></a> function instead of the module. The functional form has a boolean <code>training</code> parameter that allows you to explicitly control whether to operate in training or evaluation mode.</p>

Omroth: <p>it is torch.nn.Dropout.  Thanks Thomas.</p>
Building libtorch on different versions of Linux

RichardSmithLab: <p>After trying with Ubuntu 20, Mint 20, and Ubuntu 18, I was unable to get any of the pre-built libtorch packages to work. My target application (MorphoGraphX) only needs libc10_cuda.so  libc10.so  libtorch_cpu.so  libtorch_cuda.so  libtorch.so so that it can run traced models.</p>
<p>I was able to get things working on all those platforms by compiling it from source. However, the package for Mint 20 did not work on Ubuntu 20, and vice-versa, and I even have a case where a package built for Ubuntu 18 does not work on another Ubuntu 18 machine that has the same graphics card.</p>
<p>The error is most commonly about no kernel image being available. I have tried compiling with the architecture flags for the specific card by setting TORCH_CUDA_ARCH_LIST, but it doesn’t seem to make much difference, except that startup is faster. It works on the build machine, but not the target with the same OS and graphics card.</p>
<p>Anyone know what might be causing this problem?</p>
How to calculate L2 distance between torch.Tensor without Variable?

xwzy: <p>Is there a function to do this?</p>

Jing: <p><code>torch.norm(tensor, 2)</code></p>

xwzy: <p>I mean calculate between 2 tensors.<br>
e.g.<br>
(1, 0) and (1 ,1)  is  sqrt((1-1)^2 + (1-0)^2)</p>

xwzy: <p>torch.dist() is ok, I found answer.</p>

Jing: <p>sorry for my carelessnesss and misunderstanding .Very happy you got the right answer!</p>

mikeee: <p>torrch.dist(x, y, p) is the same as torch.norm(x - y, p) …</p>

Joonho_Park: <p>How can you calculate the distances for the list of points pair such as a=[ [1,1,1], [2,2,2]], b=[[3,3,3], [4,4,4]] ?</p>
Tensor type cast in c++

Xonobo_Xonobo: <p>In c++ side I really get hard time to find trivial things. I need to cast a LongTensor to FloatTensor. Can anyone guide me to find the solution in the c++ api documentation?</p>

Xonobo_Xonobo: <p>Here is the <a href="https://pytorch.org/cppdocs/api/function_namespaceat_1a94c746092b007b77fec572f6ba6dfe17.html#_CPPv4N2at11_cast_FloatERKN2at6TensorEb" rel="noopener nofollow ugc">link</a> and sample usage I guess:</p>
<p><code>auto my_float_tensor = at::_cast_Float(my_long_tensor);</code></p>
<p>the underscore naming of the function rises a question whether this function also changes the data type of the <code>my_long_tensor</code> (like in-place notation) but the input tensor is <code>const</code> so it can not.</p>
<p>The c++ api documentation needs a lot, I believe.</p>

Matej_Kompanek: <p>Just use</p>
<pre><code class="lang-auto">torch::Tensor longTensor = torch::randint(100, { 1 }, torch::kLong);
torch::Tensor floatTensor = longTensor.toType(torch::kFloat);
</code></pre>
<p>There are some examples over <a href="https://pytorch.org/cppdocs/notes/tensor_creation.html" rel="noopener nofollow ugc">here</a></p>
[W NNPACK.cpp:80] Could not initialize NNPACK! Reason: Unsupported hardware. error message

CracKCatZ: <p>Hey guys ,<br>
I’am running into this error. I don’t really know if this is only one error messsage.</p>
<p><strong>My OS:</strong><br>
Kali Linux</p>
<p><strong>Pyhton version:</strong><br>
3.7</p>
<p><strong>pytorch,pytorch-lightning, torchaudio, torchmetrics, torchvision version:</strong><br>
(I’ve installed pytorch, torchaudio and torchvision thru this command for cpu only<br>
conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 -c pytorch, you can find this installation command on <a href="http://pytorch.org" rel="noopener nofollow ugc">pytorch.org</a>)</p>
<p>pytorch==1.7.1<br>
pytorch-lightning==1.2.8<br>
torchaudio==0.7.2<br>
torchmetrics==0.2.0<br>
torchvision==0.8.2</p>
<p><strong>My error message:</strong></p>
<pre><code class="lang-auto">/home/anaconda3/envs/speech/lib/python3.7/site-packages/torch/functional.py:516: UserWarning: stft will require the return_complex parameter be explicitly  specified in a future PyTorch release. Use return_complex=False  to preserve the current behavior or return_complex=True to return  a complex output. (Triggered internally at  /opt/conda/conda-bld/pytorch_1607370152014/work/aten/src/ATen/native/SpectralOps.cpp:653.)
  normalized, onesided, return_complex)
/home/anaconda3/envs/speech/lib/python3.7/site-packages/torch/functional.py:516: UserWarning: The function torch.rfft is deprecated and will be removed in a future PyTorch release. Use the new torch.fft module functions, instead, by importing torch.fft and calling torch.fft.fft or torch.fft.rfft. (Triggered internally at  /opt/conda/conda-bld/pytorch_1607370152014/work/aten/src/ATen/native/SpectralOps.cpp:590.)
  normalized, onesided, return_complex)
[W NNPACK.cpp:80] Could not initialize NNPACK! Reason: Unsupported hardware.
</code></pre>
<p>After this error message is being displayed it is sort of creating an endless loop which is printing out every second a new row. So my program is being completely crashed because of this error message.</p>
<p>Is someone able to help me out with that or know what this error message is caused by?<br>
Thank’s for every suggestion and help in advance:)</p>

ptrblck: <p>Double post with potential <a href="https://discuss.pytorch.org/t/bug-w-nnpack-cpp-80-could-not-initialize-nnpack-reason-unsupported-hardware/107518/17">answer</a>.</p>
Custom gradient averaging with DDP?

sharvil: <p>DDP averages gradients by dividing by world size. Is there any mechanism (current or planned) to run a user-defined function to scale gradients instead of the default DDP behavior?</p>
<p>In my case, I have variable and uneven batch sizes on each replica and need to compute the average by <em>global</em> batch size.</p>

osalpekar: <p>The default for DDP is to use allreduce and average the gradients using the world size. You can choose to allreduce the gradients yourself as specify the reduction op yourself, but in this case you lose out on the better perf of DDP and allreduce will still not accept a custom UDF for the reduction op (though you can use the SUM op and average gradients however you want).</p>
<p>On a slightly different note, the DDP <a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel.join" rel="noopener nofollow ugc">join</a> API can help you better handle uneven inputs across ranks.</p>

sharvil: <p>Yeah, I know I could just write my own gradient averaging but I lose out on the perf wins. And it seems kind of silly to maintain a fork of DDP just to change the reduction from mean to sum. I’ve looked at the join method on DDP but it doesn’t apply to my issue. Thanks for the reply!</p>

osalpekar: <p>Sounds good! We are also working on a broader effort to make DDP more modular and configurable across the board. It seems like this definitely an interesting direction to support - feel free to make an issue on our GitHub repo or add to this RFC tracking the DDP configurability effort: <a href="https://github.com/pytorch/pytorch/issues/37002" class="inline-onebox" rel="noopener nofollow ugc">[RFC] Modularize DistributedDataParallel · Issue #37002 · pytorch/pytorch · GitHub</a></p>

rvarm1: <p>We have landed gradient compression communication hooks to be used with pytorch DDP in 1.8: <a href="https://pytorch.org/docs/stable/ddp_comm_hooks.html" class="inline-onebox" rel="noopener nofollow ugc">DDP Communication Hooks — PyTorch 1.8.1 documentation</a>. This seems like it should support your use case quite nicely, as you would be able to run a custom UDF for the gradient reduction instead of the allreduce.</p>
Is it possible to include computed gradient values into the loss?

paganpasta: <p>I have a sequence of operation performed on x to yield y_. A, B, C are learnable parameters of my system.</p>
<pre><code class="lang-auto">y_ = A(B(C(x)))
</code></pre>
<p>Is it possible to define a loss such as:</p>
<pre><code class="lang-auto">L = (y - y_) + sum(dy/dC)
</code></pre>
<p>And how to do so? Currently, the backward call requires a scalar value. Also, how to do such that I don’t overwrite the actual update gradients dL/dweights while computing dy/dC?<br>
<a class="mention" href="/u/alband">@albanD</a> any ideas?</p>
About Dropout random and How to disenable the Dropout in train step？

zs963048949: <p>Firstly, I found the training log was changed when I applied the dropout layer. I ensured other random was fixed by setting rand seed. How can I fix the dropout random?<br>
Secondly, when I want to enable dropout after <code>net.eval()</code>, I can use <code>net.apply(apply_dropout)</code>. The apply_dropout function is:</p>
<pre><code class="lang-auto">def apply_dropout(m):
    if type(m) == nn.Dropout:
        m.train()
</code></pre>
<p>However, when I wanted to disable the dropout layer in the training step using inverse operation, it did not work. Specifically, I used <code>net.apply(de_apply_dropout)</code> after <code>net.train()</code>. The de_apply_dropout function is:</p>
<pre><code class="lang-auto">def de_apply_dropout(m):
    if type(m) == nn.Dropout:
        m.eval()
</code></pre>
<p>The purpose of the above operation is that I want to enable the dropout layer in some epochs and disable it in other epochs when the network is trained. How can I solve it?<br>
Thanks in advance.</p>

ptrblck: <p>Your code seems to work correctly and I can use it to enable or disable the dropout layer:</p>
<pre><code class="lang-python">def apply_dropout(m):
    if type(m) == nn.Dropout:
        m.train()

def de_apply_dropout(m):
    if type(m) == nn.Dropout:
        m.eval()

class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.lin = nn.Linear(10, 10)
        self.drop = nn.Dropout()
        
    def forward(self, x):
        x = self.lin(x)
        x = self.drop(x)
        return x

model = MyModel()
x = torch.randn(100, 10)
out = model(x)
print((out==0.0).sum())
&gt; tensor(480)

model.eval()
out = model(x)
print((out==0.0).sum())
&gt; tensor(0)

model.train()
out = model(x)
print((out==0.0).sum())
&gt; tensor(501)

model.apply(de_apply_dropout)
out = model(x)
print((out==0.0).sum())
&gt; tensor(0)

model.apply(apply_dropout)
out = model(x)
print((out==0.0).sum())
&gt; tensor(475)
</code></pre>

zs963048949: <p>Thank you.<br>
I got it.</p>
Pytorch Async All-Reduce

sangheonlee: <p>Hi,</p>
<p>In pytorch, they provide the capability of async allreduce. And How they can control this.<br>
For example, fastest GPU0 can ready gradient of bucket size, and launch the allreduce. But other GPU are not ready for it. And how GPU0 can wait until other GPU can launch the allreduce.</p>
<p>As I know, In multi-GPU system, All-GPU launches the allreduce kernel simultaneously, But they can do asynchronously execute the allreduce. Or it control the nccl…? or pytorch.</p>
Doubt in backward() implementation in customize activation function

cbd: <p>(1) If, i write “output = torch.square(input)” in customize activation forward() in below code, i think it doesn’t require backward() implementation as it used torch function<br>
(2) But if i write “output = input * input”, it requires the implementation of backward().  Is below code is fine for customize function “output = input * input”?</p>
<p>Let me know if i am missing something.</p>
<pre><code>class my_act(Function):

# Note that both forward and backward are @staticmethods
@staticmethod
 def forward(ctx, input, weight, bias=None):
    ctx.save_for_backward(input)
    output = input * input
    return output

@staticmethod
def backward(ctx, grad_output):
   
    input = ctx.saved_tensors
    grad_input = grad_output * ( 2 * x) # DERIVATIVE OF X*X MULTIPLY WITH GRAD_OUTPUT
  
    return grad_input</code></pre>

albanD: <p>Hi,</p>
<p>Why are you using a custom Function here?<br>
If you just want the autograd to compute the gradients for you, just put that in a regular python function.</p>
<p>You should only use a custom Function if you use code that is not handled by the autograd or you want something that is not the gradient.</p>

cbd: <p>I just want to implement Kernel Density Estimator (KDE) as there is no torch function for that and if i use scikit kde, i need to detach tensor which don’t allow me to gradient flow. So for that i just clearing my doubt with demo program i shown.</p>
<p>I am very thankful if you have some idea regarding KDE. Is there any in built pytorch function OR i need to implement custom function for KDE.</p>
<p>If it required to implement custom function for KDE, have you some idea or link if it is already implemented?</p>

albanD: <p>There is no KDE implementation in core that I know of. You might want to google around as someone else might already have implemented that though in a third party lib.</p>
<p>Otherwise, for this, you indeed need a custom Function.<br>
So you will need to make sure you have the backward formula written down for the the KDE evaluation.<br>
And in that case, yes you sample above is correct.</p>

cbd: <p>OK. Thanks a lot for your guidance.</p>
How to derefernce contents of torch::nn::ModuleList?

Fetullah_Atas: <p>HI ,<br>
I am trying to use this container called  <code>torch::nn::ModuleList</code> it is frustrating that a simple container is implemented this complicated with lack of proper examples. I have looked into API, tests but I cannot figure out how to simply extract the elements inside <code>torch::nn::ModuleList</code>.<br>
What I want is very simple, fill a few Conv2D modules into a  <code>torch::nn::ModuleList</code> inside constructor and use the contents of this list in <code>forward</code> method.</p>
<pre><code class="lang-auto">   std::vector&lt;int&gt; mlp_({32, 32, 64});
  torch::nn::ModuleList mlp_convs_;
    for (int i = 0; i &lt; mlp_.size(); i++) {
      mlp_convs_-&gt;push_back(
     torch::nn::Conv2d((torch::nn::Conv2dOptions(last_channel_, mlp_.at(i), 1))));
      last_channel_ = mlp_.at(i);
    }
</code></pre>
<p>After I fill this <code>mlp_convs_</code></p>
<p>I want t extract the conv module but no way to get it done.</p>
<pre><code>for (int i = 0; i &lt; mlp_.size(); i++) {

  torch::nn::Conv2d crr_conv = *mlp_convs_-&gt;ptr(i); // fails

  auto crr_conv = *mlp_convs_[i]-&gt;as&lt;torch::nn::Conv2d&gt;(); // gets it but crr_conv is torch::nn::Conv2dImpl and I need torch::nn::Conv2d



}
</code></pre>
<p>Edit;<br>
I end up just using a std::vector for now.</p>
Packed_accessors in pytorch c++

Ipsum: <p>Hello, I am wondering if there is a way to used packed_accessors to quickly index tensors on the cuda without having to resort to frequent, hand-written cuda kernel calls.  For instance, if I have a 1-D tensor with 100 elements, and I need to get to the 50th element, using a packed_accessor in a kernel call is much faster than writing tensor[50] in my regular c++ function (along with the appropriate type casting).  I am sorry if this is an overly simplistic question, but I am new to c++ and am hoping to avoid writing many simple cuda kernels to simply access values in a time-sensitive manner.  Thank you for any help!</p>
Why does the grid_sample function limit the input values to [-1, 1]?

sunshk1227: <p>The pytorch <a href="https://pytorch.org/docs/1.4.0/nn.functional.html?highlight=grid_sample#torch.nn.functional.grid_sample" rel="noopener nofollow ugc">doc</a> says that “grid specifies the sampling pixel locations normalized by the input spatial dimensions. Therefore, it should have most values in the range of [-1, 1]”. But why do they let <code>grid_sample</code> function specify the normalized pixel locations?</p>
<p>Thank you very much for any help!</p>
torchtext0.9.0,LanguageModelingDataset does not exist

copy: <p>The model (LanguageModelingDataset) was available in the previous version, but in version 0.9, this function is no longer available. How can it be replaced.</p>
<p>AttributeError: module ‘torchtext.datasets’ has no attribute ‘LanguageModelingDataset’</p>

ptrblck: <p>You could access this class via <code>torchtext.legacy.datasets.LanguageModelingDataset</code>.</p>

copy: <p>Thank you for your help</p>
RAM Leak in IterableDataSet

tantheta01: <p>I have created a simple iterable dataset which extracts frames from a video and provides them for training. It works just fine but when I try to iterate on the dataset I observe that my CPU RAM is increasing after every iteration.</p>
<pre><code class="lang-auto">def frame_stacking(dpath, T, overlap=True):
    #T=5
    d_list = os.listdir(dpath)
    for filename in d_list:
        vidcap = cv2.VideoCapture(os.path.join(data_path, filename))
        success = True
        frame_set =[]
        while success:
            success,image = vidcap.read()

            if overlap:
                if success:
                    gray = cv2.cvtColor(image[:,40:-40, :], cv2.COLOR_BGR2GRAY)
                    gray = cv2.resize(gray, (128,128)).astype(float)/255.0
            
                    if len(frame_set)==T:
                        yield np.stack(frame_set,axis=0)  # the yield statement
        
                        tmp_frame=[]
                        for ii in range(T-1):
                            tmp_frame.append(frame_set[ii+1])
                        tmp_frame.append(gray)
                        frame_set=tmp_frame
                        tmp_frame=[]
                        
                    else:
                        frame_set.append(gray)
                
                else:
                    frame_set=[]

class MyDataset(IterableDataset):
    def __init__(self, data_root, NumFrames, isOverlapAllowed=True):
        self.samples = cycle(frame_stacking(data_root,5, isOverlapAllowed))
    
    def __iter__(self):
        return self.samples
</code></pre>
<p>Does anyone know why this could be? Am I making a mistake in creating the Dataset? I know it is caused by the dataset as it also happens in a simple situation like this.</p>
<pre><code class="lang-auto">for batch_idx, (data) in enumerate(dataloader,0):
    print("PERCENTAGE RAM USED", psutil.virtual_memory().percent)

</code></pre>

ptrblck: <p>Are you seeing the same increased memory usage, if you replace <code>cv2.VideoCapture</code> and <code>vidcap.read()</code> with random numpy arrays? This would help to narrow down, which part of the code is responsible for the increased memory usage.</p>
How to efficiently perform random array assignments and stack the results?

111414: <p>Basic requirement: suppose I have an array of [9, 8, 7], and I can generate two random assignment indices arrays, suppose they are [1, 1, 0] and [0, 2, 1], so the assignment results are [8, 8, 9] and [9, 7, 8], and finally I want to get the stacked result: [[8, 8, 9], [9, 7, 8]].</p>
<p>I can achieve the above function by the following code with a for-loop:</p>
<pre><code class="lang-auto">import torch
my_array = torch.Tensor([9, 8, 7])
for i in range(2):  # for each layer
    my_assignment_indices = torch.randint(0, 3, size=(3,))  # for each location, randomly choose a source index
    current_assignment_array = my_array[my_assignment_indices].unsqueeze(0)  # move (assign)
    if i == 0:
        result = current_assignment_array  # the first layer, create result
    else:
        result = torch.cat([result, current_assignment_array])  # the subsequent layers, stack them together
print(result)  # output my result
</code></pre>
<p>In practice, I have two parameters: N (array length), K (layer number).<br>
Notice that each index is randomly chosen from [0, …, N-1].</p>
<p>Now I am confused by the following two problems:</p>
<ol>
<li>How can I perform the above process efficiently with very large N and K on CPU or GPU?</li>
<li>How can I randomly assign K different elements to each location? The above code does not guarantee that each two indices for a location must be different.</li>
</ol>
<p>I’m waiting for a better solution, thank you for watching my problems <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:">.</p>

111414: <p>It seems that this problem is very hard… I tried several times and failed. <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>

ptrblck: <ol>
<li>
<p>Instead of using <code>torch.cat</code> you could append the results to a <code>list</code> and convert it to a tensor afterwards.</p>
</li>
<li>
<p>You could use <code>torch.randperm</code> and slice the first <code>k</code> indices. This would make sure that all indices are unique but shuffled.</p>
</li>
</ol>
Why i can not transform a tf.flags to torch type?

wuChristiana: <pre><code class="lang-auto">import tensorflow as tf

flags = tf.flags
</code></pre>
<p>i don’t want to install tensorflow</p>

ptrblck: <p><a href="https://www.tensorflow.org/api_docs/python/tf/compat/v1/flags/Flag"><code>tf.flags</code></a> seem be define TensorFlow-specific flags and since it’s defined by TF you would need to install it in order to use it.<br>
However, which flag would you like to use specifically?</p>