akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Parallelizing linear algebra and intensive computations

username: lelouedec
cooked: <p>Good evening,</p>
<p>I have been implementing some raycasting operations using numpy and thought moving them to pytorch to make use of the gpu parallelization.<br>
However I am struggling to use some functions like cross product over list/tensors of tensors.<br>
For exemple I want to compute the cross product between 10 vectors and 6000 other vectors.<br>
With numpy I would broadcast them and get thing along this:</p>
<blockquote>
<p>pvec = np.cross( directions[None,:,:],v0v2[:,None,:])</p>
</blockquote>
<p>With pytorch it seems to be a problem as cross requires same size tensors and broadcasting apparently is not available for this methods.<br>
Any idea on how to do something similar efficiently ?</p>
<p>Also what would be the a good way to do some computations in parallel in a good way ?<br>
For example the same 10 operations for a lot of vectors.<br>
In cuda I see how the kernel would work executing the same code, but directly in pytorch does not seem so clear, is it even possible ?</p>
<p>Yours Justin</p>

username: ptrblck
cooked: <p>You could manually broadcast the tensors as shown in this example:</p>
<pre><code class="lang-python"># numpy
directions = np.random.randn(3, 3)
v0v2 = np.random.randn(3, 3)
pvec = np.cross(directions[None, :, :], v0v2[:, None, :])

# PyTorch with manual broadcasting
d, v = torch.from_numpy(directions[None, :, :]), torch.from_numpy(v0v2[:, None, :])
d = d.expand(v.size(0), -1, -1)
v = v.expand(-1, d.size(1), -1)
p = torch.cross(d, v, dim=2)

# Compare
print(torch.allclose(torch.from_numpy(pvec), p))
</code></pre>
Unable to increase batch_size

username: _joker
cooked: <p>I have a particular design for testing a model where I make use of <code>torch.topk</code> which returns me the top k predictions from the classifier in reverse order. Then based on certain “If” checks I want to use the prediction out of topk predictions for each testing example.</p>
<p>Problem -<br>
When I do this for batch-size &gt; 1, it uses the whole batch for that “if check”. I am not sure how can I modify the implementation for this task.</p>
<pre><code class="lang-auto">for batch_idx, (image, label) in enumerate(testloader):
    image, label = image.to(device), label.to(device)
    # --------------- Multi Class Classifier ---------------
    predict_A = classifier_A(image)
    # --------------- Get top k probabilities ---------------
    topk = torch.topk(predict_A.data, 2)
    first_pred       = topk.indices[:,0]
    second_pred = topk.indices[:,1]

    if (threshold &lt; x):
        USE top 1 (prediction with highest probability)
    else ():
        USE top 2 (prediction with second highest probability)
</code></pre>
Build batch faster than with __getitem__?

username: veda101
cooked: <p>In a case where my data fits in memory as numpy array, I realize that batching the data using <strong>getitem</strong> from the Dataset interface is much slower than when it is indexed manually with numpy.</p>
<p>I am sure it’s because the Dataloader build batches sample by sample by calling getitem to fetch each sample.</p>
<p>Is there any workaround to build batches faster while still using the standard Dataset / Dataloader interface ?</p>

username: faizan_shaikh
cooked: <p>Hi <a class="mention" href="/u/veda101">@veda101</a>, could you clarify a small detail?</p>
<p>You mentioned that the data fits in memory, and so, do you read the entire data in <code>__init__</code>? Or do you read it lazily in <code>__getitem__</code>?</p>

username: veda101
cooked: <p>I can for instance load all the data in memory in the <code>__init__</code>, and then access each row of the dataset in the <code>__getitem__</code>, but because getitem fetch each row one by one, it is definitely slower than fetching with a slice in numpy like data[0:batch_size]</p>

username: ptrblck
cooked: <p>You could use <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.BatchSampler"><code>BatchSampler</code></a> to pass a batch of indices to <code>__getitem__</code> and create multiple samples in a single call, if that fits your use case.</p>
PyTorch Extension C++ Logs

username: leimao
cooked: <p>Hello,<br>
PyTorch extension header <code>&lt;torch/extension.h&gt;</code> provided us the macro <code>TORCH_CHECK</code> to do some runtime check. I wonder if there is something that allows us to write log information in PyTorch? Say a <code>TORCH_LOG</code>?<br>
Best,<br>
Lei</p>
Why Semantic Segmentation Results are blurry/fuzzy. Or can you use semantic segmentation to predict 3 channels rgb from an image?

username: rzp0063
cooked: <p>Hi all,<br>
I am working on semantic segmentation using the UNET architecture. I initially started off by trying to predict the 3 RGB channels from the target. I know that semantic segmentation expects a class to have binary values but gave it a shot anyways. The results I obtain are ok as shown but also blurry/fuzzy and I would like to find the reason or source of this error and also to improve it. Is there a way to improve this for prediction of RGB channels without having to split the image into multiple colors?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/c/4cc8e989bda7259f26a083385ee9134c123af90b.png" alt="pytorch_forum_Segmentation" data-base62-sha1="aXgJTDQUANTSU88b6Xdcr4xSPDJ" width="213" height="412"><br>
Note that the top picture shows the target, or label that we want to predict while the bottom shows the prediction output from UNET architecture</p>
First Linear Layer Weight Regularization - seeking classification

username: alik604
cooked: <p>I was wondering how to do First Linear Layer L1 Weight regularization, for feature engineering. <em>Out of curiosity I want to see what a MLP thinks the top N features are.</em> I read <a href="https://discuss.pytorch.org/t/how-does-one-implement-weight-regularization-l1-or-l2-manually-without-optimum/7951/2">this post</a></p>
<p>I’m probably mistaken, but this seems wrong… past answers recommend <code>for W in model.parameters()</code>, so in my case, where my model’s first Linear layer is <code>L1</code> it would be <code>for W in model.L1.parameters()</code>. But this includes the bias term!.</p>
<p>Most posts guilty of this, however 1 <a href="https://discuss.pytorch.org/t/l2-regularization-with-only-weight-parameters/56853">saw one</a> that is in line with my expectation.</p>
<p>So what’s going on here, who is mistaken? I think that regularizing the bias probably isn’t too bad, it will be tiny, and doesn’t matter if there will be normalization done directly after.</p>

username: Abhilash_Srivastava
cooked: <p>See if this works for you (applying L1 regularization for layer L1):</p>
<pre><code class="lang-auto">for name, param in model.named_parameters():
    if 'L1' in name and 'weight' in name:
        L1_reg = L1_reg + torch.norm(param, 1)
</code></pre>
<p>This can also be modified for L2 regluarization.</p>
How to apply L2 regularization to weights tensor only?

username: alan_ayu
cooked: <p>Hello everyone:<br>
Recently a paper reported that it would be better to apply L2 regularization to weights tensor only and bias should not be regularised. The implementation way I can think of is to place weights and bias tensor into two different list and use different L2 regularization hyper-parameters to these parameter list explicitly. But I found this would be very complex, Can you think of a more simple implementation?<br>
Thank you for your advice!!!</p>

username: alik604
cooked: <p>Here is what I did, based on <a href="https://discuss.pytorch.org/t/l2-regularization-with-only-weight-parameters/56853">this</a></p>
<pre><code class="lang-auto">factor = 0.00005 # reg term coefficient/multiplier/weight  
model = NeuralNet(input_size, hidden_size, num_classes).to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss() # This criterion combines nn.LogSoftmax() and nn.NLLLoss() in one single class.
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)  

l1_reg_criterion = nn.L1Loss() # size_average=False 


# Train the model
n_total_steps = len(X_train)

X_train_RFE_vals= X_train_RFE.values
y_train_vals = y_train.values.flatten()

X_test_RFE_vals= X_test_RFE.values
y_test_vals = y_test.values.flatten()

for epoch in range(1, num_epochs + 1):
    # for i in range(len(X_train_RFE_vals)//100 + 1): #, batch_size

    n_correct = 0
    n_samples = 0 
    for i in range(0, X_train_RFE_vals.shape[0], batch_size):

        x = torch.as_tensor(X_train_RFE_vals[i:i+batch_size], dtype=torch.float).to(device)
        y = torch.as_tensor(y_train_vals[i:i+batch_size], dtype=torch.long).to(device)
        
        outputs = model(x)
        loss = criterion(outputs, y)

        reg_loss = 0 
        for name, param in model.l1.state_dict().items(): # L1 is the first layer 
          if name == 'weight':
            # print(param.size())
            # print((param-param).sum().item()); throw_for_bar_after_print_stop
            reg_loss = l1_reg_criterion(param, param-param)
        
        # print(f'loss: {loss}, reg_loss: {reg_loss}'); throw_for_bar_after_print_stop
        loss = loss + (factor * reg_loss)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        _, predicted = torch.max(outputs.data, dim=1)
        n_samples += y.size(0)
        n_correct += (predicted == y).sum().item()
    print(f'Epoch [{epoch}/{num_epochs}], Step [{i+1}/{n_total_steps}], Loss: {loss.item():.4f}, Acc: {100.0 * n_correct / (n_samples+1):.4f}; -- loss: {loss}, reg_loss: {reg_loss}%')
</code></pre>
Help on TensorDataset in libtorch please

username: Chen0729
cooked: <p>Hi, All</p>
<p>I have  inquiry about conversion of TensorDataset to C++,</p>
<p>In python it is:</p>
<p>From torch.utils.data import TensorDataset</p>
<p>return TensorDataset(torch.FloatTensor(np.array(data)),<br>
torch.LongTensor(np.array(labels))</p>
<p>May I ask what is the best way to convert it in c++ please ?</p>
<p>Thanks,</p>

username: glaringlee
cooked: <p>Hi,<br>
The dataset part is straightforward, see the class here:<br>
‘<a href="https://github.com/pytorch/pytorch/blob/393ad6582d1876dc995925ef558ba94e954d115e/torch/csrc/api/include/torch/data/datasets/tensor.h" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/393ad6582d1876dc995925ef558ba94e954d115e/torch/csrc/api/include/torch/data/datasets/tensor.h</a>’</p>
<p>But you need to somehow convert the data/labels in np.array in to a c++ array or vector. Then you can use from_blob to convert them in to a tensor.</p>

username: Chen0729
cooked: <p>Thanks. I have converted them into tensor already.<br>
One issue is that from_blob needs three input argument, and the second argument are img.col and img.row which I donot have here.</p>

username: glaringlee
cooked: <p><a class="mention" href="/u/chen0729">@Chen0729</a><br>
If you previously convert it into np.array, then you can check the np array size, right?<br>
Then you know what is the col and row.<br>
Or, in python, you can do<br>
t = torch.FloatTensor(np.array(data))<br>
then<br>
t.size() to check the col and row.</p>
How to create many to one LSTM of this form?

username: rectified_sigmoid
cooked: <p>I am trying to create a 3 to 1 LSTM<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9cc44ccbebdcbd28ddcf286b1f6681dd2bf9c3b3.png" alt="Encode1" data-base62-sha1="mmP3tVhy46j2l3l8ApvsXxRvtKz" width="602" height="423"><br>
The LSTM must take sequence of 3 words, each embedded vector of size 100<br>
So, my input size is (batch_size, 3, 100). So, what is the way to achieve this? I am new to pytorch, so, please help me with this</p>

username: AnimeshSinha1309
cooked: <p>Usually LSTMs are used for inputs with varying length, for exactly 3 inputs with a total vector size of 300 you can just use a few nn.Linear layers.</p>
<p>Nevertheless, if you want to use an LSTM, you want the following:</p>
<ul>
<li>
<code>input_dim</code> is 100</li>
<li>
<code>embed_dim</code> can be anything you want, let’s say 128 here. (This will be the size of your output)</li>
<li>You only want one output but you are getting 3 (as of now). The simplest idea is just to take the last output (StackOverflow answer here.)[<a href="https://stackoverflow.com/questions/62204109/return-sequences-false-equivalent-in-pytorch-lstm" rel="noopener nofollow ugc">https://stackoverflow.com/questions/62204109/return-sequences-false-equivalent-in-pytorch-lstm</a>].</li>
</ul>
<p>Your final code will look like this:</p>
<pre><code class="lang-auto">model = torch.nn.LSTM(input_size=100, hidden_size=128, num_layers=1)
output = model(input)[-1]
</code></pre>

username: rectified_sigmoid
cooked: <p>isn’t output the first element of the tuple returned as per doc? the o/p has shape (seq_len, batch, num_directions * hidden_size). Since my model in many to 1, I will take last seq I think? so</p>
<pre><code class="lang-auto">output,_,_ = model(input)
output = output[2]
</code></pre>

username: AnimeshSinha1309
cooked: <p>Right, sorry, I forgot about the output outputs from LSTM.</p>
Getting error in using autograd function

username: PARUL_ARORA
cooked: <p>I am running a github code on google colab ‘<a href="https://github.com/locuslab/e2e-model-learning.git" rel="noopener nofollow ugc">https://github.com/locuslab/e2e-model-learning.git</a>’ .When i run the power_sched&gt;main.py file ,it gives me this error.Can you please help me in sorting this out</p>
<pre><code class="lang-auto">RuntimeError: Legacy autograd function with non-static forward method is deprecated. Please use new-style autograd function with static forward method. (Example: https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function)
&gt; /usr/local/lib/python3.6/dist-packages/torch/autograd/function.py(149)__call__()
    147     def __call__(self, *args, **kwargs):
    148         raise RuntimeError(
--&gt; 149             "Legacy autograd function with non-static forward method is deprecated. "
    150             "Please use new-style autograd function with static forward method. "
    151             "(Example: https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function)")
'''</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>I am afraid this code is several years old and was written for a very old version of pytorch. This feature (and most likely many others) are not supported anymore and you will need to update the code if you want to run it with the latest version of pytorch.</p>

username: PARUL_ARORA
cooked: <p><a class="mention" href="/u/alband">@albanD</a>…Thanks for replying.I am new to this programming and testing this code from github. Can you please tell me what to update and how ?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Given that this code is from several versions back, it will be a bit of work to update it (even more if you don’t know pytorch) as you will need to check the BC-breaking changes in the release notes and update the code accordingly.</p>
<p>If you just want to run this code to try, maybe a safer alternative is to install an old version of pytorch from here <a href="https://pytorch.org/get-started/previous-versions/">https://pytorch.org/get-started/previous-versions/</a><br>
Then if you decide that you want to use it further, you can spend the time to upgrade the code to run with newer versions.</p>

username: PARUL_ARORA
cooked: <p>Thanks a lot for your suggestion.I tried with older version too and was getting different kind of errors.<br>
I am afraid that the errors which I will get in older versions ,whether I will be able to resolve them or not.</p>
How to bypass non-differentiable modules while loss.backward()

username: Dipayan_Das
cooked: <p>Hello,</p>
<p>So I have a CNN, with a softmax output, and a mask() function. The masks are like tensor([0,1,1]), tensor([1,1,0]) etc. The CNN has a softmaxed output over three classes.</p>
<pre><code class="lang-auto">pred = cnn(input_tensor.to(device)           #Returns PDF over three classes
mask = get_mask(*args)
final_pred = pred * mask.to(device)
</code></pre>
<p>Now, the mask returned by get_mask() is a deterministically mapped mask, and doesnt require any training/gradient. However, when i use,</p>
<pre><code class="lang-auto">loss = crtierion(final_pred, labels)
loss.backward()
</code></pre>
<p>I get the error</p>
<p>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</p>
<p>I even tried making mask = autograd.Variable(mask, requries_grad=False), but still the same error. Please help!</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: albanD
cooked: <p>Hi,</p>
<p>In general, you should avoid pinging people directly as it just adds more noise…</p>
<p>For your issue, are you sure that you don’t set your cnn’s weights not to require gradients?<br>
You should make sure that <code>pred</code> requires grad as well.</p>
MAML++ implementation in Pytorch

username: Brando_Miranda
cooked: <p>I was wondering if there was an implementation of MAML++ in pytorch?</p>
<p>Ideally in higher for example.</p>
<p>See: <a href="https://github.com/facebookresearch/higher/issues/84" rel="noopener nofollow ugc">https://github.com/facebookresearch/higher/issues/84</a></p>
Softmax Function for a Probability Vector

username: mbehzad
cooked: <p>Hi,<br>
I know that the softmax function outputs probabilities with sum equal to 1. However, if we give it a probability vector (which already sums up to 1) , why does not it return the same values? For example, if I input [0.1 0.8 0.1] to softmax, it returns  [0.2491 0.5017 0.2491], isn’t this wrong in some sense?</p>

username: learner47
cooked: <p>It is because of the way softmax is calulated. When you compute <code>exp(0.1)/(exp(0.1)+exp(0.8)+exp(0.1))</code>, the value turns out to be 0.2491.</p>

username: mbehzad
cooked: <p>Thanks for the answer. Yeah yeah that I know. But my question is, isn’t it wrong in some sense?</p>

username: Abhilash_Srivastava
cooked: <p>Softmax is an activation function. The purpose is not just to ensure that the values are normalized (or rescaled) to sum = 1, but also allow to be used as input to cross-entropy loss (hence the function needs to be differentiable).</p>
<p>For your case, the inputs can be arbitrary values (not necessarily probability vectors). It is possible that there’s a mix of positive and negative values which still sum = 1 (eg: [0.3, 0.8, -0.2].<br>
Since softmax picks the class with the highest value, with the values being softly rescaled, hence the name ‘soft’-‘max’.</p>

username: KFrank
cooked: <p>Hi mbehzad!</p>
<aside class="quote no-group" data-username="mbehzad" data-post="3" data-topic="100366" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mbehzad/48/28769_2.png" class="avatar"> mbehzad:</div>
<blockquote>
<p>is, isn’t it wrong in some sense?</p>
</blockquote>
</aside>
<p>Well, I suppose it depends on what your expectations are …</p>
<p>But you might wish to base your expectations on some other functions:</p>
<p><code>x**2</code> maps <code>(-inf, inf)</code> to <code>[0.0, inf)</code>, but we don’t expect <code>x**2 = x</code><br>
to hold true for <code>x &gt;= 0.0</code>, that is for values of x in <code>[0.0, inf)</code>.</p>
<p>Or, back in the pytorch activation function world, <code>torch.sigmoid()</code> maps<br>
<code>(-inf, inf)</code> to <code>(0.0, 1.0)</code>, but <code>torch.sigmoid (torch.sigmoid())</code><br>
isn’t equal to <code>torch.sigmoid()</code>.</p>
<p>Here’s another thing to consider:<br>
<code>softmax ([0.0 + delta, 1.0 - delta])</code></p>
<p>How would you like <code>softmax()</code> to behave when a negative <code>delta</code><br>
becomes zero and then crosses over to become positive?  Bear in<br>
mind, you want this behavior to be usefully differentiable to support<br>
backpropagation.</p>
<p>K. Frank</p>
Modify the output

username: DanielC
cooked: <p>Hi there,<br>
I want to control parts of the output of my NN, but I get runtime error.<br>
here is the code:</p>
<pre><code class="lang-auto">    for i in range(epochs):
        yhat = net(inputs) 
        yhat[0] = yhat[1]   # here
        yhat[-1] = yhat[-2] # here
        loss = free_energy(yhat, x, matrix, rho, lambd) 
        loss_points.append(loss.item())
</code></pre>
<blockquote>
<p>Runtime error: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [511, 1]], which is output 0 of SigmoidBackward, is at version 2; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>
</blockquote>
<p>Appreciate any advice.</p>

username: Abhilash_Srivastava
cooked: <p>Instead of modifying inplace, can you try creating a clone of the object. Something like</p>
<pre><code class="lang-auto">yhat = net(inputs) 
yhat_clone = yhat.clone()
yhat_clone[0] = yhat[1]   # here
yhat_clone[-1] = yhat[-2] # here
loss = free_energy(yhat_clone, x, matrix, rho, lambd) 
loss_points.append(loss.item())
</code></pre>
Getting the error while converting the tensors to Images using cv2

username: Varun_Tirupathi
cooked: <p>Hi,</p>
<p>I have converted the tensors which I’m getting as an output(test set) from my UNet to images. When I’m testing the model it is throwing out of index exception which I’m not getting when I didn’t convert the tensors to images. I have 74 images in my test set but after getting tested with one image it is throwing out of index exception in my custom dataset class. any idea will there will be any problem if I do a conversion like that?</p>
<pre><code class="lang-auto"># this is where im converting
 def test_step(self, batch, batch_nb):
        x, y = batch
        y_hat = self.forward(x)
        # print(y_hat.size())
        img = y_hat.cpu().detach().numpy()
        directory = r'D:\Mobile-research project\dataset\gaussian_blur\target_outputs'
        os.chdir(directory)
        print(os.listdir(directory))
        filename = 'savedImage.jpg'
        cv2.imwrite(filename, img)
        loss = torch.nn.MSELoss()
        op_loss = loss(y_hat, y)
        return {'test_loss': op_loss}

#custom dataset

    def __getitem__(self, i):
        idx = self.ids[i]

        img_files = glob.glob(os.path.join(self.img_dir, idx+'.*'))
        mask_files = glob.glob(os.path.join(self.mask_dir, idx+'.*'))

        assert len(img_files) == 1, f'{idx}: {img_files}'
        assert len(mask_files) == 1, f'{idx}: {mask_files}'

        # use Pillow's Image to read .gif mask
        # https://answers.opencv.org/question/185929/how-to-read-gif-in-python/
        img = Image.open(img_files[0])
        mask = Image.open(mask_files[0])
        # to_tensor = transforms.ToTensor()
        # img_t = to_tensor(img)
        # a = img_t.shape
        assert img.size == mask.size, f'{img.shape} # {mask.shape}'

        return self.transforms(img),\
               self.transforms(mask)

</code></pre>
<p>I’m getting this error:</p>
<p>File “D:\unet-lightning-master -original\unet-lightning-master-original\dataset.py”, line 35, in <strong>getitem</strong><br>
assert len(img_files) == 1, f’{idx}: {img_files}’<br>
AssertionError: 0101434: []<br>
Testing:   1%|▏         | 1/72 [00:00&lt;00:40,  1.77it/s]</p>
<p>Process finished with exit code 1</p>
What Activation Function Should I use for my Output in my Autoencoder?

username: Bled_Clement
cooked: <p>Hi folks!</p>
<p>I’m writing an autoencoder using Resnet as an encoder and then some convolutions and concatenations with skip connections for the decoder. The use of this is image denoising. Following this, I’m testing a novel idea of using Pytorch to do some parallelised frequency domain filtering. It works ok.</p>
<p>However, my second stage frequency-domain filter expects my tensor values to be scaled between 0 and 1. I tried using a sigmoid output layer, however I was given the impression that this is a bad idea. I’m now considering using softmax.</p>
<p>Has anyone with more knowledge have any ideas.</p>
<p>My network backpropagates through my second stage frequency domain/empirical filter so if my autoencoder spits out values greater than 1 it makes the network trend to zero. I should also note that it is important that I stay between 0 and 1 for the autencoder because I use a measurement of noise power on this scale for the second stage of my algorithm.</p>
<p>Thanks folks.</p>
Subclassing torch.LongTensor

username: ycen
cooked: <p>Related to <a href="https://discuss.pytorch.org/t/subclassing-torch-tensor/23754" class="inline-onebox">Subclassing torch.Tensor</a><br>
But I am trying to extend tensor with dtype=torch.int64<br>
My code like this:</p>
<pre><code class="lang-python">class MyTensor(torch.LongTensor):
    @staticmethod
    def __new__(cls, data, stats, *args, **kwargs):
        return super().__new__(cls, data, *args, **kwargs)
    def __init(self, stats, *args, **kwargs):
        self._stats = stats

x = MyTensor([0,1,2], 3)
</code></pre>
<p>This works for subclassing torch.Tensor, however it not work for torch.LongTensor<br>
TypeError: type ‘torch.LongTensor’ is not an acceptable base type<br>
How to subclass torch.LongTensor</p>
<pre><code class="lang-auto"></code></pre>

username: tom
cooked: <p>Don’t use torch.LongTensor, it’s not a proper class/type but a hack!<br>
The proper thing is to subclass Tensor and have the dtype set fixed to torch.long.</p>

username: ycen
cooked: <p>I have tried to subclass torch.Tensor, however dtype cannot be param of torch.Tensor.<strong>new</strong>. I could not find out how to init dtype<br>
Here is what I have tried</p>
<pre><code class="lang-python">def __new__(self, data, stats):
    tensor = torch.as_tensor(data, dtype=torch.long)
    return torch.Tensor.__new__(self, tensor)   # Error: expected Float (got Long)
</code></pre>
<p>or like</p>
<pre><code class="lang-python">def __new__(self, data, stats):
    tensor = torch.as_tensor(data, dtype=torch.long)
    return tensor
def __init__(self, data, stats):
    self.stats = stats

x = MyTensor([1,2,3],3)   # Actually it returns a torch.LongTensor not MyTensor object
x.stats   # Error
</code></pre>

username: tom
cooked: <p>You’re diving right into the tricky bits, but:</p>
<ul>
<li>
<code>__new__</code> is a class method implicitly (don’t ask).</li>
<li>You can use <code>nn.Parameter</code> as an example. Adapted this gives:</li>
</ul>
<pre><code class="lang-python">class MyTensor(torch.Tensor):
     def __new__(cls, data, stats, requires_grad=False):
         data = torch.as_tensor(data, dtype=torch.long)
         tensor = torch.Tensor._make_subclass(cls, data, requires_grad)
         tensor.stats = stats
         return tensor
</code></pre>
<p>So I’m still not 100% sure what you’re trying to achieve, but if it is a subclass that is similar to nn.Parameter, this would be what I’d use.</p>
<p>Best regards</p>
<p>Thomas</p>
Finetuning a model on single-gpu if it was trained on multi-gpu using DistributedDataParallel

username: ishitamed
cooked: <p>Hi!<br>
I have a doubt that is if I <strong>train a model using multi-GPU, DistributedDataParallel</strong> training (say the resnet50 training on imagenet example). Later on, for some feature extraction task, if I want to use its frozen weights as it is, or just want to finetune some of its last layers, but on a <strong>single GPU with no Dataparallel or DistributedDataParallel</strong> - is that possible (if yes, then just torch.load() will work fine?)? Will there be an issue?</p>
<p>TIA</p>

username: ptrblck
cooked: <p>Yes, that’s possible and I would recommend to store the <code>state_dict</code> via:</p>
<pre><code class="lang-python">torch.save(model.module.state_dict(), path)
</code></pre>
<p>as described <a href="https://pytorch.org/docs/stable/notes/serialization.html">here</a> and <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-torch-nn-dataparallel-models">here</a>.</p>
.idx files missing. can they be generated?

username: hemanthkorrapati
cooked: <p>Hello. I have a script which processes given data and produces a pytorch binary output (.bin and .idx files). When I run this script in isolation everything works fine. But when I run multiple instances (using thread pool) of this script in parallel, I see that some of the output folders have missing .idx files. Any ideas why this could be happening ?</p>
<p>Also can we regenerate the .idx files based on the .bin files ?</p>

username: ptrblck
cooked: <p>I’m not sure if this issue is PyTorch-related or is rather an issue in your multi-threaded application.<br>
Are these files only missing, if you created them from PyTorch or also as plain Python files?</p>
RuntimeError: after reduction step 2: cudaErrorAssert: device-side assert triggered

username: chaslie
cooked: <p>Help!!!</p>
<p>I have a problem during training. I get the error message in the title when using:</p>
<pre><code class="lang-auto">import os
os.environ['CUDA_LAUNCH_BLOCKING'] = "1"
</code></pre>
<p>The error is triggered by the following line going to NaN</p>
<p>BCE = F.binary_cross_entropy(out,real_data_I,size_average=False)</p>
<p>I have tried using Binary cross entropy with Logits and this gives the same error. Since the error happens at random times, I have deduced that its something to do with the input date, however i have looked at the input data its not same input data that triggers the error.</p>
<p>Does anyone have any ideas?</p>
<p>Chaslie</p>

username: pchandrasekaran
cooked: <p>Start of by checking if you have any nan’s in the input. The fact that another loss function causes the error as well tells that there is likely a problem with the input.</p>

username: chaslie
cooked: <p>Pchandrasekaren,</p>
<p>I have checked the inputs and there are no NaN’s in the input.</p>
<p>I have re-run the model removing the sigmoid and using binary cross entropy with logits loss function it seems to be running, at the moment.</p>
<p>The model consists of 2 CVAE’s, and both run and work well with the input data and with the BCE = F.binary_cross_entropy(out,real_data_I,size_average=False) + Sigmoid and the BCE_withLogits and no sigmoid…</p>
<p><strong>Update</strong>, the model as crashed with <code> **RuntimeError: after reduction step 2: cudaErrorAssert: device-side assert triggered**</code> after 5 epochs, this is using <code>F.binary_cross_entropy_with_logits</code></p>
<p>chaslie</p>

username: chaslie
cooked: <p>I have also set batchsize to 1 and shuffle to False in dataloader, then run the model, the model failed at increments 251,286 and 145, this suggests its not a problem with the input data.</p>

username: ptrblck
cooked: <p>Which PyTorch version are you using?<br>
If you are using <code>1.5</code>, could you please update, as <code>assert</code> statements in <code>1.5.0</code> were not working properly.<br>
Also, could you post the complete stack trace you are seeing?</p>

username: chaslie
cooked: <p>Hi Ptrblk,</p>
<p>How are you? I hope you are avoiding the worst that covid is throwing at you.</p>
<p>I am using 1.4.0 version.</p>
<p>Changing the loss function from <strong>F.binary_cross_entropy_with_logits</strong> to <strong>torch.nn.BCELoss</strong> and putting the sigmoid function at end of the network seems to have resolved the error.</p>
<p>I think there maybe an issue with <strong>F.binary_cross_entropy_with_logits</strong> and <strong>binary_cross_entropy</strong>?</p>
<p>I have decreased the learning rate as well by an order of magnitude and the model is now running.</p>
<p>I will post the stack trace tomorrow once the model has finished training.</p>
<p>chaslie</p>
How to apply cross-channel pooling for a set of feature maps

username: smani
cooked: <p>I have a feature map of size 8 x 32 x 10 x 10, where 8 is the batch size, 32 is the number of channels and 10 represents the width and the height respectively.</p>
<p>I want to apply cross channel pooling over each of the 8 feature maps of 32 maps (32 maps are organized as 8,8,8,8). The output I am expecting is 8x4x10x10.</p>
<p>How can I do it?</p>
Deploy model on mobile with pure c++

username: jackyfaster
cooked: <p>Hi,<br>
I want to deploy my pytorch model to mobile(like-android, arm32 architecture), i know there is Pytorch Mobile, but java is not supported on the mobile, just supporting pure c++. I am thinking that could i transplant caffe2 to arm32, if can, there is any documentation or suggestion?<br>
Thanks!</p>
Name of the optimizer being used

username: Eulerian
cooked: <p>I’m trying to display the optimizer currently in use by the model for documentation purposes.</p>
<p>So the following snippet</p>
<pre><code class="lang-auto">optimizer = torch.optim.SGD(model.parameters(), lr=1e-5)
print(optimizer)
</code></pre>
<p>gives the following output</p>
<pre><code class="lang-auto">SGD (
Parameter Group 0
    dampening: 0
    lr: 1e-05
    momentum: 0
    nesterov: False
    weight_decay: 0
)
</code></pre>
<p>Is there a way in which I could just get <code>SGD</code> as the output (and same for other optimizers also)?<br>
I figured I could take the substring of the displayed output till the first parenthesis. Is there a cleaner way of doing this?</p>

username: KFrank
cooked: <p>Hi Nihal!</p>
<aside class="quote no-group quote-modified" data-username="Eulerian" data-post="1" data-topic="100341" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eulerian/48/30170_2.png" class="avatar"> Eulerian:</div>
<blockquote>
<pre><code class="lang-auto">print(optimizer)
</code></pre>
<p>Is there a way in which I could just get <code>SGD</code> as the output (and same for other optimizers also)?</p>
</blockquote>
</aside>
<pre data-code-wrap="python"><code class="lang-python">print (type (optimizer).__name__)
</code></pre>
<p>will do what you want.</p>
<p>(This is generally true for python objects, and is not specific to pytorch.)</p>
<p>Best.</p>
<p>K. Frank</p>
Do we really need opencv if using pytorch?

username: miner_tom
cooked: <p>I have opencv running in the same virtual environment in which I installed pytorch from source. However, when compiling pytorch I saw a warning (not an error):</p>
<p>“Excluding image processing operators due to no opencv”</p>
<p>Thank You<br>
Tom</p>
<p>Thank You<br>
Tom</p>

username: henry_Kang
cooked: <p>pytorch itself uses the PIL however I recommend to install opencv. It is easy to use and there are various function which is helpful for data augmentation</p>

username: miner_tom
cooked: <p>Thank you so much for your reply.</p>
<p>Yes, I use OpenCV and it is running in the same virtual environment. However, I do get a message while compiling pytorch</p>
<pre><code class="lang-auto">-- Excluding image processing operators due to no opencv
-- Excluding video processing operators due to no opencv

</code></pre>
<p>In a terminal, I can run</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; import cv2

What is the difference between compiling OpenCV with torch or separately?

Thank You

</code></pre>

username: henry_Kang
cooked: <p>Some of pip or conda opencv does not include the some of function because of the license or patent issue or cmake flag.<br>
I usually use the below link for opencv python.<br>
<a href="https://github.com/conda-forge/opencv-feedstock" rel="noopener nofollow ugc">https://github.com/conda-forge/opencv-feedstock</a>.<br>
I also use the miniconda system so it doesn not include large space in your storage.</p>

username: nivesh_gadipudi
cooked: <p>It entirely depends on the user. Most of the PyTorch transformations are performed on PIL images. You can convert from PIL to Tensor and vice verse and same with PIL to numpy.</p>
Use Dataloader、ImageFolder to simultaneously load the test images and ground truth from different paths

username: PytorchBeginner
cooked: <p>I need to test my model using the test set and corresponding groundtruth. There are multiple subsets in the test folder and ground truth folder. Also, I need they have one-to-one relationship. How to samutaneously load the images and groundtruth without mentioning the name of subfolders (color, poke…) but only use the roots (test, ground truth)? Many thanks for your help.</p>
<p>The structures of folders are as follows:</p>
<p>test<br>
—color<br>
------000.png<br>
------001.png<br>
------002.png<br>
—poke<br>
------000.png<br>
------001.png<br>
------002.png</p>
<p>groundtruth<br>
—color<br>
------000.png<br>
------001.png<br>
------002.png<br>
—poke<br>
------000.png<br>
------001.png<br>
------002.png</p>

username: mehdi_boston
cooked: <p>ImageFolder will build classes based on names of the folders you have under the root, so if the underlying folders and files are exactly the same you can expect a one-to-one match between datasets made with test and groundtruth root folders. You can also check .imgs attribute of each dataset which will contain full path and label for each file and see if the one-to-one match is preserved. So if you iterate over each dateset separately with shuffle=False, you should get what you need.<br>
However, I’m not sure how these two datasets need to be iterated in your testing pipeline.</p>

username: PytorchBeginner
cooked: <p>Thanks for your reply. One root contains the original images and the other contains the corresponding ground truth, which are also images (not a classification label like “dog”). So,  there is some difficulty to combine these two folders together in one dataloader.</p>
Mini batches with DataLoader and a 3D input

username: Adrian_Briceno_Aguil
cooked: <p>I have been struggling to manage and create batches for a 3D tensor. I have used it before as a way to create batches for 1D tensor. However, in my current research, I need to create batches out of a tensor with shape (1024,1024,2).</p>
<p>I created custom data to use as my input for the DataLoader method in pytorch. I created the following for the 1D array:</p>
<pre><code class="lang-auto">class CustomDataset(Dataset):
def __init__(self, x_tensor, y_tensor):
    self.xdomain = x_tensor
    self.ydomain = y_tensor
    
def __getitem__(self, index):
    return (self.xdomain[index], self.ydomain[index])

def __len__(self):
    return len(self.xdomain)
</code></pre>
<p>It works pretty well, however, I realized that this doesn’t work for tensors x_tensor and y_tensor of shape (1024,1024,2) and (1024,1024,1) respectively. I understand that I have to change the __ getitem __ and __ len __ function in a way so it can divide the tensors into batches.</p>
<p>I tried many things, but one I know it could work is that I could flatten these tensors into shapes (1024 x1024,2) and (1024x1024,1). However, I would have to not only change my NN definition but must of my code.</p>
<p>So I want to keep it as is and try to understand how to create these functions if possible. What I understand of these functions are:<br>
__ len __ so that len(dataset) returns the size of the dataset.<br>
__ getitem __ to support the indexing such that dataset[i] can be used to get ith sample.</p>
<p>With this knowledge, I created this class, that finds the indexes of the first 2 dimensions(to find the ith sample). However, this created the input of the NN to be (1024x1024,2) and output (1024x1024,1). And I want it to be (1024,1024,2) and (1024,1024,1).</p>
<p>If someone with a better understanding of Data Loader and mini-batches could explain what am I missing, that could be amazing. An first of all is this possible?</p>
<p>Thanks for reading this, sorry if this question is too basic. I hope is clear.</p>

username: Adrian_Briceno_Aguil
cooked: <p>If you don’t understand my question please ask. It will help me to get better.</p>

username: ptrblck
cooked: <p>I don’t completely understand the issue and the posted shapes and I get the expected output as <code>[batch_size, 1024, 2]</code> and <code>[batch_size, 1024, 1]</code> as seen here:</p>
<pre><code class="lang-python">class CustomDataset(Dataset):
    def __init__(self, x_tensor, y_tensor):
        self.xdomain = x_tensor
        self.ydomain = y_tensor
        
    def __getitem__(self, index):
        return (self.xdomain[index], self.ydomain[index])
    
    def __len__(self):
        return len(self.xdomain)
    
    
x_tensor = torch.randn(1024, 1024, 2)
y_tensor = torch.randn(1024, 1024, 1)
dataset = CustomDataset(x_tensor, y_tensor)
loader = DataLoader(dataset, batch_size=2)

for x, y in loader:
    print(x.shape, y.shape)
&gt; torch.Size([2, 1024, 2]) torch.Size([2, 1024, 1])
  torch.Size([2, 1024, 2]) torch.Size([2, 1024, 1])
  torch.Size([2, 1024, 2]) torch.Size([2, 1024, 1])
  [...]
</code></pre>
Binary classification but request 3 classes

username: henry_Kang
cooked: <p>Hello. I am trying to follow the pytorch tutorial for image classification.<br>
I am using the below link.<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" target="_blank" rel="noopener nofollow ugc">Transfer Learning for Computer Vision Tutorial — PyTorch Tutorials 1.6.0...</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I only changed my folder which still same binary and it should be classess number is 2.<br>
However it requires class number is 3.<br>
Does anyone know why and how can I fix it?<br>
I only change the folder of images and do not change other things<br>
Thank you.</p>
<p>BR<br>
Dong Ho Kang</p>

username: planet_pluto
cooked: <p>Interesting, can you post the stack trace of your error?</p>

username: henry_Kang
cooked: <p>Well Not the error is gone after I reboot the computer.<br>
I cannot reproduce the error.<br>
Does anyone have this experience?<br>
It is really really weird.</p>