akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Binary Classification with Sigmoid Output - Can't get above 0.3 f1 score

pascalm: <p>Hello,</p>
<p>I’m trying to predict if a mutation in a protein makes the protein go bad. My Input is ["ABCD, 0], so the mutation containing of the mutations “ABCD” make the protein go bad (0). Similarily [“ABCX”, 1] would keep it functioning.<br>
There are 21 possible amino acids, so my input is an array of length 21 with 4 ones and 17 zeros.</p>
<p>I make a dataset out of this data where I standardize the input using sklearns tools.</p>
<p>Now I try to get the probability of it being 0 or 1 i.e. good or bad using sigmoid but the probabilities “converge” towards zero. They are around [0.0, 0.2] instead of [0.0, 1.0].</p>
<p>So I use a threshold of t=0.1. So if a value is bigger than 0.1 I call it “good” and else bad. This gives me a f1 score of about ~0.30.</p>
<p>Now I won’t ever get above 30, no matter what I do. I tried for hours over hours but something is fundamentally wrong with my approach/code.</p>
<p>My code: <a href="https://bpa.st/JWPQ" class="inline-onebox" rel="noopener nofollow ugc">View paste JWPQ</a></p>
<p>Any input?</p>
Adding a residual block to standard Resnet18

Mohammed_Awney: <p>I want to use Resnet model with pre-trained weights, but I want to use additional residual  block (layer 5)  before the last FC layer is this the right way to do that:<br>
ResNet_upd = torchvision.models.resnet18(prertained= True)<br>
MyModel = nn.Sequential()</p>
<p>MyModel.add_module(“conv1”, list(ResNet_upd .children())[0])</p>
<p>MyModel.add_module(“bn1”, list(ResNet_upd .children())[1])</p>
<p>MyModel.add_module(“relu”, list(ResNet_upd .children())[2])</p>
<p>MyModel.add_module(“maxpool”, list(ResNet_upd .children())[3])</p>
<p>MyModel.add_module(“layer1”, list(ResNet_upd .children())[4])</p>
<p>MyModel.add_module(“layer2”, list(ResNet_upd .children())[5])</p>
<p>MyModel.add_module(“layer3”, list(ResNet_upd .children())[6])</p>
<p>MyModel.add_module(“layer4”, list(ResNet_upd .children())[7])</p>
<p>MyModel.add_module(“layer5”,nn.Sequential(*resnet_block(512, 512, 1, first_block=True)))</p>
<p>MyModel.add_module(“avgpool”, nn.AdaptiveAvgPool2d((1, 1)))</p>
<p>MyModel.add_module(“fc”, nn.Linear(512, 937))</p>
<p>print(MyModel)</p>
Probabilisty Sigmoid Output to Binary

pascalm: <p>Hi,</p>
<p>assume I have a tensor from a sigmoid output containing e.g. [0.1, 0.2, 0.3, 0.2, 0.4]. I now want [0, 0, 1, 1] i.e. for everything &gt; 0.2 I want a 1 and else a 0.</p>
<p>How can I do that in pyTorch?</p>

a_d: <p>Hi<br>
This can be done as follows =</p>
<pre><code class="lang-auto">output = model(input)
output = torch.sigmoid(output)
output = torch.round(output)
</code></pre>

pascalm: <p>no, round has a fixed threshold of 0.5</p>

a_d: <p>Oh sorry I misread your question,<br>
Then simply on your acitvated output do -</p>
<pre><code class="lang-auto">output = torch.where(output &gt; 0.2, 1, 0)
</code></pre>
Explain this code - Pytorch CNN

tomi_datasci: <p>These two lines of code are from the testing of a CNN model. I know of alternative ways of getting the predictions and correct predictions<br>
but I have struggled to make sense of the two lines below:</p>
<pre><code>  pred = output.data.max(1, keepdim=True)[1]
  correct += pred.eq(target.data.view_as(pred)).sum()</code></pre>

Sanjan_Das: <p>So at a high level - what your code is doing is first getting the predicted tensor <code>pred</code>, and then element-wise comparing them to the values in tensor <code>target</code>, setting them to <code>True</code> if the elements match and <code>False</code> if not. And then when you take the <code>sum()</code>, you’re simply summing over the <code>True</code> values, and that gives you the number of correct predictions.</p>
Do I have to manually loop over batches?

pascalm: <p>Hi,</p>
<p>so I have a Dataset class for my data. I use a dataloader with it. I then have e.g. a train() function that does the training loop. IIn that training loop I loop over the batches manually.</p>
<p>Is that really the way to go with pyTorch?</p>

pascal_notsawo: <p>As you can see in this <a href="https://twitter.com/_ScottCondron/status/1363494433715552259?s=08" rel="noopener nofollow ugc">little video</a>, the function of the Dataloader is to be able to return directly to you the batches of data.</p>
<p>If your Dataset class implements the method <code>__getitem__(self, index : int) -&gt; instance of your data</code>, the Dataloader is supposed to use it (efficiently) to give you directly the data batches in your loop.</p>

KFrank: <p>Hi Fancy!</p>
<aside class="quote no-group" data-username="pascalm" data-post="1" data-topic="120658" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pascalm/48/76741_2.png" class="avatar"> pascalm:</div>
<blockquote>
<p>Is that really the way to go with pyTorch?</p>
</blockquote>
</aside>
<p>Yes.  Pytorch does not offer a general-purpose training-loop function.</p>
<p>I think the issue is that for the most basic use case the training loop<br>
is quite simple and hardly a bother to write by hand.  But there are<br>
lots of different things you might want to do in a training loop as you<br>
move beyond the most basic use case so designing a general-purpose<br>
training-loop function becomes messy.</p>
<p>The training loop typically loops over epochs as well as batches.<br>
Do you want to calculate one or more accuracy measures?  Do<br>
you want to do so periodically after every <code>n</code> batches or <code>n</code> epochs?<br>
Do you want to periodically calculate losses or accuracies for a<br>
validation set?  Do you want to implement early stopping based on<br>
some validation measure?  Are you collecting statistics or printing out<br>
progress messages as you train?  Are you training multiple networks<br>
with the same batches or training an adversarial network?</p>
<p>These – and more – are some of the features a pre-packaged<br>
training-loop function might be asked to support.  Which would<br>
you implement?  What would the interface to function look like?</p>
<p>Good luck.</p>
<p>K. Frank</p>
Custom Loss Function Fails

jonathon: <p>My model trains well with a dataset using MSELoss, L1Loss, and SmoothL1Loss loss functions.  I wrote a custom loss function that could train a model on a small subsample, but does not train the model with the full dataset.  The custom loss for the data does not change over many epochs and for good measure I calculated the L1Loss at the end of each epoch and that did not change. Here is the loss function class.  Is there anything wrong with it?</p>
<p>class ProportionalLoss(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(ProportionalLoss, self).<strong>init</strong>()</p>
<p>def forward(self, inputs, targets):<br>
labels = torch.log10(targets)<br>
predictions = torch.log10(torch.clamp(inputs, min=1, max=2500))<br>
loss = F.l1_loss(predictions, labels, reduction=‘mean’)<br>
return loss</p>

ptrblck: <p>Since your model is training with a small dataset and is failing with the entire set, you could check how the gradients (and their magnitudes etc.) behave in both approaches. Since the loss is apparently not changing after a while I would guess that the gradients might converge towards zero.</p>

jonathon: <p>Thank you Patrick.  I had checked the weights of the model and they quickly became static.  Using the George Costanza theorem, I modified my code to do the exact opposite of what I was intending to do with the loss function…and it is training on the full dataset!</p>
<p>So, I confirmed that I did successfully code a loss function; there is just something wrong with my intent.  I will keep researching.</p>
Runtime error in pytorch pretrained model

Biswajit_Roy_ISGPP: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/0/4085ccb3bd38c523a915e64248a9bccdf14e2fbe.png" data-download-href="https://discuss.pytorch.org/uploads/default/4085ccb3bd38c523a915e64248a9bccdf14e2fbe" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/4085ccb3bd38c523a915e64248a9bccdf14e2fbe_2_690x389.png" alt="image" data-base62-sha1="9cNdNacB2rpmFwuQl49nwfTwOJ0" width="690" height="389" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/4085ccb3bd38c523a915e64248a9bccdf14e2fbe_2_690x389.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/4085ccb3bd38c523a915e64248a9bccdf14e2fbe_2_1035x583.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/0/4085ccb3bd38c523a915e64248a9bccdf14e2fbe.png 2x" data-dominant-color="4E4E4E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1360×768 78.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>i am facing issue in image classification with googlenet</p>

eqy: <p>Basically the shape math of the model layers needs to be checked so that the input feature size to the linear layer matches the defined weight shape. For example, this might be due to a flatten operation before the linear layer outputting a different shape than what is expected. However, since the model is pretrained you likely need to change the input shape (e.g., by resizing the image if it is an image input) to fix this.</p>

Biswajit_Roy_ISGPP: <p>ok let me try this i dont  know will it work or not</p>
Normalising images in Cityscapes using mean and std of Imagenet

887574002: <p>Hi, I am working with Cityscapes dataset. For normalising the images I used the <code>mean</code> and <code>std</code> of Imagenet. After normalising I computed <code>mean</code> and <code>std</code> for some images in the dataset. They are roughly close to 0 and 1 but not very close.</p>
<p>For example <code>mean</code> and <code>std</code> of one image after normalisation is equal to</p>
<pre><code class="lang-auto">mean = [-0.14200746, -0.07835515, -0.09254397]
std = [0.84492135, 0.8451715,  0.849345  ]
</code></pre>
<p>Are these values  for mean and std of a normalised image acceptable?</p>
<p>I also computed the <code>mean</code> and <code>std</code> over the whole dataset using the following snippet:</p>
<pre><code class="lang-auto">def compute_mean_std(dataloader):
    
    pop_mean = []
    
    pop_std = []
    
    for i, (img,mask, rgb_mask) in enumerate(dataloader):
        
        numpy_image = img.cpu().numpy()
        
        batch_mean = np.mean(numpy_image,axis=(0,2,3))
        pop_mean.append(batch_mean)
    
        batch_std = np.mean(numpy_image, axis=(0,2,3))
        pop_std.append(batch_std)
    pop_mean = np.array(pop_mean).mean(axis=0)
    pop_std = np.array(pop_std).std(axis=0)
    return(pop_mean, pop_std)
</code></pre>
<p>And I obtained values:</p>
<pre><code class="lang-auto">MEAN = [0.28660315, 0.32426634, 0.28302112]
STD = [0.00310452, 0.00292714, 0.00296411]
</code></pre>
<p>But again when I normalised images using these values and calculated the <code>mean</code> and <code>std</code> of each image the <code>mean</code> and <code>std</code> was something like this:</p>
<pre><code class="lang-auto">[-7.8684726 -4.747944  -5.4215384]
[47.001762 51.33731  50.00176 ]
</code></pre>
<p>I do not know which point I missed. I only know  these two ways to normalise images.</p>

ptrblck: <p>The ImageNet normalization is used as:</p>
<pre><code class="lang-python">normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[0.229, 0.224, 0.225])
</code></pre>
<p>While your mean is a bit different, the <code>STD</code> seems to be quite off and it also seems that you are indeed calculating the <code>stddev</code> of the <code>mean</code> of the samples, which would explain the small <code>STD</code> values and the large outputs.</p>

887574002: <p>So, dose it mean for normalising images I should use the <code>mean</code> and <code>std</code> of ImageNet?</p>

ptrblck: <p>Not necessarily and you could use your custom stats after fixing the calculation.</p>

887574002: <p>Ok, thank you for your answer.</p>
How to efficiently orthogonalize a matrix like scipy.linalg.orth()?

111414: <p>I don’t know how to orthogonalize a matrix by PyTorch efficiently on CPUs or GPUs?<br>
The function producing a random orthogonal matrix that I want may be implemented by Numpy and Scipy like this:</p>
<pre><code class="lang-auto">import torch
import numpy as np
from scipy.linalg import orth

def get_orth_matrix(N):
    m = np.random.randn(N, N)
    m = orth(m)
    return m

if __name__ == '__main__':
    M = get_orth_matrix(3)
    M = torch.Tensor(M)
    print(M.mm(M.t()))   # should be an identity matrix
</code></pre>

111414: <p>I found this code works: <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"></p>
<pre><code class="lang-auto">def get_orth_matrix(M, N):
    m = torch.randn(N, N)
    return torch.qr(m)[0].t()[:M]  # orthogonal matrix with shape of (M, N)
</code></pre>
Shuffle the order of images present in a batch

hrushi: <p>I have a tensor of shape (64, 3, 32, 32), where 64 refers to 64 images.<br>
I would like to shuffle the order of images in any random way (the resulting image would still be (64, 3, 32, 32) tensor).</p>
<p>Numpy has np.random.shuffle method, is there anything similar in pytorch?</p>

Tejan_Mehndiratta: <p>On using dataloader, use shuffle=True. That’s it.</p>
<pre><code class="lang-auto">loader = torch.utils.data.DataLoader(trainset, batch_size=batch_size, shuffle=True)
</code></pre>
<p>You can look at the docs <a href="https://pytorch.org/docs/stable/data.html" rel="noopener nofollow ugc">here</a>:</p>
<p>Thanks.</p>
Why does torch.FloatTensor change an image's appearance?

husnainali21: <p>Hello I have attached an image that explains my query ,can anyone please explain this behavior?</p>
<p>Thank you!<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/e/deae7eccc349d15f539d508d2a79b78c6d388db2.png" data-download-href="https://discuss.pytorch.org/uploads/default/deae7eccc349d15f539d508d2a79b78c6d388db2" title="Screenshot from 2021-05-08 06-03-14"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/deae7eccc349d15f539d508d2a79b78c6d388db2_2_690x339.png" alt="Screenshot from 2021-05-08 06-03-14" data-base62-sha1="vLVQSD1Pko2rPcsyEwonr23IQJI" width="690" height="339" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/deae7eccc349d15f539d508d2a79b78c6d388db2_2_690x339.png, https://discuss.pytorch.org/uploads/default/original/3X/d/e/deae7eccc349d15f539d508d2a79b78c6d388db2.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/e/deae7eccc349d15f539d508d2a79b78c6d388db2.png 2x" data-dominant-color="4A4A4A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-05-08 06-03-14</span><span class="informations">934×459 48.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p><code>matplotlib</code> raises a warning, which explains the different output:</p>
<pre><code class="lang-python">Clipping input data to the valid range for imshow with RGB data ([0..1] for floats of [0..255] for integers).
</code></pre>
<p>Based on this warning it seems the <code>FloatTensor</code> <code>image</code> contains values outside of <code>[0, 1]</code>, which will then be clipped.</p>
Implementation of ensemble of network

weiguowilliam: <p>How would you implement an ensemble of neural networks? I currently use a list like this:</p>
<pre><code class="lang-auto">list_of_models = [model] * num_model
</code></pre>
<p>I wonder if there’s a more ‘Pytorch-like’&amp;efficient way to construct an ensemble? Thank you in advance.</p>

ptrblck: <p>I assume <code>model</code> is an already initialized <code>nn.Module</code> object. If that’s the case, note that you would create <code>num_models</code> references to the same object and won’t create <code>num_models</code> different model objects. To do the latter, you would need to initialize them: <code>[MyModel() for _ in range(num_models)]</code>.<br>
With that being said, I’m not aware of cleaner/better approach, so let’s wait for others to chime in. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to use DistributedDataParallel to speed up inference?

jerly-hjzhou: <p>There are many blogs on the internet about using DistributedDataParallel to speed up training but almost no information about speed up inference. I want to know how to speed up testing using DistributedDataParallel in such a code like this</p>
<pre><code class="lang-auto">import torch
from torchvision import datasets, transforms
import time

model = torch.load('resnet-18/resnet_18-cifar_10.pth')  # 加载模型
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

model.to(device)
model.eval()
accuracy = 0.0
correctNum = 0
data_transforms = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
])
image_datasets = datasets.CIFAR10(root='./data', train=False,
                                  download=True, transform=data_transforms)
test_loader = torch.utils.data.DataLoader(image_datasets, batch_size=250,
                                          shuffle=False, num_workers=0, prefetch_factor=2)

since = time.time()
for data, target in test_loader:
    data = data.to(device)
    target = target.to(device)
    output = model(data)
    pred = output.data.max(1, keepdim=True)[1]  # get the index of the max log-probability
    correctNum += pred.eq(target.data.view_as(pred)).cpu().sum()  # 对预测正确的数据个数进行累加
accuracy = correctNum / len(test_loader.dataset)
print("accuracy is {:.4f}%".format(accuracy * 100))
time_elapsed = time.time() - since
print('Test complete in {:.0f}m {:.0f}s'.format(
    time_elapsed // 60, time_elapsed % 60))
</code></pre>
Index and replace into 3D tensor according to list of indices

gstoica: <p>Hi!</p>
<p>I have a problem where I need to select vectors from a three dimensional tensor based on a list of 2D indices. However, these indices may select different numbers of vectors per dimension.</p>
<p>To better clarify this problem, take the following example:</p>
<pre><code class="lang-python">A = torch.FloatTensor( # 3D data tensor
    [
        [
            [0, 0, 0, 0],
            [1, 1, 1, 1]
        ],
        [
            [2, 2, 2, 2],
            [3, 3, 3, 3]
        ],
        [
            [4, 4, 4, 4],
            [5, 5, 5, 5]
        ]
    ]
)
B = [(0, 0), (0, 1), (1, 0)] # 2D coordinate tensor

C = IndexSelect(A, B) # An unknown selection function
"""
C: [
           [0, 0, 0, 0],
           [1, 1, 1, 1],
           [2, 2, 2, 2]
    ]
"""
</code></pre>
<p>So C contains all vectors from A corresponding to indices from B. Note how the indices in B may select an arbitrary amount of vectors from different dimensions, and can also skip them (e,g., B skips all vectors in A[2]).</p>
<p>Is there a way to create such an “IndexSelect” function?</p>
<p>Moreover, is it possible to do this with the ability to then replace selected vectors with others?</p>
<p>For instance, piggy-backing on the first example,</p>
<pre><code class="lang-python">D = [
           [6, 6, 6, 6],
           [7, 7, 7, 7],
           [8, 8, 8, 8]
    ]
A = Replace(A, B, D)
"""
A: [
        [
            [6, 6, 6, 6],
            [7, 7, 7, 7]
        ],
        [
            [8, 8, 8, 8],
            [3, 3, 3, 3]
        ],
        [
            [4, 4, 4, 4],
            [5, 5, 5, 5]
        ]
    ]
"""
</code></pre>
<p>Where vectors in A are replaced (in the “Replace” function) by those in D according to indices in B.</p>
<p>I’ve looked at selection functions like torch.gather or torch.index_select, but my understanding is that those assume a user selects the same number of elements along the specified dimension. However in this case we may select arbitrary elements along each dimension.</p>
<p>Anything would be very greatly appreciated! Thanks for your time!</p>

ptrblck: <p>I don’t fully understand how the posted examples select “different numbers of vectors per dimension”, as it seems that <code>B</code> is used to index entire “rows” of <code>A</code>. Could you explain where the variable length is used in your example?</p>
Blocking gradient along path

kazimpal87: <p>Hi,</p>
<p>I’m wondering if there is a way to block gradients along a certain pathway without completely detaching a tensor from the graph.</p>
<p>Say I have this setup where I have a model which computes a feature F and has two classification heads A and B. A is the “main” head and I want gradients from A to flow through the model normally. B is an auxiliary head and I want it to behave as if the rest of the model is frozen, so it learns to to classify something based off the values of F without affecting the training of the rest of the model.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/8/18bf37484777aa77644198b2980e4a116a81bde4.png" alt="blockgrad" data-base62-sha1="3wV8wO4QNvBXD2xlClFQJ7MCy0I" width="575" height="378"></p>
<p>I’m not sure how to do this with detach. My understanding is that if I detach B, it just wont learn anything, but in my case I want it to learn but not to affect the rest of the model.</p>
<p>Thanks</p>

ptrblck: <p>Based on your figure you would pass the output of <code>F</code> (let’s call it <code>x_f</code>) to the modules <code>A</code> and <code>B</code>.<br>
To keep the computation graph for the <code>A</code> path you would simply pass it to <code>A</code>, calculate the loss, and call <code>backward</code> on it. However, since you only want to calculate the gradients for the parameters of <code>B</code> using this aux. path, you could detach <code>x_f</code> and pass it to <code>B</code>:</p>
<pre><code class="lang-python">x_f = F(x)
x_a = A(x_f)
x_b = B(x_f.detach())
return x_a, x_b
</code></pre>
Loss Function for Probability Output

Alex_Mathew: <p>I have a dataset in which the I/P is an image and the O/P is a probability value i.e, a decimal value in the range [0,1]. Which loss functions can I use for training a CNN with this dataset? Cross Entropy seems not to support floating points as its meant for a classification problem while the given problem is more of a regression.</p>

eqy: <p>You can use binary cross-entropy: <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html#torch.nn.BCELoss" class="inline-onebox" rel="noopener nofollow ugc">BCELoss — PyTorch 1.8.1 documentation</a>.</p>
<p>Note that you can also express the problem with cross-entropy and two classes: one of the classes is ‘0’ and the other is ‘1’. You can simply use the softmax output of the ‘1’ class as your prediction in this case.</p>
Pandas DataFrame.groupby in Dataset/Data loader

aksg87: <p>Hello all,</p>
<p>I was thinking about creating a general data loader for 3D imaging inputs (volumes).  I want to eventually have a lot of flexibility for how I load the images (i.e. what percentage of the volume I’m inputting), and control for sampling-based (and metrics) based on certain attributes which could easily be stored in a DataFrame.</p>
<p>I was considering using the pandas.DataFrame.groupby object as a core of the data loader. It was would be relatively simple to initialize the core parameters that define the image coordinates of interest.</p>
<p>Is this a good strategy? Alternatively, I could have a custom data structure that maps across the image volumes and the spatial coordinates.</p>
<p>Any feedback is appreciated before I dive into this <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Competition for pipe resources while trying to receive PyTorch tensors

zhangxiaochuan: <p>I met a problem that sub-processes in python will compete for pipe resources when one subprocess tries to receive PyTorch tensors and another subprocess requests a different pipe connection.</p>
<p>Here is my code (run on Ubuntu 18.04):</p>
<pre><code class="lang-auto">import time
from multiprocessing import Process, Pipe
import torch


def send(conn):
    a, b = Pipe()
    data = [torch.rand(1) for _ in range(20)]
    print('start sending', time.time())
    conn.send(data)
    print('finish sending', time.time())

    while True:
        if a.poll():
            pass
        # else:
        #    time.sleep(0.1)


def recv(conn):
    while True:
        if conn.poll():
            print('start receiving', time.time())
            conn.recv()
            print('finish receiving', time.time())
            break
        else:
            time.sleep(0.1)


if __name__ == '__main__':
    recv_conn, send_conn = Pipe()

    send_proc = Process(target=send, args=(send_conn,))
    recv_proc = Process(target=recv, args=(recv_conn,))

    send_proc.start()
    recv_proc.start()

    recv_proc.join()
    send_proc.terminate()
</code></pre>
<p>This code needs several seconds to receive the torch tensors. But if I uncomment the last two lines in the <code>send</code> method, the speed of receiving the torch tensors will be very quick.</p>
<p>And this problem only appears while sending torch tensors.</p>
<p>Does anyone know the reason for this problem?</p>

rvarm1: <p>cc <a class="mention" href="/u/colesbury">@colesbury</a> <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a> regarding multiprocessing with torch.tensors</p>

zhangxiaochuan: <p>Hi Rohan, thanks for your reply. I’ve solved this problem, and I just want to know the reason for it. Can you give me some advice?</p>
How scale data to 0 to 1 range after each layer

shivarama93: <p>Hello,<br>
I am building an image restoration CNN. After each layer, I need the data to be scaled between 0 to 1. Is there any built-in normalization/scaling technique? Also, it would be sufficient if I can adjust the nn.batchnorm2d() to scale the data to 0 to 1 range. I am just a beginner to pyTorch.</p>

eqy: <p>There are many ways to do this, but a simple method is to just do something like <code>a = a/torch.max(a)</code>, perhaps adding some epsilon to the divisor for numerical stability.</p>

shivarama93: <p>Thank you for the reply. I will try to use this solution.</p>
Semantic segmentation loss function questions

KKKcat: <p>Hi there, I’m new to segmentation model.<br>
I would like to use the <code>deeplabv3_resnet50</code> model.<br>
My image has shape (256, 256, 3) and my label has shape (256, 256). Each pixel in my label has a class value(0-4). And the batch size set in the DataLoader is 32.<br>
The shape of my input batch is <code>[32, 3, 256, 256]</code> and the shape of corresponding target is <code>[32, 256, 256]</code>. I believe this is correct.<br>
I was trying to use <code>nn.BCEWithLogitsLoss()</code>.</p>
<ol>
<li>Is this the correct loss function for my case? Or should I use <code>CrossEntropy</code> instead?</li>
<li>If this is the right one, the output of my model is <code>[32, 5, 256, 256]</code>. Each image prediction has the shape <code>[5, 256, 256]</code>, does layer 0 means the unnomarlized probabilities of class 0? In order to make a <code>[32, 256, 256]</code> tensor to match the target to feed into the <code>BCEWithLogitsLoss</code>,  do I need to transform the unnomarlized probabilities to classes?</li>
</ol>
<p>Thank you everyone.</p>
Conv1d implementation using torch.nn.functional.unfold

cll: <p>Hi,</p>
<p>I am trying to implement 1d convolution using the unfold and matrix multiplication operations. Convolution is equivalent to the unfold, matrix multiplication, and fold (or view) operations, and a 2d convolutional layer can be implemented equivalently as follows:</p>
<pre><code class="lang-auto">inp = torch.randn(1, 3, 10, 12)
w = torch.randn(2, 3, 4, 5)
inp_unf = torch.nn.functional.unfold(inp, (4, 5)).squeeze()
mat_a = inp_unf.T
mat_b = w.view(w.size(0), -1)
mat_b = mat_b.t()
out_unf = torch.matmul(mat_a, mat_b).T
out = out_unf.view(1, 2, 7, 8)
print((torch.nn.functional.conv2d(inp, w) - out).abs().max())
</code></pre>
<p>Unfortunately, this does not appear to generalize to 1d convolutional layers:</p>
<pre><code class="lang-auto">inp = torch.randn(1, 5, 10)
w = torch.randn(8, 5, 2)
inp_unf = torch.nn.functional.unfold(inp.unsqueeze(0), 2).squeeze()
mat_a = inp_unf.T
mat_b = w.view(w.size(0), -1)
mat_b = mat_b.t()
out_unf = torch.matmul(mat_a, mat_b).T
out = out_unf.view(...) # Reshape out_unf
print((torch.nn.functional.conv1d(inp, w) - out).abs().max())
</code></pre>
<p>as <em>mat_a</em> and <em>mat_b</em> cannot be multiped (36x4 and 10x8). Currently torch.nn.functional.unfold only supports 4D tensors.</p>
<p>Any help is greatly appreciated!</p>

Ning_Ning: <p>Any updates? I’m having the same issue here.</p>

wliang: <p>This is my implementation for Conv1d that supports groups and bias, you need to unsqueeze input to be 4d</p>
<pre><code class="lang-auto">
groups = 3
batch = 4
out_channels = 6
in_channels = 6
kernel = 5
bias = torch.rand(out_channels)
inp = torch.randn(batch, in_channels, 1, 12)
w = torch.randn(out_channels, in_channels // groups, 1, kernel)
inp_unf = torch.nn.functional.unfold(inp.view(batch * groups, in_channels // groups, 1, 12), (1, kernel))
# inp_unf.size()   # 10 is kernel * in_channels // groups, 8 is conv output width
# torch.Size([12, 10, 8])
out_unf = inp_unf.view(batch, groups, inp_unf.size(1), inp_unf.size(2)).transpose(2, 3).matmul(w.view(1, groups, out_channels // groups, -1).permute(0, 1, 3, 2)).transpose(2,3)
# out_unf.size()
# torch.Size([4, 3, 2, 8])
e_out = torch.nn.functional.conv2d(inp, w, bias=bias, groups=3)
# e_out.size()
# torch.Size([4, 6, 1, 8])
(out_unf.reshape(batch, out_channels, 1, inp_unf.size(2)) + bias.unsqueeze(0).unsqueeze(2).unsqueeze(3) - e_out).abs().max()
# tensor(4.7684e-07)
</code></pre>
Reset optimizer parameters

KickItLikeShika: <p>I have a model and optimizer already trained, then I used a learning rate finder and a method in the LRFinder API to apply this optimal LR to my optimizer <code>optimizer.param_groups[0]['lr] = optimal_lr</code>, now I want to create a new_model instance with new parameters and use the optimizer with applied learning rate, to do that I should reset the parameters of the optimizer like this <code>optimizer.param_groups[0]['params'] = new_model.parameters()</code>, but this doesn’t work, It doesn’t give good results, and the new_model doesn’t converge.</p>
<pre><code class="lang-auto">model = ...
optimizer = optim.SGD(model.parameters(), lr=3e-4, momentum=0.9)

# train them, then decide to find the optimal lr
lr_finder.apply_suggested_lr(optimizer)
new_model = ....
optimizer.param_groups[0]['params'] = new_model.parameters()
</code></pre>
<p>And the problem is not the LR, when I try to manually create a new optimizer with this new optimal LR, I get better results.</p>

ptrblck: <aside class="quote no-group" data-username="KickItLikeShika" data-post="1" data-topic="120501">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kickitlikeshika/48/35443_2.png" class="avatar"> KickItLikeShika:</div>
<blockquote>
<p>And the problem is not the LR, when I try to manually create a new optimizer with this new optimal LR, I get better results.</p>
</blockquote>
</aside>
<p>This approach sounds right, as I don’t think that assigning the <code>parameters()</code> generator would directly work. The optimizer assigns the parameters <a href="https://github.com/pytorch/pytorch/blob/e9c3ce30d44abdab82feba7cfec6eb8470afe135/torch/optim/optimizer.py#L33-L54">here</a> so you could either try to reproduce this method or just create a new optimizer and set the found learning rate afterwards.</p>
How do I ensure that optimization does not find the trivial solution by setting weights to 0?

Warrior: <p>I am trying to train a neural network which takes as input (<code>input_t0</code>) and an initial hidden state (call it <code>s_t0</code>) and produces a new hidden state (<code>s_t1</code>) by transforming the input via a series of transformations (neural network layers). At the next time step, a transformed input (<code>input_t1</code>) and the hidden state from the previous time step (<code>s_t1</code>) is passed to the same model. This process keeps repeating for a couple of steps.</p>
<p>The goal of optimization is to ensure the distance between <code>s_t0</code> and <code>s_t1</code> is small through self-supervision, as <code>s_t1</code> is supposed to be an transformed version of <code>s_t0</code>. In other words, I want <code>s_t1</code> to only carry new information in the new input. My intuition tells me taking the norm of the weights and ensuring the norm does not go to zero (is this even possible?) would be one way to achieve this. However, I’m afraid won’t be the best thing to do necessarily, as it might not encourage the model to update the state vector with new information.</p>
<p>Currently the way I train the model is by taking the absolute distance between <code>s_t0</code> and <code>s_t1</code> via <code>loss = torch.abs(s_t1 - s_t0).mean(dim=1)</code>. Then I call <code>loss.backward()</code> and <code>optimizer.step()</code> which changes the weights. Note that the reason that I use <code>abs()</code> is that the hidden states are produced after applying ReLU, so the only hold positive values. So what is the best way to achieve this and ensure the weights don’t go to 0?  Would I be able to somehow use mutual information for this?</p>
<p>However, I noticed that optimization quickly finds the trivial solution by setting weights to 0. This causes both <code>s_t0</code> and <code>s_t1</code> get smaller and smaller until their difference is 0, which satisfies the constraint but does not yield the behavior I expect. Is there a way to ensure weights do not go to zero during optimization?</p>
Best way to change model submodule

julianolm: <p>Hello, I am wondering wich of these two ways is the best for changing a module’s submodule:</p>
<p>Here I want to change model’s “conv” module to a new convolution (different in and out sizes)</p>
<pre><code class="lang-auto">with torch.no_grad():
    new_conv = nn.Conv2d(...)
    W = nn.Parameter(new_conv.weight, requires_grad=True)
    model.conv1.weight = W
</code></pre>
<pre><code class="lang-auto">with torch.no_grad():
    new_conv = nn.Conv2d(...)
    model.conv1 = new_conv
</code></pre>
<p>Are both ways equivalent or there’s a better one? (in the case that none of the convolutions have a bias)</p>

ptrblck: <p>The approaches would be a bit different, since the first one would only reassign the <code>weight</code> parameter and keep all other attributes of <code>model.conv1</code> equal (stride, dilation, groups, etc.), while the latter one would reassign the entire module with all its attributes.</p>
Is that possible to train the weights in CrossEntropyLoss?

deJQK: <p>I am trying to train the weight used in CrossEntropyLoss, and my code is something like this:</p>
<pre><code class="lang-auto">...
loss_weights = nn.Parameter(nn.Softmax()(torch.randn(n_classes, device=device, requires_grad=True)))
criterion = nn.CrossEntropyLoss(weight=loss_weights, size_average=False)
optimizer_ft = optim.SGD(list(mdl.parameters()) + list(loss_weights), lr=0.001, momentum=0.9)
...
</code></pre>
<p>But is report ‘can’t optimized non-leaf Tensor’ error.</p>
<p>How is that possible to do this?</p>

richard: <aside class="quote no-group" data-username="deJQK" data-post="1" data-topic="19944">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dejqk/48/4965_2.png" class="avatar"> deJQK:</div>
<blockquote>
<p>loss_weights = nn.Parameter(nn.Softmax()(torch.randn(n_classes, device=device, requires_grad=True)))</p>
</blockquote>
</aside>
<p>The problem with this statement is that a leaf tensor is being created (<code>torch.randn(..., requires_grad=True)</code>) and then it is being hidden because nn.Softmax() returns a new tensor.</p>
<p>To make this work, try something like:</p>
<pre><code class="lang-auto">initial_weights = nn.Softmax()(torch.randn(n_classes, device=device)
loss_weights = torch.zeros_like(initial_weights, requires_grad=True)
# Don't record the following operation in autograd
with torch.no_grad():
    loss_weights.copy_(initial_weights)
</code></pre>
<p>and then proceed to optimize <code>loss_weights</code></p>

deJQK: <p>Hi <a class="mention" href="/u/richard">@richard</a>, thanks for your kind help. Finally I changed my code in some other way but got another error saying <code>'the derivative for 'weight' is not implemented'</code>, so it seems impossible to optimize the weights in CrossEntropyLoss function, and on the official website, it is said <code>manual rescaling weight</code>. Maybe I need to use some log function from torch to implement the cross entropy loss by myself.</p>

GUPTA_NAMAN: <p>you cannot have them trainable. your weightage might go to negative infinity.</p>

deJQK: <p>Hi Naman,</p>
<p>Thanks. The softmax operation prevents infinity.</p>
GraphModule.to_folder produces bad syntax in module.py

amrmartini: <p>I’m trying to leverage some new FX features, and part of the pipeline I have in mind would involve the to_folder method. However, it produces sometimes syntactically incorrect outputs. Here’s an example:</p>
<pre><code class="lang-auto">from torch.fx import symbolic_trace
import torchvision.models as tvm

model = tvm.resnet18()
model_fx = symbolic_trace(model)
model_fx.to_folder('debug', 'Resnet18Transformed')
from debug import Resnet18Transformed
</code></pre>
<p>I get the following error:</p>
<pre><code class="lang-auto">    def forward(self, x : torch.Tensor) -&gt; torch.Tensor:
    ^
IndentationError: unexpected indent
</code></pre>
<p>It seems like because the forward function has a dependency on the torch module (it calls torch.flatten), FX tries to insert an “import torch” near the dependency, even if that insertion breaks syntax and despite that there’s already another “import torch” in the beginning.</p>
<p>Any better idea on why this is happening, and what does it take to fix fundamentally? Thanks</p>
Combine LSTM and CNN in the final layer to get the final result

Young19: <p>Is it possible to build such a model structure to get the final result:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/f/7fcabde70020d50db2466ac8f99588aee32c22fd.png" data-download-href="https://discuss.pytorch.org/uploads/default/7fcabde70020d50db2466ac8f99588aee32c22fd" title="Screen Shot 2021-05-07 at 12.19.10 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7fcabde70020d50db2466ac8f99588aee32c22fd_2_648x500.png" alt="Screen Shot 2021-05-07 at 12.19.10 PM" data-base62-sha1="iev0GwF0eAYMl7bKzrP7eZJostn" width="648" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7fcabde70020d50db2466ac8f99588aee32c22fd_2_648x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7fcabde70020d50db2466ac8f99588aee32c22fd_2_972x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7fcabde70020d50db2466ac8f99588aee32c22fd_2_1296x1000.png 2x" data-dominant-color="EDF1F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-05-07 at 12.19.10 PM</span><span class="informations">1296×1000 89.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Tejan_Mehndiratta: <p>Absolutely.<br>
Notation: N is batch_size, h is lstm hidden_size</p>
<p>The output of CNN would be of shape [N, C, W, H].<br>
Then reshape it to [N, W * H, C].</p>
<p>The output of the LSTM at the last time step would be [N, 1, h].<br>
Now, on this output of LSTM, you can do one thing is do <code>.repeat(1, W * H, 1)</code>,<br>
so that the output would now be of shape [N,  W * H, h]</p>
<p>Now, you can concatenate the output of CNN and LSTM at dim=2 so that the overall output shape would be: [N, W * H, C + h].</p>
<p>Now, you can pass it to the linear layer and do whatever you want to do.</p>

Young19: <p>Thank you very much. This is the first time I am going to combine these two models, thank you for your suggestions.</p>
2 * 1024 input vector

ali_ebrahimi: <p>hello guys!</p>
<p>I want to design a binary classifier,<br>
I have a dataset with size 5000 and 3 columns (<br>
‘text_a’ a vector with 1024 numbers(this is the output of Roberta),<br>
‘text_b’ a vector with 1024 numbers,<br>
‘label’ 0/1)</p>
<p>in each row, if the text_a and text_b be similar the label would be 1 otherwise it would be 0.</p>
<p>I want to design a classifier that gives 2 inputs, each input with size 1024 but I can’t.</p>
<p>could you help me how should be the first layer? or the whole model?</p>

Tejan_Mehndiratta: <p>Not clearly understood the problem statement. Please write something extra or in different language for me to help you out.</p>

ali_ebrahimi: <p>thanks for your reply,</p>
<p>I am a beginner in Paytorch. And I only work with networks with one input.<br>
Here we have two inputs for a classifier and I do not know what the network architecture and input layer should be like?</p>

Tejan_Mehndiratta: <p>Okay. So lets do one thing. Since I am not able to grasp your task, so you can share your dataset with me via email. I will give it time and update you sometime tomorrow.</p>
<p>email: <a href="mailto:tejanmehndiratta15@gmail.com">tejanmehndiratta15@gmail.com</a></p>

ali_ebrahimi: <p>ok, thank you very much</p>

Tejan_Mehndiratta: <p>And lets chat on private msg regarding this.</p>
RuntimeError: mat1 and mat2 shapes cannot be multiplied (4x5645376 and 121x192

I192121_Tayyab_Iqbal: <p>lass Net(nn.Module):</p>
<pre><code>def __init__(self):

    super(Net, self).__init__()

    self.conv1 = nn.Conv2d(3, 192, 3)

    self.pool = nn.MaxPool2d(2, 2)

    self.conv2 = nn.Conv2d(192, 921, 3)

  

    self.conv3 = nn.Conv2d(921, 212, 3)

    self.conv4 = nn.Conv2d(212, 121, 3)

    

    self.fc1 = nn.Linear(121, 192)

    self.fc2 = nn.Linear(121, 192)

    self.fc3 = nn.Linear(192, 120)

def forward(self, x):

    x = F.relu(self.conv1(x))

    x = F.relu(self.conv2(x))

    x = F.relu(self.conv3(x))

    x = F.relu(self.conv4(x))

    

    x = x.view(x.size(0), -1)

    x = F.relu(self.fc1(x))

    x = F.relu(self.fc2(x))

    x = self.fc1(x)

    return x</code></pre>

eqy: <p>The input feature size must be matched to the linear layer size. (Consider the output shape after the last conv layer and the expected input size to the first fully connected layer)</p>

Tejan_Mehndiratta: <p>Change this:</p>
<pre><code class="lang-auto">self.fc1 = nn.Linear (5645376, 192)
</code></pre>