akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Passing output image from top neural network to below, probabilities from below to top

vainaijr: <p>top neural network outputs an image, which needs to be fed to two below neural networks, which feed image to below two neural networks, and then return probability back to the top, this implementation uses JoinProbs class to compute probability, but this probability needs to be done within Top, Base, Middle classes, when I attempt at calling forward method of Middle from Top, it does not give desired result.</p>
<pre><code class="lang-auto">class Top(nn.Module):
  def __init__(self):
    super().__init__()
    self.encoder = nn.Sequential(nn.Conv2d(1, 1, 1), nn.Dropout(0.5))
    self.lin = nn.Linear(10, 10)
  def forward(self, x):
    return self.encoder(x), self.encoder(x)

net = Top()

outp1, outp2 = net(torch.randn(1, 1, 28, 28))

outp1.shape, outp2.shape

class Middle(nn.Module):
  def __init__(self):
    super().__init__()
    self.encoder = nn.Sequential(nn.Conv2d(1, 1, 1), nn.Dropout(0.5))
    self.lin = nn.Linear(10, 10)
  def forward(self, x):
    return self.encoder(x), self.encoder(x)

net2 = Middle()

outp11, outp12, outp21, outp22 = *net2(outp1), *net2(outp2)

y = map(lambda x: x.shape, (outp11, outp12, outp21, outp22))

list(y)

class Base(nn.Module):
  def __init__(self):
    super().__init__()
    self.lin = nn.Linear(784, 10)
  def forward(self, x):
    out = (x.view(x.size(0), -1))
    return self.lin(out)

net3 = Base()

prob11, prob12, prob21, prob22 = net3(outp11), net3(outp12), net3(outp21), net3(outp22)

y = map(lambda x: x.shape, (prob11, prob12, prob21, prob22))

list(y)

class JoinProbs(nn.Module):
  def __init__(self):
    super().__init__()
    self.lin = nn.Linear(20, 10)
  def forward(self, *x):
    out = torch.cat(x, dim=-1)
    return self.lin(out)

net4 = JoinProbs()

a, b = net4(prob11, prob12), net4(prob21, prob22)

a, b

net4(a, b)
</code></pre>

vainaijr: <p>I modified it to use ‘up’ or ‘down’ direction</p>
<pre><code class="lang-auto">from IPython.core.debugger import set_trace

class Top(nn.Module):
  def __init__(self, direction):
    super().__init__()
    self.encoder = nn.Sequential(nn.Conv2d(1, 1, 1), nn.Dropout(0.5))
    self.lin = nn.Linear(10, 10)
    self.direction = direction
    if self.direction == 'down':
      self.childone = Bottom()
      self.childtwo = Bottom()
  def forward(self, x):
    if self.direction == 'down':
      return self.childone(self.encoder(x)), self.childtwo(self.encoder(x))
    elif self.direction == 'up':
      return self.lin(x)

class Bottom(nn.Module):
  def __init__(self):
    super().__init__()
    self.lin_one = nn.Linear(784, 100)
    self.lin_two = nn.Linear(100, 10)
    self.parent = Top('up')
  def forward(self, x):
    return self.parent(self.lin_two(self.lin_one(x.view(x.size(0), -1))))
</code></pre>
<p>but this also has problem, as it gives output to Top class twice, so finally I get tuple instead of one output.</p>
Current torch.min() does not support multiple dimensions?

Kevin96: <p>Let say I have a 3 dimensional tensor.</p>
<blockquote>
<p>a = torch.arange(8).reshape(2, 2, 2)</p>
</blockquote>
<blockquote>
<p>&gt;&gt;&gt; a<br>
tensor(<br>
[[[0, 1],<br>
[2, 3]],<br>
[[4, 5],<br>
[6, 7]]])</p>
</blockquote>
<p>I want to compute the minimum over the dimension 1 and 2, to get</p>
<blockquote>
<p>tensor([0, 4])</p>
</blockquote>
<p>However, torch.min() does not support multiple dimensions. In contrast, torch.sum() does support? Is there any particular reason for that?</p>
<p>So for this specific case, currently the most efficient way to take minimum over multiple dimensions is to first reshape the tesnor</p>
<blockquote>
<p>a.reshape(2, 2 * 2)</p>
</blockquote>
<p>and then take the minimum over the second axis?</p>

spanev: <p>Hi <a class="mention" href="/u/kevin96">@Kevin96</a>,</p>
<p>You should do:</p>
<pre><code class="lang-auto">a.view(-1,2*2).min(axis=1)
</code></pre>
<p>where <code>view</code> <a href="https://discuss.pytorch.org/t/in-pytorch-0-4-is-it-recommended-to-use-reshape-than-view-when-it-is-possible/17034/2">guarantees</a> no memory copy, and almost no overhead.</p>

mru4913: <p>HI kaiwen,</p>
<p>not very memory-efficient but “easy”?</p>
<pre><code class="lang-auto">def mul_min(x, axis, keepdim=False):
  axis = reversed(sorted(axis))
  min_x = x
  for i in axis:
    min_x, _ = min_x.min(i, keepdim)
  return min_x
</code></pre>
Can pytorch by-pass python gil?

coincheung: <p>Hi,</p>
<p>Suppose there is an occasion where I use python multi-thread to do inference with two models. That is thread 1 use model 1 to infer picture 1, and thread 2 use model 2 to infer picture 2. Suppose python gil switches among the two threads every 2ms, and the inference time is 200ms. Will the whole time of running the two models concurrently be 400ms or less than that ? Will the inference on one thread be interrupted and turned into waiting mode when another thread is active, or will it not be affected?</p>

albanD: <p>Hi,</p>
<p>We do release the GIL as soon as we get out of python code. So for pytorch ops, it’s more or less all of them.<br>
The backward, unless you implement custom Functions, will run completely out of the GIL.</p>
Which function is invoked when do slicing on Tensor?

Crazyai: <p>When two tensors are added together, for example,</p>
<pre><code class="lang-auto">c = a + b
</code></pre>
<p>the function <code>tensor.__add__</code> is invoked to really “add” two tensors.</p>
<p>When a tensor is sliced on a given axis, which function is invoked?</p>
<pre><code class="lang-auto"># slice on axis=1
b = a[:, 1:2]
</code></pre>
<p>I searched for <code>*slice</code> on the pytorch reference documentation but found nothing.</p>

albanD: <p>Hi,</p>
<p>The <a href="https://docs.python.org/3/reference/datamodel.html#object.__getitem__" rel="nofollow noopener"><code>__getitem__</code></a> method of that class is called.<br>
If you’re interested of how pytorch implements this, it is routed to <a href="https://github.com/pytorch/pytorch/blob/58a0dee749e1bbac92bdedf0379ac89158ca93d3/torch/csrc/autograd/python_variable_indexing.cpp#L284" rel="nofollow noopener">this</a> function.</p>
ImageFolder vs custom dataset for image classification

Krithika_Challani: <p>Hello,</p>
<p>I need to do image classifcation and also use the same images alsong with another sensor measurement into a lstm. Therefore, i have created my own custom dataset class following this <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">tutorial</a><br>
However, image classification by fine tuning image classifier models like resnet50, vgg19 etc does not perform well with custom dataset class. If i use <a href="https://pytorch.org/docs/stable/torchvision/datasets.html#imagefolder" rel="nofollow noopener">ImageFolder</a> , the classifier works well with accuracy above 95 %. Is there a way to use custom dataset  class for image classification</p>

albanD: <p>Hi,</p>
<p>ImageFolder or a custom Dataset are just two different ways to read the data from disk.<br>
If you observe wildly different results when using one or the other, then there is a bug in some of the code.</p>
Hardcoded numbers in image transforms?

shresta: <p>Hi,</p>
<p>I am getting started with pytorch by browsing code and in a lot of example code i see magic numbers used away without explanation or comments. What do these numbers mean. I come from a C/C++ background where it is considered a horrible practice to ever use numbers in the code, preferring <span class="hashtag">#defines</span> on top to explain the values. But these numbers do not seem specific for the project or an author idiosyncrasy and seems more like a pytorch norm.<br>
For example, In the pytorch tutorial page the parameters to the Normalise function,</p>
<pre><code class="lang-auto">        transforms.Normalize(mean=[0.485, 0.456, 0.406],
                             std=[0.229, 0.224, 0.225])
</code></pre>
<p>Other examples are permute and view functions in one of the authors on github.</p>
<pre><code class="lang-auto">    im = cv2.imread(dirpath + '/' + file)
    im = torch.Tensor(im).permute(2, 0, 1).view(1, 3, 224, 224).double()
    model.eval()
    im -= torch.Tensor(np.array([129.1863, 104.7624, 93.5940])).double().view(1, 3, 1, 1)
</code></pre>
<p>Thanks for the help!</p>

ptrblck: <p>These numbers are taken from the ImageNet dataset (<a href="https://pytorch.org/docs/stable/torchvision/models.html" rel="nofollow noopener">info about pretrained models</a>).<br>
As these stats were used to pretraind the models, they should also work for finetuning.<br>
However, if your dataset samples come from another domain (e.g. medical images), you would most likely want to use the stats from your training dataset.</p>

shresta: <p>Great thanks for the quick response. Useful link. That helps me understand Normalise and the first example fully.</p>
<p>Regarding the 2nd example, Any convention regarding transforms like permute or view?<br>
When data is given by the dataloader does it apply any of these transforms automatically? And because this examples is reading the image from memory without using a data loading abstraction these transforms are done by the author? I do not have any other popular repositories at the moment. I am looking at this example currently. <a href="https://github.com/prlz77/vgg-face.pytorch/blob/master/models/vgg_face.py" rel="nofollow noopener">Repo Link</a> But I can get back with more examples where a lot of image manipulation seems to be done by operations that seems obvious for the pytorch community but not for an outsider.</p>

polarbaryon: <p>Permute: OpenCV uses BGR rather than RGB, so to import the data correctly, it has to permute the channels.</p>
<p>View is just the right way of viewing that imported data correctly.</p>
Forward() missing 2 required positional arguments: 'input' and 'target'

Tsakunelson: <p>Hello everyone,<br>
I am quite new with Skorch, and I am in the process of applying K-fold cross-validation using data Loaders from a directory of images.<br>
After creating the train and validation loaders, I user the <code>fit(X,y)</code> method from NeuralNet class, with <code>X = current_train_loader</code>, and <code>y = current_train_target</code>. As soon as the <code>fit(X,y)</code> executes, the following error occurs:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/5915971a21a46131de61f20e1032d8dd63c6c2f5.png" data-download-href="https://discuss.pytorch.org/uploads/default/5915971a21a46131de61f20e1032d8dd63c6c2f5" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/5/5915971a21a46131de61f20e1032d8dd63c6c2f5.png" alt="image" data-base62-sha1="cI4KBJlCsr7Mj3cnXvNRDkzTx9b" width="690" height="194" data-dominant-color="0F0F0F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">987×278 12.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Whereas, I fill in the input and target exactly as mentioned. Is there anything wrong with my procedure? Following is what my training phase looks like:</p>

Tsakunelson: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/8/8701972d25e0309eba35c682033ec99dc5c0eb51.png" data-download-href="https://discuss.pytorch.org/uploads/default/8701972d25e0309eba35c682033ec99dc5c0eb51" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/8/8701972d25e0309eba35c682033ec99dc5c0eb51.png" alt="image" data-base62-sha1="jgjRHImu5nTcRJLp2eH0kolrVAZ" width="588" height="500" data-dominant-color="0D0D0E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">643×546 21.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>thanks</p>

Eddy_Hsu: <p>This also happened to me when I initialized the criterion. For example,</p>
<p>criterion = torch.nn.BCELoss()</p>
<p>I solve this problem by:</p>
<p>criterion = torch.nn.BCELoss</p>
Can pytorch dataloader prefetch training samples

coincheung: <p>Hi,</p>
<p>I am trying to understand how the dataloader works, and I come in to a question: can pytorch prefetch the training samples during training, so that the model do not need to wait for the data before one round of training?</p>
<p>If pytorch has this function, how could I switch on or switch off it?</p>

ptrblck: <p>If you are using multiple workers (<code>num_workers&gt;=1</code>), the next batches will be prefetched once the code enters the <code>DataLoader</code> loop.</p>
Existing implementation of ResNet on a MLP setting?

mikeaalv: <p>I’m start to use Pytorch and trying to implement ResNet for my problem, which is not related to images or NLP. The implementation of ResNet in Pytorch has convolutional layers which I don’t need and don’t want to include at the beginning. Are you guys aware of existing implementation of ResNet in Pytorch in a MLP type of setting? Or what I really need to do is reimplement the ResNet in Pytorch?</p>
<p>Thank you.</p>

ptrblck: <p>If you are removing the conv layers in the ResNet implementation, I guess you would like to create a fully connected model with some skip connections?<br>
If so, you could have a look at the current implementation of <a href="https://github.com/pytorch/vision/blob/e4d5003956db97d4e4bc1055ec8b045c39ee4882/torchvision/models/resnet.py#L35" rel="nofollow noopener"><code>BasicBlock</code></a> and <a href="https://github.com/pytorch/vision/blob/e4d5003956db97d4e4bc1055ec8b045c39ee4882/torchvision/models/resnet.py#L76" rel="nofollow noopener"><code>BottleNeck</code></a> and try to adapt these modules to your code.</p>

mikeaalv: <p>Thank you. That’s actually what I’m doing currently. I try read and modify the code in github: pytorch/vision/blob/master/torchvision/models/resnet.py.</p>
Get validation set out of training set

Andrea_Rosasco: <p>I noticed that when similar questions are asked the commonly given advice is to use <code>SubsetRandomSampler</code> or <code>random.split</code>. Isn’t it wrong, though?</p>
<p>I always thought that, during validation, the validation set should be the same through all the different runs, since, changing it every time wouldn’t allow getting a good estimate of the error given the hyperparameters.</p>
<p>I guess the solution is as simple as setting a manual seed before calling <code>random.split</code> and then setting it back to its initial value. I just wonder if my idea of how validation works is correct.</p>

ptrblck: <p>I would say it depends a bit on your use case.<br>
If you are running some quick experiments, setting the seed and splitting the datasets randomly might be sufficient to reproduce your results.<br>
However, if you really need the same splits for each run, you could sample the split indices once, store them locally, and use them in a <code>Subset</code> to get the same training, validation (and test) sets.</p>
Trained model using DataParallel cannot be loaded for inference on 1 gpu

kirk86: <p>Hi folks,<br>
I have question regarding <code>DataParallel</code>. When I use it to train a model on 2 gpus and then try to load the model only on gpu:0 for prediction I keep getting a lot of mismatch keys from the state dictionary. When I inspect the state dictionary I see that all params of the model are on gpu:0, so I don’t understand why I keep getting these mismatch keys?</p>
<p>Instead if I load the model for prediction with <code>DataParallel</code> everything seems to work fine. My initial thought while getting mismatch keys was that maybe some parts of the model are on one gpu and others on the other, but that’s not the case since after loading the checkpoint and inspecting all params everything seems to be on gpu:0.</p>

JuanFMontesinos: <p>When you save a model from Dataparallel it adds an additional module which is reflected on the keywords such that 1st word is always module.</p>
<pre><code class="lang-auto">        new_state_dict = OrderedDict()

        for k, v in state_dict.items():
            if k[:7] == 'module.':
                name = k[7:]  # remove `module.`
            else:
                name = k
            new_state_dict[name] = v
</code></pre>

kirk86: <p>Thank you, Juan<br>
that’s refreshing to know especially when these things are not reflected in any documentation.<br>
If you don’t mind me asking do you have any idea on what is the proper usage of <code>distributed.DataParallel</code>?</p>
<p>For instance <a href="https://discuss.pytorch.org/t/error-upon-using-distributed-data-parallel/55439">here</a> I’m explaining how I’ve used it and still end up getting error during execution.</p>

ptrblck: <p>Some information regarding the serialization of a data parallel model is given <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-torch-nn-dataparallel-models" rel="nofollow noopener">here</a>. Where have you been looking and where would you like to see another mention of this snippet?</p>

kirk86: <p>Oops, my bad I overlooked that, I guess I didn’t scroll down all the way to the end of the page.</p>
Intel MKL Error when using torch.symeig

bbrighttaer: <p>Hi,<br>
I have a symmetric matrix <code>X</code> which is constructed using the outputs of network for each batch. I then find the eigenvalues and eigenvectors of <code>X</code> using</p>
<pre><code class="lang-python">e, V = torch.symeig(X, True)
</code></pre>
<p>While this works initially, I get the error below after some time. I think this may be due to the outputs of the network. Pls, is there any approach I can use to avoid this error?</p>
<pre><code class="lang-auto">Intel MKL ERROR: Parameter 4 was incorrect on entry to SLASCL.

Intel MKL ERROR: Parameter 4 was incorrect on entry to SLASCL.
info: 99
, submat: 0
info: 901

symeig_cuda: the algorithm failed to converge; 901 off-diagonal elements of an intermediate tridiagonal form did not converge to zero.
</code></pre>