akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Size of input images/tensor to mobilenet and other pretrained models

username: John_J_Watson
cooked: <p>I am relatively new to transfer learning and I have been working through tutorials on pytorch’s webpages. I notice that they all say that the minimum size they expect is 224x224 and I get why - the kernels are trained on 224x224 to extract valuable features.</p>
<p>However, I just tried to use mobilnetv2 with a dimension of 128x128 and it worked fine and the results are also impressive. To be clear, my input tensor is of size 3,128,128. Why? Shouldn’t it error out since the minimum dimension size is not met?</p>
<p>Does that now mean that if I manage to feed it 224x224 (memory was an issue and hence I downsized it) the result ill be better?</p>

username: Alpha
cooked: <p>I guess the last pooling layer in your implementation is <code>AdaptivePool</code>, which could deal with any size and the output is <code>(b, c, 1, 1)</code>. So, it is ok to use <code>112x112</code>.</p>
<p>Besides, because the Mobilenetv2 has a downsample rate of <code>32</code> ,  your input size is OK when it is larger than <code>32</code>.</p>
<p><code>224x224</code> is a commonly used size for imagenet classification task  and  I think <code>224</code> is very likely better than <code>112</code>.</p>

username: John_J_Watson
cooked: <p>Thank you <a class="mention" href="/u/alpha">@Alpha</a> for your reply. It makes sense.</p>
Multiple (2) optimizers for a single model

username: John_J_Watson
cooked: <p>I have a use case, where I think using two optimisers would help the model learn better. Let me explain: so, I have one model like so:</p>
<p><code>my_model=Net()</code></p>
<p>which is trained over <code>N</code> samples belonging to training_set1. Now, I use results of this trained model (embeddings to be precise) to generate another training_set2, which is a much <code>harder_to_learn</code> training set than training_set1 (btw: I cannot generate training_set2 without training on training_set1).</p>
<p>So, my plan is to use two optimisers like so:</p>
<pre><code class="lang-auto">my_model=Net()
optimiser1=Adam(my_model.params()) # rough syntax ;)
optimiser2=Adam(my_model.params()) # rough syntax ;)

for epoch in epochs:
    # train my_model on train_set1 using optimiser1

    # generate training_set2, much harder samples
    # and run a small number of epochs on these harder samples

    for training_set2_epoch in training_set2_epochs:
          # train my_model using optimiser2
</code></pre>
<p>is this a valid method to train a single model?</p>
<p>When I use two optimisers like above, my results are much better than when I just use a single optimiser to train the model. Is this fluke ?</p>

username: ptrblck
cooked: <p>Your approach should be valid code-wise.<br>
The optimizers would only apply their <code>step()</code> method to all parameters, which have a valid gradient, so there shouldn’t be a problem.</p>

username: John_J_Watson
cooked: <p>thank you for your time <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
API Rest with several models loaded using GPU but not at same time

username: WaterKnight
cooked: <p>I am creating an API Rest that does inference with several models given a list.</p>
<p>The problem is that all the models don’t fit at GPU at the same time.</p>
<p>Is there a way of loading a model into GPU make inference with that model and move it to CPU and load next model to GPU for inference then to CPU…</p>

username: ptrblck
cooked: <p>You could push a single model to the device, perform the forward pass to get the prediction, and push it back to the host using a loop like:</p>
<pre><code class="lang-python">models = [...] # define a list of all models on the CPU

input = ... # get your input
for model in models:
    model.to('cuda')
    pred = model(input)
    model.to('cpu')
</code></pre>

username: WaterKnight
cooked: <p>Thank you for comming to this issue <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>
<p>I have tried that approach, the problem that I have seen is that if I don’t delete <code>pred</code> variable after using, it keeps memory in use and when I load other model cuda memory error is thrown!</p>

username: ptrblck
cooked: <p>For inference only, you should wrap the forward pass in a <code>with torch.no_grad()</code> block, which will not store the intermediate tensors, which would be needed for for <code>backward</code> call.</p>

username: WaterKnight
cooked: <p>Good to know. Do I need to delete the predictions too?</p>

username: ptrblck
cooked: <p>If you’ve wrapped them in the <code>no_grad</code> block, then it’ll be just a tensor on the current device (and would probably not take much memory).<br>
You could delete it anyway, if you want to save this memory, but I don’t think it’s necessary.</p>

username: WaterKnight
cooked: <p>Ok, thank you for the infor!</p>
What are the advantages of using torch.sparse api

username: Mainul
cooked: <p>Hi,<br>
Lets say, my input is a sparse vector of 5000 dim. If I declare a normal linear layer</p>
<p><code>L=nn.Linear(5000,500)</code><br>
<code>output=L(input_sparse_vector)</code></p>
<p>and optimize it as usual instead it using torch.sparse, what problem I might have?</p>
<p>will the gradient calculation in every step will be the same?</p>
CosineEmbeddingLoss() strange behaviour

username: aroibu1
cooked: <p>I have been trying for a while to use CosineEmbeddingLoss() as the loss function for a network that I am working on. However, when I tried a simple example to check some of it’s features, I am running into some strange behaviour, as below.</p>
<p>If, for example, I do the following, then the behaviour is as expected…</p>
<pre><code class="lang-auto">&gt;&gt;&gt; loss = torch.nn.CosineEmbeddingLoss()
&gt;&gt;&gt; input1 = torch.rand(1,2,10,10)
&gt;&gt;&gt; y3 = torch.ones(input1.size())
&gt;&gt;&gt; loss(input1,input1,y3)
tensor(0.)
</code></pre>
<p>However, if I define a random tensor and a tensor mask, and try again, I am geetting several weird results, as follows:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; loss = torch.nn.CosineEmbeddingLoss()
&gt;&gt;&gt; a = torch.rand(3,3)
&gt;&gt;&gt; a
tensor([[0.3415, 0.3722, 0.7526],
        [0.4574, 0.1331, 0.1791],
        [0.0287, 0.1917, 0.7043]])
&gt;&gt;&gt; b = torch.tensor([[0.,0.,0.],[0.,1.,0.],[0.,0.,0.]])
&gt;&gt;&gt; a * b
tensor([[0.0000, 0.0000, 0.0000],
        [0.0000, 0.1331, 0.0000],
        [0.0000, 0.0000, 0.0000]])
&gt;&gt;&gt; y = torch.ones(a.size())
&gt;&gt;&gt; loss(a,a,y)
tensor(0.)
&gt;&gt;&gt; loss(a*b, a*b, y)
tensor(0.6667)
</code></pre>
<p>When comparing a x b with a x b, we should be looking at exactly the same tensors in this case,hence the loss should be 0. However, it is not… Similarly, I get an identical answer if I do the following:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = torch.tensor([[0,0,0],[0,1,0],[0,0,0]])
&gt;&gt;&gt; b = torch.tensor([[0,0,0],[0,5,0],[0,0,0]])
&gt;&gt;&gt; y = torch.ones(a.size())
&gt;&gt;&gt; loss(a,a,y)
tensor(0.6667)
&gt;&gt;&gt; loss(a,b,y)
tensor(0.6667)
</code></pre>
<p>Can someone please help me understand why this behaviour is occuring? Thank you!</p>

username: ptrblck
cooked: <p>The problem arises, as the cosine similarity between zero vectors is undefined.<br>
In PyTorch <code>nn.CosineEmbeddingLoss</code> returns apparently a loss of 1 for zero tensors:</p>
<pre><code class="lang-python">loss = torch.nn.CosineEmbeddingLoss(reduction='none')
b = torch.tensor([[0.,0.,0.],[0.,1.,0.],[0.,0.,0.]])
print(loss(b, b, torch.ones_like(b)))
&gt; tensor([[1., 0., 1.],
        [1., 0., 1.],
        [1., 0., 1.]])
</code></pre>
<p>which would return <code>0.6667</code> if you calculate its mean.</p>

username: aroibu1
cooked: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>! You are a legend! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Visualization of a multidimensional tensor

username: morrohype
cooked: <p>Hello,</p>
<p>I’m looking for ways to visualize a multidimensional tensor. In my example the tensor is of size: torch.Size([32, 32, 4, 4])</p>
<p>I would be happy with any kind of help!</p>

username: aktgpt
cooked: <p>Since you’ve not specified which dimensions you want to visualize, I’m assuming your data is in the format of <code>(B x C x H x W)</code>. Then you can just use,</p>
<pre><code class="lang-auto">torchvision.utils.make_grid(tensor)
</code></pre>
<p>which will expand your tensor in B and C to give you a tensor of <code>((BxC) x H x W)</code> which you can then convert to numpy array and view.</p>
<p>For more information, you can refer to:<br>
<a href="https://pytorch.org/docs/stable/torchvision/utils.html#torchvision-utils" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/torchvision/utils.html#torchvision-utils</a></p>
Euclidean Distance

username: julioeu99
cooked: <p>Hi, can someone, explain me better this formula: <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.euclidean.html#scipy.spatial.distance.euclidean" rel="nofollow noopener">https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.euclidean.html#scipy.spatial.distance.euclidean</a></p>
<p>I don´t understand if the formula is just this: ||u-v||^2<br>
Or the second formula.<br>
Thanks.</p>

username: harsha_g
cooked: <p>The formula is ||u-v||_2 (note this has a square root included) and the weights being 1 in all dimensions. Please note that you have a ^2 in your formula. And the second one (in the documentation) with a <em>w</em> implies ‘if the weights are specified’. Hope this makes sense.</p>