akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Best practice on transfer learning

temi92: <p>HI all,<br>
I would like some advice on best practice on transfer learning.<br>
Supposing i have a dataset A, and i have done some training with that dataset to produce optimum weights W1, If i have an additional dataset B, and i decide to start training from weights W1,  in an effort to produce optimum weights W2 that best maps my data.  is it best to train with just the additional dataset B or should i resume training from W1  with dataset A+B.</p>

ptrmcl: <p>How similar are the two datasets? Do they have the same labels?</p>

temi92: <aside class="quote no-group" data-username="ptrmcl" data-post="2" data-topic="121023">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrmcl/48/32259_2.png" class="avatar"> ptrmcl:</div>
<blockquote>
<p>Do they have the same labels?</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/ptrmcl">@ptrmcl</a> thank you for taking the time out to reply. and yes assuming dataset A and dataset B have the same labels and are relatively similar. I would like to know when applying transfer learning with a learned set of weights that was trained on dataset A. In order to get good results on dataset B, do i need to feed my model just dataset B alone or dataset A +B</p>

ptrmcl: <p>So are the data in both datasets something you may see when using your model? In that case, I think I’d use both if you have the computational resources.</p>
<p>Would be good to get others’ opinions too.</p>

davidweb: <p>Hey, <a class="mention" href="/u/temi92">@temi92</a>!<br>
I have applied transfer learning once, and did it that way:</p>
<ul>
<li>I used a pretrained VGG on Imagenet (your dataset A with weights W1)</li>
<li>Then, I used this VGG to train on Imagenette (a similar dataset, your dataset B)</li>
<li>So in my case, it was best to just continue training with dataset B, however, note that the VGG already converged on the dataset A.</li>
</ul>
<p>Let me know if you have any questions.<br>
BR, David</p>

ptrmcl: <p>So yeah frankly in the case of Imagenet it makes sense since it is a large dataset. If both datasets are small I think it may be better to train on A+B?</p>

davidweb: <p>Yes, might be. But depending on the architecture used, and the dataset size, I would also worry about potential overfitting. Why not try both ways, using a validation set during training and testing what works best.</p>
Fusion of operations when lots of redundant computations are present?

Haxxardoux: <p>I have a very strange task that requires pairs of inputs be passed through a neural network, where the pairs create many-to-many relationship, and each “many” is really “very many”. This means that for each input tensor from set 1, there will be lots of redundant calculations made for each comparison made with each tensor from set 2. I believe one could do operation fusion to speed this up - fusion of a repetitive input and a similar layer seems conceptually similar to fusion of a conv and BN layer, however i cannot seem to find a good way to do this. Most of the ideas I came up with are stupid, and do not actually do fusion.</p>
<p>Does anyone know of a good way to do this - or - if it is even possible natively in pytorch?</p>
<p>Additionally, do people believe my hypothesis is correct, and this would boost performance measurably?</p>
Can Pytorch0.4.1 support cuda10.0

Robby: <p>I use RTX2080 to run my code on pytorch0.4.1<br>
But I found that RTX2080 just support cuda 10.0,while pytorch 0.4.1 can’t support cuda 10.0.<br>
Do you know how to fix it up?</p>

ptrblck: <p>You could install CUDA10.0 locally and build PyTorch from source as described <a href="https://github.com/pytorch/pytorch" rel="nofollow noopener">here</a>. If I’m not mistaken, CUDA10 wasn’t released yet when PyTorch 0.4 came out.<br>
What’s the reason you are using such an old PyTorch version?</p>

Robby: <p>Thanks! I will have a try. I use 0.4.1 to run some paper’s codes  which are based on torch 0.4.1.<br>
When I use the higher version to run codes,there are so many errors</p>

WorkHard_For_Mom: <p>Hello! Do you fix you problem?<br>
I use 0.4.1 to run some paper’s codes which are based on torch 0.4.1<br>
But I only have RTX2070</p>

Robby: <p>I didn’t solve this problem. Finally I used pytorch1.0 to run the code based on pytorch 0.4.1.</p>
Backward pass for hyperbolic tangent?

3c06d7576e3434b36c48: <p>Hello I am trying to build a custom loss function which include using tanh.<br>
if forward pass is y=tanhx<br>
than how should be the backward pass?<br>
In other word what should be the relationship between gradOutput and gradInput?</p>

albanD: <p>I guess you’re implementing the backward by hand? Otherwise it will work with the autograd.</p>
<p>The derivative for tanh is quite common and can be found <a href="http://math2.org/math/derivatives/more/hyperbolics.htm" rel="nofollow noopener">here</a> for example. The C implementation of the backward pass by pytorch is <a href="https://github.com/pytorch/pytorch/blob/bd19dd4b90aa94359b84986b0f9a298538ac7889/aten/src/THNN/generic/Tanh.c#L29" rel="nofollow noopener">here</a>.</p>

hsong: <p><a class="mention" href="/u/alband">@albanD</a>  Thanks for the reply. I’m a bit confused by the autograd process. The document says it computes the vector-Jacobian result, which usually returns a vector.</p>
<p>However, the tanh function is element-wise, that is, Y_{ij} = nn.tanh(X_{ij}), where X and Y are both matrices.</p>
<p>In this case, we have a matrix-valued function and the gradient of Y wrt X would be a 4-D Jacobian matrix.</p>
<p>The C-souce code is like this:<br>
<code>*gradInput_data = *gradOutput_data * (1. - z*z);</code><br>
It seems so perform an elementwise multiplication: Y_ij * (1-X_ij^2)</p>
<p>Could you please explain how the auto-grad actually handle this?</p>
<p>P.S.: Sorry if it’s inappropriate to reply such an old post.</p>

albanD: <p>As you mentioned, autograd just does the vector jacobian product.<br>
In this case, you multiply the 2d matrix of gradients wrt the output (grad_output) again a 4D Jacobian matrix for that op.<br>
The nice thing is that the op is element-wise and so the 4D Jacibian is actually diagonal and what this vector jacobian product boils down to is multiply each element of the input 2D matrix with each element in the diagonal of the Jacobian.<br>
Here, <code>(1. - z*z)</code> is actually the “diagonal” of the jacobian and the result is computed by doing an element-wise product with the gradient flowing back to get the gradient wrt the input.</p>

hsong: <p>Thank you so much for your reply!</p>
Piecewise constraint in Pytorch

whoab: <p>Let’s say I am predicting a quaternion. How do I implement this piecewise constraint that is conditioned on the scalar/real “w” (for quaternion = [x,y,z,w]).</p>
<p>In Python:</p>
<pre><code>def positive_real_quat(quat):
    """
    Flip quaternion to positive real hemisphere so there is a single quaternion describing each rotation

    :param quat: [x,y,z,w]
    :return:
    """
    x, y, z, w =quat
    if w &gt;= 0:
        return quat
    else:
        return -quat
</code></pre>
<p>But I need the above code to be differentiable as the final function in my neural network. How do I do it?</p>

ptrblck: <p>Since you are only multiplying with <code>-1</code> for a negative <code>w</code>, your function should not break the computation graph.<br>
I’ve changed the input to a tensor and it seems to work fine:</p>
<pre><code class="lang-python">def positive_real_quat(quat):
    x, y, z, w = quat.split(1, 1)
    if w &gt;= 0:
        return quat
    else:
        return -quat
    
# Positive
x1 = torch.tensor([[1., 1., 1., 1.]], requires_grad=True)
out1 = positive_real_quat(x1)
out1.mean().backward()
# Check grad
print(x1.grad)
&gt; tensor([[0.2500, 0.2500, 0.2500, 0.2500]])

# Negative
x2 = torch.tensor([[1., 1., 1., -1.]], requires_grad=True)
out2 = positive_real_quat(x2)
out2.mean().backward()
# Check grad
print(x2.grad)
&gt; tensor([[-0.2500, -0.2500, -0.2500, -0.2500]])
</code></pre>

whoab: <p>It seems to be saying that you can always “differentiate” through if else conditionals because you only pass the gradient through the conditional that is selected… kind of like how max is differentiable because you only pass the gradient through the maximum value…</p>
<p>Are there any cases where program logic is not differentiable?</p>

ptrblck: <p>Note that <code>torch.max</code> and your approach are a bit different.<br>
While <code>torch.max</code> works on a tensor and will pass the gradient to the max value (as you’ve explained), your approach would create different computation graphs.<br>
Since PyTorch creates the computation graph (which is used to calculate the gradients in the backward pass) dynamically, you can use plain Python conditions, loops, etc. in your code.</p>

whoab: <p>I guess what I’m looking for is some rigorous argument that gradient descent will minimize a computation graph that contains Python conditions and logic.</p>

veghans: <p>Hi, I am also very interested in this question. We are trying to implement a “custom” (to our problem) piecewise function using simple if/else logic. Simplified example; we create torch parameters (a,b,c and d)  and say that e.g. “if input &lt; 5: value = a<em>input + b, else: value = c</em>input + d”.</p>
<p>The backpropagation and optimization seems to only give gradients for some of the values though, although we know that some of the inputs are in the different “ranges” (we have inputs both over and under 5, following the example above).</p>
<p>Will Pytorch accept such a function and the backpropagation + optimization (SGD) algorithm actually work for our function?</p>
Displaying Colormap over Image

davidweb: <p>Hey everyone,</p>
<p>I tried to display a heatmap over an image both using <code>matplotlib</code> - which I was able to get done now. This is the result:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/2/02e1bac2e3cc3b7691b95cdf8b3ef67d4eebeab7.png" alt="Screenshot 2021-05-13 at 13.30.40" data-base62-sha1="puzQLf9K6zapxOkRrpRx3hvgkD" width="309" height="304"></p>
<p>More particularly, I have:</p>
<ul>
<li>a tensor of shape <code>(7, 7)</code> with floats between 0 and 1 - let’s call it the <code>activation_map</code></li>
<li>the <code>image</code> tensor of shape <code>(3, 224, 224)</code></li>
</ul>
<p>I now upsampled the <code>activation_map</code> to <code>(224, 224)</code> and overlayed it with a custom <code>colormap</code> that is based on the <code>gray</code> colormap, just making high values transparent instead of white.</p>
<p>However, when I plot multiple of these pictures with <code>activation_map</code>, I always want them to show the same amount of pixels of the picture. Basically, I always want the highest 10.000 values of the <code>activation_map</code> to be ‘1’ and the other ones to be ‘0’. How can I achieve that?</p>
<p>I know I need to squeeze the <code>activation_map</code> to an array, sort it, then make the biggest values to ‘1’, the smallest ones to ‘0’. But how do I get the values back in the same order as they were in the original <code>activation_map</code>? I need to work with the indices but I am struggling to do so…</p>
<p>I would be happy to receive any help, overall it is for research as this will be used in an experiment.<br>
BR, David</p>

davidweb: <p>Well, sometimes it helps to just approach the problem again after you explained it to others… Here is the solution I got the problem solved with:</p>
<pre><code class="lang-auto"># make a copy of the original activation_map
nact = act.copy()

#Reshape the activation map to a 1D Matrix / Array
nact = nact.reshape(-1)

#Get the indices that would sort the array from low to high
sort_ind = np.argsort(nact)

# indices of the values I want to set to 0
sort_zero = sort_ind[:47175]

#indices of the values I want to set to 1
sort_ones = sort_ind[47176:]

#set values to zero
nact[sort_zero] = 0.0
# set values to one
nact[sort_ones] = 1.0

#Reshape activation_map back to original size
d_act = nact.reshape(224, 224)

#Visualize it over the image
plt.imshow(img)
plt.imshow(d_act, alpha=1.0, cmap=my_cmap)
plt.axis('off'); plt.tight_layout(); plt.show()
</code></pre>
<p>Result:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/a/7ab64a7ed4d9f9fa6b1adef900d3589a19298599.png" alt="Screenshot 2021-05-13 at 14.36.28" data-base62-sha1="hvyO6arQOtOx01OtInxyERS6wY1" width="344" height="346"></p>
Adding trainable parameters to each parameter of another model

sroutray: <p>I have a model given by</p>
<pre><code class="lang-auto">class M2(nn.Module):
    def __init__( self, dim_x=28*28, dim_y=10):
        super(M2, self).__init__()
        self.phi_y = nn.Sequential(nn.Linear(dim_x, 500), nn.ReLU(inplace=True),
                                   nn.Linear(500, 500), nn.ReLU(inplace=True),
                                   nn.Linear(500, dim_y))

    def forward(self, x):
        pass
</code></pre>
<p>I want to add trainable parameters to every parameter in <code>self.phi_y</code>. Then, I would like to use the resulting new network, and use it to make prediction and calculate loss in the forward function. During training, I would like to optimize the parameters of <code>self.phi_y</code> as well as the added parameters.</p>
<p>Thanks!</p>