akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Activation Functions in PyTorch Examples

username: CRWayman
cooked: <p>Hello, I’m very new to machine learning and PyTorch. I’m looking at the Learning PyTorch with Examples page.</p>
<p><a href="https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-custom-nn-modules" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-custom-nn-modules</a></p>
<p>I’m confused about the use of activation functions in this example code below. It doesn’t seem like there’s any kind of activation function being used here, like a ReLU, as in previous examples. Am I completely missing something here? If I want to apply this code to my project, do I need to introduce a ReLU at some point? I see that a h_relu variable is created in the forward function, but this doesn’t seem like the same thing.</p>
<p>Any explanations would be greatly appreciated.</p>
<pre><code class="lang-auto">class TwoLayerNet(torch.nn.Module):
    def __init__(self, D_in, H, D_out):
        """
        In the constructor we instantiate two nn.Linear modules and assign them as
        member variables.
        """
        super(TwoLayerNet, self).__init__()
        self.linear1 = torch.nn.Linear(D_in, H)
        self.linear2 = torch.nn.Linear(H, D_out)

    def forward(self, x):
        """
        In the forward function we accept a Tensor of input data and we must return
        a Tensor of output data. We can use Modules defined in the constructor as
        well as arbitrary operators on Tensors.
        """
        h_relu = self.linear1(x).clamp(min=0)
        y_pred = self.linear2(h_relu)
        return y_pred
</code></pre>

username: RicCu
cooked: <p>Hi!<br>
The method <code>clamp(min=0)</code> is functionally equivalent to ReLU. All ReLU does is to set all negative values to zero and keep all positive values unchanged, which is what is being done in that example with the use of clamp set to <code>min=0</code>.</p>
<p>Here’s the documentation for <a href="https://pytorch.org/docs/stable/torch.html#torch.clamp" rel="nofollow noopener">torch.clamp</a>.</p>

username: CRWayman
cooked: <p>So in this case models “activation function” is ReLU?</p>
<p>Thank you for the response!</p>

username: ajhanwar
cooked: <p>yup in this case it is exactly like the relu activation</p>
__new__() missing 1 required positional argument: 'aux_logits'

username: chenf99
cooked: <p>I was fine-tuning inception_v3 from torchvision, but when I use DataParallel I met a TypeError:<code>__new__() missing 1 required positional argument: 'aux_logits'</code> ,  here are my code:</p>
<pre><code class="lang-python">model = inception_v3(pretrained=True)
class_num = 8
channel_in = model.fc.in_features
model.AuxLogits.fc = nn.Linear(768, class_num)
model.fc = nn.Sequential(nn.Dropout(p=0.2),
                         nn.Linear(channel_in, class_num))

optimizer = torch.optim.Adam(model.parameters(), lr=lr)
loss_fn = nn.CrossEntropyLoss()

model = nn.DataParallel(model)
model.aux_logits = True
model = model.cuda()
loss_fn = loss_fn.cuda()
</code></pre>
<pre><code class="lang-python">outputs, aux_outputs = model(batch_x)
loss1 = loss_fn(outputs, batch_y)
loss2 = loss_fn(aux_outputs, batch_y)
loss = loss1 + 0.4*loss2
</code></pre>
<p>could anyone tell me how to solve this problem?Thank you!</p>

username: druvpatel
cooked: <p>Hello, I am having the same issue. Did you figure out a workaround for this other than disabling the aux_logits output in the model?</p>
About Pytorch Capabilities

username: sajjad
cooked: <p>Hi all,<br>
I am totally new to pytorch and I want to know its capabilities in developing new optimization ideas.<br>
More precisely, I have previously written a paper entitled “a new framework to train autoencoders using nonsmooth regularization” where the codes where implemented in Matlab  from scratch. Now I want to use the capabilities of pytorch in developing my idea in the above paper.<br>
Especially I want to know the following items:<br>
1- Can I define a dynamic cost function in pytorch? In other words consider I have a term in cost function like ||w-v||_F^2 where w is the weight matrix for a special layer and v is a fix matrix which is changed in each epoch based on a specific rule.<br>
2- Can we evaluate the gradient of dynamic cost function with respect to network parameters?<br>
3- Can we change the point where the gradient is evaluated? In other word consider I want to evaluate the gradient in a point where the value of weight matrix w is w+a*v where a is a constant scalar and v is a constant matrix.<br>
Thanks.</p>

username: tom
cooked: <ol>
<li>yes</li>
<li>yes</li>
<li>Not without re-evaluating the model. Most if not all auto differentiation (as opposed to symbolic differentiation) offer to evaluate the derivative at the point where you evaluated the function. This is because they need intermediate outputs etc. You can, of course, evaluate the model twice.</li>
</ol>

username: sajjad
cooked: <p>Thank you Tom for your answer.</p>
Why the model fail to converge without a manul weight initialization?

username: fricative
cooked: <p>hi all,</p>
<p>I am trying to create a most simple 1 linear layer network to fit a linear regression. Just to help myself better understand how Pytorch works. However, I encountered a strange issue with the model training.</p>
<p>in my model’s <strong>init</strong>() method, I have to add a manual initialization step(shown below) to have the model quickly converge to my regression function. (the weight value 2, 3 are random number, I could put any value here and the model will still converge)</p>
<pre><code class="lang-auto">self.layer1.weight = torch.nn.Parameter(torch.Tensor([2, 3]))
</code></pre>
<p>Without this line, the model never converge, the training loss just randomly oscillates in the range of hundreds of thousands. With this line, it quickly decreases to near 1.</p>
<p>I have postulated that it is because default initial weight parameters were too small if I do not initialize them to be far away from zero. Then I changed the initial values and found out the convergence always work as long as I have this line, the exact value I set does not matter. Could someone explain what is going on behind the scene here? Thanks.</p>
<p>My entire script:</p>
<pre><code class="lang-auto">import torch
import numpy as np

class Net(torch.nn.Module):
    
    def __init__(self, input_dim, output_dim):
        super(Net, self).__init__()
        self.layer1 = torch.nn.Linear(input_dim, output_dim, bias=False)
        self.layer1.weight = torch.nn.Parameter(torch.Tensor([2, 3]))
        
    def forward(self, x):
        x = self.layer1(x)
        x.squeeze()
        return x
        
# generate data using the linear regression setup y = 5 * x1 + 3 * x2

sample_size = 10000
input_dim = 2
output_dim = 1
epoch = 30
bs = 100

data = np.random.randn(sample_size, 3)
data[:, :2] = data[:, :2] * 100
# add a normal noise term
data[:, 2] = 5 * data[:, 0] + 3 * data[:, 1] + np.random.randn(sample_size)
data = torch.Tensor(data)
train_x = data[:, :input_dim]
train_y = data[:, input_dim]

net = Net(input_dim, output_dim)
net.zero_grad()
criterion = torch.nn.MSELoss()
optimizer = torch.optim.RMSprop(net.parameters(), lr=.01)

for i in range(epoch):
    
    batch = 0
    while batch * bs &lt; train_x.shape[0]:
        
        batch_x = train_x[batch * bs : (batch + 1) * bs, :]
        batch_y = train_y[batch * bs : (batch + 1) * bs]
        
        pred_y = net.forward(batch_x)
        loss = criterion(pred_y, batch_y)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
        if batch % 100 == 0:
            #print(f"{i} {batch} {loss}")
            print(net.layer1.weight)
        batch += 1
</code></pre>
Default behavior for padded data without setting padding_idx when calling nn.Embedding with pretrained weights

username: pytorch_learner
cooked: <p>These should be a novice question but unfortunately I couldn’t find an exact answer after some search, and hope to find some help here.</p>
<p>I am trying to train a simple LSTM model (using nn.LSTM) for sentiment classification with pretrained GloVe embeddings trained on my corpus. I padded the training data with zeros to align the timesteps within each batchs during training, but it came to me that there were no mapping vector contained for the padded zeros in the pretrained embedding, and I did not set padding_idx=0. However the model was able to train. I am therefore wondering what is the behavior of the model taking the input when in runs into indices without an embedding mapping when a set of pretrained weight is used (i.e. are those initiated with some random vector, or just omitted, or any others)?</p>
<p>In addition, the documentation stated if padding_idx is assigned some value, the gradient for that value will always be zero. Does that imply the model can be trained correctly (and as efficient) without having to pack the model with pack_pad_sequences and then invert, given the zero gradients of the padded values?</p>
<p>Thank you and any suggestions are highly appreciated.</p>